Running out of inodes, no space left on device, php not cleaning sessions

Today I run into a weird problem. When I tried to open my (Serendipity) blog I got this strange error:

Query failed: 

SELECT 
                    
	e.id,
	...
	ORDER BY timestamp DESC
	LIMIT 15

/ Can't create/write to file '/tmp/#sql_6ee_0.MYI' (Errcode: 28)

I was kind of puzzled. All other sites were working fine (such us wiki and list of pim prototypes). I sshed to a server and checked the disk usage:

myusername@pim:~$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             3.8G  3.0G  685M  82% / 

Everything fine here. I checked the /tmp folder

myusername@pim:/tmp# ls -la 
total 32
drwxrwxrwt  4 root root 20480 Apr 18 12:42 .
drwxr-xr-x 22 root root  4096 Apr 18 11:20 ..
drwxrwxrwt  2 root root  4096 Mar 24 22:42 .ICE-unix

Nothing strange here either. I tried to login to mysql:

myusername@pim:/tmp$ mysql -u myusername -p
Enter password: 
Welcome to the MySQL monitor

Everything worked fine. So I checked the ERROR 28 the web page was giving me

myusername@pim:/tmp$ perror 28
OS error code  28:  No space left on device

I tried to create a file with no success:

myusername@pim:/tmp$ touch bla.txt
touch: cannot touch `bla.txt': No space left on device

Then I googled a bit and found out that I might be running out of inodes.

myusername@pim:/tmp$ df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1             250976  250957      19  100% /

Yep, that was it. Now I had to find which folder had a lot of files that were taking up the inodes (as root). Not all results are shown in the below output.

myusername@pim:/var$ cd /
myusername@pim:/$ sudo bash 
root@pim:/$ for i in /*; do echo $i; find $i -type f | wc -l; done
/bin 91
/boot 23
/dev 346
/etc 745
/home 5
/lib 1709
/media 0
/proc 11435
/root 5
/sbin 103
/sys 3863
/tmp 0
/usr 45188
/var    186774

I went to /var and run the same command

root@pim:/var$ for i in ; do echo $i; find $i -type f | wc -l ;done
backups 1
cache 55
lib 173682
log 47
mail 2
run 16
spool 19
www 12950

So I needed to check /var/lib

mkljun@pim:/var/lib$ for i in ; do echo $i; sudo find $i -type f | wc -l ;done
apt 9
doc-base 37
dpkg 2342
misc 2
mlocate 1
mysql 665
php5 170453
screen-profiles 21
ucf 23
xml-core 2

The folder /var/lib/php5 contained 170k files! It was kind of strange why all these files were not deleted after a session timeout. I certainly did not have 170k users visiting my pages (I wished this to be true though).

root@pim:/var/lib# cd php5
root@pim:/var/lib/php5# ls -l | wc -l
170453

I checked the maximum time set for a php5 sessions to live.

root@pim:/var/lib/php5# /usr/lib/php5/maxlifetime
24

24 minutes. I decided to delete older files.

root@pim:/var/lib/php5# find /var/lib/php5/ -type f -cmin +$(/usr/lib/php5/maxlifetime) -print0 -exec rm {} \;
root@pim:/var/lib/php5# ls -la
total 11768
drwx-wx-wt  2 root     root     12017664 Apr 18 12:35 .
drwxr-xr-x 38 root     root         4096 Sep 22  2010 ..
-rw-------  1 www-data www-data      168 Apr 18 12:23 sess_389b8191e2563e8bb43c41e20e015c0d
-rw-------  1 www-data www-data      118 Apr 18 12:11 sess_52c9acc7779f5cfd7e7e92a900fe7f19
-rw-------  1 www-data www-data      160 Apr 18 12:27 sess_bb8cfa842ceb0027cc9e4082ad8b0419

I checked the inodes usage.

root@pim:/tmp# df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1             250976   80524  170452   33% /

That was it.

Why PHP is not deleting expired sessions is another question. I went to my php.ini file and found this:

; After this number of seconds, stored data will be seen as 'garbage' and
; cleaned up by the garbage collection process.
session.gc_maxlifetime = 1440

; NOTE: If you are using the subdirectory option for storing session files
;       (see session.save_path above), then garbage collection does not
;       happen automatically.  You will need to do your own garbage
;       collection through a shell script, cron entry, or some other method.

So all I needed to do was to create a cron job (I decided to rut it every hour as root)

root@pim:/tmp# crontab -e 

0      /usr/bin/find /var/lib/php5/ -type f -cmin +$(/usr/lib/php5/maxlifetime) -print0 -exec /bin/rm {} \;

Hopefully this will help some other soul out there.

Trackbacks

Trackback specific URI for this entry

This link is not meant to be clicked. It contains the trackback URI for this entry. You can use this URI to send ping- & trackbacks from your own blog to this entry. To copy the link, right click and select "Copy Shortcut" in Internet Explorer or "Copy Link Location" in Mozilla.

No Trackbacks

Comments

Display comments as Linear | Threaded

jeroen on :

thanks so much. this was exactly the problem i ran into. except i had a couple millions of old session files...

michael on :

Thank you. Just the answer I was looking for.

Mukesh on :

Thanks Matjaž! Solved a huge headache.

Webdesigner on :

Hey, just wanted to say thank you, because you made my day with your help!

Cheers

botika on :

*A million thanks Matjaž*!

We've had the same problem with the million inodes in our server, which was incomprehensible until we found this to perfectly solve it.

Two years and a half later your post can still "help some other soul out there" ;)

Matjaž on :

I'm glad you have solved this problem as well. Two years ago it took me quite a while to crack it and I thought that there are other soulsout there than might benefit from my experience. Internet is such a great tool.

lovek on :

Solved a huge headache, otherwise the only option was to create new virtual server,
THANKS BUDDY

mk on :

Glad it helped

Ruben on :

Wow, thanks for the help. i had 9.9million used inodes in a Magento sessions folder.... took me 5 hours to rm them all. Did not notice them because the size was not that big...

Mike on :

Thank you!!

Zahid Hussain on :

Simply Superb!!!
I got what I needed and resolved my problem.

Thank you very much. Matjaž.

Corine on :

Hi! I'm at work browsing your blog from my new iphone 3gs!
Just wanted to say I love reading your blog and look forward to all your posts!
Keep up the fantastic work!

Add Comment

Enclosing asterisks marks text as bold (*word*), underscore are made via _word_.
E-Mail addresses will not be displayed and will only be used for E-Mail notifications.

To prevent automated Bots from commentspamming, please enter the string you see in the image below in the appropriate input box. Your comment will only be submitted if the strings match. Please ensure that your browser supports and accepts cookies, or your comment cannot be verified correctly.
CAPTCHA

Can you please write (or copy/paste) this text in the field below: i h a t e s p a m