{"id":165,"date":"2011-04-20T12:38:00","date_gmt":"2011-04-20T12:38:00","guid":{"rendered":"https:\/\/pim.famnit.upr.si\/wp\/?p=165"},"modified":"2021-11-17T11:21:43","modified_gmt":"2021-11-17T11:21:43","slug":"running-out-of-inodes-no-space-left-on-device-php-not-cleaning-sessions","status":"publish","type":"post","link":"https:\/\/pim.famnit.upr.si\/wp\/?p=165","title":{"rendered":"Running out of inodes, no space left on device, php not cleaning sessions"},"content":{"rendered":"

\nToday I run into a weird problem. When I tried to open my<\/strong> (Serendipity) blog I got<\/strong> this strange error<\/strong>:\n<\/p>\n

Query failed: \r\n\r\nSELECT \r\n                    \r\n\te.id,\r\n\t...\r\n\tORDER BY timestamp DESC\r\n\tLIMIT 15\r\n\r\n\/ Can't create\/write to file '\/tmp\/#sql_6ee_0.MYI' (Errcode: 28)\r\n<\/pre>\n
\nI was kind of puzzled. All other sites were working fine (such us wiki and list of pim prototypes). I sshed to a server and checked the disk usage<\/strong>:\n<\/p>\n
myusername@pim:~$ df -h\r\nFilesystem            Size  Used Avail Use% Mounted on\r\n\/dev\/sda1             3.8G  3.0G  685M  82% \/ \r\n<\/pre>\n
\nEverything fine here. I checked the \/tmp<\/strong> folder\n<\/p>\n
myusername@pim:\/tmp# ls -la \r\ntotal 32\r\ndrwxrwxrwt  4 root root 20480 Apr 18 12:42 .\r\ndrwxr-xr-x 22 root root  4096 Apr 18 11:20 ..\r\ndrwxrwxrwt  2 root root  4096 Mar 24 22:42 .ICE-unix\r\n<\/pre>\n
\nNothing strange here either. I tried to login to mysql<\/strong>:\n<\/p>\n
myusername@pim:\/tmp$ mysql -u myusername -p\r\nEnter password: \r\nWelcome to the MySQL monitor\r\n<\/pre>\n
\nEverything worked fine. So I checked the ERROR 28<\/strong> the web page was giving me\n<\/p>\n
myusername@pim:\/tmp$ perror 28\r\nOS error code  28:  No space left on device\r\n<\/pre>\n
\nI tried to create a file<\/strong> with no success:\n<\/p>\n
myusername@pim:\/tmp$ touch bla.txt\r\ntouch: cannot touch `bla.txt': No space left on device\r\n<\/pre>\n
\nThen I googled a bit and found out that I might be running out of inodes<\/strong>.\n<\/p>\n
myusername@pim:\/tmp$ df -i\r\nFilesystem            Inodes   IUsed   IFree IUse% Mounted on\r\n\/dev\/sda1             250976  250957      19  100% \/\r\n<\/pre>\n
\nYep, that was it. Now I had to find which folder had a lot of files that were taking up the inodes<\/strong> (as root). Not all results are shown in the below output.\n<\/p>\n
myusername@pim:\/var$ cd \/\r\nmyusername@pim:\/$ sudo bash \r\nroot@pim:\/$ for i in \/*; do echo $i; find $i -type f | wc -l; done\r\n\/bin 91\r\n\/boot 23\r\n\/dev 346\r\n\/etc 745\r\n\/home 5\r\n\/lib 1709\r\n\/media 0\r\n\/proc 11435\r\n\/root 5\r\n\/sbin 103\r\n\/sys 3863\r\n\/tmp 0\r\n\/usr 45188\r\n\/var    186774\r\n<\/pre>\n
\nI went to \/var<\/strong> and run the same command\n<\/p>\n
root@pim:\/var$ for i in *; do echo $i; find $i -type f | wc -l ;done\r\nbackups 1\r\ncache 55\r\nlib 173682\r\nlog 47\r\nmail 2\r\nrun 16\r\nspool 19\r\nwww 12950\r\n<\/pre>\n
\nSo I needed to check \/var\/lib<\/strong> <\/p>\n
mkljun@pim:\/var\/lib$ for i in *; do echo $i; sudo find $i -type f | wc -l ;done\r\napt 9\r\ndoc-base 37\r\ndpkg 2342\r\nmisc 2\r\nmlocate 1\r\nmysql 665\r\nphp5 170453\r\nscreen-profiles 21\r\nucf 23\r\nxml-core 2\r\n<\/pre>\n
\nThe folder \/var\/lib\/php5 contained 170k files<\/strong>! It was kind of strange why all these files were not deleted after a session timeout. I certainly did not have 170k users visiting my pages (I wished this to be true though).\n<\/p>\n
root@pim:\/var\/lib# cd php5\r\nroot@pim:\/var\/lib\/php5# ls -l | wc -l\r\n170453\r\n<\/pre>\n
\nI checked the maximum time set for a php5 sessions to live<\/strong>.\n<\/p>\n
root@pim:\/var\/lib\/php5# \/usr\/lib\/php5\/maxlifetime\r\n24\r\n<\/pre>\n
\n24 minutes. I decided to delete older files.<\/strong> <\/p>\n
root@pim:\/var\/lib\/php5# find \/var\/lib\/php5\/ -type f -cmin +$(\/usr\/lib\/php5\/maxlifetime) -print0 -exec rm {} \\;\r\nroot@pim:\/var\/lib\/php5# ls -la\r\ntotal 11768\r\ndrwx-wx-wt  2 root     root     12017664 Apr 18 12:35 .\r\ndrwxr-xr-x 38 root     root         4096 Sep 22  2010 ..\r\n-rw-------  1 www-data www-data      168 Apr 18 12:23 sess_389b8191e2563e8bb43c41e20e015c0d\r\n-rw-------  1 www-data www-data      118 Apr 18 12:11 sess_52c9acc7779f5cfd7e7e92a900fe7f19\r\n-rw-------  1 www-data www-data      160 Apr 18 12:27 sess_bb8cfa842ceb0027cc9e4082ad8b0419\r\n<\/pre>\n
\nI checked the inodes usage.<\/strong> <\/p>\n
root@pim:\/tmp# df -i\r\nFilesystem            Inodes   IUsed   IFree IUse% Mounted on\r\n\/dev\/sda1             250976   80524  170452   33% \/\r\n<\/pre>\n
 That was it.
\n<\/strong><\/p>\n
Why PHP is not deleting expired sessions<\/strong> is another question. I went to my php.ini file<\/strong> and found this:\n<\/p>\n
; After this number of seconds, stored data will be seen as 'garbage' and\r\n; cleaned up by the garbage collection process.\r\nsession.gc_maxlifetime = 1440\r\n\r\n; NOTE: If you are using the subdirectory option for storing session files\r\n;       (see session.save_path above), then garbage collection does *not*\r\n;       happen automatically.  You will need to do your own garbage\r\n;       collection through a shell script, cron entry, or some other method.\r\n<\/pre>\n
\nSo all I needed to do was to create a cron job<\/strong> (I decided to rut it every hour as root)\n<\/p>\n
root@pim:\/tmp# crontab -e \r\n\r\n0 * * * *  \/usr\/bin\/find \/var\/lib\/php5\/ -type f -cmin +$(\/usr\/lib\/php5\/maxlifetime) -print0 -exec \/bin\/rm {} \\;\r\n<\/pre>\n
\nHopefully this will help some other soul out there.<\/p>\n","protected":false},"excerpt":{"rendered":"
Today I run into a weird problem. When I tried to open my (Serendipity) blog I got this strange error: Query failed: SELECT e.id, … ORDER BY timestamp DESC LIMIT 15 \/ Can’t create\/write to file ‘\/tmp\/#sql_6ee_0.MYI’ (Errcode: 28) I was kind of puzzled. All other sites were working fine (such us wiki and list...<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5],"tags":[],"class_list":["post-165","post","type-post","status-publish","format-standard","hentry","category-4-software-hints"],"_links":{"self":[{"href":"https:\/\/pim.famnit.upr.si\/wp\/index.php?rest_route=\/wp\/v2\/posts\/165","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/pim.famnit.upr.si\/wp\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/pim.famnit.upr.si\/wp\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/pim.famnit.upr.si\/wp\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/pim.famnit.upr.si\/wp\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=165"}],"version-history":[{"count":1,"href":"https:\/\/pim.famnit.upr.si\/wp\/index.php?rest_route=\/wp\/v2\/posts\/165\/revisions"}],"predecessor-version":[{"id":813,"href":"https:\/\/pim.famnit.upr.si\/wp\/index.php?rest_route=\/wp\/v2\/posts\/165\/revisions\/813"}],"wp:attachment":[{"href":"https:\/\/pim.famnit.upr.si\/wp\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=165"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/pim.famnit.upr.si\/wp\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=165"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/pim.famnit.upr.si\/wp\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=165"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}