Hello, I've am running a rather large site off an AMD64 box with 2GB RAM, and switched from Apache to Lighttpd several months ago. I never had any real issues with it, it was certainly smaller in memory and much much faster with serving pages. Now, for no apparent reason, Lighttpd has become practically unusable for me: At random points in time, it just stops serving pages. The lighttpd process uses almost no cpu but over 1.3 GB memory! The associated php processes spawned by fastcgi don't show any anormality whatsoever. I'm running Debian Sarge and tried both upgrading to the latest precompiled binary, and even compiled my own. The symptoms have remained exactly the same. Sometimes it takes a few hours for the "explosion" to happen, other times it happens almost right away. And whenever it happens, I can watch lighttpd grow in "top" from it's 3-5 megabytes to over a gigabytes within a few dozen seconds! Before this happens, lighttpd runs as fast as usual. EDIT: No error reports show up in the errorlog, so certainly nothing about processes dying or handles running out or something. Since I am quite certain that nothing else changed on my box, is there any new exploit in the wild that I should know about? I was forced to switch back to Apache for the moment since a slow server is better than no server at all. I'd seriously appreciate any and all clues or reports of similar incidents if available. UPDATE: Apache just couldn't handle the load (thus making CPU cycles the critical resource, not RAM) and so I switched back to a fresh install of lighttpd where I made sure to clean out any old libraries that lingered around in /usr/local/lib. It ran at around 6MB mem usage for half an hour, then "exploded" to 1.1GB, but it still runs without any noticeable hitch for about 4 hours now. UPDATE2: Alright, it stopped serving pages during the night. Same as always, lighttpd used up over 1.7 GB of memory.
on 07.01.2006 18:20
on 08.01.2006 21:35
Tried using the latest version compiled from the SVN source, no joy. Same result: -------- top - 21:33:07 up 6:01, 3 users, load average: 19.68, 9.56, 4.12 Tasks: 146 total, 3 running, 143 sleeping, 0 stopped, 0 zombie Cpu(s): 55.6% us, 35.5% sy, 0.0% ni, 0.0% id, 7.2% wa, 0.3% hi, 1.3% si Mem: 2054544k total, 2047504k used, 7040k free, 4460k buffers Swap: 9839804k total, 2816312k used, 7023492k free, 202712k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 25414 www-data 16 0 3871m 1.3g 1856 D 4.0 67.8 0:25.83 lighttpd 2939 mysql 16 0 262m 33m 6552 S 0.0 1.7 0:01.56 mysqld 2296 root 18 0 27904 17m 4816 S 0.0 0.8 0:03.11 spamd 25444 www-data 15 0 13620 10m 4180 S 1.1 0.5 0:08.17 php 25427 www-data 15 0 12804 9.9m 4180 S 0.0 0.5 0:08.97 php 25457 www-data 15 0 12768 9.9m 4180 S 1.1 0.5 0:09.79 php -------- Error log shows no problems. I've come up with a small script to check if the server is still serving requests at a reasonable speed, and this has successfully noticed whenever the blowup happens, killing and restarting the server without any need for manual work. It would much prefer NOT having to resort to such measures though. I don't like the idea of my webserver dying ever few hours... Especially since Lighttpd has been working without any hitch for months before! -------- #!/bin/bash links -dump -receive-timeout 10 http://hoxdna.org/yiffstar/phpbb/index.php>/dev/null 2>&1 && sleep 1 ||{ echo "Server is down!" /etc/init.d/lighttpd stop sleep 3 killall -9 lighttpd sleep 1 killall -9 php sleep 1 /etc/init.d/lighttpd start /sbin/checkproc /usr/sbin/lighttpd && echo "Server back up again" || echo "Server cannot be restarted!" } --------
on 13.01.2006 22:09
Just a heads-up that I "solved" the problem. I upgraded from PHP4 to PHP5, as well as got up to speed to the latest version of MySQL. Since then, the effect above is no longer happening on that machine. Memory usage stays at around 11MB for the lighttpd process and the thing is stable again. I guess I'll never know what the real reason was though, which is especially strange since it has been running without problems for months, with the crashes appearing out of nowhere.
on 14.01.2006 03:42
toumal wrote: > Just a heads-up that I "solved" the problem. I upgraded from PHP4 to > PHP5, as well as got up to speed to the latest version of MySQL. Since > then, the effect above is no longer happening on that machine. Memory > usage stays at around 11MB for the lighttpd process and the thing is > stable again. > > I guess I'll never know what the real reason was though, which is > especially strange since it has been running without problems for > months, with the crashes appearing out of nowhere. Hi, although i guess you did check this already, but when you state the lighty does not provide any pages anymore, do you talk about dynamic pages (php) or do we talk about static pages as well?! i ask, because lighty has "still" no capability to fight against most simple DOS attacks. if a kid starting a little perl program which opens hundreds of concurrent requests to any php file (e.g. index.php) your fastcgi sockets are wasted and lighty stops providing fastcgi pages (e.g. php). only static files work then... Have you checked the connections when this happened? Simon
on 14.01.2006 17:08
> do you talk about dynamic > pages (php) or do we talk about static pages as well?! It affected both dynamic and static pages. > i ask, because lighty has "still" no capability to fight against most > simple DOS attacks. Yep. That's why I built in a hammering-protection into my pages. If an IP slams the server with more than 60 page hits in 120 seconds, he'll be kicked out for half an hour. Works wonderfully. > Have you checked the connections when this happened? Yep. Around 50-80 connections. No new connections are being added and the old ones aren't being worked on. But eh, with the new PHP all is working fine now. No memory usage growth at all.
on 17.01.2006 00:13
I had this kind of problem with PHP 4.4.1 I reinstalled PHP 4.4.1 I found out that I had mistyped some configuration commands in PHP 4.4.1 regarding the CGI compilation. I also found out I mistyped some configuration commands in the Alternative PHP Cache. Everything is working fine now.
on 19.01.2006 14:03
http://lighttpd.net/documentation/fastcgi.html#fastcgi Hi we have tried this and the site is working smoothly.. fastcgi.server = ( ".php" => (( "socket" => "/tmp/php-fastcgi.socket", "bin-path" => "/usr/local/bin/php", "bin-environment" => ( "PHP_FCGI_CHILDREN" => "16", "PHP_FCGI_MAX_REQUESTS" => "10000" ), "bin-copy-environment" => ( "PATH", "SHELL", "USER" ), "broken-scriptfilename" => "enable" )) ) this is from this link http://lighttpd.net/documentation/fastcgi.html#fastcgi
on 25.01.2006 13:13
toumal wrote: > Just a heads-up that I "solved" the problem. I upgraded from PHP4 to > PHP5, as well as got up to speed to the latest version of MySQL. Since > then, the effect above is no longer happening on that machine. Memory > usage stays at around 11MB for the lighttpd process and the thing is > stable again. > > I guess I'll never know what the real reason was though, which is > especially strange since it has been running without problems for > months, with the crashes appearing out of nowhere. Weird. I've had a similar problem, also with phpbb. I'll try PHP5 :) Cheers
on 04.04.2006 16:24
anish wrote: > http://lighttpd.net/documentation/fastcgi.html#fastcgi > > Hi we have tried this and the site is working smoothly.. > > fastcgi.server = ( ".php" => > (( "socket" => "/tmp/php-fastcgi.socket", > "bin-path" => "/usr/local/bin/php", > "bin-environment" => ( > "PHP_FCGI_CHILDREN" => "16", > "PHP_FCGI_MAX_REQUESTS" => "10000" ), > "bin-copy-environment" => ( > "PATH", "SHELL", "USER" ), > "broken-scriptfilename" => "enable" > )) > ) > > this is from this link > http://lighttpd.net/documentation/fastcgi.html#fastcgi I will second the fact that lighty will sometimes use up all memory. Right now my lighty install (with PHP5 installed under fcgi) is using so much memory that the oom killer gets rid of it (leaving my server... web serverless). I'm going to try the above fix so, we'll see what happens ^_^
on 04.04.2006 20:02
Jhon Adams wrote: > anish wrote: >> http://lighttpd.net/documentation/fastcgi.html#fastcgi >> >> Hi we have tried this and the site is working smoothly.. >> >> fastcgi.server = ( ".php" => >> (( "socket" => "/tmp/php-fastcgi.socket", >> "bin-path" => "/usr/local/bin/php", >> "bin-environment" => ( >> "PHP_FCGI_CHILDREN" => "16", >> "PHP_FCGI_MAX_REQUESTS" => "10000" ), >> "bin-copy-environment" => ( >> "PATH", "SHELL", "USER" ), >> "broken-scriptfilename" => "enable" >> )) >> ) >> >> this is from this link >> http://lighttpd.net/documentation/fastcgi.html#fastcgi > > I will second the fact that lighty will sometimes use up all memory. > Right now my lighty install (with PHP5 installed under fcgi) is using so > much memory that the oom killer gets rid of it (leaving my server... web > serverless). I'm going to try the above fix so, we'll see what happens > ^_^ I tried to fix the configuration with the above line and with server.network-backend = "writev" (since I am serving up concurrent connections to several people with large files) However after the server was running for ~4 hours I noticed that it was no longer responding (the actual time it died is somewhere before that time however I do not know what it is) The same problem as before, dmesg reports out of memory and kills lighttpd.
on 04.04.2006 20:32
Please try the latest version of Lighttpd in svn. Based on the timestamps in this thread, it seems that they were posted long before many of the memory leak cleanups. Speaking of which...it would be wonderful if we had Lighttpd-1.4.12 released. It contains enough fixes for many of us to want to use in production.
on 04.04.2006 21:25
Lighty-1.4.11 wrote: > Please try the latest version of Lighttpd in svn. > > Based on the timestamps in this thread, it seems that they were posted > long before many of the memory leak cleanups. > > Speaking of which...it would be wonderful if we had Lighttpd-1.4.12 > released. It contains enough fixes for many of us to want to use in > production. I upgraded my install to the latest from subversion, watching vmstat my free memory goes from 1.3GB down to 33MB once I start downloading two 600mb files (OS CDs in this case) top shows lighttpd using 90% of memory. I have tried using server.network-backend = "writev" and I have tried with local buffering enabled and disabled with the same results... I was able to get several users downloading CDs at once with my apache install but under lighty that seems to be not really working out o_O Thanks again for your help...
on 04.04.2006 21:54
Jhon Adams wrote: > Lighty-1.4.11 wrote: >> Please try the latest version of Lighttpd in svn. >> >> Based on the timestamps in this thread, it seems that they were posted >> long before many of the memory leak cleanups. >> >> Speaking of which...it would be wonderful if we had Lighttpd-1.4.12 >> released. It contains enough fixes for many of us to want to use in >> production. > > I upgraded my install to the latest from subversion, watching vmstat my > free memory goes from 1.3GB down to 33MB once I start downloading two > 600mb files (OS CDs in this case) top shows lighttpd using 90% of > memory. I have tried using server.network-backend = "writev" and I have > tried with local buffering enabled and disabled with the same results... > I was able to get several users downloading CDs at once with my apache > install but under lighty that seems to be not really working out o_O > Thanks again for your help... Sorry the latest svn has the same problem. Hope future fixes will correct this. One workaround might be to use vsftpd if the ftp protocol is an option for your requirements.
on 06.04.2006 16:17
Jhon Adams wrote: > Lighty-1.4.11 wrote: >> Please try the latest version of Lighttpd in svn. >> >> Based on the timestamps in this thread, it seems that they were posted >> long before many of the memory leak cleanups. >> >> Speaking of which...it would be wonderful if we had Lighttpd-1.4.12 >> released. It contains enough fixes for many of us to want to use in >> production. > > I upgraded my install to the latest from subversion, watching vmstat my > free memory goes from 1.3GB down to 33MB once I start downloading two > 600mb files (OS CDs in this case) top shows lighttpd using 90% of > memory. I have tried using server.network-backend = "writev" and I have > tried with local buffering enabled and disabled with the same results... > I was able to get several users downloading CDs at once with my apache > install but under lighty that seems to be not really working out o_O > Thanks again for your help... 'free memory' ? Can you please provide real numbers as 'free memory' is usually only a indicator that you have too much memory. You want that you memory is used for the fs-cache, internal buffers, ... everything that is reducing the io-load on the disk. Please try again and attach the numbers from: $ ps axu | grep lighttpd $ vmstat -n 1 5 $ free As example, this is my system: $ free total used free shared buffers cached Mem: 709768 675088 34680 0 37116 125884 -/+ buffers/cache: 512088 197680 Swap: 524280 195696 328584 35Mb is unused/free. The rest is used by the OS as cache. Jan
on 07.04.2006 07:33
Jan Kneschke wrote: > > ... > > 'free memory' ? > > Can you please provide real numbers as 'free memory' is usually only a > indicator that you have too much memory. You want that you memory is > used for the fs-cache, internal buffers, ... everything that is reducing > the io-load on the disk. > > Please try again and attach the numbers from: > > $ ps axu | grep lighttpd > $ vmstat -n 1 5 > $ free > > > As example, this is my system: > > $ free > total used free shared buffers > cached > Mem: 709768 675088 34680 0 37116 > 125884 > -/+ buffers/cache: 512088 197680 > Swap: 524280 195696 328584 > > 35Mb is unused/free. The rest is used by the OS as cache. > > Jan The free memory in vmstat went to 33-ish mb and top showed lighttpd using the memory. Unfortunately, I downgraded my server to Apache and since it is currently in use I cannot provide the information you requested. If an opportunity arrises, I shall retrieve it.
on 17.08.2006 11:53
Is anyone still having problems in this area? I'm noticing my lighttpd nodes growing slowly in size over time until they have to be restarted. When lighttpd is restarted, memory usage is low again. Since the php-serving nodes are much more affected than the static content nodes, it appears there is a problem within PHP or fastcgi handling.
on 14.05.2008 09:26
Serhat wrote: > Is anyone still having problems in this area? > > I'm noticing my lighttpd nodes growing slowly in size over time until > they have to be restarted. When lighttpd is restarted, memory usage is > low again. Since the php-serving nodes are much more affected than the > static content nodes, it appears there is a problem within PHP or > fastcgi handling. I think that problem be happen again! Now, I use lighttpd 1.4.19 + fastcgi with php5. When I started lighttpd process, It used only 30MB for memory. But in only 30 minute it used up to 100MB and go to 600MB in 2-3 days. I must restart lighttpd everytime I think server slower (before server die). Does you know about this problem or how to fix them?
on 14.05.2008 12:21
bigaaa wrote:
> Does you know about this problem or how to fix them?
It's a problem when you're serving large files directly from PHP. Use
x-sendfile header.
