Any ideas? Traffic is substantial, but not extreme (I was running Apache prior to this, and it was handling the traffic with less CPU load...) Load averages have been anywhere from 0.5 to 30 - I can't understand the spikes in CPU load, they're not correlated to spikes in traffic. Seems like the only way to get the CPU load back down is to reboot the server. Any ideas? #top -b -n 1 top - 09:16:57 up 12:46, 2 users, load average: 17.58, 19.03, 21.02 Tasks: 183 total, 19 running, 164 sleeping, 0 stopped, 0 zombie Cpu(s): 33.1%us, 6.3%sy, 0.0%ni, 59.2%id, 0.4%wa, 0.2%hi, 0.8%si, 0.0%st Mem: 964160k total, 769800k used, 194360k free, 43588k buffers Swap: 2104440k total, 68k used, 2104372k free, 89384k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16165 root 16 0 147m 16m 11m R 9.6 1.8 0:07.80 php5-cgi 16099 root 16 0 147m 17m 12m R 7.7 1.8 0:07.49 php5-cgi 16146 root 16 0 149m 19m 11m R 7.7 2.0 0:07.64 php5-cgi 16220 root 16 0 149m 19m 12m R 7.7 2.1 0:08.27 php5-cgi 16127 root 16 0 149m 19m 12m R 5.8 2.1 0:04.55 php5-cgi 16161 root 16 0 150m 20m 12m R 5.8 2.1 0:07.69 php5-cgi 16092 root 15 0 25400 4248 908 S 3.8 0.4 0:09.93 lighttpd 16098 root 15 0 149m 19m 11m R 3.8 2.0 0:07.44 php5-cgi 16130 root 16 0 149m 20m 12m R 3.8 2.1 0:08.36 php5-cgi 16181 root 15 0 149m 20m 12m R 3.8 2.2 0:07.90 php5-cgi 16196 root 15 0 147m 16m 11m S 3.8 1.8 0:07.44 php5-cgi 16215 root 16 0 149m 19m 11m R 3.8 2.0 0:09.33 php5-cgi 16221 root 16 0 150m 20m 12m R 3.8 2.1 0:06.96 php5-cgi 16245 root 15 0 10728 1276 864 R 3.8 0.1 0:00.03 top 16133 root 16 0 149m 18m 11m R 1.9 2.0 0:07.21 php5-cgi 16148 root 16 0 149m 19m 12m R 1.9 2.1 0:26.19 php5-cgi 16094 root 25 0 146m 10m 6388 S 0.0 1.1 0:00.13 php5-cgi [there are approximately 110 more php5-cgi processes] #vim /etc/lighttpd/lighttpd.conf #256MB ram / 4 * 256 = 16,384 (http://docs.sun.com/source/816-7159-10/pt_chap6.html) server.max-fds = 16384 #http://trac.lighttpd.net/trac/wiki/server.max-connectionsDetails #16384 / 3 = 5461 server.max-connections = 5461 $HTTP["host"] =~ "^(www\.|)bluwiki\.(com|org)$" { url.rewrite-once = ( "^/blog(.*)" => "/sam/blog$1", #blog redirects "^/(images|skins|sam)(.*)" => "$0", #static folders "^/(.*).(php|htm|html|js)(.*)" => "$0", #static files "^/go/(.*)" => "index.php?title=$1", "^/user/(.*)" => "index.php?title=User:$1", "^/wiki/(.*)" => "index.php?title=$1", "^/clean/(.*)" => "index.php?title=$1&printable=yes", "^/site/(.*)" => "index.php?title=$1&printable=yes", ) fastcgi.server = ( ".php" => ( #Added for PHP support based on this howto: http://www.howtoforge.com/lighttpd_mysql_php_debian_etch "localhost" => ( "bin-path" => "/usr/bin/php5-cgi", "socket" => "/tmp/php.socket" + var.PID, "max-procs" => 2, "idle-timeout" => 20, "bin-environment" => ( "PHP_FCGI_CHILDREN" => "64", "PHP_FCGI_MAX_REQUESTS" => "10000" ), "bin-copy-environment" => ( "PATH", "SHELL", "USER" ), "broken-scriptfilename" => "enable" ) ) ) server.document-root = "/var/www/bluwiki-com/htdocs/" } [ and more misc defaults ...] # php5-cgi -v PHP 5.2.0-8+etch10 (cgi-fcgi) (built: Jan 18 2008 18:27:42) Copyright (c) 1997-2006 The PHP Group Zend Engine v2.2.0, Copyright (c) 1998-2006 Zend Technologies with XCache v2.0.0-dev, Copyright (c) 2005-2007, by mOo
on 03.05.2008 05:17
on 03.05.2008 15:05
Sam wrote: > [there are approximately 110 more php5-cgi processes] > "max-procs" => 2, > "PHP_FCGI_CHILDREN" => "64", Why so many?
on 03.05.2008 20:19
Olaf van der Spek wrote: > Sam wrote: >> [there are approximately 110 more php5-cgi processes] >> "max-procs" => 2, >> "PHP_FCGI_CHILDREN" => "64", > > Why so many? I'm new to Lightty (obviously) and was just following a tutorial. After googling, is 4 or 8 a more reasonable value? I have a single core 64-bit processor. I also define 6 separate fast-cgi processes for 6 different hosts. IE, the following code is repeated 6 times: $HTTP["host"] =~ "host_URL" { fastcgi.server = ( ".php" => ( #Added for PHP support based on this howto: "localhost" => ( "max-procs" => 2, "bin-environment" => ( "PHP_FCGI_CHILDREN" => "8", ), ) ) ) Thanks for your help.
on 03.05.2008 22:14
Sam wrote: > Olaf van der Spek wrote: >> Sam wrote: >>> [there are approximately 110 more php5-cgi processes] >>> "max-procs" => 2, >>> "PHP_FCGI_CHILDREN" => "64", >> >> Why so many? > > I'm new to Lightty (obviously) and was just following a tutorial. After > googling, is 4 or 8 a more reasonable value? I have a single core > 64-bit processor. > I also define 6 separate fast-cgi processes for 6 different hosts. IE, > the following code is repeated 6 times: So did you have 6 * 2 * 64 = 768? I hope not. > $HTTP["host"] =~ "host_URL" { > fastcgi.server = ( > ".php" => ( #Added for PHP support based on this howto: > "localhost" => ( > "max-procs" => 2, > "bin-environment" => ( > "PHP_FCGI_CHILDREN" => "8", > ), > ) > ) > ) > > > Thanks for your help. That's still 96. I'd got for maybe 6 * 1 * 4 or even lower.
on 04.05.2008 01:10
> That's still 96. I'd got for maybe 6 * 1 * 4 or even lower.
I've reduced it to a total of 22 processes max (two 2 * 4 sites and
three 1 * 2 sites) and the process list looks much healthier. However
CPU load is still crazy high:
#uptime
05:18:32 up 19:39, 1 user, load average: 14.10, 14.60, 15.04
Should I reduce it even more?
# top -b -n 1
top - 05:20:03 up 19:41, 1 user, load average: 8.73, 12.67, 14.31
Tasks: 61 total, 11 running, 50 sleeping, 0 stopped, 0 zombie
Cpu(s): 81.5%us, 14.1%sy, 0.0%ni, 2.0%id, 0.0%wa, 0.5%hi, 1.8%si,
0.0%st
Mem: 1996836k total, 321244k used, 1675592k free, 49692k buffers
Swap: 2104440k total, 0k used, 2104440k free, 135460k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4189 root 15 0 148m 18m 11m R 30.7 0.9 0:10.76 php5-cgi
4190 root 16 0 149m 19m 12m R 21.1 1.0 0:09.71 php5-cgi
4183 root 16 0 149m 20m 13m R 17.3 1.1 0:08.72 php5-cgi
4185 root 16 0 149m 19m 12m R 11.5 1.0 0:11.55 php5-cgi
2345 root 15 0 46420 37m 640 R 5.8 1.9 18:31.65 memcached
4184 root 16 0 150m 22m 14m R 3.8 1.2 0:09.42 php5-cgi
4191 root 15 0 149m 20m 12m R 3.8 1.1 0:11.39 php5-cgi
4180 root 15 0 24672 3508 896 S 1.9 0.2 0:00.99 lighttpd
4186 root 16 0 149m 20m 12m R 1.9 1.0 0:11.66 php5-cgi
1 root 18 0 6124 688 568 S 0.0 0.0 0:01.41 init
2 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
3 root 34 19 0 0 0 R 0.0 0.0 0:00.00 ksoftirqd/0
4 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
5 root 10 -5 0 0 0 S 0.0 0.0 0:00.10 events/0
6 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 khelper
7 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kthread
11 root 10 -5 0 0 0 S 0.0 0.0 0:00.07 kblockd/0
12 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 kacpid
101 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 khubd
103 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kseriod
149 root 25 0 0 0 0 S 0.0 0.0 0:00.00 pdflush
150 root 15 0 0 0 0 S 0.0 0.0 0:11.05 pdflush
151 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 kswapd0
152 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 aio/0
605 root 19 -5 0 0 0 S 0.0 0.0 0:00.00 ata/0
607 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 ata_aux
685 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_0
690 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_1
994 root 10 -5 0 0 0 S 0.0 0.0 0:00.66 md0_raid1
998 root 10 -5 0 0 0 S 0.0 0.0 0:13.25 md1_raid1
1002 root 10 -5 0 0 0 S 0.0 0.0 8:03.64 md2_raid1
1072 root 10 -5 0 0 0 S 0.0 0.0 0:00.17 reiserfs/0
1263 root 21 -4 10648 824 348 S 0.0 0.0 0:00.19 udevd
1605 root 18 -5 0 0 0 S 0.0 0.0 0:00.00 kpsmoused
1666 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 kgameportd
1922 root 13 -5 0 0 0 S 0.0 0.0 0:00.00 kmirrord
1971 root 13 -5 0 0 0 S 0.0 0.0 0:00.00 kjournald
2028 daemon 16 0 4820 400 276 S 0.0 0.0 0:00.00 portmap
2263 root 15 0 3856 596 456 S 0.0 0.0 0:00.45 syslogd
2269 root 15 0 2780 392 308 S 0.0 0.0 0:00.02 klogd
2324 Debian-e 15 0 22308 1092 732 S 0.0 0.1 0:00.00 exim4
2351 root 25 0 5912 592 476 S 0.0 0.0 0:00.00 inetd
2373 root 18 0 25840 1256 800 S 0.0 0.1 0:00.00 sshd
2403 statd 24 0 5916 752 616 S 0.0 0.0 0:00.00 rpc.statd
2414 root 15 0 7356 924 416 S 0.0 0.0 0:00.61 mdadm
2424 daemon 18 0 9176 412 276 S 0.0 0.0 0:00.00 atd
2431 root 15 0 11620 936 716 S 0.0 0.0 0:00.07 cron
2456 root 18 0 2656 540 452 S 0.0 0.0 0:00.00 getty
2457 root 18 0 2652 540 452 S 0.0 0.0 0:00.00 getty
2458 root 18 0 2656 540 452 S 0.0 0.0 0:00.00 getty
2459 root 18 0 2652 536 452 S 0.0 0.0 0:00.00 getty
2460 root 18 0 2652 536 452 S 0.0 0.0 0:00.00 getty
2461 root 18 0 2656 540 452 S 0.0 0.0 0:00.00 getty
3742 root 15 0 36788 2728 2160 S 0.0 0.1 0:00.07 sshd
3751 root 15 0 11552 2028 1436 S 0.0 0.1 0:00.04 bash
3759 root 15 0 37064 2796 2128 S 0.0 0.1 0:00.07 sshd
3762 root 15 0 21116 1748 1216 S 0.0 0.1 0:00.08 sftp-server
4181 root 25 0 146m 10m 6388 S 0.0 0.5 0:00.07 php5-cgi
4187 root 25 0 146m 10m 6388 S 0.0 0.5 0:00.06 php5-cgi
4188 root 16 0 150m 23m 15m R 0.0 1.2 0:12.69 php5-cgi
4193 root 15 0 10592 1164 864 R 0.0 0.1 0:00.00 top
Thanks for your help.
on 04.05.2008 10:08
Sam wrote: >> That's still 96. I'd got for maybe 6 * 1 * 4 or even lower. > > I've reduced it to a total of 22 processes max (two 2 * 4 sites and > three 1 * 2 sites) and the process list looks much healthier. However > CPU load is still crazy high: > #uptime > 05:18:32 up 19:39, 1 user, load average: 14.10, 14.60, 15.04 > > Should I reduce it even more? I doubt that would reduce the CPU usage. Have you installed a PHP accelerator already? If Apache was handling the same traffic with less load, something weird is going on. Find out why. ;)
on 04.05.2008 20:52
Olaf van der Spek wrote: > Have you installed a PHP accelerator already? > If Apache was handling the same traffic with less load, something weird > is going on. Find out why. ;) Thanks. I'm running xcache - do you recommend something else? I think my next step will be to distribute the load among multiple back-ends. The apache machine was slightly faster, which is maybe why it was handling the load better (it was a dual core machine, I'm not running lightty on a single core machine). According to my traffic stats, the apache machine was handling ~5x the load with CPU usage at 10-40 (not %, 20-30 processes queued). Currently, the lightty machine's CPU usage is between 8-15 (with significantly less traffic). I guess it's just not the jump in performance I was looking for. By the way, I'm also getting the following errors in my log file... any idea if they're related? 2008-05-04 23:03:28: (server.c.1149) NOTE: a request for URL_HERE timed out after writing 11680 bytes. We waited 360 seconds. If this a problem increase server.max-write-idle 2008-05-04 19:21:01: (request.c.1086) POST-request, but content-length missing -> 411 From what I can tell, there just a result of high server load...
on 04.05.2008 21:10
Sam wrote: > Thanks. I'm running xcache - do you recommend something else? I think I use xcache as well. I don't really know the difference between all the accelerators. > my next step will be to distribute the load among multiple back-ends. > > The apache machine was slightly faster, which is maybe why it was > handling the load better (it was a dual core machine, I'm not running > lightty on a single core machine). > > According to my traffic stats, the apache machine was handling ~5x the > load with CPU usage at 10-40 (not %, 20-30 processes queued). > Currently, the lightty machine's CPU usage is between 8-15 (with > significantly less traffic). I guess it's just not the jump in > performance I was looking for. I'm not sure, but I think load numbers aren't meaningful. CPU usage in % is what's important. But note that switching to FastCGI means mainly a reduction in memory usage due to less PHP processes. It won't really reduce CPU usage (AFAIK). > By the way, I'm also getting the following errors in my log file... any > idea if they're related? > > 2008-05-04 23:03:28: (server.c.1149) NOTE: a request for URL_HERE timed > out after writing 11680 bytes. We waited 360 seconds. If this a problem > increase server.max-write-idle > 2008-05-04 19:21:01: (request.c.1086) POST-request, but content-length > missing -> 411 > > From what I can tell, there just a result of high server load... No, it doesn't depend on load and it's normal.
