I am using lighttpd 1.4.11 in a low-traffic virtual hosting setup and am having the problem that it seems to consume large amounts of memory and starts filling swap space after running for a day: -- max@mx2:~> ps aux |grep lighttpd www 13612 0.0 46.5 1176388 476588 ? S Sep10 1:25 /usr/local/sbin/lighttpd -f ./lighttpd.conf Server-Status Uptime 1 days 9 hours 25 min 18 s Started at 2006-09-10 16:45:45 absolute (since start) Requests 69 kreq Traffic 1.12 Gbyte average (since start) Requests 0 req/s Traffic 9.72 kbyte/s average (5s sliding average) Requests 1 req/s Traffic 7.13 kbyte/s 5 connections rrhrr lighttpd 1.4.11 Server-Features RegEx Conditionals enabled Network Engine fd-Event-Handler linux-sysepoll Config-File-Settings Loaded Modules indexfile access status fastcgi simple_vhost cgi compress accesslog evasive dirlisting staticfile -- The strange thing is that I also run lighty on a different server for a high-traffic website, yet it does not seem to have this problem. === max@lion:~> ps aux |grep lighttpd nobody 11822 1.0 7.8 175880 156164 ? S Aug05 586:59 /usr/local/sbin/lighttpd -f lighttpd.conf Server-Status Uptime 37 days 3 hours 41 min 9 s Started at 2006-08-05 22:47:08 absolute (since start) Requests 63 Mreq Traffic 696.10 Gbyte average (since start) Requests 19 req/s Traffic 227.38 kbyte/s average (5s sliding average) Requests 13 req/s Traffic 67.49 kbyte/s 48 connections hhhhrhhrrrrrrrrrrrrrrrrhrrrrrrrrrrrrWrrrrrrrrrWh === The difference in configuration is that the first setup uses mod_simple_vhost and runs PHP through CGI instead of FastCGI. Are there any known memory leaks or other problems with these modules?
on 12.09.2006 00:53
on 14.10.2006 19:02
Floris Bos wrote: > I am using lighttpd 1.4.11 in a low-traffic virtual hosting setup and am > having the problem that it seems to consume large amounts of memory and > starts filling swap space after running for a day: > > -- > max@mx2:~> ps aux |grep lighttpd > www 13612 0.0 46.5 1176388 476588 ? S Sep10 1:25 > /usr/local/sbin/lighttpd -f ./lighttpd.conf Which OS and PHP version are you using. I have same problem on CentOS with PHP 5.1.6 but it works well on Fedora Core 4. It also works if I use PHP 5.1.2 So there are possibly some issues with new PHP Fast-CGI Interface, which are also platform specific :(
on 28.10.2006 17:12
Peter Zaitsev wrote:
> Which OS and PHP version are you using.
Have these issues with both Suse and Ubuntu. PHP 5.1.6
I think I have found a possible cause.
Lighty seems to buffer the output of scripts indefinitely.
This means that if one of your users has a php script without proper
error handling among the lines of:
<?
...
while ( !feof($non_existing_fd) )
{
...
}
?>
causing an endless loop outputting "Warning: feof(): supplied argument
is not a valid stream resource" until the execution times out, your
memory will be filled fast.
It seems that this buffer memory is NOT freed, or the memory is not
properly returned to the OS, even after all client
connections to the server have been closed:
root@Host5:/home/max/etc# netstat -anp |grep lightt
tcp 0 0 0.0.0.0:80 0.0.0.0:*
LISTEN 5604/lighttpd
root@Host5:/home/max/etc# ps aux |grep lightt
root 5604 15.7 16.5 690080 688976 ? S 00:28 1:30
/usr/local/sbin/lighttpd -f ./lighttpd.conf
on 02.11.2006 03:51
Ok, figured it out.
Lighttpd does free the memory, but it is never returned to the Operating
system.
In the above situation in which PHP is in an endless loop, lighty seems
to do millions of very small memory allocations (chunks in the
con->write_queue) to buffer the output.
Small memory allocations are placed in a heap by glibc's memory
allocation routines.
Because the top of this heap is fragmented, it cannot return this memory
to the operating system when it is freed. Instead it is placed in a free
list, for reuse by the application.
Large memory allocations above 256 KB in size, are not placed in this
heap but in a seperate (mmap) memory region, and do get returned to the
operating system when they are freed.
I modified chunk.c a bit, so that it combines small chunks into chunks
+/- 512 KB in size:
---
int chunkqueue_append_buffer(chunkqueue *cq, buffer *mem) {
chunk *c;
if (mem->used == 0) return 0;
if (!cq->unused && cq->last && cq->last->type == MEM_CHUNK &&
cq->last->offset == 0 && cq->last->mem->used < 524288)
{
buffer_append_string_buffer(cq->last->mem, mem);
}
else
{
c = chunkqueue_get_unused_chunk(cq);
c->type = MEM_CHUNK;
c->offset = 0;
buffer_copy_string_buffer(c->mem, mem);
chunkqueue_append_chunk(cq, c);
}
return 0;
}
int chunkqueue_append_mem(chunkqueue *cq, const char * mem, size_t len)
{
chunk *c;
if (len == 0) return 0;
if (!cq->unused && cq->last && cq->last->type == MEM_CHUNK &&
cq->last->offset == 0 && cq->last->mem->used < 524288)
{
buffer_append_string_len(cq->last->mem, mem, len-1);
}
else
{
c = chunkqueue_get_unused_chunk(cq);
c->type = MEM_CHUNK;
c->offset = 0;
buffer_copy_string_len(c->mem, mem, len - 1);
chunkqueue_append_chunk(cq, c);
}
return 0;
}
---
Now lighty still consumes a lot of memory DURING the execution of a
broken php script:
# ps aux |grep lighttpd
root 14439 4.3 9.0 375824 374780 ? S 09:52 1:13
/usr/local/sbin/lighttpd -f ./lighttpd.conf
But afterwards it does return the memory to the OS:
# ps aux |grep lighttpd
root 14439 4.1 0.0 2324 1280 ? S 09:52 1:13
/usr/local/sbin/lighttpd -f ./lighttpd.conf
That seems a lot better.
However during the execution of the script, memory usage is still high
and therefore there is a risk that the OS starts swapping or kills
lighty.
There should really be an option to limit the amount of data that is
buffered in the first place.
