[mod_python] memory leak in mod_python?

stietke at diw.de stietke at diw.de
Tue Apr 13 12:55:34 EST 2004


Hi guys,

thank you for this extremely cool piece of software. I am very
happy with it. However there is one little problem I encountered
while playing with it. On my computer, the apache process seems
to leak memory when used with mod_python in certain configurations.

In particular, it seems to leak memory when I use PythonOption
directives in a <Directory>...</Directory> context.

I tried to figure out where the memory disapears using valgrind,
but the hundreds of lines valgrind spit out left me rather clueless.

Maybe somebody here has any idea what is going on.

Thank you in advance + all the best,
Stefan


summary:
--------
Some mod_python directives, including PythonOption and PythonInterpreter,
seem to cause apache or mod_python to lose a certain, constant amount
of memory for each request mod_python handles. The number of bytes
lost each request seems to scale with the number of directives used.

The apache process seems to lose memory when the mod_python module
is loaded with a certain configuration, independently of the actual
request beeing handled by mod_python. (see results 6 for details)

versions:
---------
	apache 2.0.49
	mod_python 3.1.3
	python 2.3.3
	linux kernel 2.4.19
	libc6 2.3.1
	debian woody with some updates

apache, python and mod_python are compiled from source.

apache is compiled with prefork mpm to make testing easier (no threads),
but tests with worker mpm showed similar results in earlier tests.

httpd.config:
-------------
$ diff conf/httpd.conf conf/httpd-std.conf
119,122c119,122
< StartServers         1
< MinSpareServers      1
< MaxSpareServers      1
< MaxClients           1
---
> StartServers         5
> MinSpareServers      5
> MaxSpareServers     10
> MaxClients         150
219c219
< Listen 127.0.0.1:10080
---
> Listen 80
233d232
< LoadModule python_module
/home/stefan/apache_test/apache_prefork/modules/mod_python.so
362,372d360
<     #
<     # mod_python
<     #
<     AddHandler mod_python .py
<     PythonHandler testdummy
<     #PythonInterpreter pylet_interp
<     #PythonDebug On
<     #PythonOption x a
<     #PythonOption y b
<     #PythonOption z c
<
493c481
< LogLevel debug
---
> LogLevel warn

mod_python handler:
-------------------
$ cat testdummy.py
def handler(req):
        return 0

testprocedure:
--------------
1) warm up: ../apache/bin/ab -n 50000 -k http://localhost:10080/test.py

2) test: cat /proc/2315/status > httpd_status.txt;
	 for x in 1 2 3 4 5; do
		../apache/bin/ab -n 100000 -k http://localhost:10080/test.py;
		cat /proc/2315/status >> httpd_status.txt;
	 done

3) see if the worker process grows in size: grep VmSize: httpd_status.txt

results:
--------
1) with the mod_python configuration from above, no growth.
2) with the line "PythonOption x a" enabled:
	VmSize:     8776 kB
	VmSize:    10348 kB	---> +1572
	VmSize:    11916 kB	---> +1568
	VmSize:    13488 kB	---> +1572
	VmSize:    15056 kB	---> +1568
	VmSize:    16624 kB	---> +1568
   total: ((16624 - 8776) * 1024) / 500000 = 16.07270 bytes/request
3) with the PythonOption replaced with "PythonOption
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
(that is 100*x and 100*a):
	result: exactly the same as 2).
4) with "PythonOption x a" and "PythonOption y b" enabled:
	VmSize:     9956 kB	---> +3924
	VmSize:    13880 kB	---> +3924
	VmSize:    17804 kB	---> +3924
	VmSize:    21728 kB	---> +3932
	VmSize:    25660 kB	---> +3924
	VmSize:    29584 kB
   total: ((29584 - 9956) * 1024) / 500000 = 40.19814 bytes/request
5) with additionally "PythonOption z c" enabled:
	VmSize:    11532 kB
	VmSize:    18620 kB	---> +7088
	VmSize:    25708 kB	---> +7088
	VmSize:    32792 kB	---> +7084
	VmSize:    39880 kB	---> +7088
	VmSize:    46968 kB	---> +7088
   total: ((46968 - 11532) * 1024) / 500000 = 72.57292 bytes/request
6) with all PythonOptions disabled, but with
   "PythonInterpreter pylet_interp" enabled:
	result: exactly the same as 2).
   BUT: after this test, i ran
   ../apache/bin/ab -n 300000 -k http://localhost:10080/index.html.en
   without restarting apache, and the VmSize grew again to 21328 kB.
   ((21328 - 16624) * 1024) / 300000 = 16.05632
   Apparently, apache loses memory already when the mod_python
   module is loaded but does not handle the request at hand.





More information about the Mod_python mailing list