[mod_python] Apache/mod_python process sizes. Help wanted.

Jim Gallacher jpg at jgassociates.ca
Sat Mar 3 19:31:34 EST 2007


Graham Dumpleton wrote:
> 
> On 04/03/2007, at 9:30 AM, Graham Dumpleton wrote:
> 
>> The result for mod_python request is (pid 405):
>>
>>   410 httpd        0.0%  0:00.00   1     8    39   212K  2.87M   888K  
>> 30.4M
>>   408 httpd        0.0%  0:00.00   1     8    39   212K  2.87M   764K  
>> 30.4M
>>   407 httpd        0.0%  0:00.00   1     8    39   216K  2.87M   768K  
>> 30.4M
>>   406 httpd        0.0%  0:00.00   1     8    39   216K  2.87M   768K  
>> 30.4M
>>   405 httpd        0.0%  0:00.37   1    10    86  2.40M  2.75M  3.94M  
>> 32.0M
>>   404 httpd        0.0%  0:00.00   1     8    39   212K  2.87M   768K  
>> 30.4M
>>   403 httpd        0.0%  0:00.06   1    11    39    32K  2.87M  1.30M  
>> 30.4M
>>
>> Thus, private memory use by Apache child process jumps up to 2.40M.
>> Ie., about an extra 2.2M.
>>
>> For a mod_wsgi request however, the result is (pid 407):
>>
>>   410 httpd        0.0%  0:00.00   1     8    39   212K  2.87M   816K  
>> 30.4M
>>   408 httpd        0.0%  0:00.00   1     8    39   212K  2.87M   764K  
>> 30.4M
>>   407 httpd        0.0%  0:00.01   1    10    39   568K  2.87M  1.64M+ 
>> 30.4M
>>   406 httpd        0.0%  0:00.00   1     8    39   216K  2.87M   768K  
>> 30.4M
>>   405 httpd        0.0%  0:00.37   1    10    86  2.40M  2.75M  3.95M  
>> 32.0M
>>   404 httpd        0.0%  0:00.00   1     8    39   212K  2.87M   768K  
>> 30.4M
>>   403 httpd        0.0%  0:00.06   1    11    39    32K  2.87M  1.30M  
>> 30.4M
>>
>> That is, private memory use by Apache child process only jumps up to 
>> 568K.
>> Ie., about an extra 350K.
>>
>> This is quite a big difference and really makes one wander what 
>> mod_python
>> is actually doing. This will also have to be investigated, but 
>> probably still
>> want to pursue this shared versus static library thing first.
> 
> Part of the jump is the imports in mod_python.apache:
> 
> import sys
> import traceback
> import time
> import os
> import pdb
> import stat
> import imp
> import types
> import cgi
> import _apache
> import threading
> 
> I had been thinking they would have been done when mod_python loaded
> for first interpreter, but they aren't done until time of first request.
> 
> If I import the same modules into mod_wsgi example, then get (pid 429):
> 
>   432 httpd        0.0%  0:00.00   1     8    39   212K  2.87M   904K  
> 30.4M
>   430 httpd        0.0%  0:00.00   1     8    39   212K  2.87M   776K  
> 30.4M
>   429 httpd        0.0%  0:00.18   1    10    67  1.95M  2.88M  3.33M  
> 31.5M
>   428 httpd        0.0%  0:00.00   1     8    39   216K  2.87M   780K  
> 30.4M
>   427 httpd        0.0%  0:00.00   1     8    39   216K  2.87M   780K  
> 30.4M
>   426 httpd        0.0%  0:00.00   1     8    39   212K  2.87M   780K  
> 30.4M
>   425 httpd        0.0%  0:00.07   1    11    39    32K  2.87M  1.34M  
> 30.4M
> 
> Thus these modules alone causes it to jump up to 1.95M. This works out
> about an extra 1.4MB. Because mod_wsgi is all in C code aren't relying on
> any Python modules so base overhead is less.
> 
> Interesting. How many of these modules could be avoided in mod_python
> if we tried hard?

I have no idea, but it's worth a look.

Attached is a fun little script that uses pmap to examine the changes. 
You'll need to modify the $URL in the script so it points to an 
appropriate mod_python script.

Results for before and after running a helloworld.py script:

--- before/30006.log	2007-03-03 18:59:20.000000000 -0500
+++ after/30006.log	2007-03-03 18:59:20.000000000 -0500
@@ -2,7 +2,40 @@
  Address   Kbytes Mode  Offset           Device    Mapping
  08048000     364 r-x-- 0000000000000000 003:00004 apache2
  080a3000      12 rw--- 000000000005a000 003:00004 apache2
-080a6000    1792 rw--- 00000000080a6000 000:00000   [ anon ]
+080a6000    2592 rw--- 00000000080a6000 000:00000   [ anon ]
+a9cd4000       4 r-x-- 0000000000000000 003:00004 _weakref.so
+a9cd5000       4 rw--- 0000000000000000 003:00004 _weakref.so
+a9cd6000       4 r-x-- 0000000000000000 003:00004 _bisect.so
+a9cd7000       4 rw--- 0000000000001000 003:00004 _bisect.so
+a9cd8000     260 rw--- 00000000a9cd8000 000:00000   [ anon ]
+a9d19000      12 r-x-- 0000000000000000 003:00004 sha.so
+a9d1c000       4 rw--- 0000000000002000 003:00004 sha.so
+a9d1d000       8 r-x-- 0000000000000000 003:00004 md5.so
+a9d1f000       4 rw--- 0000000000002000 003:00004 md5.so
+a9d20000     388 rw--- 00000000a9d20000 000:00000   [ anon ]
+a9d81000      56 r-x-- 0000000000000000 003:00004 datetime.so
+a9d8f000      12 rw--- 000000000000d000 003:00004 datetime.so
+a9d92000     260 rw--- 00000000a9d92000 000:00000   [ anon ]
+a9dd3000      16 r-x-- 0000000000000000 003:00004 struct.so
+a9dd7000       4 rw--- 0000000000004000 003:00004 struct.so
+a9dd8000      12 r-x-- 0000000000000000 003:00004 collections.so
+a9ddb000       4 rw--- 0000000000003000 003:00004 collections.so
+a9ddc000       8 r-x-- 0000000000000000 003:00004 fcntl.so
+a9dde000       4 rw--- 0000000000002000 003:00004 fcntl.so
+a9ddf000       8 r-x-- 0000000000000000 003:00004 _random.so
+a9de1000       4 rw--- 0000000000002000 003:00004 _random.so
+a9de2000      12 r-x-- 0000000000000000 003:00004 binascii.so
+a9de5000       8 rw--- 0000000000002000 003:00004 binascii.so
+a9de7000      12 r-x-- 0000000000000000 003:00004 math.so
+a9dea000       4 rw--- 0000000000002000 003:00004 math.so
+a9deb000     260 rw--- 00000000a9deb000 000:00000   [ anon ]
+a9e2c000      12 r-x-- 0000000000000000 003:00004 _ssl.so
+a9e2f000       4 rw--- 0000000000003000 003:00004 _ssl.so
+a9e30000      36 r-x-- 0000000000000000 003:00004 _socket.so
+a9e39000      12 rw--- 0000000000009000 003:00004 _socket.so
+a9e3c000      20 r-x-- 0000000000000000 003:00004 itertools.so
+a9e41000       8 rw--- 0000000000004000 003:00004 itertools.so
+a9e43000     260 rw--- 00000000a9e43000 000:00000   [ anon ]
  a9e84000      36 r-x-- 0000000000000000 003:00004 libgcc_s.so.1
  a9e8d000       4 rw--- 0000000000009000 003:00004 libgcc_s.so.1
  a9e8e000       4 ----- 00000000a9e8e000 000:00000   [ anon ]
@@ -70,6 +103,12 @@
  b777b000     888 r-x-- 0000000000000000 003:00004 libpython2.4.so.1.0
  b7859000     156 rw--- 00000000000de000 003:00004 libpython2.4.so.1.0
  b7880000      16 rw--- 00000000b7880000 000:00000   [ anon ]
+b7884000       4 r-x-- 0000000000000000 003:00004 syslog.so
+b7885000       4 rw--- 0000000000001000 003:00004 syslog.so
+b7886000      16 r-x-- 0000000000000000 003:00004 strop.so
+b788a000       8 rw--- 0000000000003000 003:00004 strop.so
+b788c000      12 r-x-- 0000000000000000 003:00004 time.so
+b788f000       8 rw--- 0000000000002000 003:00004 time.so
  b7891000      92 r-x-- 0000000000000000 003:00004 mod_python.so
  b78a8000      12 rw--- 0000000000017000 003:00004 mod_python.so
  b78ab000     136 r-x-- 0000000000000000 003:00004 
libsvn_fs_base-1.so.0.0.0
@@ -82,6 +121,8 @@
  b7914000       4 rw--- 0000000000006000 003:00004 libsvn_delta-1.so.0.0.0
  b7915000     112 r-x-- 0000000000000000 003:00004 libsvn_repos-1.so.0.0.0
  b7931000       4 rw--- 000000000001b000 003:00004 libsvn_repos-1.so.0.0.0
+b7932000      12 r-x-- 0000000000000000 003:00004 cStringIO.so
+b7935000       4 rw--- 0000000000003000 003:00004 cStringIO.so
  b7936000       8 r-x-- 0000000000000000 003:00004 mod_userdir.so
  b7938000       4 rw--- 0000000000001000 003:00004 mod_userdir.so
  b7939000       8 r-x-- 0000000000000000 003:00004 libutil-2.3.6.so
@@ -161,4 +202,4 @@
  bfb4b000      80 rwx-- 00000000bfb4b000 000:00000   [ stack ]
  bfb5f000       8 rw--- 00000000bfb5f000 000:00000   [ anon ]
  ffffe000       4 ----- 0000000000000000 000:00000   [ anon ]
-mapped: 232452K    writeable/private: 224444K    shared: 208K
+mapped: 235048K    writeable/private: 226776K    shared: 208K


So we are increasing memory usage by 4928K (2596K mapped + 2332K 
writable/private) which is in line with what Graham found using top.

Jim

-------------- next part --------------
A non-text attachment was scrubbed...
Name: quickie-profile.sh
Type: application/x-shellscript
Size: 510 bytes
Desc: not available
Url : http://mm_cfg_has_not_been_edited_to_set_host_domains/pipermail/mod_python/attachments/20070303/038809bc/quickie-profile-0001.bin


More information about the Mod_python mailing list