[mod_python] Apache/mod_python process sizes. Help wanted.

On 04/03/2007, at 9:30 AM, Graham Dumpleton wrote:

> The result for mod_python request is (pid 405):
>
>   410 httpd        0.0%  0:00.00   1     8    39   212K  2.87M    
> 888K  30.4M
>   408 httpd        0.0%  0:00.00   1     8    39   212K  2.87M    
> 764K  30.4M
>   407 httpd        0.0%  0:00.00   1     8    39   216K  2.87M    
> 768K  30.4M
>   406 httpd        0.0%  0:00.00   1     8    39   216K  2.87M    
> 768K  30.4M
>   405 httpd        0.0%  0:00.37   1    10    86  2.40M  2.75M   
> 3.94M  32.0M
>   404 httpd        0.0%  0:00.00   1     8    39   212K  2.87M    
> 768K  30.4M
>   403 httpd        0.0%  0:00.06   1    11    39    32K  2.87M   
> 1.30M  30.4M
>
> Thus, private memory use by Apache child process jumps up to 2.40M.
> Ie., about an extra 2.2M.
>
> For a mod_wsgi request however, the result is (pid 407):
>
>   410 httpd        0.0%  0:00.00   1     8    39   212K  2.87M    
> 816K  30.4M
>   408 httpd        0.0%  0:00.00   1     8    39   212K  2.87M    
> 764K  30.4M
>   407 httpd        0.0%  0:00.01   1    10    39   568K  2.87M   
> 1.64M+ 30.4M
>   406 httpd        0.0%  0:00.00   1     8    39   216K  2.87M    
> 768K  30.4M
>   405 httpd        0.0%  0:00.37   1    10    86  2.40M  2.75M   
> 3.95M  32.0M
>   404 httpd        0.0%  0:00.00   1     8    39   212K  2.87M    
> 768K  30.4M
>   403 httpd        0.0%  0:00.06   1    11    39    32K  2.87M   
> 1.30M  30.4M
>
> That is, private memory use by Apache child process only jumps up  
> to 568K.
> Ie., about an extra 350K.
>
> This is quite a big difference and really makes one wander what  
> mod_python
> is actually doing. This will also have to be investigated, but  
> probably still
> want to pursue this shared versus static library thing first.

Part of the jump is the imports in mod_python.apache:

import sys
import traceback
import time
import os
import pdb
import stat
import imp
import types
import cgi
import _apache
import threading

I had been thinking they would have been done when mod_python loaded
for first interpreter, but they aren't done until time of first request.

If I import the same modules into mod_wsgi example, then get (pid 429):

   432 httpd        0.0%  0:00.00   1     8    39   212K  2.87M    
904K  30.4M
   430 httpd        0.0%  0:00.00   1     8    39   212K  2.87M    
776K  30.4M
   429 httpd        0.0%  0:00.18   1    10    67  1.95M  2.88M   
3.33M  31.5M
   428 httpd        0.0%  0:00.00   1     8    39   216K  2.87M    
780K  30.4M
   427 httpd        0.0%  0:00.00   1     8    39   216K  2.87M    
780K  30.4M
   426 httpd        0.0%  0:00.00   1     8    39   212K  2.87M    
780K  30.4M
   425 httpd        0.0%  0:00.07   1    11    39    32K  2.87M   
1.34M  30.4M

Thus these modules alone causes it to jump up to 1.95M. This works out
about an extra 1.4MB. Because mod_wsgi is all in C code aren't  
relying on
any Python modules so base overhead is less.

Interesting. How many of these modules could be avoided in mod_python
if we tried hard?

Graham