[mod_python] Apache/mod_python process sizes. Help wanted.

Graham Dumpleton grahamd at dscpl.com.au
Fri Mar 2 19:14:53 EST 2007

This email is to enlist the help of mod_python users out there to  
disprove a possible misconception that exists and which is often used
to malign mod_python and label it a bad solution to the problem of  
web applications. If intrigued, keep reading, especially as the  
results may
end up showing that you might be able to improve the memory use profile
of your own Apache installation.

The actual claim which would be nice to disprove is that mod_python
consumes huge amounts of memory.

The first part of this issue I want to look at and get data on is how  
memory is used by an Apache child process immediately after Apache is
started and before any actual request or additional Python modules are

What I am trying to determine is by how much is the process size caused
not by mod_python, but by the fact that standard Python installations  
not to be built with the '-enable-shared' option. To be more  
specific, that the
Python library is static rather than being a shared library.

The reason this can be an issue is that when creating the mod_python.so
module which is loaded into Apache, if the Python library is static  
than shared, then libtool will actually incorporate the contents of  
the Python
static library into mod_python.so itself. This alone will cause the  
size of the
mod_python.so file to be much larger.

The next factor is in what way particular operating systems will  
treat the
memory consumed by the mod_python.so file when loaded into memory
as shared data

The best scenario would be that the memory would be shared between all
Apache child processes. Thus, even if mod_python.so is bigger than it
needs to be due to there only being a static Python library, at least  
there is
only one copy in memory.

The worst case and one which apparently can happen on some platforms,
is that because the objects in the Python static library are not  
objects, when mapped into memory, the required address translations
will result in a copy on write of the memory used by the Python library
object files in memory. This will result in the Python library parts  
of the
mod_python.so file not being able to be shared across all Apache child
processes and thus Apache as a whole could consume huge amounts as
each Apache child process would effectively have its own copy in memory
of the Python library.

What I am therefore after is a selection of data for various  
platforms and
the different ways that Python can be built, showing size of  
whether Python library is static or shared, how much memory Apache child
processes take when mod_python not loaded and then how much when
it is loaded.

Note that I am not interested in this for Win32 or MacOS systems, but  
Linux/BSD/Solaris/HPUX platforms.


1. Operating System

For example:

   Mac OS X 10.4 (PowerPC)

2. Size of mod_python.so.

For example:

$ ls -las mod_python.so
744 -rwxr-xr-x   1 grahamd  admin  378280 Dec 10 17:24 mod_python.so

3. Size of Python library. This should exist in the 'lib/pythonX.Y/ 
directory of your Python installation. Ie., at same directory level  
as your
'site-packages' directory. It would be called 'libpythonX.Y.a',  
or 'libpythonX.Y.sl'.

MacOS is a bit different as it uses what it calls frameworks, ie., a  
fancy shared
library, and they are named differently, but as an example:

$ ls -las Python
2144 -rwxr-xr-x   1 root  wheel  1097268 Sep 13 12:12 Python

On UNIX platforms, use 'ls -las libpython*'.

4. Confirm whether mod_python.so uses Python library as static  
library or
shared library. On UNIX systems you can use the 'ldd' command on
mod_python.so to determine this.

MacOS is a bit different again here and uses 'otool' command instead and
output looks a fair bit different, but as example:

	$ otool -L mod_python.so mod_python.so:
         /usr/lib/libSystem.B.dylib (compatibility version 1.0.0,  
current version 88.1.6)
Python (compatibility version 2.3.0, current version 2.3.5)
CoreServices (compatibility version 1.0.0, current version 18.0.0)
Foundation (compatibility version 300.0.0, current version 567.27.0)
         /usr/lib/libmx.A.dylib (compatibility version 1.0.0, current  
version 92.0.0)

5. Work out how much memory Apache uses when mod_python is not being
loaded. One can use 'top' for this. Hopefully others may suggest  
better ways.

For example:

   277 httpd        0.0%  0:00.00   1     8    26    72K  1.88M    
496K  29.3M
   276 httpd        0.0%  0:00.00   1     8    26    76K  1.88M    
500K  29.3M
   275 httpd        0.0%  0:00.00   1     8    26    76K  1.88M    
500K  29.3M
   274 httpd        0.0%  0:00.00   1     8    26    76K  1.88M    
500K  29.3M
   273 httpd        0.0%  0:00.00   1     8    26    72K  1.88M    
500K  29.3M
   272 httpd        0.0%  0:00.01   1    11    27    76K  1.89M   
1.07M  29.3M

6. Work out how much memory Apache uses when mod_python is loaded
but before any requests have occurred. Ensure that no PythonImport
directives have been used.

For example:

   291 httpd        0.0%  0:00.00   1     8    36   120K  2.77M    
700K  30.3M
   290 httpd        0.0%  0:00.00   1     8    36   124K  2.77M    
704K  30.3M
   289 httpd        0.0%  0:00.00   1     8    36   124K  2.77M    
704K  30.3M
   288 httpd        0.0%  0:00.00   1     8    36   124K  2.77M    
704K  30.3M
   287 httpd        0.0%  0:00.00   1     8    36   120K  2.77M    
704K  30.3M
   286 httpd        0.0%  0:00.07   1    11    36    40K  2.77M   
1.81M  30.3M

Thus on MacOS where shared library for Python is used, process specific
memory use does go up, but no but that much. One sees a bigger jump
in shared memory use because of shared Python library.

If eager, use a hello world mod_python example and trigger one request
against it and see how size changes.

7. If you are also loading lots of other Apache modules, such as  
module for PHP etc, list any significant ones so we might understand  
else may be using memory. Ideally, do the tests with as little  
modules loaded
as possible.

So, what do other people get?

Is that fact that Python installations often only provide a static  
library the culprit?

Is there a better way of analysing memory use than this?

It would be great if someone has the time and resources to actually  
up two Python versions, one with static and shared libraries, then build
mod_python.so and compare for each. Being on the same platform
may help in comparing them. I can't really do this on MacOS and don't
have access to other platforms to test.

Thanks in advance.


More information about the Mod_python mailing list