[mod_python] Apache/mod_python process sizes. Help wanted.

Greg Fawcett greg at vig.co.nz
Fri Mar 2 23:31:58 EST 2007


Hi Graham -

First, thanks for your frequent and insightful contributions to this list.

1. Ubuntu Linux 6.06 LTS

2. ls -las /usr/lib/apache2/modules/mod_python.so
        84 -rw-r--r-- 1 root root 80072 2006-03-20 21:29
/usr/lib/apache2/modules/mod_python.so

3. ls -las /usr/lib/libpython2.4.so.1.0
        1052 -rw-r--r-- 1 root root 1070260 2006-10-06 21:35
/usr/lib/libpython2.4.so.1.0

4. ldd /usr/lib/apache2/modules/mod_python.so
        linux-gate.so.1 =>  (0xffffe000)
        libpython2.4.so.1.0 => /usr/lib/libpython2.4.so.1.0 (0xb7e1b000)
        libpthread.so.0 => /lib/tls/i686/cmov/libpthread.so.0 (0xb7e08000)
        libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0xb7e05000)
        libutil.so.1 => /lib/tls/i686/cmov/libutil.so.1 (0xb7e02000)
        libm.so.6 => /lib/tls/i686/cmov/libm.so.6 (0xb7de0000)
        libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7cb1000)
        /lib/ld-linux.so.2 (0x80000000)

5. Remove all references to mod_python in sites-available/*
        sudo a2dismod mod_python
        sudo /etc/init.d/apache2 force-reload
        top -U www-data

                top - 17:25:30 up 100 days,  3:01,  1 user,  load average:
0.00, 0.00, 0.00
                Tasks:  76 total,   2 running,  74 sleeping,   0 stopped,
0 zombie
                Cpu(s):  0.0% us,  0.3% sy,  0.0% ni, 99.7% id,  0.0% wa,
0.0% hi,  0.0% si
                Mem:    255824k total,   222824k used,    33000k free,
84288k buffers
                Swap:  1951800k total,     4320k used,  1947480k free,
52072k cached

                  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+
COMMAND
                 8138 www-data  18   0 19628 3236  868 S  0.0  1.3
0:00.00apache2
                 8139 www-data  19   0 19628 3236  868 S  0.0  1.3
0:00.00apache2
                 8140 www-data  19   0 19628 3236  868 S  0.0  1.3
0:00.00apache2
                 8141 www-data  19   0 19628 3236  868 S  0.0  1.3
0:00.00apache2
                 8142 www-data  19   0 19628 3236  868 S  0.0  1.3
0:00.00apache2

6. Add mod_python references again...
        sudo a2enmod mod_python
        sudo /etc/init.d/apache2 force-reload
        top -U www-data

                top - 17:29:05 up 100 days,  3:04,  1 user,  load average:
0.00, 0.00, 0.00
                Tasks:  76 total,   2 running,  74 sleeping,   0 stopped,
0 zombie
                Cpu(s):  0.0% us,  0.0% sy,  0.0% ni, 100.0% id,  0.0% wa,
0.0% hi,  0.0% si
                Mem:    255824k total,   224292k used,    31532k free,
84408k buffers
                Swap:  1951800k total,     4320k used,  1947480k free,
52224k cached

                  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+
COMMAND
                 8223 www-data  20   0 21580 4136  980 S  0.0  1.6
0:00.00apache2
                 8224 www-data  20   0 21580 4136  980 S  0.0  1.6
0:00.00apache2
                 8225 www-data  22   0 21580 4136  980 S  0.0  1.6
0:00.01apache2
                 8226 www-data  21   0 21580 4136  980 S  0.0  1.6
0:00.00apache2
                 8227 www-data  21   0 21580 4136  980 S  0.0  1.6
0:00.00apache2

Hope that makes more sense to you than it does to me. Let me know if there
is anything else I should have done.

Cheers!
Greg Fawcett.

On 03/03/07, Graham Dumpleton <grahamd at dscpl.com.au> wrote:
>
> This email is to enlist the help of mod_python users out there to
> hopefully
> disprove a possible misconception that exists and which is often used
> to malign mod_python and label it a bad solution to the problem of
> hosting
> web applications. If intrigued, keep reading, especially as the
> results may
> end up showing that you might be able to improve the memory use profile
> of your own Apache installation.
>
> The actual claim which would be nice to disprove is that mod_python
> consumes huge amounts of memory.
>
> The first part of this issue I want to look at and get data on is how
> much
> memory is used by an Apache child process immediately after Apache is
> started and before any actual request or additional Python modules are
> imported.
>
> What I am trying to determine is by how much is the process size caused
> not by mod_python, but by the fact that standard Python installations
> tend
> not to be built with the '-enable-shared' option. To be more
> specific, that the
> Python library is static rather than being a shared library.
>
> The reason this can be an issue is that when creating the mod_python.so
> module which is loaded into Apache, if the Python library is static
> rather
> than shared, then libtool will actually incorporate the contents of
> the Python
> static library into mod_python.so itself. This alone will cause the
> size of the
> mod_python.so file to be much larger.
>
> The next factor is in what way particular operating systems will
> treat the
> memory consumed by the mod_python.so file when loaded into memory
> as shared data
>
> The best scenario would be that the memory would be shared between all
> Apache child processes. Thus, even if mod_python.so is bigger than it
> needs to be due to there only being a static Python library, at least
> there is
> only one copy in memory.
>
> The worst case and one which apparently can happen on some platforms,
> is that because the objects in the Python static library are not
> relocatable
> objects, when mapped into memory, the required address translations
> will result in a copy on write of the memory used by the Python library
> object files in memory. This will result in the Python library parts
> of the
> mod_python.so file not being able to be shared across all Apache child
> processes and thus Apache as a whole could consume huge amounts as
> each Apache child process would effectively have its own copy in memory
> of the Python library.
>
> What I am therefore after is a selection of data for various
> platforms and
> the different ways that Python can be built, showing size of
> mod_python.so,
> whether Python library is static or shared, how much memory Apache child
> processes take when mod_python not loaded and then how much when
> it is loaded.
>
> Note that I am not interested in this for Win32 or MacOS systems, but
> the
> Linux/BSD/Solaris/HPUX platforms.
>
> Thus:
>
> 1. Operating System
>
> For example:
>
>    Mac OS X 10.4 (PowerPC)
>
> 2. Size of mod_python.so.
>
> For example:
>
> $ ls -las mod_python.so
> 744 -rwxr-xr-x   1 grahamd  admin  378280 Dec 10 17:24 mod_python.so
>
> 3. Size of Python library. This should exist in the 'lib/pythonX.Y/
> config'
> directory of your Python installation. Ie., at same directory level
> as your
> 'site-packages' directory. It would be called 'libpythonX.Y.a',
> 'libpythonX.Y.so'
> or 'libpythonX.Y.sl'.
>
> MacOS is a bit different as it uses what it calls frameworks, ie., a
> fancy shared
> library, and they are named differently, but as an example:
>
> $ ls -las Python
> 2144 -rwxr-xr-x   1 root  wheel  1097268 Sep 13 12:12 Python
>
> On UNIX platforms, use 'ls -las libpython*'.
>
> 4. Confirm whether mod_python.so uses Python library as static
> library or
> shared library. On UNIX systems you can use the 'ldd' command on
> mod_python.so to determine this.
>
> MacOS is a bit different again here and uses 'otool' command instead and
> output looks a fair bit different, but as example:
>
>         $ otool -L mod_python.so mod_python.so:
>          /usr/lib/libSystem.B.dylib (compatibility version 1.0.0,
> current version 88.1.6)
>          /System/Library/Frameworks/Python.framework/Versions/2.3/
> Python (compatibility version 2.3.0, current version 2.3.5)
>          /System/Library/Frameworks/CoreServices.framework/Versions/A/
> CoreServices (compatibility version 1.0.0, current version 18.0.0)
>          /System/Library/Frameworks/Foundation.framework/Versions/C/
> Foundation (compatibility version 300.0.0, current version 567.27.0)
>          /usr/lib/libmx.A.dylib (compatibility version 1.0.0, current
> version 92.0.0)
>
> 5. Work out how much memory Apache uses when mod_python is not being
> loaded. One can use 'top' for this. Hopefully others may suggest
> better ways.
>
> For example:
>
>    PID COMMAND      %CPU   TIME   #TH #PRTS #MREGS RPRVT  RSHRD
> RSIZE  VSIZE
>    277 httpd        0.0%  0:00.00   1     8    26    72K  1.88M
> 496K  29.3M
>    276 httpd        0.0%  0:00.00   1     8    26    76K  1.88M
> 500K  29.3M
>    275 httpd        0.0%  0:00.00   1     8    26    76K  1.88M
> 500K  29.3M
>    274 httpd        0.0%  0:00.00   1     8    26    76K  1.88M
> 500K  29.3M
>    273 httpd        0.0%  0:00.00   1     8    26    72K  1.88M
> 500K  29.3M
>    272 httpd        0.0%  0:00.01   1    11    27    76K  1.89M
> 1.07M  29.3M
>
> 6. Work out how much memory Apache uses when mod_python is loaded
> but before any requests have occurred. Ensure that no PythonImport
> directives have been used.
>
> For example:
>
>           PID COMMAND      %CPU   TIME   #TH #PRTS #MREGS RPRVT  RSHRD
> RSIZE  VSIZE
>    291 httpd        0.0%  0:00.00   1     8    36   120K  2.77M
> 700K  30.3M
>    290 httpd        0.0%  0:00.00   1     8    36   124K  2.77M
> 704K  30.3M
>    289 httpd        0.0%  0:00.00   1     8    36   124K  2.77M
> 704K  30.3M
>    288 httpd        0.0%  0:00.00   1     8    36   124K  2.77M
> 704K  30.3M
>    287 httpd        0.0%  0:00.00   1     8    36   120K  2.77M
> 704K  30.3M
>    286 httpd        0.0%  0:00.07   1    11    36    40K  2.77M
> 1.81M  30.3M
>
> Thus on MacOS where shared library for Python is used, process specific
> memory use does go up, but no but that much. One sees a bigger jump
> in shared memory use because of shared Python library.
>
> If eager, use a hello world mod_python example and trigger one request
> against it and see how size changes.
>
> 7. If you are also loading lots of other Apache modules, such as
> mod_perl,
> module for PHP etc, list any significant ones so we might understand
> what
> else may be using memory. Ideally, do the tests with as little
> modules loaded
> as possible.
>
> So, what do other people get?
>
> Is that fact that Python installations often only provide a static
> Python
> library the culprit?
>
> Is there a better way of analysing memory use than this?
>
> It would be great if someone has the time and resources to actually
> build
> up two Python versions, one with static and shared libraries, then build
> mod_python.so and compare for each. Being on the same platform
> may help in comparing them. I can't really do this on MacOS and don't
> have access to other platforms to test.
>
> Thanks in advance.
>
> Graham
>
> _______________________________________________
> Mod_python mailing list
> Mod_python at modpython.org
> http://mailman.modpython.org/mailman/listinfo/mod_python
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mm_cfg_has_not_been_edited_to_set_host_domains/pipermail/mod_python/attachments/20070303/5fbe4d54/attachment-0001.html


More information about the Mod_python mailing list