|
Greg Fawcett
greg at vig.co.nz
Fri Mar 2 23:31:58 EST 2007
Hi Graham -
First, thanks for your frequent and insightful contributions to this list.
1. Ubuntu Linux 6.06 LTS
2. ls -las /usr/lib/apache2/modules/mod_python.so
84 -rw-r--r-- 1 root root 80072 2006-03-20 21:29
/usr/lib/apache2/modules/mod_python.so
3. ls -las /usr/lib/libpython2.4.so.1.0
1052 -rw-r--r-- 1 root root 1070260 2006-10-06 21:35
/usr/lib/libpython2.4.so.1.0
4. ldd /usr/lib/apache2/modules/mod_python.so
linux-gate.so.1 => (0xffffe000)
libpython2.4.so.1.0 => /usr/lib/libpython2.4.so.1.0 (0xb7e1b000)
libpthread.so.0 => /lib/tls/i686/cmov/libpthread.so.0 (0xb7e08000)
libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0xb7e05000)
libutil.so.1 => /lib/tls/i686/cmov/libutil.so.1 (0xb7e02000)
libm.so.6 => /lib/tls/i686/cmov/libm.so.6 (0xb7de0000)
libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7cb1000)
/lib/ld-linux.so.2 (0x80000000)
5. Remove all references to mod_python in sites-available/*
sudo a2dismod mod_python
sudo /etc/init.d/apache2 force-reload
top -U www-data
top - 17:25:30 up 100 days, 3:01, 1 user, load average:
0.00, 0.00, 0.00
Tasks: 76 total, 2 running, 74 sleeping, 0 stopped,
0 zombie
Cpu(s): 0.0% us, 0.3% sy, 0.0% ni, 99.7% id, 0.0% wa,
0.0% hi, 0.0% si
Mem: 255824k total, 222824k used, 33000k free,
84288k buffers
Swap: 1951800k total, 4320k used, 1947480k free,
52072k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
8138 www-data 18 0 19628 3236 868 S 0.0 1.3
0:00.00apache2
8139 www-data 19 0 19628 3236 868 S 0.0 1.3
0:00.00apache2
8140 www-data 19 0 19628 3236 868 S 0.0 1.3
0:00.00apache2
8141 www-data 19 0 19628 3236 868 S 0.0 1.3
0:00.00apache2
8142 www-data 19 0 19628 3236 868 S 0.0 1.3
0:00.00apache2
6. Add mod_python references again...
sudo a2enmod mod_python
sudo /etc/init.d/apache2 force-reload
top -U www-data
top - 17:29:05 up 100 days, 3:04, 1 user, load average:
0.00, 0.00, 0.00
Tasks: 76 total, 2 running, 74 sleeping, 0 stopped,
0 zombie
Cpu(s): 0.0% us, 0.0% sy, 0.0% ni, 100.0% id, 0.0% wa,
0.0% hi, 0.0% si
Mem: 255824k total, 224292k used, 31532k free,
84408k buffers
Swap: 1951800k total, 4320k used, 1947480k free,
52224k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
8223 www-data 20 0 21580 4136 980 S 0.0 1.6
0:00.00apache2
8224 www-data 20 0 21580 4136 980 S 0.0 1.6
0:00.00apache2
8225 www-data 22 0 21580 4136 980 S 0.0 1.6
0:00.01apache2
8226 www-data 21 0 21580 4136 980 S 0.0 1.6
0:00.00apache2
8227 www-data 21 0 21580 4136 980 S 0.0 1.6
0:00.00apache2
Hope that makes more sense to you than it does to me. Let me know if there
is anything else I should have done.
Cheers!
Greg Fawcett.
On 03/03/07, Graham Dumpleton <grahamd at dscpl.com.au> wrote:
>
> This email is to enlist the help of mod_python users out there to
> hopefully
> disprove a possible misconception that exists and which is often used
> to malign mod_python and label it a bad solution to the problem of
> hosting
> web applications. If intrigued, keep reading, especially as the
> results may
> end up showing that you might be able to improve the memory use profile
> of your own Apache installation.
>
> The actual claim which would be nice to disprove is that mod_python
> consumes huge amounts of memory.
>
> The first part of this issue I want to look at and get data on is how
> much
> memory is used by an Apache child process immediately after Apache is
> started and before any actual request or additional Python modules are
> imported.
>
> What I am trying to determine is by how much is the process size caused
> not by mod_python, but by the fact that standard Python installations
> tend
> not to be built with the '-enable-shared' option. To be more
> specific, that the
> Python library is static rather than being a shared library.
>
> The reason this can be an issue is that when creating the mod_python.so
> module which is loaded into Apache, if the Python library is static
> rather
> than shared, then libtool will actually incorporate the contents of
> the Python
> static library into mod_python.so itself. This alone will cause the
> size of the
> mod_python.so file to be much larger.
>
> The next factor is in what way particular operating systems will
> treat the
> memory consumed by the mod_python.so file when loaded into memory
> as shared data
>
> The best scenario would be that the memory would be shared between all
> Apache child processes. Thus, even if mod_python.so is bigger than it
> needs to be due to there only being a static Python library, at least
> there is
> only one copy in memory.
>
> The worst case and one which apparently can happen on some platforms,
> is that because the objects in the Python static library are not
> relocatable
> objects, when mapped into memory, the required address translations
> will result in a copy on write of the memory used by the Python library
> object files in memory. This will result in the Python library parts
> of the
> mod_python.so file not being able to be shared across all Apache child
> processes and thus Apache as a whole could consume huge amounts as
> each Apache child process would effectively have its own copy in memory
> of the Python library.
>
> What I am therefore after is a selection of data for various
> platforms and
> the different ways that Python can be built, showing size of
> mod_python.so,
> whether Python library is static or shared, how much memory Apache child
> processes take when mod_python not loaded and then how much when
> it is loaded.
>
> Note that I am not interested in this for Win32 or MacOS systems, but
> the
> Linux/BSD/Solaris/HPUX platforms.
>
> Thus:
>
> 1. Operating System
>
> For example:
>
> Mac OS X 10.4 (PowerPC)
>
> 2. Size of mod_python.so.
>
> For example:
>
> $ ls -las mod_python.so
> 744 -rwxr-xr-x 1 grahamd admin 378280 Dec 10 17:24 mod_python.so
>
> 3. Size of Python library. This should exist in the 'lib/pythonX.Y/
> config'
> directory of your Python installation. Ie., at same directory level
> as your
> 'site-packages' directory. It would be called 'libpythonX.Y.a',
> 'libpythonX.Y.so'
> or 'libpythonX.Y.sl'.
>
> MacOS is a bit different as it uses what it calls frameworks, ie., a
> fancy shared
> library, and they are named differently, but as an example:
>
> $ ls -las Python
> 2144 -rwxr-xr-x 1 root wheel 1097268 Sep 13 12:12 Python
>
> On UNIX platforms, use 'ls -las libpython*'.
>
> 4. Confirm whether mod_python.so uses Python library as static
> library or
> shared library. On UNIX systems you can use the 'ldd' command on
> mod_python.so to determine this.
>
> MacOS is a bit different again here and uses 'otool' command instead and
> output looks a fair bit different, but as example:
>
> $ otool -L mod_python.so mod_python.so:
> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0,
> current version 88.1.6)
> /System/Library/Frameworks/Python.framework/Versions/2.3/
> Python (compatibility version 2.3.0, current version 2.3.5)
> /System/Library/Frameworks/CoreServices.framework/Versions/A/
> CoreServices (compatibility version 1.0.0, current version 18.0.0)
> /System/Library/Frameworks/Foundation.framework/Versions/C/
> Foundation (compatibility version 300.0.0, current version 567.27.0)
> /usr/lib/libmx.A.dylib (compatibility version 1.0.0, current
> version 92.0.0)
>
> 5. Work out how much memory Apache uses when mod_python is not being
> loaded. One can use 'top' for this. Hopefully others may suggest
> better ways.
>
> For example:
>
> PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD
> RSIZE VSIZE
> 277 httpd 0.0% 0:00.00 1 8 26 72K 1.88M
> 496K 29.3M
> 276 httpd 0.0% 0:00.00 1 8 26 76K 1.88M
> 500K 29.3M
> 275 httpd 0.0% 0:00.00 1 8 26 76K 1.88M
> 500K 29.3M
> 274 httpd 0.0% 0:00.00 1 8 26 76K 1.88M
> 500K 29.3M
> 273 httpd 0.0% 0:00.00 1 8 26 72K 1.88M
> 500K 29.3M
> 272 httpd 0.0% 0:00.01 1 11 27 76K 1.89M
> 1.07M 29.3M
>
> 6. Work out how much memory Apache uses when mod_python is loaded
> but before any requests have occurred. Ensure that no PythonImport
> directives have been used.
>
> For example:
>
> PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD
> RSIZE VSIZE
> 291 httpd 0.0% 0:00.00 1 8 36 120K 2.77M
> 700K 30.3M
> 290 httpd 0.0% 0:00.00 1 8 36 124K 2.77M
> 704K 30.3M
> 289 httpd 0.0% 0:00.00 1 8 36 124K 2.77M
> 704K 30.3M
> 288 httpd 0.0% 0:00.00 1 8 36 124K 2.77M
> 704K 30.3M
> 287 httpd 0.0% 0:00.00 1 8 36 120K 2.77M
> 704K 30.3M
> 286 httpd 0.0% 0:00.07 1 11 36 40K 2.77M
> 1.81M 30.3M
>
> Thus on MacOS where shared library for Python is used, process specific
> memory use does go up, but no but that much. One sees a bigger jump
> in shared memory use because of shared Python library.
>
> If eager, use a hello world mod_python example and trigger one request
> against it and see how size changes.
>
> 7. If you are also loading lots of other Apache modules, such as
> mod_perl,
> module for PHP etc, list any significant ones so we might understand
> what
> else may be using memory. Ideally, do the tests with as little
> modules loaded
> as possible.
>
> So, what do other people get?
>
> Is that fact that Python installations often only provide a static
> Python
> library the culprit?
>
> Is there a better way of analysing memory use than this?
>
> It would be great if someone has the time and resources to actually
> build
> up two Python versions, one with static and shared libraries, then build
> mod_python.so and compare for each. Being on the same platform
> may help in comparing them. I can't really do this on MacOS and don't
> have access to other platforms to test.
>
> Thanks in advance.
>
> Graham
>
> _______________________________________________
> Mod_python mailing list
> Mod_python at modpython.org
> http://mailman.modpython.org/mailman/listinfo/mod_python
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mm_cfg_has_not_been_edited_to_set_host_domains/pipermail/mod_python/attachments/20070303/5fbe4d54/attachment-0001.html
|