[mod_python] mod_python + mod_authz_host w/o mod_auth_digest causes ~1Mb leak per apache graceful restart

Aseem Mohanty aseem at metaweb.com
Thu Jun 28 17:40:16 EDT 2007


Jim Gallacher wrote:
> I've confirmed that mod_python leaks on graceful restart, but I don't 
> see the dependency on the auth* modules. Apache always seems to leak 
> when mod_python is loaded, regardless of the other modules. My test 
> setup is a little different: On Debian with apache 2.2.3, mpm-worker, 
> mod_python 3.3.1 running within VMWare.

Hmm. The reason I felt that the dependency existed was the difference in the 
size of the leak when the two mod_auth* modules were loaded. In addition, the 
use of mod_authz_host alone did not leak at all, which led me to this 
conclusion. I will try out your config and see if the leak size remains the same 
as just using mod_python only.

> 
> My gut tells me that this may be related to 
> http://issues.apache.org/jira/browse/MODPYTHON-109.
> 
> This issue also reminds me of 
> http://issues.apache.org/jira/browse/MODPYTHON-195.
> 
> I think at this point we should log this in JIRA and move the discussion 
> over to python-dev.
> 
> The obvious (but suboptimal) temporary solution for Aseem is to avoid 
> using graceful restart.

They sure look like good candidates, so should I be looking at mod_wsgi in 
daemon mode (we are serving WSGI apps via an adapter). The reason I ask is that 
we need the graceful restarts and cannot really do without it for now.

Thanks.
AM

> 
> Jim
> 
> Aseem Mohanty wrote:
>> Graham Dumpleton wrote:
>>> Is the version of mod_python you are using one you compiled yourself,
>>> or one from Ubuntu packages?
>>
>> I have used both and it does not seem to matter. When compiling from 
>> source, I did the following:
>> ./configure --prefix=/opt --with-apxs=/opt/share/apache2/bin/apxs
>> make install_dso
>> to use the apache build from source. Apache was build with the 
>> following config:
>> ./configure --prefix=/opt --enable-modules=all 
>> --enable-mods-shared=all --enable-so --with-included-apr
>>
>>>
>>> Is the version of Python you are using one you compiled yourself, or
>>> one from Ubuntu packages?
>>
>> Python-2.5.1 is the one from the ubuntu packages.05:22 PM $ python
>> Python 2.5.1 (r251:54863, May  2 2007, 16:56:35)
>> [GCC 4.1.2 (Ubuntu 4.1.2-0ubuntu4)] on linux2
>> Type "help", "copyright", "credits" or "license" for more information.
>>
>>>
>>> What is the size of mod_python.so? Does it use a shared Python library
>>> or static? Use 'ldd' on mod_python.so to see dependencies.
>>
>> mod_python from ubuntu packages:
>> v 3.2.10
>> 05:24 PM $ ll mod_python.so
>> -rw-r--r-- 1 root root 99K 2007-01-16 04:44 mod_python.so
>> 05:24 PM $ ldd mod_python.so
>>         linux-gate.so.1 =>  (0xffffe000)
>>         libpython2.5.so.1.0 => /usr/lib/libpython2.5.so.1.0 (0xb7e4f000)
>>         libpthread.so.0 => /lib/tls/i686/cmov/libpthread.so.0 
>> (0xb7e38000)
>>         libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0xb7e33000)
>>         libutil.so.1 => /lib/tls/i686/cmov/libutil.so.1 (0xb7e2f000)
>>         libm.so.6 => /lib/tls/i686/cmov/libm.so.6 (0xb7e08000)
>>         libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7cc7000)
>>         /lib/ld-linux.so.2 (0x80000000)
>>
>> mod_python built from source:
>> v 3.3.1
>> 05:25 PM $ ll mod_python.so
>> -rwxr-xr-x 1 aseem aseem 343K 2007-06-27 09:50 mod_python.so*
>> aseem @ narada: /opt/share/apache2/modules
>> 05:25 PM $ ldd mod_python.so
>>         linux-gate.so.1 =>  (0xffffe000)
>>         libpython2.5.so.1.0 => /usr/lib/libpython2.5.so.1.0 (0xb7e30000)
>>         libpthread.so.0 => /lib/tls/i686/cmov/libpthread.so.0 
>> (0xb7e19000)
>>         libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0xb7e14000)
>>         libutil.so.1 => /lib/tls/i686/cmov/libutil.so.1 (0xb7e10000)
>>         libm.so.6 => /lib/tls/i686/cmov/libm.so.6 (0xb7de9000)
>>         libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7ca8000)
>>         /lib/ld-linux.so.2 (0x80000000)
>>
>> In both cases they do seem to use shared python libs.
>>
>> Thanks.
>> AM
>>
>>>
>>> Graham
>>>
>>> On 28/06/07, Aseem Mohanty <aseem at metaweb.com> wrote:
>>>> Yes, hence the exotic phraseology. Results for the various 
>>>> combinations are below:
>>>>
>>>> 1. mod_python only, no mod_auth* modules:                   leaks
>>>> 2. mod_auth_digest only, no other mod_auth*, no mod_python: does not 
>>>> leak
>>>> 3. mod_authz_host only, no other mod_auth*, no mod_python:  does not 
>>>> leak
>>>> 4. mod_python + mod_authz_host, no mod_auth_digest:         leaks
>>>> 5. mod_python + mod_auth_digest, no mod_authz_host:         leaks < 
>>>> (1) or (4)
>>>> 6. mod_python + mod_auth_digest + mod_authz_host:           does not 
>>>> leak
>>>>
>>>> NOTE: 'does not leak' implies ~50-100K leakage per graceful restart
>>>>        (5) leaks ~500-600K per graceful restart
>>>>        zero load on server
>>>>
>>>> As far as I can tell there is definitely some sinister interaction 
>>>> going on
>>>> between these three modules. Unfortunately not being conversant with 
>>>> apache
>>>> module development, I cannot really say what.
>>>>
>>>> Thanks.
>>>> AM
>>>>
>>>>
>>>> Jim Gallacher wrote:
>>>> > Aseem Mohanty wrote:
>>>> >> This may sound quite exotic
>>>> >
>>>> >
>>>> > Why yes, it does? Interaction between these modules resulting in a
>>>> > memory leak sure sounds weird. Did you try it with *only* mod_python
>>>> > loaded? (ie, without mod_authz_host)
>>>> >
>>>> > Jim
>>>> >
>>>> >> but here it is:
>>>> >>
>>>> >> In a vanilla apache2.2.x configuration if I only load mod_authz_host
>>>> >> and none of
>>>> >> the other mod_auth* modules, every graceful restart causes ~1Mb leak
>>>> >> in the
>>>> >> parent process. This is with mod_python loaded. Without 
>>>> mod_python the
>>>> >> memory profile remains unchanged.
>>>> >>
>>>> >> Steps to reproduce:
>>>> >> Gentoo:
>>>> >> 1. In /etc/apache2/httpd.conf comment out all "LoadModule auth*"
>>>> >> directives except
>>>> >> LoadModule authz_host_module modules/mod_authz_host.so
>>>> >>
>>>> >> Ubuntu Feisty (repos):
>>>> >> 1. In /etc/apache2/mods-enabled rename auth*.load files except
>>>> >> authz_host.load
>>>> >> to auth*.load.not
>>>> >>
>>>> >> Ubuntu Feisty (httpd built from source):
>>>> >> 1. In conf/httpd.conf comment out all "LoadModule auth*" directives
>>>> >> except
>>>> >> LoadModule authz_host_module modules/mod_authz_host.so
>>>> >>
>>>> >> 2. Start apache
>>>> >> 3. Track apache parent process: top -d1 -p $(pgrep -o apache/httpd)
>>>> >> 4. Hit apache instances with graceful restarts:
>>>> >> for i in $(seq 150); do echo $i; kill -USR1 $(pgrep -o 
>>>> apache/httpd);
>>>> >> sleep 2; done
>>>> >>
>>>> >> In this case the parent will leak ~1Mb per restart. If however in 
>>>> each
>>>> >> of the
>>>> >> configurations mod_auth_digest.so is also loaded the leak is only
>>>> >> about 150-200K
>>>> >> or so per graceful restart. This is on a server with 0 load. In my
>>>> >> case it
>>>> >> caused apache instances that were restarted every 55 minutes to grow
>>>> >> up to 300Mb
>>>> >> in a week.
>>>> >>
>>>> >> Tested on:
>>>> >> 1. Ubuntu Feisty (built from source httpd-2.2.4, mod_python-3.3.1):
>>>> >> ./configure --prefix=/opt --enable-modules=all
>>>> >> --enable-mods-shared=all --enable-so --with-included-apr
>>>> >> Server version: Apache/2.2.4 (Unix)
>>>> >> Server built:   Jun 27 2007 09:34:16
>>>> >> Server's Module Magic Number: 20051115:4
>>>> >> Server loaded:  APR 1.2.8, APR-Util 1.2.8
>>>> >> Compiled using: APR 1.2.8, APR-Util 1.2.8
>>>> >> Architecture:   32-bit
>>>> >> Server MPM:     Prefork
>>>> >>   threaded:     no
>>>> >>     forked:     yes (variable process count)
>>>> >> Server compiled with....
>>>> >>  -D APACHE_MPM_DIR="server/mpm/prefork"
>>>> >>  -D APR_HAS_SENDFILE
>>>> >>  -D APR_HAS_MMAP
>>>> >>  -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
>>>> >>  -D APR_USE_SYSVSEM_SERIALIZE
>>>> >>  -D APR_USE_PTHREAD_SERIALIZE
>>>> >>  -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
>>>> >>  -D APR_HAS_OTHER_CHILD
>>>> >>  -D AP_HAVE_RELIABLE_PIPED_LOGS
>>>> >>  -D DYNAMIC_MODULE_LIMIT=128
>>>> >>  -D HTTPD_ROOT="/opt/share/apache2"
>>>> >>  -D SUEXEC_BIN="/opt/share/apache2/bin/suexec"
>>>> >>  -D DEFAULT_PIDLOG="logs/httpd.pid"
>>>> >>  -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
>>>> >>  -D DEFAULT_LOCKFILE="logs/accept.lock"
>>>> >>  -D DEFAULT_ERRORLOG="logs/error_log"
>>>> >>  -D AP_TYPES_CONFIG_FILE="conf/mime.types"
>>>> >>  -D SERVER_CONFIG_FILE="conf/httpd.conf"
>>>> >>
>>>> >>
>>>> >> 2. gentoo (64 bit dual proc AMD):
>>>> >> Linux host1 2.6.16-gentoo-r9 #7 SMP Fri Mar 16 17:35:26 UTC 2007
>>>> >> x86_64 AMD
>>>> >> Opteron(tm) Processor 248 AuthenticAMD GNU/Linux
>>>> >>
>>>> >> Server version: Apache/2.2.4 (Unix)
>>>> >> Server built:   Jun 26 2007 19:29:33
>>>> >> Server's Module Magic Number: 20051115:4
>>>> >> Server loaded:  APR 1.2.8, APR-Util 1.2.8
>>>> >> Compiled using: APR 1.2.8, APR-Util 1.2.8
>>>> >> Architecture:   64-bit
>>>> >> Server MPM:     Prefork
>>>> >>   threaded:     no
>>>> >>     forked:     yes (variable process count)
>>>> >> Server compiled with....
>>>> >>  -D APACHE_MPM_DIR="server/mpm/prefork"
>>>> >>  -D APR_HAS_SENDFILE
>>>> >>  -D APR_HAS_MMAP
>>>> >>  -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
>>>> >>  -D APR_USE_SYSVSEM_SERIALIZE
>>>> >>  -D APR_USE_PTHREAD_SERIALIZE
>>>> >>  -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
>>>> >>  -D APR_HAS_OTHER_CHILD
>>>> >>  -D AP_HAVE_RELIABLE_PIPED_LOGS
>>>> >>  -D DYNAMIC_MODULE_LIMIT=128
>>>> >>  -D HTTPD_ROOT="/usr"
>>>> >>  -D SUEXEC_BIN="/usr/sbin/suexec"
>>>> >>  -D DEFAULT_PIDLOG="/var/run/httpd.pid"
>>>> >>  -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
>>>> >>  -D DEFAULT_LOCKFILE="/var/run/accept.lock"
>>>> >>  -D DEFAULT_ERRORLOG="logs/error_log"
>>>> >>  -D AP_TYPES_CONFIG_FILE="/etc/apache2/mime.types"
>>>> >>  -D SERVER_CONFIG_FILE="/etc/apache2/httpd.conf"
>>>> >>
>>>> >> 3. Ubuntu Feisty (32 bit Intel core duo - ubuntu repos):
>>>> >> Linux host2 2.6.20-16-generic #2 SMP Thu Jun 7 20:19:32 UTC 2007 
>>>> i686
>>>> >> GNU/Linux
>>>> >>
>>>> >> Server version: Apache/2.2.3
>>>> >> Server built:   Jan 15 2007 18:14:50
>>>> >> Server's Module Magic Number: 20051115:3
>>>> >> Server loaded:  APR 1.2.7, APR-Util 1.2.7
>>>> >> Compiled using: APR 1.2.7, APR-Util 1.2.7
>>>> >> Architecture:   32-bit
>>>> >> Server MPM:     Prefork
>>>> >>   threaded:     no
>>>> >>     forked:     yes (variable process count)
>>>> >> Server compiled with....
>>>> >>  -D APACHE_MPM_DIR="server/mpm/prefork"
>>>> >>  -D APR_HAS_SENDFILE
>>>> >>  -D APR_HAS_MMAP
>>>> >>  -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
>>>> >>  -D APR_USE_SYSVSEM_SERIALIZE
>>>> >>  -D APR_USE_PTHREAD_SERIALIZE
>>>> >>  -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
>>>> >>  -D APR_HAS_OTHER_CHILD
>>>> >>  -D AP_HAVE_RELIABLE_PIPED_LOGS
>>>> >>  -D DYNAMIC_MODULE_LIMIT=128
>>>> >>  -D HTTPD_ROOT=""
>>>> >>  -D SUEXEC_BIN="/usr/lib/apache2/suexec"
>>>> >>  -D DEFAULT_PIDLOG="/var/run/apache2.pid"
>>>> >>  -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
>>>> >>  -D DEFAULT_LOCKFILE="/var/run/apache2/accept.lock"
>>>> >>  -D DEFAULT_ERRORLOG="logs/error_log"
>>>> >>  -D AP_TYPES_CONFIG_FILE="/etc/apache2/mime.types"
>>>> >>  -D SERVER_CONFIG_FILE="/etc/apache2/apache2.conf"
>>>> >>
>>>> >> Bug or just an unexpected apache configuration?? I actually filed a
>>>> >> bug against apache core
>>>> >> (http://issues.apache.org/bugzilla/show_bug.cgi?id=42749) initially
>>>> >> not realizing that it could be a combination of factors that was
>>>> >> causing it and not just soe module not being loaded.
>>>> >>
>>>> >> Thanks.
>>>> >> AM
>>>> >> _______________________________________________
>>>> >> Mod_python mailing list
>>>> >> Mod_python at modpython.org
>>>> >> http://mailman.modpython.org/mailman/listinfo/mod_python
>>>> >>
>>>> >
>>>> _______________________________________________
>>>> Mod_python mailing list
>>>> Mod_python at modpython.org
>>>> http://mailman.modpython.org/mailman/listinfo/mod_python
>>>>
>> _______________________________________________
>> Mod_python mailing list
>> Mod_python at modpython.org
>> http://mailman.modpython.org/mailman/listinfo/mod_python
>>
> 


More information about the Mod_python mailing list