[mod_python] mod_python + mod_authz_host w/o mod_auth_digest causes ~1Mb leak per apache graceful restart

Jim Gallacher jpg at jgassociates.ca
Thu Jun 28 17:27:20 EDT 2007


I've confirmed that mod_python leaks on graceful restart, but I don't 
see the dependency on the auth* modules. Apache always seems to leak 
when mod_python is loaded, regardless of the other modules. My test 
setup is a little different: On Debian with apache 2.2.3, mpm-worker, 
mod_python 3.3.1 running within VMWare.

My gut tells me that this may be related to 
http://issues.apache.org/jira/browse/MODPYTHON-109.

This issue also reminds me of 
http://issues.apache.org/jira/browse/MODPYTHON-195.

I think at this point we should log this in JIRA and move the discussion 
over to python-dev.

The obvious (but suboptimal) temporary solution for Aseem is to avoid 
using graceful restart.

Jim

Aseem Mohanty wrote:
> Graham Dumpleton wrote:
>> Is the version of mod_python you are using one you compiled yourself,
>> or one from Ubuntu packages?
> 
> I have used both and it does not seem to matter. When compiling from 
> source, I did the following:
> ./configure --prefix=/opt --with-apxs=/opt/share/apache2/bin/apxs
> make install_dso
> to use the apache build from source. Apache was build with the following 
> config:
> ./configure --prefix=/opt --enable-modules=all --enable-mods-shared=all 
> --enable-so --with-included-apr
> 
>>
>> Is the version of Python you are using one you compiled yourself, or
>> one from Ubuntu packages?
> 
> Python-2.5.1 is the one from the ubuntu packages.05:22 PM $ python
> Python 2.5.1 (r251:54863, May  2 2007, 16:56:35)
> [GCC 4.1.2 (Ubuntu 4.1.2-0ubuntu4)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> 
>>
>> What is the size of mod_python.so? Does it use a shared Python library
>> or static? Use 'ldd' on mod_python.so to see dependencies.
> 
> mod_python from ubuntu packages:
> v 3.2.10
> 05:24 PM $ ll mod_python.so
> -rw-r--r-- 1 root root 99K 2007-01-16 04:44 mod_python.so
> 05:24 PM $ ldd mod_python.so
>         linux-gate.so.1 =>  (0xffffe000)
>         libpython2.5.so.1.0 => /usr/lib/libpython2.5.so.1.0 (0xb7e4f000)
>         libpthread.so.0 => /lib/tls/i686/cmov/libpthread.so.0 (0xb7e38000)
>         libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0xb7e33000)
>         libutil.so.1 => /lib/tls/i686/cmov/libutil.so.1 (0xb7e2f000)
>         libm.so.6 => /lib/tls/i686/cmov/libm.so.6 (0xb7e08000)
>         libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7cc7000)
>         /lib/ld-linux.so.2 (0x80000000)
> 
> mod_python built from source:
> v 3.3.1
> 05:25 PM $ ll mod_python.so
> -rwxr-xr-x 1 aseem aseem 343K 2007-06-27 09:50 mod_python.so*
> aseem @ narada: /opt/share/apache2/modules
> 05:25 PM $ ldd mod_python.so
>         linux-gate.so.1 =>  (0xffffe000)
>         libpython2.5.so.1.0 => /usr/lib/libpython2.5.so.1.0 (0xb7e30000)
>         libpthread.so.0 => /lib/tls/i686/cmov/libpthread.so.0 (0xb7e19000)
>         libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0xb7e14000)
>         libutil.so.1 => /lib/tls/i686/cmov/libutil.so.1 (0xb7e10000)
>         libm.so.6 => /lib/tls/i686/cmov/libm.so.6 (0xb7de9000)
>         libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7ca8000)
>         /lib/ld-linux.so.2 (0x80000000)
> 
> In both cases they do seem to use shared python libs.
> 
> Thanks.
> AM
> 
>>
>> Graham
>>
>> On 28/06/07, Aseem Mohanty <aseem at metaweb.com> wrote:
>>> Yes, hence the exotic phraseology. Results for the various 
>>> combinations are below:
>>>
>>> 1. mod_python only, no mod_auth* modules:                   leaks
>>> 2. mod_auth_digest only, no other mod_auth*, no mod_python: does not 
>>> leak
>>> 3. mod_authz_host only, no other mod_auth*, no mod_python:  does not 
>>> leak
>>> 4. mod_python + mod_authz_host, no mod_auth_digest:         leaks
>>> 5. mod_python + mod_auth_digest, no mod_authz_host:         leaks < 
>>> (1) or (4)
>>> 6. mod_python + mod_auth_digest + mod_authz_host:           does not 
>>> leak
>>>
>>> NOTE: 'does not leak' implies ~50-100K leakage per graceful restart
>>>        (5) leaks ~500-600K per graceful restart
>>>        zero load on server
>>>
>>> As far as I can tell there is definitely some sinister interaction 
>>> going on
>>> between these three modules. Unfortunately not being conversant with 
>>> apache
>>> module development, I cannot really say what.
>>>
>>> Thanks.
>>> AM
>>>
>>>
>>> Jim Gallacher wrote:
>>> > Aseem Mohanty wrote:
>>> >> This may sound quite exotic
>>> >
>>> >
>>> > Why yes, it does? Interaction between these modules resulting in a
>>> > memory leak sure sounds weird. Did you try it with *only* mod_python
>>> > loaded? (ie, without mod_authz_host)
>>> >
>>> > Jim
>>> >
>>> >> but here it is:
>>> >>
>>> >> In a vanilla apache2.2.x configuration if I only load mod_authz_host
>>> >> and none of
>>> >> the other mod_auth* modules, every graceful restart causes ~1Mb leak
>>> >> in the
>>> >> parent process. This is with mod_python loaded. Without mod_python 
>>> the
>>> >> memory profile remains unchanged.
>>> >>
>>> >> Steps to reproduce:
>>> >> Gentoo:
>>> >> 1. In /etc/apache2/httpd.conf comment out all "LoadModule auth*"
>>> >> directives except
>>> >> LoadModule authz_host_module modules/mod_authz_host.so
>>> >>
>>> >> Ubuntu Feisty (repos):
>>> >> 1. In /etc/apache2/mods-enabled rename auth*.load files except
>>> >> authz_host.load
>>> >> to auth*.load.not
>>> >>
>>> >> Ubuntu Feisty (httpd built from source):
>>> >> 1. In conf/httpd.conf comment out all "LoadModule auth*" directives
>>> >> except
>>> >> LoadModule authz_host_module modules/mod_authz_host.so
>>> >>
>>> >> 2. Start apache
>>> >> 3. Track apache parent process: top -d1 -p $(pgrep -o apache/httpd)
>>> >> 4. Hit apache instances with graceful restarts:
>>> >> for i in $(seq 150); do echo $i; kill -USR1 $(pgrep -o apache/httpd);
>>> >> sleep 2; done
>>> >>
>>> >> In this case the parent will leak ~1Mb per restart. If however in 
>>> each
>>> >> of the
>>> >> configurations mod_auth_digest.so is also loaded the leak is only
>>> >> about 150-200K
>>> >> or so per graceful restart. This is on a server with 0 load. In my
>>> >> case it
>>> >> caused apache instances that were restarted every 55 minutes to grow
>>> >> up to 300Mb
>>> >> in a week.
>>> >>
>>> >> Tested on:
>>> >> 1. Ubuntu Feisty (built from source httpd-2.2.4, mod_python-3.3.1):
>>> >> ./configure --prefix=/opt --enable-modules=all
>>> >> --enable-mods-shared=all --enable-so --with-included-apr
>>> >> Server version: Apache/2.2.4 (Unix)
>>> >> Server built:   Jun 27 2007 09:34:16
>>> >> Server's Module Magic Number: 20051115:4
>>> >> Server loaded:  APR 1.2.8, APR-Util 1.2.8
>>> >> Compiled using: APR 1.2.8, APR-Util 1.2.8
>>> >> Architecture:   32-bit
>>> >> Server MPM:     Prefork
>>> >>   threaded:     no
>>> >>     forked:     yes (variable process count)
>>> >> Server compiled with....
>>> >>  -D APACHE_MPM_DIR="server/mpm/prefork"
>>> >>  -D APR_HAS_SENDFILE
>>> >>  -D APR_HAS_MMAP
>>> >>  -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
>>> >>  -D APR_USE_SYSVSEM_SERIALIZE
>>> >>  -D APR_USE_PTHREAD_SERIALIZE
>>> >>  -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
>>> >>  -D APR_HAS_OTHER_CHILD
>>> >>  -D AP_HAVE_RELIABLE_PIPED_LOGS
>>> >>  -D DYNAMIC_MODULE_LIMIT=128
>>> >>  -D HTTPD_ROOT="/opt/share/apache2"
>>> >>  -D SUEXEC_BIN="/opt/share/apache2/bin/suexec"
>>> >>  -D DEFAULT_PIDLOG="logs/httpd.pid"
>>> >>  -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
>>> >>  -D DEFAULT_LOCKFILE="logs/accept.lock"
>>> >>  -D DEFAULT_ERRORLOG="logs/error_log"
>>> >>  -D AP_TYPES_CONFIG_FILE="conf/mime.types"
>>> >>  -D SERVER_CONFIG_FILE="conf/httpd.conf"
>>> >>
>>> >>
>>> >> 2. gentoo (64 bit dual proc AMD):
>>> >> Linux host1 2.6.16-gentoo-r9 #7 SMP Fri Mar 16 17:35:26 UTC 2007
>>> >> x86_64 AMD
>>> >> Opteron(tm) Processor 248 AuthenticAMD GNU/Linux
>>> >>
>>> >> Server version: Apache/2.2.4 (Unix)
>>> >> Server built:   Jun 26 2007 19:29:33
>>> >> Server's Module Magic Number: 20051115:4
>>> >> Server loaded:  APR 1.2.8, APR-Util 1.2.8
>>> >> Compiled using: APR 1.2.8, APR-Util 1.2.8
>>> >> Architecture:   64-bit
>>> >> Server MPM:     Prefork
>>> >>   threaded:     no
>>> >>     forked:     yes (variable process count)
>>> >> Server compiled with....
>>> >>  -D APACHE_MPM_DIR="server/mpm/prefork"
>>> >>  -D APR_HAS_SENDFILE
>>> >>  -D APR_HAS_MMAP
>>> >>  -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
>>> >>  -D APR_USE_SYSVSEM_SERIALIZE
>>> >>  -D APR_USE_PTHREAD_SERIALIZE
>>> >>  -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
>>> >>  -D APR_HAS_OTHER_CHILD
>>> >>  -D AP_HAVE_RELIABLE_PIPED_LOGS
>>> >>  -D DYNAMIC_MODULE_LIMIT=128
>>> >>  -D HTTPD_ROOT="/usr"
>>> >>  -D SUEXEC_BIN="/usr/sbin/suexec"
>>> >>  -D DEFAULT_PIDLOG="/var/run/httpd.pid"
>>> >>  -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
>>> >>  -D DEFAULT_LOCKFILE="/var/run/accept.lock"
>>> >>  -D DEFAULT_ERRORLOG="logs/error_log"
>>> >>  -D AP_TYPES_CONFIG_FILE="/etc/apache2/mime.types"
>>> >>  -D SERVER_CONFIG_FILE="/etc/apache2/httpd.conf"
>>> >>
>>> >> 3. Ubuntu Feisty (32 bit Intel core duo - ubuntu repos):
>>> >> Linux host2 2.6.20-16-generic #2 SMP Thu Jun 7 20:19:32 UTC 2007 i686
>>> >> GNU/Linux
>>> >>
>>> >> Server version: Apache/2.2.3
>>> >> Server built:   Jan 15 2007 18:14:50
>>> >> Server's Module Magic Number: 20051115:3
>>> >> Server loaded:  APR 1.2.7, APR-Util 1.2.7
>>> >> Compiled using: APR 1.2.7, APR-Util 1.2.7
>>> >> Architecture:   32-bit
>>> >> Server MPM:     Prefork
>>> >>   threaded:     no
>>> >>     forked:     yes (variable process count)
>>> >> Server compiled with....
>>> >>  -D APACHE_MPM_DIR="server/mpm/prefork"
>>> >>  -D APR_HAS_SENDFILE
>>> >>  -D APR_HAS_MMAP
>>> >>  -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
>>> >>  -D APR_USE_SYSVSEM_SERIALIZE
>>> >>  -D APR_USE_PTHREAD_SERIALIZE
>>> >>  -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
>>> >>  -D APR_HAS_OTHER_CHILD
>>> >>  -D AP_HAVE_RELIABLE_PIPED_LOGS
>>> >>  -D DYNAMIC_MODULE_LIMIT=128
>>> >>  -D HTTPD_ROOT=""
>>> >>  -D SUEXEC_BIN="/usr/lib/apache2/suexec"
>>> >>  -D DEFAULT_PIDLOG="/var/run/apache2.pid"
>>> >>  -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
>>> >>  -D DEFAULT_LOCKFILE="/var/run/apache2/accept.lock"
>>> >>  -D DEFAULT_ERRORLOG="logs/error_log"
>>> >>  -D AP_TYPES_CONFIG_FILE="/etc/apache2/mime.types"
>>> >>  -D SERVER_CONFIG_FILE="/etc/apache2/apache2.conf"
>>> >>
>>> >> Bug or just an unexpected apache configuration?? I actually filed a
>>> >> bug against apache core
>>> >> (http://issues.apache.org/bugzilla/show_bug.cgi?id=42749) initially
>>> >> not realizing that it could be a combination of factors that was
>>> >> causing it and not just soe module not being loaded.
>>> >>
>>> >> Thanks.
>>> >> AM
>>> >> _______________________________________________
>>> >> Mod_python mailing list
>>> >> Mod_python at modpython.org
>>> >> http://mailman.modpython.org/mailman/listinfo/mod_python
>>> >>
>>> >
>>> _______________________________________________
>>> Mod_python mailing list
>>> Mod_python at modpython.org
>>> http://mailman.modpython.org/mailman/listinfo/mod_python
>>>
> _______________________________________________
> Mod_python mailing list
> Mod_python at modpython.org
> http://mailman.modpython.org/mailman/listinfo/mod_python
> 



More information about the Mod_python mailing list