|
Aseem Mohanty
aseem at metaweb.com
Thu Jun 28 17:40:16 EDT 2007
Jim Gallacher wrote: > I've confirmed that mod_python leaks on graceful restart, but I don't > see the dependency on the auth* modules. Apache always seems to leak > when mod_python is loaded, regardless of the other modules. My test > setup is a little different: On Debian with apache 2.2.3, mpm-worker, > mod_python 3.3.1 running within VMWare. Hmm. The reason I felt that the dependency existed was the difference in the size of the leak when the two mod_auth* modules were loaded. In addition, the use of mod_authz_host alone did not leak at all, which led me to this conclusion. I will try out your config and see if the leak size remains the same as just using mod_python only. > > My gut tells me that this may be related to > http://issues.apache.org/jira/browse/MODPYTHON-109. > > This issue also reminds me of > http://issues.apache.org/jira/browse/MODPYTHON-195. > > I think at this point we should log this in JIRA and move the discussion > over to python-dev. > > The obvious (but suboptimal) temporary solution for Aseem is to avoid > using graceful restart. They sure look like good candidates, so should I be looking at mod_wsgi in daemon mode (we are serving WSGI apps via an adapter). The reason I ask is that we need the graceful restarts and cannot really do without it for now. Thanks. AM > > Jim > > Aseem Mohanty wrote: >> Graham Dumpleton wrote: >>> Is the version of mod_python you are using one you compiled yourself, >>> or one from Ubuntu packages? >> >> I have used both and it does not seem to matter. When compiling from >> source, I did the following: >> ./configure --prefix=/opt --with-apxs=/opt/share/apache2/bin/apxs >> make install_dso >> to use the apache build from source. Apache was build with the >> following config: >> ./configure --prefix=/opt --enable-modules=all >> --enable-mods-shared=all --enable-so --with-included-apr >> >>> >>> Is the version of Python you are using one you compiled yourself, or >>> one from Ubuntu packages? >> >> Python-2.5.1 is the one from the ubuntu packages.05:22 PM $ python >> Python 2.5.1 (r251:54863, May 2 2007, 16:56:35) >> [GCC 4.1.2 (Ubuntu 4.1.2-0ubuntu4)] on linux2 >> Type "help", "copyright", "credits" or "license" for more information. >> >>> >>> What is the size of mod_python.so? Does it use a shared Python library >>> or static? Use 'ldd' on mod_python.so to see dependencies. >> >> mod_python from ubuntu packages: >> v 3.2.10 >> 05:24 PM $ ll mod_python.so >> -rw-r--r-- 1 root root 99K 2007-01-16 04:44 mod_python.so >> 05:24 PM $ ldd mod_python.so >> linux-gate.so.1 => (0xffffe000) >> libpython2.5.so.1.0 => /usr/lib/libpython2.5.so.1.0 (0xb7e4f000) >> libpthread.so.0 => /lib/tls/i686/cmov/libpthread.so.0 >> (0xb7e38000) >> libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0xb7e33000) >> libutil.so.1 => /lib/tls/i686/cmov/libutil.so.1 (0xb7e2f000) >> libm.so.6 => /lib/tls/i686/cmov/libm.so.6 (0xb7e08000) >> libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7cc7000) >> /lib/ld-linux.so.2 (0x80000000) >> >> mod_python built from source: >> v 3.3.1 >> 05:25 PM $ ll mod_python.so >> -rwxr-xr-x 1 aseem aseem 343K 2007-06-27 09:50 mod_python.so* >> aseem @ narada: /opt/share/apache2/modules >> 05:25 PM $ ldd mod_python.so >> linux-gate.so.1 => (0xffffe000) >> libpython2.5.so.1.0 => /usr/lib/libpython2.5.so.1.0 (0xb7e30000) >> libpthread.so.0 => /lib/tls/i686/cmov/libpthread.so.0 >> (0xb7e19000) >> libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0xb7e14000) >> libutil.so.1 => /lib/tls/i686/cmov/libutil.so.1 (0xb7e10000) >> libm.so.6 => /lib/tls/i686/cmov/libm.so.6 (0xb7de9000) >> libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7ca8000) >> /lib/ld-linux.so.2 (0x80000000) >> >> In both cases they do seem to use shared python libs. >> >> Thanks. >> AM >> >>> >>> Graham >>> >>> On 28/06/07, Aseem Mohanty <aseem at metaweb.com> wrote: >>>> Yes, hence the exotic phraseology. Results for the various >>>> combinations are below: >>>> >>>> 1. mod_python only, no mod_auth* modules: leaks >>>> 2. mod_auth_digest only, no other mod_auth*, no mod_python: does not >>>> leak >>>> 3. mod_authz_host only, no other mod_auth*, no mod_python: does not >>>> leak >>>> 4. mod_python + mod_authz_host, no mod_auth_digest: leaks >>>> 5. mod_python + mod_auth_digest, no mod_authz_host: leaks < >>>> (1) or (4) >>>> 6. mod_python + mod_auth_digest + mod_authz_host: does not >>>> leak >>>> >>>> NOTE: 'does not leak' implies ~50-100K leakage per graceful restart >>>> (5) leaks ~500-600K per graceful restart >>>> zero load on server >>>> >>>> As far as I can tell there is definitely some sinister interaction >>>> going on >>>> between these three modules. Unfortunately not being conversant with >>>> apache >>>> module development, I cannot really say what. >>>> >>>> Thanks. >>>> AM >>>> >>>> >>>> Jim Gallacher wrote: >>>> > Aseem Mohanty wrote: >>>> >> This may sound quite exotic >>>> > >>>> > >>>> > Why yes, it does? Interaction between these modules resulting in a >>>> > memory leak sure sounds weird. Did you try it with *only* mod_python >>>> > loaded? (ie, without mod_authz_host) >>>> > >>>> > Jim >>>> > >>>> >> but here it is: >>>> >> >>>> >> In a vanilla apache2.2.x configuration if I only load mod_authz_host >>>> >> and none of >>>> >> the other mod_auth* modules, every graceful restart causes ~1Mb leak >>>> >> in the >>>> >> parent process. This is with mod_python loaded. Without >>>> mod_python the >>>> >> memory profile remains unchanged. >>>> >> >>>> >> Steps to reproduce: >>>> >> Gentoo: >>>> >> 1. In /etc/apache2/httpd.conf comment out all "LoadModule auth*" >>>> >> directives except >>>> >> LoadModule authz_host_module modules/mod_authz_host.so >>>> >> >>>> >> Ubuntu Feisty (repos): >>>> >> 1. In /etc/apache2/mods-enabled rename auth*.load files except >>>> >> authz_host.load >>>> >> to auth*.load.not >>>> >> >>>> >> Ubuntu Feisty (httpd built from source): >>>> >> 1. In conf/httpd.conf comment out all "LoadModule auth*" directives >>>> >> except >>>> >> LoadModule authz_host_module modules/mod_authz_host.so >>>> >> >>>> >> 2. Start apache >>>> >> 3. Track apache parent process: top -d1 -p $(pgrep -o apache/httpd) >>>> >> 4. Hit apache instances with graceful restarts: >>>> >> for i in $(seq 150); do echo $i; kill -USR1 $(pgrep -o >>>> apache/httpd); >>>> >> sleep 2; done >>>> >> >>>> >> In this case the parent will leak ~1Mb per restart. If however in >>>> each >>>> >> of the >>>> >> configurations mod_auth_digest.so is also loaded the leak is only >>>> >> about 150-200K >>>> >> or so per graceful restart. This is on a server with 0 load. In my >>>> >> case it >>>> >> caused apache instances that were restarted every 55 minutes to grow >>>> >> up to 300Mb >>>> >> in a week. >>>> >> >>>> >> Tested on: >>>> >> 1. Ubuntu Feisty (built from source httpd-2.2.4, mod_python-3.3.1): >>>> >> ./configure --prefix=/opt --enable-modules=all >>>> >> --enable-mods-shared=all --enable-so --with-included-apr >>>> >> Server version: Apache/2.2.4 (Unix) >>>> >> Server built: Jun 27 2007 09:34:16 >>>> >> Server's Module Magic Number: 20051115:4 >>>> >> Server loaded: APR 1.2.8, APR-Util 1.2.8 >>>> >> Compiled using: APR 1.2.8, APR-Util 1.2.8 >>>> >> Architecture: 32-bit >>>> >> Server MPM: Prefork >>>> >> threaded: no >>>> >> forked: yes (variable process count) >>>> >> Server compiled with.... >>>> >> -D APACHE_MPM_DIR="server/mpm/prefork" >>>> >> -D APR_HAS_SENDFILE >>>> >> -D APR_HAS_MMAP >>>> >> -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) >>>> >> -D APR_USE_SYSVSEM_SERIALIZE >>>> >> -D APR_USE_PTHREAD_SERIALIZE >>>> >> -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT >>>> >> -D APR_HAS_OTHER_CHILD >>>> >> -D AP_HAVE_RELIABLE_PIPED_LOGS >>>> >> -D DYNAMIC_MODULE_LIMIT=128 >>>> >> -D HTTPD_ROOT="/opt/share/apache2" >>>> >> -D SUEXEC_BIN="/opt/share/apache2/bin/suexec" >>>> >> -D DEFAULT_PIDLOG="logs/httpd.pid" >>>> >> -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" >>>> >> -D DEFAULT_LOCKFILE="logs/accept.lock" >>>> >> -D DEFAULT_ERRORLOG="logs/error_log" >>>> >> -D AP_TYPES_CONFIG_FILE="conf/mime.types" >>>> >> -D SERVER_CONFIG_FILE="conf/httpd.conf" >>>> >> >>>> >> >>>> >> 2. gentoo (64 bit dual proc AMD): >>>> >> Linux host1 2.6.16-gentoo-r9 #7 SMP Fri Mar 16 17:35:26 UTC 2007 >>>> >> x86_64 AMD >>>> >> Opteron(tm) Processor 248 AuthenticAMD GNU/Linux >>>> >> >>>> >> Server version: Apache/2.2.4 (Unix) >>>> >> Server built: Jun 26 2007 19:29:33 >>>> >> Server's Module Magic Number: 20051115:4 >>>> >> Server loaded: APR 1.2.8, APR-Util 1.2.8 >>>> >> Compiled using: APR 1.2.8, APR-Util 1.2.8 >>>> >> Architecture: 64-bit >>>> >> Server MPM: Prefork >>>> >> threaded: no >>>> >> forked: yes (variable process count) >>>> >> Server compiled with.... >>>> >> -D APACHE_MPM_DIR="server/mpm/prefork" >>>> >> -D APR_HAS_SENDFILE >>>> >> -D APR_HAS_MMAP >>>> >> -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) >>>> >> -D APR_USE_SYSVSEM_SERIALIZE >>>> >> -D APR_USE_PTHREAD_SERIALIZE >>>> >> -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT >>>> >> -D APR_HAS_OTHER_CHILD >>>> >> -D AP_HAVE_RELIABLE_PIPED_LOGS >>>> >> -D DYNAMIC_MODULE_LIMIT=128 >>>> >> -D HTTPD_ROOT="/usr" >>>> >> -D SUEXEC_BIN="/usr/sbin/suexec" >>>> >> -D DEFAULT_PIDLOG="/var/run/httpd.pid" >>>> >> -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" >>>> >> -D DEFAULT_LOCKFILE="/var/run/accept.lock" >>>> >> -D DEFAULT_ERRORLOG="logs/error_log" >>>> >> -D AP_TYPES_CONFIG_FILE="/etc/apache2/mime.types" >>>> >> -D SERVER_CONFIG_FILE="/etc/apache2/httpd.conf" >>>> >> >>>> >> 3. Ubuntu Feisty (32 bit Intel core duo - ubuntu repos): >>>> >> Linux host2 2.6.20-16-generic #2 SMP Thu Jun 7 20:19:32 UTC 2007 >>>> i686 >>>> >> GNU/Linux >>>> >> >>>> >> Server version: Apache/2.2.3 >>>> >> Server built: Jan 15 2007 18:14:50 >>>> >> Server's Module Magic Number: 20051115:3 >>>> >> Server loaded: APR 1.2.7, APR-Util 1.2.7 >>>> >> Compiled using: APR 1.2.7, APR-Util 1.2.7 >>>> >> Architecture: 32-bit >>>> >> Server MPM: Prefork >>>> >> threaded: no >>>> >> forked: yes (variable process count) >>>> >> Server compiled with.... >>>> >> -D APACHE_MPM_DIR="server/mpm/prefork" >>>> >> -D APR_HAS_SENDFILE >>>> >> -D APR_HAS_MMAP >>>> >> -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) >>>> >> -D APR_USE_SYSVSEM_SERIALIZE >>>> >> -D APR_USE_PTHREAD_SERIALIZE >>>> >> -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT >>>> >> -D APR_HAS_OTHER_CHILD >>>> >> -D AP_HAVE_RELIABLE_PIPED_LOGS >>>> >> -D DYNAMIC_MODULE_LIMIT=128 >>>> >> -D HTTPD_ROOT="" >>>> >> -D SUEXEC_BIN="/usr/lib/apache2/suexec" >>>> >> -D DEFAULT_PIDLOG="/var/run/apache2.pid" >>>> >> -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" >>>> >> -D DEFAULT_LOCKFILE="/var/run/apache2/accept.lock" >>>> >> -D DEFAULT_ERRORLOG="logs/error_log" >>>> >> -D AP_TYPES_CONFIG_FILE="/etc/apache2/mime.types" >>>> >> -D SERVER_CONFIG_FILE="/etc/apache2/apache2.conf" >>>> >> >>>> >> Bug or just an unexpected apache configuration?? I actually filed a >>>> >> bug against apache core >>>> >> (http://issues.apache.org/bugzilla/show_bug.cgi?id=42749) initially >>>> >> not realizing that it could be a combination of factors that was >>>> >> causing it and not just soe module not being loaded. >>>> >> >>>> >> Thanks. >>>> >> AM >>>> >> _______________________________________________ >>>> >> Mod_python mailing list >>>> >> Mod_python at modpython.org >>>> >> http://mailman.modpython.org/mailman/listinfo/mod_python >>>> >> >>>> > >>>> _______________________________________________ >>>> Mod_python mailing list >>>> Mod_python at modpython.org >>>> http://mailman.modpython.org/mailman/listinfo/mod_python >>>> >> _______________________________________________ >> Mod_python mailing list >> Mod_python at modpython.org >> http://mailman.modpython.org/mailman/listinfo/mod_python >> >
|