[mod_python] mod_python + mod_authz_host w/o mod_auth_digest causes ~1Mb leak per apache graceful restart

Aseem Mohanty aseem at metaweb.com
Fri Jun 29 14:40:44 EDT 2007


With the help of a slightly modified version of Jim's script (thanks!) I ran 
some tests with various combinations of mod_python/wsgi/auth_digest/authz_host. 
It turns out that (as Jim mentioned in a previous email) that mod_authz_host is 
a red herring. However mod_auth_digest _definitely_ seems to mitigate the 
effects of the leak caused by mod_python.

OTOH mod_wsgi behaves exactly the same as mod_python in terms memory leakage and 
interaction with mod_auth_digest. Summary statistics of the tests are provided 
below. Detailed logs and script and my initial httpd.conf file are attached.

Test environment:
11:31 AM $ uname -a
Linux host1 2.6.20-16-generic #2 SMP Thu Jun 7 20:19:32 UTC 2007 i686 GNU/Linux
Ubuntu Feisty

11:33 AM $ python
Python 2.5.1 (r251:54863, May  2 2007, 16:56:35)
[GCC 4.1.2 (Ubuntu 4.1.2-0ubuntu4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
 >>>

apache 2.2.4 (downloaded yesterday) built from source with:
(built from source httpd-2.2.4, mod_python-3.3.1):
./configure --prefix=/opt --enable-modules=all --enable-mods-shared=all 
--enable-so --with-included-apr

mod_python 3.3.1 built from source with:
./configure --prefix=/opt --with-apxs=/opt/share/apache2/bin/apxs
make install_dso

mod_wsgi (svn-trunk) built from source with:
./configure --prefix=/opt --with-apxs=/opt/share/apache2/bin/apxs

ldd for both modules:
11:37 AM $ ldd mod_python.so
         linux-gate.so.1 =>  (0xffffe000)
         libpython2.5.so.1.0 => /usr/lib/libpython2.5.so.1.0 (0xb7df6000)
         libpthread.so.0 => /lib/tls/i686/cmov/libpthread.so.0 (0xb7ddf000)
         libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0xb7dda000)
         libutil.so.1 => /lib/tls/i686/cmov/libutil.so.1 (0xb7dd6000)
         libm.so.6 => /lib/tls/i686/cmov/libm.so.6 (0xb7daf000)
         libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7c6e000)
         /lib/ld-linux.so.2 (0x80000000)
11:37 AM $ ldd mod_wsgi.so
         linux-gate.so.1 =>  (0xffffe000)
         libpython2.5.so.1.0 => /usr/lib/libpython2.5.so.1.0 (0xb7dd0000)
         libpthread.so.0 => /lib/tls/i686/cmov/libpthread.so.0 (0xb7db9000)
         libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0xb7db4000)
         libutil.so.1 => /lib/tls/i686/cmov/libutil.so.1 (0xb7db0000)
         libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7c6f000)
         libm.so.6 => /lib/tls/i686/cmov/libm.so.6 (0xb7c48000)
         /lib/ld-linux.so.2 (0x80000000)

Modules	Loaded		Res Mem Start		Res Mem End
------------------------------------------------------------
None of the 4 loaded	2336			2424
============================================================
mod_python		5028			72M
------------------------------------------------------------
mod_python		5044			72M
mod_authz_host
------------------------------------------------------------
mod_python		5052			30M
mod_auth_digest
------------------------------------------------------------
mod_python		5056			28M
mod_authz_host
mod_auth_digest
============================================================
mod_wsgi		4984			72M
------------------------------------------------------------
mod_wsgi		5000			72M
mod_authz_host
------------------------------------------------------------
mod_wsgi		5024			22M
mod_auth_digest
------------------------------------------------------------
mod_wsgi		5020			30M
mod_authz_host
mod_auth_digest
============================================================

For now I am content with adding the LoadModule directive for mod_auth_digest 
since it a pretty good workaround with an almost non-existent overhead.

Thanks.
AM

Graham Dumpleton wrote:
> On 29/06/07, Graham Dumpleton <graham.dumpleton at gmail.com> wrote:
>> On 29/06/07, Aseem Mohanty <aseem at metaweb.com> wrote:
>> > They sure look like good candidates, so should I be looking at 
>> mod_wsgi in
>> > daemon mode (we are serving WSGI apps via an adapter). The reason I 
>> ask is that
>> > we need the graceful restarts and cannot really do without it for now.
>>
>> Maybe not. When I look at it, mod_wsgi is also leaking on 'graceful'
>> and 'restart' as well. If these leaks are from Python itself not
>> cleaning things up properly, not sure what I am going to do. Unlike
>> mod_python I actually Py_Finalize() the interpreter in the main
>> process so that everything is properly reset back to a clean slate, so
>> I may actually be more susceptible to it than mod_python.
>>
>> I'll investigate the mod_wsgi problem and maybe that will throw some
>> light on why mod_python also has problems.
> 
> Its okay to try mod_wsgi now if you want to see whether you get memory
> leaks there as well. Just make sure you update from repository if you
> already had it checked out. Obviously, don't load mod_python at the
> same time.
> 
> The change I made in mod_wsgi was to comment out trigger for recycling
> Python interpreter in main Apache process. This was only being done
> for Apache 2.X and not Apache 1.3, so I know not doing it doesn't
> cause a problem. I do still see very minor leaks, but the same ones I
> get without mod_wsgi.
> 
> I'll try now and see what happens with mod_wsgi when I add the auth
> modules you are talking about. If that is okay with mod_wsgi, can try
> with mod_python and see if I can work out what the differences are.
> 
> Graham
-------------- next part --------------
A non-text attachment was scrubbed...
Name: logs+script.tgz
Type: application/x-compressed-tar
Size: 10213 bytes
Desc: not available
Url : http://mm_cfg_has_not_been_edited_to_set_host_domains/pipermail/mod_python/attachments/20070629/310a9da0/logsscript-0001.bin
-------------- next part --------------
#
# This is the main Apache HTTP server configuration file.  It contains the
# configuration directives that give the server its instructions.
# See <URL:http://httpd.apache.org/docs/2.2> for detailed information.
# In particular, see 
# <URL:http://httpd.apache.org/docs/2.2/mod/directives.html>
# for a discussion of each configuration directive.
#
# Do NOT simply read the instructions in here without understanding
# what they do.  They're here only as hints or reminders.  If you are unsure
# consult the online docs. You have been warned.  
#
# Configuration and logfile names: If the filenames you specify for many
# of the server's control files begin with "/" (or "drive:/" for Win32), the
# server will use that explicit path.  If the filenames do *not* begin
# with "/", the value of ServerRoot is prepended -- so "logs/foo.log"
# with ServerRoot set to "/opt/share/apache2" will be interpreted by the
# server as "/opt/share/apache2/logs/foo.log".

#
# ServerRoot: The top of the directory tree under which the server's
# configuration, error, and log files are kept.
#
# Do not add a slash at the end of the directory path.  If you point
# ServerRoot at a non-local disk, be sure to point the LockFile directive
# at a local disk.  If you wish to share the same ServerRoot for multiple
# httpd daemons, you will need to change at least LockFile and PidFile.
#
ServerRoot "/opt/share/apache2"

#
# Listen: Allows you to bind Apache to specific IP addresses and/or
# ports, instead of the default. See also the <VirtualHost>
# directive.
#
# Change this to Listen on specific IP addresses as shown below to 
# prevent Apache from glomming onto all bound IP addresses.
#
#Listen 12.34.56.78:80
Listen 8080

#
# Dynamic Shared Object (DSO) Support
#
# To be able to use the functionality of a module which was built as a DSO you
# have to place corresponding `LoadModule' lines at this location so the
# directives contained in it are actually available _before_ they are used.
# Statically compiled modules (those listed by `httpd -l') do not need
# to be loaded here.
#
# Example:
# LoadModule foo_module modules/mod_foo.so
#
#LoadModule authn_file_module modules/mod_authn_file.so
#LoadModule authn_dbm_module modules/mod_authn_dbm.so
#LoadModule authn_anon_module modules/mod_authn_anon.so
#LoadModule authn_dbd_module modules/mod_authn_dbd.so
#LoadModule authn_default_module modules/mod_authn_default.so
#LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
#LoadModule authz_user_module modules/mod_authz_user.so
#LoadModule authz_dbm_module modules/mod_authz_dbm.so
#LoadModule authz_owner_module modules/mod_authz_owner.so
#LoadModule authz_default_module modules/mod_authz_default.so
#LoadModule auth_basic_module modules/mod_auth_basic.so
#LoadModule authz_host_module modules/mod_authz_host.so
#LoadModule auth_digest_module modules/mod_auth_digest.so

LoadModule dbd_module modules/mod_dbd.so
LoadModule dumpio_module modules/mod_dumpio.so
LoadModule ext_filter_module modules/mod_ext_filter.so
LoadModule include_module modules/mod_include.so
LoadModule filter_module modules/mod_filter.so
LoadModule deflate_module modules/mod_deflate.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule log_forensic_module modules/mod_log_forensic.so
LoadModule logio_module modules/mod_logio.so
LoadModule env_module modules/mod_env.so
LoadModule mime_magic_module modules/mod_mime_magic.so
LoadModule cern_meta_module modules/mod_cern_meta.so
LoadModule expires_module modules/mod_expires.so
LoadModule headers_module modules/mod_headers.so
LoadModule ident_module modules/mod_ident.so
LoadModule usertrack_module modules/mod_usertrack.so
LoadModule unique_id_module modules/mod_unique_id.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule version_module modules/mod_version.so
LoadModule mime_module modules/mod_mime.so
LoadModule dav_module modules/mod_dav.so
LoadModule status_module modules/mod_status.so
LoadModule autoindex_module modules/mod_autoindex.so
LoadModule asis_module modules/mod_asis.so
LoadModule info_module modules/mod_info.so
LoadModule cgi_module modules/mod_cgi.so
LoadModule dav_fs_module modules/mod_dav_fs.so
LoadModule vhost_alias_module modules/mod_vhost_alias.so
LoadModule negotiation_module modules/mod_negotiation.so
LoadModule dir_module modules/mod_dir.so
LoadModule imagemap_module modules/mod_imagemap.so
LoadModule actions_module modules/mod_actions.so
LoadModule speling_module modules/mod_speling.so
LoadModule userdir_module modules/mod_userdir.so
LoadModule alias_module modules/mod_alias.so
LoadModule rewrite_module modules/mod_rewrite.so

#LoadModule python_module      modules/mod_python.so
#LoadModule wsgi_module        modules/mod_wsgi.so

<IfModule !mpm_netware_module>
#
# If you wish httpd to run as a different user or group, you must run
# httpd as root initially and it will switch.  
#
# User/Group: The name (or #number) of the user/group to run httpd as.
# It is usually good practice to create a dedicated user and group for
# running httpd, as with most system services.
#
User daemon
Group daemon
</IfModule>

# 'Main' server configuration
#
# The directives in this section set up the values used by the 'main'
# server, which responds to any requests that aren't handled by a
# <VirtualHost> definition.  These values also provide defaults for
# any <VirtualHost> containers you may define later in the file.
#
# All of these directives may appear inside <VirtualHost> containers,
# in which case these default settings will be overridden for the
# virtual host being defined.
#

#
# ServerAdmin: Your address, where problems with the server should be
# e-mailed.  This address appears on some server-generated pages, such
# as error documents.  e.g. admin at your-domain.com
#
ServerAdmin you at example.com

#
# ServerName gives the name and port that the server uses to identify itself.
# This can often be determined automatically, but we recommend you specify
# it explicitly to prevent problems during startup.
#
# If your host doesn't have a registered DNS name, enter its IP address here.
#
#ServerName www.example.com:80

#
# DocumentRoot: The directory out of which you will serve your
# documents. By default, all requests are taken from this directory, but
# symbolic links and aliases may be used to point to other locations.
#
DocumentRoot "/opt/share/apache2/htdocs"

#
# Each directory to which Apache has access can be configured with respect
# to which services and features are allowed and/or disabled in that
# directory (and its subdirectories). 
#
# First, we configure the "default" to be a very restrictive set of 
# features.  
#
<Directory />
    Options FollowSymLinks
    AllowOverride None
    #Order deny,allow
    #Deny from all
</Directory>

#
# Note that from this point forward you must specifically allow
# particular features to be enabled - so if something's not working as
# you might expect, make sure that you have specifically enabled it
# below.
#

#
# This should be changed to whatever you set DocumentRoot to.
#
<Directory "/opt/share/apache2/htdocs">
    #
    # Possible values for the Options directive are "None", "All",
    # or any combination of:
    #   Indexes Includes FollowSymLinks SymLinksifOwnerMatch ExecCGI MultiViews
    #
    # Note that "MultiViews" must be named *explicitly* --- "Options All"
    # doesn't give it to you.
    #
    # The Options directive is both complicated and important.  Please see
    # http://httpd.apache.org/docs/2.2/mod/core.html#options
    # for more information.
    #
    Options Indexes FollowSymLinks

    #
    # AllowOverride controls what directives may be placed in .htaccess files.
    # It can be "All", "None", or any combination of the keywords:
    #   Options FileInfo AuthConfig Limit
    #
    AllowOverride None

    #
    # Controls who can get stuff from this server.
    #
    #Order allow,deny
    #Allow from all

</Directory>

#
# DirectoryIndex: sets the file that Apache will serve if a directory
# is requested.
#
<IfModule dir_module>
    DirectoryIndex index.html
</IfModule>

#
# The following lines prevent .htaccess and .htpasswd files from being 
# viewed by Web clients. 
#
<FilesMatch "^\.ht">
    #Order allow,deny
    #Deny from all
    Satisfy All
</FilesMatch>

#
# ErrorLog: The location of the error log file.
# If you do not specify an ErrorLog directive within a <VirtualHost>
# container, error messages relating to that virtual host will be
# logged here.  If you *do* define an error logfile for a <VirtualHost>
# container, that host's errors will be logged there and not here.
#
ErrorLog logs/error_log

#
# LogLevel: Control the number of messages logged to the error_log.
# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
#
LogLevel warn

<IfModule log_config_module>
    #
    # The following directives define some format nicknames for use with
    # a CustomLog directive (see below).
    #
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
    LogFormat "%h %l %u %t \"%r\" %>s %b" common

    <IfModule logio_module>
      # You need to enable mod_logio.c to use %I and %O
      LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
    </IfModule>

    #
    # The location and format of the access logfile (Common Logfile Format).
    # If you do not define any access logfiles within a <VirtualHost>
    # container, they will be logged here.  Contrariwise, if you *do*
    # define per-<VirtualHost> access logfiles, transactions will be
    # logged therein and *not* in this file.
    #
    CustomLog logs/access_log common

    #
    # If you prefer a logfile with access, agent, and referer information
    # (Combined Logfile Format) you can use the following directive.
    #
    #CustomLog logs/access_log combined
</IfModule>

<IfModule alias_module>
    #
    # Redirect: Allows you to tell clients about documents that used to 
    # exist in your server's namespace, but do not anymore. The client 
    # will make a new request for the document at its new location.
    # Example:
    # Redirect permanent /foo http://www.example.com/bar

    #
    # Alias: Maps web paths into filesystem paths and is used to
    # access content that does not live under the DocumentRoot.
    # Example:
    # Alias /webpath /full/filesystem/path
    #
    # If you include a trailing / on /webpath then the server will
    # require it to be present in the URL.  You will also likely
    # need to provide a <Directory> section to allow access to
    # the filesystem path.

    #
    # ScriptAlias: This controls which directories contain server scripts. 
    # ScriptAliases are essentially the same as Aliases, except that
    # documents in the target directory are treated as applications and
    # run by the server when requested rather than as documents sent to the
    # client.  The same rules about trailing "/" apply to ScriptAlias
    # directives as to Alias.
    #
    ScriptAlias /cgi-bin/ "/opt/share/apache2/cgi-bin/"

</IfModule>

<IfModule cgid_module>
    #
    # ScriptSock: On threaded servers, designate the path to the UNIX
    # socket used to communicate with the CGI daemon of mod_cgid.
    #
    #Scriptsock logs/cgisock
</IfModule>

#
# "/opt/share/apache2/cgi-bin" should be changed to whatever your ScriptAliased
# CGI directory exists, if you have that configured.
#
<Directory "/opt/share/apache2/cgi-bin">
    AllowOverride None
    Options None
    #Order allow,deny
    #Allow from all
</Directory>

#
# DefaultType: the default MIME type the server will use for a document
# if it cannot otherwise determine one, such as from filename extensions.
# If your server contains mostly text or HTML documents, "text/plain" is
# a good value.  If most of your content is binary, such as applications
# or images, you may want to use "application/octet-stream" instead to
# keep browsers from trying to display binary files as though they are
# text.
#
DefaultType text/plain

<IfModule mime_module>
    #
    # TypesConfig points to the file containing the list of mappings from
    # filename extension to MIME-type.
    #
    TypesConfig conf/mime.types

    #
    # AddType allows you to add to or override the MIME configuration
    # file specified in TypesConfig for specific file types.
    #
    #AddType application/x-gzip .tgz
    #
    # AddEncoding allows you to have certain browsers uncompress
    # information on the fly. Note: Not all browsers support this.
    #
    #AddEncoding x-compress .Z
    #AddEncoding x-gzip .gz .tgz
    #
    # If the AddEncoding directives above are commented-out, then you
    # probably should define those extensions to indicate media types:
    #
    AddType application/x-compress .Z
    AddType application/x-gzip .gz .tgz

    #
    # AddHandler allows you to map certain file extensions to "handlers":
    # actions unrelated to filetype. These can be either built into the server
    # or added with the Action directive (see below)
    #
    # To use CGI scripts outside of ScriptAliased directories:
    # (You will also need to add "ExecCGI" to the "Options" directive.)
    #
    #AddHandler cgi-script .cgi

    # For type maps (negotiated resources):
    #AddHandler type-map var

    #
    # Filters allow you to process content before it is sent to the client.
    #
    # To parse .shtml files for server-side includes (SSI):
    # (You will also need to add "Includes" to the "Options" directive.)
    #
    #AddType text/html .shtml
    #AddOutputFilter INCLUDES .shtml
</IfModule>

#
# The mod_mime_magic module allows the server to use various hints from the
# contents of the file itself to determine its type.  The MIMEMagicFile
# directive tells the module where the hint definitions are located.
#
#MIMEMagicFile conf/magic

#
# Customizable error responses come in three flavors:
# 1) plain text 2) local redirects 3) external redirects
#
# Some examples:
#ErrorDocument 500 "The server made a boo boo."
#ErrorDocument 404 /missing.html
#ErrorDocument 404 "/cgi-bin/missing_handler.pl"
#ErrorDocument 402 http://www.example.com/subscription_info.html
#

#
# EnableMMAP and EnableSendfile: On systems that support it, 
# memory-mapping or the sendfile syscall is used to deliver
# files.  This usually improves server performance, but must
# be turned off when serving from networked-mounted 
# filesystems or if support for these functions is otherwise
# broken on your system.
#
#EnableMMAP off
#EnableSendfile off

# Supplemental configuration
#
# The configuration files in the conf/extra/ directory can be 
# included to add extra features or to modify the default configuration of 
# the server, or you may simply copy their contents here and change as 
# necessary.

# Server-pool management (MPM specific)
#Include conf/extra/httpd-mpm.conf

# Multi-language error messages
#Include conf/extra/httpd-multilang-errordoc.conf

# Fancy directory listings
#Include conf/extra/httpd-autoindex.conf

# Language settings
#Include conf/extra/httpd-languages.conf

# User home directories
#Include conf/extra/httpd-userdir.conf

# Real-time info on requests and configuration
#Include conf/extra/httpd-info.conf

# Virtual hosts
#Include conf/extra/httpd-vhosts.conf

# Local access to the Apache HTTP Server Manual
#Include conf/extra/httpd-manual.conf

# Distributed authoring and versioning (WebDAV)
#Include conf/extra/httpd-dav.conf

# Various default settings
#Include conf/extra/httpd-default.conf

# Secure (SSL/TLS) connections
#Include conf/extra/httpd-ssl.conf
#
# Note: The following must must be present to support
#       starting without SSL on platforms with no /dev/random equivalent
#       but a statically compiled-in mod_ssl.
#
<IfModule ssl_module>
SSLRandomSeed startup builtin
SSLRandomSeed connect builtin
</IfModule>


More information about the Mod_python mailing list