[mod_python] Mod_Python not releasing memory?

> -----Original Message-----
> From: Graham Dumpleton [mailto:graham.dumpleton at gmail.com]
> Sent: Monday, August 18, 2008 9:30 PM
> To: Chapman, Scott Earl
> Cc: mod_python at modpython.org
> Subject: Re: [mod_python] Mod_Python not releasing memory?
> 
> 
> 2008/8/19  <Scott.Chapman at verizonwireless.com>:
> > I have Apache/2.2.8 (Unix) mod_python/3.3.1 Python/2.4.5 on Red Hat
> > Enterprise Linux AS release 4 (Nahant Update 6).
> > Apache compiled:
> > ./configure --with-mpm=prefork --enable-mods-shared="most proxy
> > charset_lite" --with-expat=system
> >
> > Mod Python:
> > ./configure --with-apxs=/usr/local/apache2/bin/apxs
> > --with-python=/usr/local/bin/python2.4
> >
> > Python:
> > ./configure --enable-shared
> >
> > My application is using Cheetah to generate a very large 
> HTML table and
> > sending it back to the client, after retrieving a bunch of 
> database rows.
> >
> > One hit to Apache to generate one of these big tables runs 
> memory usage up
> > to 30.6% of system RAM:
> >
> > $ ps aux | grep 30316 | grep -v grep
> > USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   
> TIME COMMAND
> > web      30316  2.7 27.7 1559948 1122772 ?   Rl   12:48   1:21
> > /usr/local/apache2/bin/httpd -k start
> >
> > $ ps aux | grep 30316 | grep -v grep
> > USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   
> TIME COMMAND
> > web      30316  2.7 30.6 1559948 1237948 ?   Rl   12:48   1:21
> > /usr/local/apache2/bin/httpd -k start
> >
> > After the query is done and the results delivered to the 
> browser, the Apache
> > process is still holding about 25% of system memory.
> >
> > I don't see any way to free that memory.
> > Once I "return result" to the browser, I can't do 
> "result=None" so I'm
> > relying on Python to release the memory and it is taking 
> way too long for
> > that to happen.  I do set the database result set to None 
> as soon as I can
> > and that helps some.
> >
> > If I get a number of these big hits, OOM starts killing 
> PIDs which is a "Bad
> > Thing".
> >
> > Any clues on how to make that memory release sooner?
> 
> This is normal behaviour. You need to do some reading/research on how
> memory is used in operating systems.
> 
> In short though, in general once a process needs to allocate memory
> from the operating system it is then marked as being in use by that
> process for the life of the process, even if the individual memory
> fragments are freed by code. Such freed memory can be reused within
> the same process, but isn't released back to operating system for
> other processes to use.

If that were true, how would a processes memory usage ever spike and then go back down? I have seen this behavior plenty of times.

Scott


The information contained in this message and any attachment may be
proprietary, confidential, and privileged or subject to the work
product doctrine and thus protected from disclosure.  If the reader
of this message is not the intended recipient, or an employee or
agent responsible for delivering this message to the intended
recipient, you are hereby notified that any dissemination,
distribution or copying of this communication is strictly prohibited.
If you have received this communication in error, please notify me
immediately by replying to this message and deleting it and all
copies and backups thereof.  Thank you.