[mod_python] Mod_Python not releasing memory?

Hmm i had similar trouble with mysql. Turned out to be the connection  
to mysql must not be made global. My fix was to open it in my handler  
and to close it in the fixup by using req.register_cleanup

The connection to the db should be made within the req object: req.db  
= (make connection to your db) and like so with a cursor.

Also think 'stream'

Do not cursor.fetchall but fetchone, write the output after each fetch

If you load with fetchall and build the entire page, the data is I'm  
memory twice!! With streaming only one record is in apache process,  
you can handle way more request in the same memory and it greatly  
improves user experiance.

Hope it helps,
Martijn



Op 19 aug 2008 om 16:06 heeft "Scott.Chapman at VerizonWireless.com" <Scott.Chapman at VerizonWireless.com 
 > het volgende geschreven:\

>
>
>> -----Original Message-----
>> From: Graham Dumpleton [mailto:graham.dumpleton at gmail.com]
>> Sent: Monday, August 18, 2008 9:30 PM
>> To: Chapman, Scott Earl
>> Cc: mod_python at modpython.org
>> Subject: Re: [mod_python] Mod_Python not releasing memory?
>>
>>
>> 2008/8/19  <Scott.Chapman at verizonwireless.com>:
>>> I have Apache/2.2.8 (Unix) mod_python/3.3.1 Python/2.4.5 on Red Hat
>>> Enterprise Linux AS release 4 (Nahant Update 6).
>>> Apache compiled:
>>> ./configure --with-mpm=prefork --enable-mods-shared="most proxy
>>> charset_lite" --with-expat=system
>>>
>>> Mod Python:
>>> ./configure --with-apxs=/usr/local/apache2/bin/apxs
>>> --with-python=/usr/local/bin/python2.4
>>>
>>> Python:
>>> ./configure --enable-shared
>>>
>>> My application is using Cheetah to generate a very large
>> HTML table and
>>> sending it back to the client, after retrieving a bunch of
>> database rows.
>>>
>>> One hit to Apache to generate one of these big tables runs
>> memory usage up
>>> to 30.6% of system RAM:
>>>
>>> $ ps aux | grep 30316 | grep -v grep
>>> USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START
>> TIME COMMAND
>>> web      30316  2.7 27.7 1559948 1122772 ?   Rl   12:48   1:21
>>> /usr/local/apache2/bin/httpd -k start
>>>
>>> $ ps aux | grep 30316 | grep -v grep
>>> USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START
>> TIME COMMAND
>>> web      30316  2.7 30.6 1559948 1237948 ?   Rl   12:48   1:21
>>> /usr/local/apache2/bin/httpd -k start
>>>
>>> After the query is done and the results delivered to the
>> browser, the Apache
>>> process is still holding about 25% of system memory.
>>>
>>> I don't see any way to free that memory.
>>> Once I "return result" to the browser, I can't do
>> "result=None" so I'm
>>> relying on Python to release the memory and it is taking
>> way too long for
>>> that to happen.  I do set the database result set to None
>> as soon as I can
>>> and that helps some.
>>>
>>> If I get a number of these big hits, OOM starts killing
>> PIDs which is a "Bad
>>> Thing".
>>>
>>> Any clues on how to make that memory release sooner?
>>
>> This is normal behaviour. You need to do some reading/research on how
>> memory is used in operating systems.
>>
>> In short though, in general once a process needs to allocate memory
>> from the operating system it is then marked as being in use by that
>> process for the life of the process, even if the individual memory
>> fragments are freed by code. Such freed memory can be reused within
>> the same process, but isn't released back to operating system for
>> other processes to use.
>
> If that were true, how would a processes memory usage ever spike and  
> then go back down? I have seen this behavior plenty of times.
>
> Scott
>
>
> The information contained in this message and any attachment may be
> proprietary, confidential, and privileged or subject to the work
> product doctrine and thus protected from disclosure.  If the reader
> of this message is not the intended recipient, or an employee or
> agent responsible for delivering this message to the intended
> recipient, you are hereby notified that any dissemination,
> distribution or copying of this communication is strictly prohibited.
> If you have received this communication in error, please notify me
> immediately by replying to this message and deleting it and all
> copies and backups thereof.  Thank you.
>
>
>
> _______________________________________________
> Mod_python mailing list
> Mod_python at modpython.org
> http://mailman.modpython.org/mailman/listinfo/mod_python