[mod_python] mod_python 3.3.1 memory leak?

All I can say is that is simply how Apache works. It is not a problem
with mod_python. To try and explain it, look at simpler code of
ap_rflush() which is one of two Apache functions called when you call
req.write().

AP_DECLARE(int) ap_rflush(request_rec *r)
{
    conn_rec *c = r->connection;
    apr_bucket_brigade *bb;
    apr_bucket *b;

    bb = apr_brigade_create(r->pool, c->bucket_alloc);
    b = apr_bucket_flush_create(c->bucket_alloc);
    APR_BRIGADE_INSERT_TAIL(bb, b);
    if (ap_pass_brigade(r->output_filters, bb) != APR_SUCCESS)
        return -1;

    return 0;
}

The important thing here is r->pool. This is a memory pool which
exists for the life of a request. When you allocate memory from the
memory pool, it will only be reclaimed at the end of the request. Ie.,
it isn't like malloc/free where you can give back a block of memory
using free and it will be reusable straight away.

Now, every time a flush is occurring, it needs to create a bucket
brigade which holds a special flush object. This is then passed down
through the output filters. In this case it causes any pending data to
be flushed out.

Now because the memory for those bucket structures is only reclaimed
at the end of the request, it means if you have a very long running
request which outputs data in small blocks, then there will be an
incremental use of memory because of the need to create the bucket
structures. This memory will be unavailable for use by anything else
until the requests ends.

FWIW, I didn't realise Apache did this either. I can see there are
potentially good reasons for it being done this way, but still was a
surprise.

Graham

On 03/07/07, yubing <trueice at gmail.com> wrote:
> oh, sorry, that's my typo in the mail, it should have been:
> ------------------------------
> def pump_file(req):
>     fp = open("/dev/zero", "r")
>     while(True):
>         buf = fp.read(4096)
>         try:
>             req.write(buf)
>             time.sleep(0.1)
>         except:
>             fp.close()
>             break
> -------------------------------
> you can still observe the memory going up slowly
> BTW: I'm using the prefork mpm of apache
>
> On 7/3/07, Graham Dumpleton <graham.dumpleton at gmail.com > wrote:
> > On 03/07/07, yubing <trueice at gmail.com > wrote:
> > > Our project has a live HTTP video streamer written in python, which
> keeps
> > > pumping a stream out to the client.
> > > The HTTP serving module is a simple mod_python request handler running
> on
> > > Apache 2.2.4 with mod_python 3.3.1 (Python 2.5.1).
> > > The stream is read out of our streaming server via TCP socket, and the
> > > python script just do some simple processing like header building, each
> > > allocated buffer is del-ed after being used.
> > >
> > > The problem is:
> > > We observed that after its running serveral hours, its memory occupation
> > > grows up to serveral hundreds of megabytes and keeps growing in 4k-8k
> > > increment every 1-2 seconds.
> > >
> > > Below is a simple testing scriptlet, the memory leaking issue is not so
> > > serious as our live serving module, but you can still observe 4k memory
> > > growing every serveral seconds.
> > >
> > > Could anyone help me to figure out the root cause of this issue?
> > >
> > > --------------------------
> > > import time
> > >
> > > def pump_file(req):
> > >     while(True):
> > >         fp = open("/dev/zero", "r")
> > >         buf = fp.read(4096)
> > >         try:
> > >             req.write(buf)
> > >             del buf
> > >             time.sleep(0.1)
> > >         except:
> > >             fp.close()
> > >             break
> >
> > BTW, you do realise that you open the file in the loop when it should
> > be outside.
> >
> > Graham
> >
>
>
>
> --
> truly yours
> ice