[mod_python] Execute python code while having generated page content already sent to browser - possible ?

On 05/02/2008, Mateusz Korniak <mateusz at ant.gliwice.pl> wrote:
> Hi !
> Is there any way to continue execute python code while having generated page
> content already sent to browser ?
>
> What I would like to implement is:
>
> content = generate_page_content() # It takes 0.02[s]
> sent_to_browser(content) # We are not going to send anything more to browser
> lazy_processing_of_request() # May take serveral seconds.
>
> I do not want make browser to wait util lazy_processing_of_request() finishes
> to start receiving page contents.
>
> Is it possible ?

You can use:

    req.status = apache.HTTP_OK
    req.content_type = "..."
    ....
    req.write("....")

    # do processing

    return apache.OK

The problem with doing this is that although the browser gets the
response and can display it while you do your processing, Apache will
not close the socket connection or be able to reuse it if keep alive
is enabled, until the processing is complete. This could result in
Apache holding lots of connections open longer than it needs to.

Other option is to create a background thread to do the processing, so
long as you give it data which is independent of the request object.
In other words, if you try and access the request object after having
returned from the request, Apache will most likely crash. This is
because mod_python doesn't adequate protect against this happening.
Problem with threads is controlling how many there are, as such you
might need to implement a thread pool and job queuing systems for
processing data so the number of threads doesn't blow out in number.

What is probably the preferred way, is to run a long running back end
process and communicate to it using XML-RPC, passing any data across
which can then be processed by that back end process. Once the handler
has handed off the data, it can just return immediately. Using a back
end process also has the benefit that when Apache is running as a
multiple process server, then subsequent requests could ask about the
status of something by supplying some unique value returned by the
original request. To do this though, the back end process would have
to have some concurrency mechanism to allow processing in parallel
with incoming requests to get status etc.

Graham