[mod_python] Mod_Python, multicore and concurrency

2009/5/31 John Lister <john.lister at kickstone.com>:
> Graham Dumpleton wrote:
>>
>> If you are using mod_python sessions at the same time, they will block
>> another request using same session at the same time. It can also block
>> other distinct sessions as well because of the limited number of
>> shared locks used to mediate global session locking.
>>
>
> Cheers, just noticed your blog and having a read about apache processes and
> mod_python. I still have a couple of questions - which may in part be due to
> my confusion:
> You blog states that the parent apache process initialises the python
> interpreter, does mod_python therefore run in the main apache process with a
> number of sub-interpreters and the child apache processes farm off requests
> to this one? Or does each child apache process get its own couple of a
> python interpreter - which is what i thought and other parts of your blog
> seem to suggest this.

The main Python interpreter is initialised within the parent. That is
inherited by child processes when Apache forks its child server
processes. Any additional sub interpreters as required, are then only
created within the context of the child server processes. All Python
request handling occurs in the child server processes.

> I'm not sure what you mean by session above.

I specifically mean if you were using the Session class provided with
mod_python directly, or indirectly by using 'session' attribute in a
PSP page.

> For example say i have a quad
> core system with the default apache settings for pre-fork and i make 4
> distinct requests for a python page at the same time, am i wrong to think
> that 4 child processes will run (ideally one per core) . At the moment i see
> a spike on all the cores for the first request, but if i repeat the above,
> only one core seems to be maxed out at 100%.

All I can suggest to try and debug relationships between requests by
printing out to the error log the time of the request, the os.getpid()
of the process handling the request and the Apache req.connection.id
of the inbound connection. The latter to ascertain whether keep alive
connections may be part of the issue.

Graham

>> Other than that guess, it is impossible to answer without know exactly
>> what your code does.
>>
>
> The code is relatively straightforward in that it does a series of cpu bound
> calculations before returning the result. These generally take in the order
> of seconds rather than milliseconds which is why i'd like to spread them
> across multiple cores, the handler is something like this:
>
> def handler(req):
>   html=generateHtml()
>   req.write(html)
>   return apache.OK
>
> Thanks
>
> JOHN
>
>