[mod_python] Re: Bug in mod_python.c causing multiple/redundant interpreter creation.

Graham Dumpleton grahamd at dscpl.com.au
Sun Oct 17 00:14:36 EDT 2004

On 16/10/2004, at 11:04 PM, Graham Dumpleton wrote:

> On 09/10/2004, at 6:10 PM, Nicolas Lehuen wrote:
>> Hi Graham, what are the pros and cons of using the imp module versus 
>> an
>> execfile call or an exec call ?
>> Using Apache 2.0.52 on win32, with a multithreaded MPM, I have 
>> noticed a few
>> problem with mod_python.publisher which reloads the same module many 
>> times
>> when many threads try to load it concurrently. It seems that either
>> mod_python.publisher or the imp module does not properly handle this
>> situation. The result is that I have many instances of the same 
>> module in
>> memory, so I got many connection pools instead of only one and so on. 
>> That's
>> why I decided to fix this and other problems (the famous 'only one 
>> index.py
>> in your application') by reimplementing my own variant of the 
>> publisher.
> Found it.
> From what I can tell, there is actually a bug in mod_python.c which is
> causing the problem you describe as "reloads the same module many times
> when many threads try to load it concurrently".
> I am going to describe it quickly as it is getting late for me and I 
> need
> some sleep. Have attached a patch for people to evaluate and try. Note 
> that
> the patch also includes a Mac OS X specific fix. That fix is harmless 
> in
> itself so just leave it in. The important changes are where APR mutex 
> is
> created and then locked/unlocked.
> The basic problem revolves around the Python dictionary used to hold 
> the
> set of interpreters. The code in mod_python.c is trying to use the 
> Python
> GIL to provide exclusive access to that dictionary and any subsequent
> creation of an interpreter.
> The only catch is that in creating a new interpreter, the Python core 
> is,
> in someway I don't understand, swapping thread states at some point 
> which
> is allowing other threads to acquire the GIL. This is allowing other 
> threads
> to check the set of interpreters for the same interpreter that is 
> currently
> being constructed and since it isn't there yet, then also goes onto 
> create
> it as well.

Actually, the GIL could get taken away for a few reasons. First, once 
start executing actual Python code, it can get taken away. Also, if the
Python code calls back out to C code and does a potentially blocking
operation, it will voluntarily give up the GIL so something else can 
Since the Python apache.import_module() function is going to be called
to load your content handler, and that has to do file i/o, would be a 
good candidate for the GIL being given up and thus letting the other 
in to be able to check to see if the interpreter needs creating.

Graham Dumpleton (grahamd at dscpl.com.au)

More information about the Mod_python mailing list