[mod_python] Re: Bug in mod_python.c causing multiple/redundant interpreter creation.

Graham Dumpleton grahamd at dscpl.com.au
Sun Oct 17 00:14:36 EDT 2004


On 16/10/2004, at 11:04 PM, Graham Dumpleton wrote:

>
> On 09/10/2004, at 6:10 PM, Nicolas Lehuen wrote:
>
>> Hi Graham, what are the pros and cons of using the imp module versus 
>> an
>> execfile call or an exec call ?
>>
>> Using Apache 2.0.52 on win32, with a multithreaded MPM, I have 
>> noticed a few
>> problem with mod_python.publisher which reloads the same module many 
>> times
>> when many threads try to load it concurrently. It seems that either
>> mod_python.publisher or the imp module does not properly handle this
>> situation. The result is that I have many instances of the same 
>> module in
>> memory, so I got many connection pools instead of only one and so on. 
>> That's
>> why I decided to fix this and other problems (the famous 'only one 
>> index.py
>> in your application') by reimplementing my own variant of the 
>> publisher.
>
> Found it.
>
> From what I can tell, there is actually a bug in mod_python.c which is
> causing the problem you describe as "reloads the same module many times
> when many threads try to load it concurrently".
>
> I am going to describe it quickly as it is getting late for me and I 
> need
> some sleep. Have attached a patch for people to evaluate and try. Note 
> that
> the patch also includes a Mac OS X specific fix. That fix is harmless 
> in
> itself so just leave it in. The important changes are where APR mutex 
> is
> created and then locked/unlocked.
>
> The basic problem revolves around the Python dictionary used to hold 
> the
> set of interpreters. The code in mod_python.c is trying to use the 
> Python
> GIL to provide exclusive access to that dictionary and any subsequent
> creation of an interpreter.
>
> The only catch is that in creating a new interpreter, the Python core 
> is,
> in someway I don't understand, swapping thread states at some point 
> which
> is allowing other threads to acquire the GIL. This is allowing other 
> threads
> to check the set of interpreters for the same interpreter that is 
> currently
> being constructed and since it isn't there yet, then also goes onto 
> create
> it as well.

Actually, the GIL could get taken away for a few reasons. First, once 
you
start executing actual Python code, it can get taken away. Also, if the
Python code calls back out to C code and does a potentially blocking
operation, it will voluntarily give up the GIL so something else can 
run.
Since the Python apache.import_module() function is going to be called
to load your content handler, and that has to do file i/o, would be a 
very
good candidate for the GIL being given up and thus letting the other 
threads
in to be able to check to see if the interpreter needs creating.

--
Graham Dumpleton (grahamd at dscpl.com.au)



More information about the Mod_python mailing list