|
Graham Dumpleton
grahamd at dscpl.com.au
Sun Oct 17 00:14:36 EDT 2004
On 16/10/2004, at 11:04 PM, Graham Dumpleton wrote: > > On 09/10/2004, at 6:10 PM, Nicolas Lehuen wrote: > >> Hi Graham, what are the pros and cons of using the imp module versus >> an >> execfile call or an exec call ? >> >> Using Apache 2.0.52 on win32, with a multithreaded MPM, I have >> noticed a few >> problem with mod_python.publisher which reloads the same module many >> times >> when many threads try to load it concurrently. It seems that either >> mod_python.publisher or the imp module does not properly handle this >> situation. The result is that I have many instances of the same >> module in >> memory, so I got many connection pools instead of only one and so on. >> That's >> why I decided to fix this and other problems (the famous 'only one >> index.py >> in your application') by reimplementing my own variant of the >> publisher. > > Found it. > > From what I can tell, there is actually a bug in mod_python.c which is > causing the problem you describe as "reloads the same module many times > when many threads try to load it concurrently". > > I am going to describe it quickly as it is getting late for me and I > need > some sleep. Have attached a patch for people to evaluate and try. Note > that > the patch also includes a Mac OS X specific fix. That fix is harmless > in > itself so just leave it in. The important changes are where APR mutex > is > created and then locked/unlocked. > > The basic problem revolves around the Python dictionary used to hold > the > set of interpreters. The code in mod_python.c is trying to use the > Python > GIL to provide exclusive access to that dictionary and any subsequent > creation of an interpreter. > > The only catch is that in creating a new interpreter, the Python core > is, > in someway I don't understand, swapping thread states at some point > which > is allowing other threads to acquire the GIL. This is allowing other > threads > to check the set of interpreters for the same interpreter that is > currently > being constructed and since it isn't there yet, then also goes onto > create > it as well. Actually, the GIL could get taken away for a few reasons. First, once you start executing actual Python code, it can get taken away. Also, if the Python code calls back out to C code and does a potentially blocking operation, it will voluntarily give up the GIL so something else can run. Since the Python apache.import_module() function is going to be called to load your content handler, and that has to do file i/o, would be a very good candidate for the GIL being given up and thus letting the other threads in to be able to check to see if the interpreter needs creating. -- Graham Dumpleton (grahamd at dscpl.com.au)
|