|
michael bayer
mike_mp at zzzcomputing.com
Sun Jun 19 11:58:34 EDT 2005
I just searched around for this, and hey youre right, it does. Myghty imports "in-memory" modules in a similar way (i.e. via imp.new_module + exec)....but I hadnt thought of just bypassing sys.modules altogether. A "request-level" module really doesnt need to be generally importable so that now makes sense. I would most like to find a way to do this while also taking advantage of .pyc files (seems like, create .pyc file if it doesnt exist, else if it exists, read four bytes + one long value, then use marshal.load() to create a code object, then exec that). On Jun 19, 2005, at 4:19 AM, Nicolas Lehuen wrote: > Just as a note, those issues have been thoroughly covered by Graham > Dumpleton in his Vampire project, and some of his ideas have been > retrofitted into the development version of mod_python. > > Regards, > Nicolas > > 2005/6/18, michael bayer <mike_mp at zzzcomputing.com>: > >> I use the "two-check" approach to this, which allows multiple threads >> to get at an already-imported resource without blocking each other, >> while allowing the actual import to occur in a threadsafe >> manner...suppose every access of a module calls "verify_module" which >> reloads the module if needed, using the functions _needs_reload to >> perform a check and _reload_module to do the actual reload: >> >> import threading >> mutex = threading.RLock() >> >> def verify_module(module): >> if _needs_reload(module): >> mutex.acquire() >> try: >> if _needs_reload(module): >> _reload_module(module) >> finally: >> mutex.release() >> >> in the normal case, threads will call _needs_reload() without >> synchronization and return, without any bottleneck occuring. when >> the reload is actually required, threads will line up onto the >> mutex.acquire() and re-check once entering the critical section, thus >> allowing only the first thread inside to actually perform the reload. >> >> an additional optimization is to use a unique mutex for each >> module...that way multiple threads that are reloading several >> different modules also don't block each other. >> >> the issue of other threads executing code against the module while >> its being reloaded is not really addressed here. as a normal reload >> () or imp.load_module() doesnt remove any of the old module values, >> its usually (but not always) the case that nothing really >> happens....unless there are persistent pointers to the module's >> globals lying around elsewhere, which then become outdated. I did >> see a reload() implementation that loads a new module under a >> temporary name, then moved it's dictionary over, which attempts to be >> more atomic...I havent tried it but its over at http:// >> lists.canonical.org/pipermail/kragen-hacks/2002-January/000302.html . >> >> On Jun 17, 2005, at 3:09 PM, Dan Eloff wrote: >> >> >>> Well this is the biggest problem in my otherwise threadsafe code. >>> From >>> what I've read, importing/reloading modules in a threaded >>> environment >>> is just plain dangerous. There could be any number of requests >>> executing code from and accessing global variables in a module >>> when it >>> changes and gets reloaded. Then you have the two level problem of >>> what >>> happens to the executing requests, as well as what happens when >>> multiple incoming requests all spot that the module has changed, and >>> reload it multiple times in quick succession, re-executing code >>> in the >>> module scope (this is pretty much worst case) >>> >>> I know Graham knows a lot more about this than I do (help!). >>> >>> I do have a lot of flexibility here, since I'm using my own >>> publisher >>> I do not have to use apache.import_module. In fact I would prefer to >>> not have the whole importing/reloading business affect my code >>> outside >>> of the publisher, that way I can change things internally at will. >>> Clearly one option is to create a simple import function that >>> imports >>> once only in a production environment, and every time in a >>> development >>> env, but I'm still interested in exploring the options. >>> >>> -Dan >>> >>> _______________________________________________ >>> Mod_python mailing list >>> Mod_python at modpython.org >>> http://mailman.modpython.org/mailman/listinfo/mod_python >>> >>> >> _______________________________________________ >> Mod_python mailing list >> Mod_python at modpython.org >> http://mailman.modpython.org/mailman/listinfo/mod_python >> >
|