[mod_python] Udate python modules without restarting Apache

Hi Graham, what are the pros and cons of using the imp module versus an
execfile call or an exec call ?

Using Apache 2.0.52 on win32, with a multithreaded MPM, I have noticed a few
problem with mod_python.publisher which reloads the same module many times
when many threads try to load it concurrently. It seems that either
mod_python.publisher or the imp module does not properly handle this
situation. The result is that I have many instances of the same module in
memory, so I got many connection pools instead of only one and so on. That's
why I decided to fix this and other problems (the famous 'only one index.py
in your application') by reimplementing my own variant of the publisher.

My variant uses its own module cache to load pages, based on my recipe here
:

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/302997

The recipe shows a ModuleCache which inherits from FileCache, which in turns
uses fstat to check whether a file is changed and reload it if necessary.
The cache my publisher uses is similar but uses req.finfo to save an fstat
call. This is not rocket science, but the Cache class carefully locks
whatever needs to be locked to ensure a proper behaviour in a multithreaded
context. 

Anyway, here is what I do when I want to "load" a module :

(here, opened is an opened file object, and Module is a placeholder object,
see code)

        try:
            module = Module(name)
            exec opened in module.__dict__
            return module
        finally:
            opened.close()

Do you think this is acceptable ? Are there any drawbacks to this method ?

Regards,

Nicolas

> -----Message d'origine-----
> De : mod_python-bounces at modpython.org 
> [mailto:mod_python-bounces at modpython.org] De la part de 
> Graham Dumpleton
> Envoyé : samedi 9 octobre 2004 04:26
> À : Daniel Popowich
> Cc : mod_python at modpython.org
> Objet : Re: [mod_python] Udate python modules without 
> restarting Apache
> 
> 
> On 09/10/2004, at 12:23 AM, Daniel Popowich wrote:
> 
> >> BTW, mpservlet doesn't use "imp" module to do module importing of 
> >> .mps files but instead uses execfile(). By using execfile() it 
> >> effectively throws away the prior module before reimporting it.
> >
> > The main reason I used execfile is because I wanted to 
> build a cache 
> > of servlet classes keyed on the full filename.  This means I avoid 
> > problems of /somedir/index.mps and /somedir/subdir/index.mps 
> > colliding.  Also, there are so many gnarly issues with using the 
> > import system, I believe I've avoided many a head-ache 
> (e.g., dealing 
> > with pythonpath; wanting to import files that end in .mps).
> 
> The ability to distinguish between index.py in two different 
> directories is possible with "imp" module functions. The 
> trick is that when you finally call imp.load_module, rather 
> than passing the name of the module as the first argument, 
> you pass it some encoded name which incorporates the full 
> pathname to the module code file. In doing this you have to 
> ensure though to get rid of any magic characters that might 
> cause problems. I have also a long long time ago had issues 
> with the length of the name used, thus a scheme to shorten it 
> is possibly a good idea as well.
> 
> But then, I think you perhaps know this, as I recollect 
> mpservlets not using execfile() in an older version, or was 
> that some other package for mod_python I am thinking of.
> 
> FWIW, to see how Vampire mangles the names so as to still be 
> able to use the "imp" module, check out:
> 
>     
> http://www.dscpl.com.au/projects/vampire/examples-01/python-
> modules.html
> 
> The "label" field is the name which the module appears as in 
> sys.modules.
> Those starting with "_vampire_" are the mangled names which 
> Vampire uses in its module importing and caching system and 
> which are getting passed as first argument to 
> imp.load_module. It is a strange combination of md5 hashing 
> and base64 encoding of the full pathname.
> 
> BTW, with the latest version of Vampire, I show how you can 
> integrate use of mpservlets package. Unfortunately my public 
> web site isn't Apache 2.0 so can't host a working example. 
> You can see the source code though at for the mpservlets demo at:
> 
>     
> http://www.dscpl.com.au/projects/vampire/source/examples-06/servlet-
> demo.py
> 
> When using Vampire, your code is in ".py" file and not ".mps" 
> and you don't have to access it as ".mps" either. If you want 
> extensions, you can set it up to use ".html" or ".mps". If 
> you want no extension to be used, you can do that also. Thus 
> a bit more flexible.
> 
> Anyway, I will be making a announcement about latest version 
> of Vampire shortly.
> 
> --
> Graham Dumpleton (grahamd at dscpl.com.au)
> 
> _______________________________________________
> Mod_python mailing list
> Mod_python at modpython.org
> http://mailman.modpython.org/mailman/listinfo/mod_python
>