[mod_python] How to stop reimporting modules

Hi Barry,

Barry Pearce wrote:
> Hi Jim,
> 
>>> This is an interesting issue - I have my own handler but I have a 
>>> problem where mod_python does NOT cause modules to be reloaded - 
>>> despite having changed on the disk I would have expected the module 
>>> to reload...but it stays the same until I restart the web server - 
>>> perhaps there is something Im doing wrong!!
>>>
>>> I really dont want to restart the apache server to upgrade my 
>>> software - it kills all current sessions and downloads....
>>>
>>> Any ideas?
>>
>>
>>
>> My impression is that the handler specified in the apache 
>> PythonHandler directive does not get reloaded when changed. Of course 
>> I may be completely off the mark here.
> 
> 
> Ah. OK....thats why everything that it in turn imports also remains 
> fixed in memory...its just a pain in the bum when I change something 
> that the re-import does not take place.
> 
>> For my code, the PythonHandler cms.publisher is just a stub that does 
>> a little pre-processing and then uses apache.import_module() to get my 
>> real handler. It's apache.import_module that does the reloading magic, 
>> and so I avoid restarting apache when my handler code changes. I 
>> figure once my code has pretty much settled down, I can change the 
>> PythonHandler to specify the real handler.
> 
> 
> The issue comes during upgrade of live sites with many users with 
> concurrent sessions and no 'quiet' period per say where a restart could 
> be sensibly placed.
> 
> Interestingly I have been using a large number of python files and using 
> the standard python import - however, if the top level module is not 
> reimported it will never execute any of the other import statements.
> 
> Interesting...sorry im rambling...
> 
> Perhaps I need to re-evaluate my import strategy and call 
> apache.import_module() instead of just relying on the python import...
> 
> Now I dont want it all checked on each request - but maybe I can write a 
> handler that I can send a request to which re-evalutes all imported 
> modules and then appropriately reloads them...

I don't think this will work in the apache prefork model. (I have no 
idea about the threaded model). In the prefork, each child process has 
it's own copy of the python interpreter and modules. Your call to the 
special handler will only cause the child that handles that request to 
reload. All the other children will still have a copy of the old 
modules. You have to make sure that each child gets a new copy - which 
is what happens when you restart apache. All the child processes are 
killed and then new ones created which will import the new modules.

So maybe your main handler can check a flag defined by a PythonOption 
directive and reload as necessary? Thus each child process will be 
forced to reload it's modules.

def handler(req):
    opts = req.get_options()
    reload_modules = int(opts.get('RELOAD_MODULES', '0'))

    if reload_modules:
         do_magic_module_reload()

    ... handle request ...

Edit your apache conf file for the site:
   PythonOption RELOAD_MODULES 1

Run 'apache reload' to read the config file changes. Subsequent requests 
will see the flag and reload of any modified modules. Once everything 
seems to be working change the flag back to 0 and run 'apache reload'.

Or how about having a version variable defined with module scope and a 
corresponding PythonOption directive.

# your publisher module
__version__ = 41

def handler(req):
    opts = req.get_options()
    if opts.has_key('MODULE_VERSION'):
        version = int(opts['MODULE_VERSION'])
        if __version__ < version:
            do_magic_module_reload()
            __version__ = version

    ... handle request ...

And in your apache config:
PythonOption MODULE_VERSION 42

There, all you need to do now is write the code for 
do_magic_module_reload(). Oh, and send me copy when you're done. :)

But seriously, apache.import_module won't do what you want, but it looks 
like an interesting place to start. Python introspection is a new topic 
for me, but I don't think a solution to this problem is too difficult.

Regards,
Jim