[mod_python] Udate python modules without restarting Apache

On Oct 04 23:03, berry groenendijk <berry.groenendijk at gmail.com> wrote:
>
> Subject: Re: [mod_python] Udate python modules without restarting Apache
>
> Indeed when I change .py modules in my developement environment, I
> need to restart Apache to get the changes to work. So, I was also
> wondering how to update .py modules on a server at a hosting provider
> that does not allow you to restart the Apache server proces.
> 
> On Mon, 4 Oct 2004 15:06:34 +0100, MJR <pynode at centrum.cz> wrote:
> > Hi,
> > 
> > I'm running mp servlets on Apache server I don't have control over, i.e. I
> > cannot restart it when I change my .py modules imported by my .mps scripts.
> > 
> > I'm just wondering whether there is any hack for this. How people deal with
> > it in a production environment.

I made a few comments on this at the bottom of a recent message, but I'll
try and cover the choices and some of the issues. One of these days I'll
write a proper little discussion piece about module reloading issues and
put it on my web site. There is a lot to it which isn't obvious nor documented
anywhere. ;-)

The basic problem is that when you use the "import" statement in Python to
bring a module into code executing under Python, if you later what to change
and reinstall that module, about the only thing you can do is to restart Apache.
In an environment where you don't have control over Apache to do that, that
becomes a problem.

The first and quickest thing you can do is a mega hack and may not be a
good thing to do in a production environment, but if you are desperate it
can be a life saver.

What you can do here is to explicitly delete the modules out of sys.modules
and touch all handler code files which import the modules so that they will
get reloaded by mod_python the next time they are accessed. You will need
to ensure that PythonAutoReload is On.

To do this, create a separate directory somewhere which is setup for using
mod_python.publisher. Ie.,

  SetHandler python-program
  PythonHandler mod_python.publisher

In that directory create a file "purge.py" which deletes all the modules
you need to delete. Ie., contains something like:

  def doit():
    try: del sys.modules["HTMLTemplate"]
    except: pass

    try: del sys.modules["vampire"]
    except: pass

    for key in sys.modules.keys():
      if string.find(key,"vampire.") == 0:
        del sys.modules[key]

  modules = sys.modules.keys()
  modules.sort()

  return modules

Then access the page "purge/doit". Note that if Apache is running in prefork
mode, you will have to keep reloading the page until you think you have
managed to invoked the code in all separate forked Apache processes.
Not sure about threaded servers or where separate interpreters are used.

As I said it is a mega hack, but if you are desperate... Be warned that It is
entirely feasible that it may cause everything to stuff up completely. Thus,
if you are talking high volume production site where the pages are being
accessed constantly and you are not the only one with mod_python hosted
code then don't do it. If it is a production web server where your mod_python
code is the only such code and hardly anyone uses it anyway, can be used
with caution.

Note that I would only suggest this be done for code modules which are
specifically intended to be imported as part of your mod_python application.
Ie., don't do it for modules which are used by other mod_python applications.
That way you limit damage to your own stuff and you only have yourself
to blame for the problems. :-)

Moving towards a proper solution, simply don't use "import" at all for your
common code modules in your mod_python application. To achieve this,
you can use mod_python.apache.import_module() method. Thus, instead
of using:

  from mymodules import mymodules

use:

  from mod_python import apache
  module1 = apache.import_module("mymodules.module1")

There are a few issues with the import_module() method though that you
have to be mindful of and you may still have to poke at things a bit to get
it to work.

The first is that even though "mymodules.module1" will now be reloaded,
it will only be done so if this loading of the module is down from inside a
method defined within the content handler and called as part of request
handling.

That is, if you stick this at global scope in your content handler code file,
it will only be called once when the content handler code file is imported.
If you still want it at global scope, you would also need to touch the content
handler code file so that its modification time is updated. That way the
content handler will be reloaded and the module reloaded as well.

Note that the import_module() method will though only reload a module
when the modification time is newer than what it was before. This means
that if you restored an old copy of a module such that its modification time
was actually older, it will not reload. You will need to touch the file to make
its modification time newer.

The next issue is that this is probably useless if "module1" uses "import" to
bring in "modules2" from your "mymodules" directory. That is, any change
to "modules2" will not be reloaded. Thus, any imports amongst your own
modules should also use "import_module()". To find "module2" in this case,
I am not sure if "module1" would need to load it as "mymodules.module2"
or just "module2". I would suggest that "mymodules.module2" would be
preferred and may actually be necessary.

Now if "module2" was imported at global scope in "module1", you end up
with the same problem that even if it is modified it will not be reloaded.
You basically end up with a whole chaining problem and if you do this you
end up still having to touch higher up files in the chain to update the
modification times so that they are reloaded.

So, although extra poking is required in the way of touching some files to
update modification times, you can at least trick mod_python into reloading
things without restarting Apache.

Now, some of the above is based on theory only, so if I am wrong, people
please correct me.

BTW, if you want to see my attempt at an alternate module loading system
which tries to deal with the chaining issue. I have developed a package
for mod_python called Vampire. When used properly, you can change
"module2" and without having to touch the timestamp on any files,
"module1" and any content handlers which use "module1" or "module2"
would be automatically reloaded the next time they are accessed.
Obviously there will be a performance hit with the extra checking, but
if you want maximum flexibility in the way of making changes...

Version 1.0 of Vampire, lacking any documentation on the module loading
system, can be found at:

  http://www.dscpl.com.au/projects/vampire

I have the code for Vampire 1.1 ready, which adds more nice features, but
am still fiddling with the slightly better documentation and examples. I have
not tried integrating it with mpservlets yet, but 1.1 does show at a high level
how it can be used in conjunction with psp. Note that it doesn't try and address
some of the chaining issues of making changes in psp code itself though.

--
Graham Dumpleton (grahamd at dscpl.com.au)