[mod_python] telling mod_python to exit its children

Martin Blais wrote ..
>
> thanks for the pointers.  it seems from your description that Vampire
> is doing the same as i am with my reloader module.  from the details
> you mention, it seems you're also running against the similar
> weirdness.  i'll check it out soon.  however all i can hope to find
> there is a less buggy version of the approach i'm already trying.

That you talk about fiddling with sys.modules means that although you
might be trying to achieve the same thing you aren't doing it in the same way.
I am not experiencing any wierdness in the stuff I work with, yes there
will be some obscure limitations that can be avoided, but other than that
it works fine and generally only need to restart Apache when I change
modules installed into site-packages directory.

Continuing to use the default "import" and "reload" or the import routines
in the "imp" module, whereby things are still stored in sys.modules as you
appear to be do, does not work very well and you will always have problems
with it. The default module loader in mod_python uses the "imp" module for
loading modules and has various problems, especially where someone tries
to use the same module name in different places for representing a request
handler.

Relying on the "imp" module, unless you resort to fiddles, also has problems
with the fact it reloads on top of existing modules and doesn't start fresh.
This can result in stale data being preserved and the inability to remove
completely functions or classes from a module. Reloading on top of an
existing module can also be a recipe for disaster in a multithreaded server
as you could be reloading a module on top of the old while another request
handler is executing in it.

Vampire thus does not use the "imp" module except for creating empty
modules into which distinct imports are done by using execfile(). Vampire
does not reload on top of existing modules, thus avoiding screwing up
requests being managed in a separate thread and ensuring that stale
data doesn't persist along with any removed classes or functions.

Because however preservation of data is sometimes required when reloading
a module, Vampire also provides a means of supplying hook functions which
are executed when a module is reloading so as to allow you to create the
initial environment that the new module loads into. This means you can copy
data from the old module into the new at point of reloading. Because you
have control of the hook function which does this, you can perform any
necessary thread locking in it to ensure that when copying data you don't
cause problems for other threads still executing in the code.

There is also a hook function which is called when an old module is being
purged so that you can do some cleanup. Thus, although a new module might
create a new database pool, you may still have to properly shutdown the old
one.

I don't know to what extent you have thought about these problems, but
there are lot more issues than most would appreciate when one looks at
the wider context of how people may use mod_python.

> but what i'm trying to do now, is different, is to bypass all that,
> really.  i want to find out how to communicate to mod_python to exit
> all its current pool of children (perhaps not all at the same time,
> but at least to mark them as "not runnable").  it's a trivial approach
> to that problem, which will certainly work well in all cases (in a
> non-threaded apache).

There is no mechanism that I know of for controlling the life time of
the interpreters that mod_python creates or even Apache processes
as a whole (bar causing seg faults). I would suggest it would be far
from trivial to consider even doing such a thing and may serve only
to introduce a level of fragility into the whole system.

> if it's not in there already, i will probably add that feature,
> because it would be a much cleaner and easier way to implement this
> automatic reload,

Good luck then. Having delved into the internal C code of mod_python
I don't personally feel it is a practical solution and don't believe it would
be easier.

> and it would work with all frameworks, and perhaps
> it could even find other uses.

The main issue I have found as far as making it work with other frameworks
is simply the desire not to have to change the code of those frameworks
to be making explicit calls into a special module loading system. This
is made harder when the framework uses code generation, eg. Cheetah
templates, and it hard codes use of "import" for module imports for pages
that derive from other pages.

In the latest version of Vampire I have in subversion repository (not released
as official version yet but still available), I use special "import" hooks so
that within certain situations use of "import" automatically maps onto the
Vampire module importing system. This means that one can avoid having
to change the code and get the benefit of automatic reloading for existing
code.

At the moment I have only made this work for importing of file based
modules and not packages and works only at global scope within a module
because of what I am trying to achieve and in what context I anticipate
it should be used. Ie., I do not believe that module reloading should be
applied to any sort of standard module which is installed as part of Python
or in the site-packages directory. It should only really be used for modules
supporting actual request handlers associated with web page delivery.

It might be used with web application specific utility modules as well
but there should never in effect be an overlap between sys.path and any
directories for which the automatic module loader is active. If there is
an overlap and a single module is imported using standard "import" and
the module loader at the same time, all sorts of problems could then
arise.

Anyway, getting a module importing system to work well is an interesting
exercise. Just make sure you think outside of your own requirements and
see how others might use mod_python and thus what other things you
may have to consider.

I'll see if I can get all the Vampire module importing stuff documented
over the next week along with all the issues which have influenced it.
Keep an eye out on the articles section of the Vampire web site.

Graham