[mod_python] The new module loader

Graham Dumpleton wrote:
> Graham Dumpleton wrote ..
>> The new module importer completely ignores packages as it is practically
>> impossible to get any form of automatic module reloading to work
>> correctly with them when they are more than trivial. As such, packages
>> are handed off to standard Python __import__ to deal with. That it even
>> finds the package means that you have it installed in sys.path. Even if
>> it was a file based module, because it is on sys.path and thus likely to
>> be installed in a standard location, the new module importer would again
>> ignore it as it leaves all sys.path modules up to Python __import__
>> as too dangerous to be mixing importing schemes.
>>
>> Anyway, that all only applies if you were expecting PyServer.pyserver to
>> automatically reload upon changes.

Graham, can you enumerate the different ways packages are handled, or is 
it enough to say that packages are never reloaded? In this thread, you 
explain that when a package is imported via PythonHandler, mod_python 
uses the conventional Python __import__, requiring an apache restart to 
reliably reload the package, as in the past.

This also implies that if a published module imports a package, and the 
published module is touched or modified, then the module will be 
reloaded, but not the package. Is this correct?

> BTW, that something outside of the document tree, possibly in sys.path,
> is dealt with by Python __import__ doesn't mean you can't have module
> reloading on stuff outside of the document tree. The idea is that if it is
> part of the web application and needs to be reloadable, that it doesn't
> really belong in standard Python directories anyway. People only install
> it there at present because it is convenient.

There are security benefits to not putting your code in the 
DocumentRoot. It's also useful to develop generic utilities that are 
used in multiple apps (not just mod_python), but that you don't want 
available globally on the system. I prefer extremely minimal frontends 
in the DocumentRoot, with most of my code stored elsewhere. Will the new 
importer support reloading modules outside of the DocumentRoot without 
putting them in sys.path?

> The better way of dealing with this with the new module importer is to
> put your web application modules elsewhere, ie., not on sys.path. You then
> specify an absolute path to the actual .py file in the handler directive.
> 
>  <Directory />
>      SetHandler mod_python
>      PythonHandler /path/to/web/application/PyServer/pserver.py
>      ...

How arbitrary is this path? Must it be within the DocumentRoot?

> Most cases I have seen is that people use packages purely to create a
> namespace to group the modules. With the new module importer that
> doesn't really need to be done anymore. That is because you can
> directly reference an arbitrary module by its path. When you use the
> "import" statement in files in that directory, one of the places it will
> automatically look, without that directory needing to be in sys.path,
> is the same directory the file is in. This achieves the same result as
> what people are using packages for now but you can still have module
> reloading work.

Does it (the initial loading, not the reloading) also apply to packages 
in that directory? Or will it only work with standalone single file 
modules in the root of that directory?

This is all very nifty, because it implies that a mod_python application 
can now be easily distributed by inflating a tarball and specifying the 
PythonHandler accordingly.

If the new importer works outside of the DocumentRoot, and Location is 
used instead of Directory, no files need to be created in the 
DocumentRoot at all. Or is this currently impossible, in regards to 
automatic module reloading? I already do this for some handlers I've 
written, and really like the flexibility provided by the virtualization.