[mod_python] Articles on module importing.

Graham Dumpleton wrote:
> Jorey Bump wrote ..
> 
>>Most (but not all) of the issues you raise concern writing handlers. I'd
>>hate to see someone leave with the wrong impression about how to deal 
>>with these issues in their applications, when the advice given is more
>>appropriate at the handler level.
> 
> If you feel that this primarily only affects those who are writing their own
> high level handlers, as opposed to those who might instead be using the
> mod_python.publisher handler, then I would suggest you are wrong.

No, I understand that an application inherits the problems of the 
handler on which it was based.

> The current mod_python.publisher handler uses the import_module()
> function internally to load modules and so it is just as much afflicted by many
> of these problems, more so when within a published module the code uses
> the import_module() explicitly and even if only the "import" statement is
> used within that module.

Understood. My point is that alternate import mechanisms such as 
import_module() shouldn't be encouraged in *published* modules or 
modules that are based on other handlers. I think we both agree that we 
want the average Python programmer to be able to start using mod_python 
  as seamlessly as possible. Unfortunately, they need to choose a 
handler, and usually aren't any more interested in writing their own 
handler than a PHP programmer would be. I like to use Publisher, and I 
have worked around every limitation except one: After editing modules 
imported by a published module, apache must be restarted. So I agree 
that this is a serious problem that must be fixed, and not only in the 
Publisher handler.

>>I think that defining the higher level problems and goals now will help
>>to arrive at solutions that might not necessarily involve fixing current
>>components.
>>
>>For example, let's imagine that all of mod_python's import problems can
>>be solved with some kind of magic bullet:
>>
>>1. We want the PythonPath extended automatically based on the location
>>of a file.
> 
> This is not a goal or requirement, it is indicative of a solution. To provide
> the desired behaviour doesn't require PythonPath be extended as there are
> actually better ways of achieving the same thing and still maintain the
> required transparency that you require.
> 
> The real goal you are probably alluding to is that within a Python code
> file serving as a handler you want the "import" statement to be able to
> pick up a module that resides in the same directory. Yes?

No. I think it's poor practice to put any kind of library files in a 
directory that is accessible to browsers, even with other languages. The 
risk is too high that someone with knowledge of your directory structure 
can call your support modules directly, causing unintended side effects.

What I would like to see is path extension that is private to the 
application (not just the handler) but is otherwise indistinguishable 
from other parts of the search path. Naturally, it should come first, 
somewhat *emulating* the way the current directory is added to sys.path 
when running python from the command line, but without the risk of 
placing the modules/packages where they can be exposed to a browser. 
Because mod_python runs as the apache user, and not the user who owns 
the files, it's impossible to establish a convention that places this 
extended path outside the DocumentRoot. But a convention is required, 
hence my .htPython suggestion.

> At the moment, if using mod_python.publisher this only works where the
> published module is in the same directory as where the PythonHandler
> directive was specified, it does not work where the published module is
> in a subdirectory. Thus, should the ability to use the "import" statement
> in this way also work for a published module in a subdirectory of where
> the handler directive was originally defined?

No. For backwards compatibility, I'd leave this behaviour as-is. We use 
the phrase "published module" to refer to a module that we intend a 
browser to access, but mod_python.publisher makes no such distinction. I 
don't want to see mod_python automagically package subdirectories or add 
them to the path because it gives newbies more rope to hang themselves. 
How can mod_python know if a subdirectory contains published or 
importable modules? It's better to check for a specially named directory 
and let the handler add it to the path. And it's not enough to restrict 
this special directory to the location where the PythonHandler was 
specified, because we want to be able to easily install the application 
elsewhere. Therefore, it has to be in the same directory as the 
published module. For example, http://host/app/ might contain this:

  .htPython/module1.py
  .htPython/module2.py
  app1.py
  app2.py

#app1.py

import module1
import module2

def index
     a = module1.do(something)
     return module2.show(a)

app1 imports modules from the special directory, but app2 doesn't. No 
big deal. What's nice is that app.tgz could be untarred anywhere that 
the appropriate PythonHandler is defined, and it wouldn't be necessary 
to manually extend the PythonPath with an explicit file specification.

The only precaution left for the developer is to name imported 
modules/packages appropriately to prevent collisions, but this must be 
done for all Python apps, anyway. Ideally, the path would be modified 
per "published" module, but I don't know if that's even possible.

>>2. We want imported modules to be unloaded/reloaded automatically if 
>>modified or replaced in any way.
> 
> What sort of imported modules though? Should this only apply to modules
> imported by import_module() whether as a high level handler or explicitly
> by the user or within something like mod_python.publisher? Should it also
> apply to modules imported using the "import" statement for (1) above?
> Should there be depth based dependency checking such that if some
> grand child module imported at global scope within a high level handler
> is changed, that all parents of the module up to the high level handler
> should be automatically reloaded?

Depth is certainly an issue, but one could assume (or at least establish 
the convention) that the custom modules in .htPython mostly import 
stable modules that are unlikely to change, such as those in the 
standard library. The mere possibility of specifying a conventional and 
portable location for custom modules goes a long way towards solving the 
import problem.

>>3. Imports of application code should be handled in a way that supports
>>the packaging and portablility of an application.
> 
> Packages are a big problem when it comes to making automatic module
> reloading work properly. I have probably put in a few days worth of
> effort just into this particular issue of late and still don't have a workable
> solution. I got to the point where I could finally see what might yield
> a workable solution and then stopped as the code was getting overly
> complicated, so much so that one had to question whether it was worth
> the trouble.

Sorry, by packaging I meant simply creating a tarball or zipfile for 
distribution, not a Python package.

> The problem with packages...

I currently develop most of my Publisher applications as packages that 
are stored in a directory prefixed to PythonPath. It works fine and I 
couldn't bear to lose that functionality (although I obviously still 
need to restart apache after editing package code). Unfortunately, I 
don't know enough about the import mechanism to understand the 
difference between reloading modules vs. packages, so I don't know if my 
scheme offers any solution here (or even if it's feasible).