[mod_python] mod_python 3.3 importer with "import" statement

On 31/05/07, Ben Hoyt <ben at micropledge.com> wrote:
> Hi guys,
>
> We're starting a web site running on mod_python, and we've run into import
> problems, specifically using
>
> from DIR import FILE
>
>  apache.import_modulewhere DIR is a directory (not a module or package) and
> file is a FILE.py file. It works with standard Python importing, but now
> that we have multiple virtual servers, we need to use, but we can't quite.
>
> Here's the long gory story:
> ---------------------
>
> Up till now we've just been using the standard import statement, and it's
> worked fine. But now we're running two copies of our Python code-base on two
> virtual servers. So there are two files called " code.py" and suchlike in
> two different directories, and the imports get confused if the two are out
> of sync. All as the docs explain ... because of how sys.modules is stored
> without full paths.

By default VirtualHosts use distinct Python interpreter instances.
Thus, unless you have explicitly set PythonInterpreter explicitly so
the same interpreters is used for both hosts there shouldn't be a
problem if the two virtual hosts use the same module/package name but
different versions obtained from different sys.path directories.

> So I've just been trying to use the new apache.import_module(), but I'm
> running into various problems because we have multiple directories in our
> code structure, which looks something like this:
>
> code/
> code/code.py  # main file for mod_python usage, imports everything
> code/cli.py  # command line "test harness" loader
> code/logic/dbase.py
> code/logic/errors.py
> code/handlers/home.py
> code/handlers/users.py
> ...  # and heaps more files in each of the subdirs
>
> "logic" and "handlers" aren't actual packages, so we thought we'd be okay
> using apache.import_module() on code.py, and all the import statements
> scattered throughout the code would just work (am I correct in this? if the
> top-level import uses import_module, the rest will automatically?). code.py
> is actually run by modpython_gateway.py, a WSGI wrapper for mod_python that
> we modified to use import_module instead of __import__.
>
> But the problem is that it bombs out on statements like:
>
> from logic import errors
>
> with "ImportError: No module named logic"
>
> I guess this is because logic isn't a file/module, it's a directory. Is this
> correct? (In .htaccess, I have PythonOption mod_python.importer.path
> "['c:/www/mpledge/code']" so it definitely knows where to go, but the import
> statement fallback probably doesn't. It won't help if we set sys.path,
> because then we're back to the old import problems.)
>
> In short, we're wondering the best way forward using mod_python. The options
> we see are:
>
> 1) Only ever run one code base. Non-ideal.
>
>  2) Override __builtin__.__import__ with our own one that checks if it's a
> directory and calls your import_module on the result, e.g.,
> import_module('logic/errors'). Would this work?

Overriding import is more complicated than just replacing __import__. ;-)

> 3) Write a patch to modpython's import_module that adds that functionality.
> But I'm guessing there's some good reason it doesn't support this -- is that
> correct?

It wasn't possible to support true Python packages. But then, as
documented apache.import_module() does support pseudo packages using
relative paths. I cant remember why I didn't support the pseudo
packages through import, most likely because the __init__.py file if
it did exist wouldn't get imported and as a result felt that that
might confuse users.

> 4) Change all our import statements to import_module calls. We have a lot of
> them in different forms, so this would be messy. Particularly because we
> want it to work in test/command-line mode too. And at the moment with import
> statements it works just as well from Apache with code.py and from the
> command line with "python -i cli.py". So we'd have to do if/else conditions
> around all the imports, for whether it's in cli/test mode or not.
>
> 5) Look into FastCGI or something ... but we're not sure we want to go there
> at this stage.

I'd suggest other things before that, maybe mod_wsgi perhaps. :-)

> Oh, and one more thing, we're also using Cheetah, so I guess we'll have to
> change its __import__s to import_module calls too...
>
>  What do you think is the best way?

Since your application is designed for WSGI, I would recommend against
using any features which are specific to the mod_python module
importer, whether that be implicit or explicit. The reason for that is
that by doing so your application would then no longer be portable to
other WSGI hosting solutions which cause you more work if you want to
change later. I do acknowledge though that by making it a portable
WSGI application, you have to give up automatic module reloading and
thus would have to restart Apache whenever changes are made.

That said, to make it a portable WSGI application, you would just need
to turn your whole collection of modules into a package. The
__init__.py files can though be all empty. Thus:

micropledge/

micropledge/__init__.py
micropledge/code.py  # main file for mod_python usage, imports everything
micropledge/cli.py  # command line "test harness" loader

micropledge/logic/__init__.py
micropledge/logic/dbase.py
micropledge/logic/errors.py

micropledge/handlers/__init__.py
micropledge/handlers/home.py
micropledge/handlers/users.py

By being a package everything is then using a full dotted path. For
example 'micropledge.logic.errors'. Thus same module name in separate
subdirectories of the one package not a problem.

As for keeping the different versions in different virtual hosts
separate, the use of distinct Python sub interpreters should manage
that as long as the sys.path in each only refers to its own version
and not that of the other virtual host.

Thus, in the first instance, would be investigating why your virtual
hosts would be using the same Python sub interpreter.

Graham