[mod_python] How can I turn off apparent caching of python code?

Graham Dumpleton grahamd at dscpl.com.au
Sun Aug 29 21:42:15 EDT 2004

On 25/08/2004, at 12:20 PM, Graham Dumpleton wrote:

> On 24/08/2004, at 9:54 AM, Gregory (Grisha) Trubetskoy wrote:
>> On Mon, 23 Aug 2004, Tobiah wrote:
>>>>  DEBUG = 1
>>>>  import mymodulea
>>>>  import mymoduleb
>>>>  if DEBUG:
>>>>         mymodulea = reload(mymodulea)
>>>>         mymoduleb = reload(mymoduleb)
>>>> That is O.K., but still, unwieldy.
>> This method is pretty old, now a simpler way to do it is:
>> mymodulea = apache.import_module("mymodulea")
>> mymoduleb = apache.import_module("mymoduleb")
> No time right now for an extended description, will send one later, 
> but Vampire which I
> described recently and which is available at 
> "http://www.dscpl.com.au/projects/vampire"
> has a module caching system which has explicit module importing 
> similar to the
> mod_python.apache module, but what it will do is track relationships 
> between module
> imports and is able to do a reload even when child imports are changed 
> and not the
> immediate module you are loading.
> What this means is if that A imports B which imports C and similarly D 
> imports C, if C
> is then changed, all the modules which depend on C, ie., A, B and D 
> will be automatically
> reloaded when requested, rather than them being grabbed from the 
> cache. If all your
> application modules are in one place and you always use the module 
> importer in Vampire
> for importing any application module from a page, but also one 
> application module from
> another, any change to an application module should see anything that 
> depends on it
> automatically reloaded.

I said I would follow up later with a better description. Being on 
holidays in a
foreign land this has turned out to the first opportunity to do so.

The module caching mechanism in Vampire is used as:

   import vampire.cache

   cache = vampire.cache.ModuleCache()
   c = cache.importModule("c",".")

The first argument to importModule() is the name of the module to load 
and the second
argument is the specific directory to look in. Specifically, the 
contents of the
sys.path variable is NOT used to find the module, instead you have to 
tell it where
it resides.

The reason for doing this is so you know you pick up the exact module 
you want. The
idea is that all the modules which relate to the application component 
of your web
pages would reside in one place. The name of that root directory would 
be the second
argument you use to importModule(). This avoids any problems where you 
can't be
certain about the order of sys.path when multiple applications are 
running within
the same web server and each different application uses the same module 

Another important feature of the module cache is the encoding of the 
actual path to
the module into the lookup key for the module as it is listed in 
sys.modules. The
module cache in mpservlets does a similar thing, although in Vampire, 
for reasons I
can't quite remember, I encode the name to a greater degree. I vaguely 
having some problems with really long module names in some older 
version of Python.

For example, the module "c" above would appear in sys.modules under the 


The path recorded in the loaded module would be "./c.py" or "./c.pyc" 
as appropriate.

The reason for this is to work around a problem often described on the 
mailing list
whereby using the same name in multiple locations will potentially 
cause reloading
of the handler every time it is accessed from the different locations 
even no changes
have been made.

To explain the tracking of parent/child relationships between modules, 
now consider
the following Python code files.

   # c.py

   print "c"

   # b.py

   import vampire.cache
   print "b 1"
   cache = vampire.cache.ModuleCache()
   c = cache.importModule("c",".")
   print "b 2"

   # a.py

   import vampire.cache
   print "a 1"
   cache = vampire.cache.ModuleCache()
   b = cache.importModule("b",".")
   print "a 2"

   # test.py

   import vampire.cache
   import time

   cache = vampire.cache.ModuleCache()

   for i in range(100):
     print "test",i
     a = cache.importModule("a",".")
     for n in cache.cachedModules():
       c = cache.moduleInfo(n)

First thing to be noted is that a normal "import" is not used for any 
module which
would be in your application module directory. Any access to such 
modules is mediated
through the module cache.

Now when the test script is run, it will on the first iteration 
generate the output:

   test 0
   a 1
   b 1
   b 2
   a 2
   c _vampire__2b_yg4_2b_XZhHdEtPHG6AmTz9w_3 1 ./c.py 1093773189 
1093774844.22 2 0
   a _vampire_LqOp5S04uKeR2wBXKPT61w_3 3 ./a.py 1093773270 1093774844.23 
1 0
   b _vampire_XcjTxdeaT7e4Z8DD4M2Axg_3 2 ./b.py 1093773255 1093774844.22 
1 0

The code in each of the modules "a.py", "b.py" and "c.py" has been 
executed upon loading
as demonstrated by print statements outputs.

On the subsequent iteration, the output of the test program is:

   test 1
   c _vampire__2b_yg4_2b_XZhHdEtPHG6AmTz9w_3 1 ./c.py 1093773189 
1093774849.24 2 1
   a _vampire_LqOp5S04uKeR2wBXKPT61w_3 3 ./a.py 1093773270 1093774849.24 
2 0
   b _vampire_XcjTxdeaT7e4Z8DD4M2Axg_3 2 ./b.py 1093773255 1093774849.24 
1 1

That is, no modules were reloaded, instead they were obtained from the 
cache. The output
here is information maintained by the cache about modification and 
access times, number
of access and overall generation snapshot of the cache as a whole at 
the point the
module was loaded.

Note that the test script only attempts to load module "a". If we touch 
the file "a.py",
will now see the iteration of test script following point when file was 
modified showing:

   test 19
   a 1
   a 2
   c _vampire__2b_yg4_2b_XZhHdEtPHG6AmTz9w_3 1 ./c.py 1093773189 
1093774939.5 3 19
   a _vampire_LqOp5S04uKeR2wBXKPT61w_3 4 ./a.py 1093774935 1093774939.5 
1 0
   b _vampire_XcjTxdeaT7e4Z8DD4M2Axg_3 2 ./b.py 1093773255 1093774939.5 
2 18

Because only "a.py" was modified, there is no need to reload "b" and 
"c" from disk and
they instead come from the cache.

If instead we modify "b.py" and then "c.py" we will see the output:

   test 22
   a 1
   b 1
   b 2
   a 2
   c _vampire__2b_yg4_2b_XZhHdEtPHG6AmTz9w_3 1 ./c.py 1093773189 
1093774954.54 5 21
   a _vampire_LqOp5S04uKeR2wBXKPT61w_3 6 ./a.py 1093774935 1093774954.55 
1 0
   b _vampire_XcjTxdeaT7e4Z8DD4M2Axg_3 5 ./b.py 1093774951 1093774954.55 
1 0

   test 25
   a 1
   b 1
   b 2
   a 2
   c _vampire__2b_yg4_2b_XZhHdEtPHG6AmTz9w_3 7 ./c.py 1093774966 
1093774969.64 2 1
   a _vampire_LqOp5S04uKeR2wBXKPT61w_3 9 ./a.py 1093774935 1093774969.65 
1 0
   b _vampire_XcjTxdeaT7e4Z8DD4M2Axg_3 8 ./b.py 1093774951 1093774969.65 
1 0

In these cases, although "a.py" was not modified, but "b.py" and then 
"c.py" were,
the module cache knows that "a.py" depends on "b.py" and that "b.py" 
depends on
"c.py" and thus that it should force a reload of "a.py" anyway in case 
the latter
files have changed in some important way.

As an example, imagine that "c.py" contained configuration information. 
module cache will ensure that modules which use that configuration are 
to pick up the changed information. This auto reloading would also be 
good where
a servlet style approach was used and each file contained a servlet 
which was
part of a hierarchy. If the lowest base class were changed and it 
important site layout code, those servlets derived from it would be 
reloaded when next used.

 From what I can see, this module cache will solve the sorts of problems 
are talking about with how to force reloading of modules when changed. 
Unless I
am missing something, the other suggestions are okay when there is only 
one level
of module importing, but if there are multiple levels of imports from 
your set
of application modules, they will not detect when the file associated 
with a child
import is changed and thus that an unchanged parent should still be 

I will apologise in advance for not answering any followup questions, 
if any, for a
few days as am about to travel to yet another country for a few days 
and will not be
taking my laptop with me on this part of my trip.

FWIW, Vampire can currently be obtained from:


Graham Dumpleton (grahamd at dscpl.com.au)

More information about the Mod_python mailing list