[mod_python] Caching issues

Graham Dumpleton graham.dumpleton at gmail.com
Mon Jan 12 00:19:52 EST 2009


Now you do realise you don't need to convert 'import' to
import_module() everywhere?

The 'import' directive when used inside a file which was already
managed by mod_python calls import_module() internally anyway.

You just need to ensure you set mod_python's own module search path,
ie. mod_python.importer.path

Graham

2009/1/12 Tim Valenta <tonightslastsong at gmail.com>:
> Okay, I think I'm on the same page as you now-- I've been playing
> around with it and it seems that when I leave my PythonPath to find my
> import modules, I definitely get no auto reloading goodness, while
> when I specify the root path to find them as files, rather than as
> modules (as explained in that documentation you linked to), things
> seem to work as expected.
>
> I wasn't getting any warnings in my apache log, and after putting
> together the details, I shouldn't be expecting any of that behavior
> with two of the same module in memory.  My issue seems to be simply in
> avoiding letting my PythonPath find my code.
>
> Thank you much,
> Tim
>
> On Sun, Jan 11, 2009 at 9:39 PM, Graham Dumpleton
> <graham.dumpleton at gmail.com> wrote:
>> 2009/1/12 Tim Valenta <tonightslastsong at gmail.com>:
>>> Okay, that makes sense.  My PythonPath system var definitely includes
>>> my www directory.
>>
>> That in itself may not be enough to cause problem. You should actually
>> see a warning in Apache error log complaining you have done this. That
>> is, overlapped sys.path with directories that mod_python module
>> importer uses.
>>
>> The real problem with doing this overlapping is that a module which
>> wasn't one managed by mod_python, ie., standard Python module, imports
>> something from the www directory, which also happens to be a handler
>> module, there will actually be too copies in memory and so can give
>> unexpected results.
>>
>> Graham
>>
>>> I had been using a forced path in my apache conf
>>> file to point to '~', for simple reference for the time being.
>>>
>>> I think my problem is simply that the modules I'm trying to import
>>> aren't 'candidates' for reloading.  Other than that, all other modules
>>> should already have been chained together with these
>>> apache.load_module calls.  My test case was pretty simple-- just two
>>> scripts, the first loading the second.
>>>
>>> Tim
>>>
>>> On Sun, Jan 11, 2009 at 8:25 PM, Graham Dumpleton
>>> <graham.dumpleton at gmail.com> wrote:
>>>> 2009/1/12 Tim Valenta <tonightslastsong at gmail.com>:
>>>>> Apologies for more trouble on the caching issue-- I've been adapting
>>>>> some of my scripts over to this apache.load_module function, but I
>>>>> don't think I'm seeing resolution to the issue.  If I've read the
>>>>> documentation properly, it seems that the PythonOption for
>>>>> auto-reloading is set to On by default, so to be clear, I haven't put
>>>>> a line for that in my httpd.conf file.  Are there any other tricks, or
>>>>> have I simply misread the documentation?  I still have to restart the
>>>>> Apache service in order to push the changes.
>>>>
>>>> Reloading does not work for any Python modules/packages installed in
>>>> standard Python module search path. Thus, if you have set PythonPath
>>>> to some directory where you store your modules, or if they are in
>>>> system site-packages directory, they are not candidates for reloading.
>>>>
>>>> Where are the modules you are modifying and expecting to be reloading
>>>> located? Is that location on sys.path, or have you explicitly setup
>>>> mod_python module importer path to tell it where they are?
>>>>
>>>> From within running application, print out __name__ from the module
>>>> you have loaded and are expecting to be reloadable, What is it? Does
>>>> it look like a normal module name or something magic?
>>>>
>>>> How was that module imported? There needs to be a chain from
>>>> mod_python handler for imports right down through to all modules you
>>>> want reloadable. If there aren't, for example, you are using
>>>> import_module() from a normal Python module which isn't a candidate
>>>> for reloading, you will not see the depth checking for reloading
>>>> occurring as you might expect.
>>>>
>>>> Graham
>>>>
>>>>> Tim
>>>>>
>>>>> On Sun, Jan 11, 2009 at 2:32 PM, Tim Valenta <tonightslastsong at gmail.com> wrote:
>>>>>> Despite the jab at Windows, the site will be deployed on a Unix
>>>>>> machine.  I just prefer Windows for ease of use, especially since my
>>>>>> wife has zero experience on a unix terminal or in the dearth of truly
>>>>>> professional software found on the Unix/Linux platform.  So yes, in
>>>>>> the end it will be on a "real" operating system,  But for now I'm
>>>>>> willingly sticking to a more user-friendly environment for the simple
>>>>>> folk.
>>>>>>
>>>>>> Given the characteristics of PHP that allow it to circumvent this
>>>>>> caching stuff, I still think a nice little friendly red-flag reminder
>>>>>> on the main mod_python tutorial should point out this limitation of an
>>>>>> Apache module like this, for those of us like me, who would have gone
>>>>>> many months more before having every accidentally found this manual
>>>>>> import method.
>>>>>>
>>>>>> On Sun, Jan 11, 2009 at 2:23 PM, Graham Dumpleton
>>>>>> <graham.dumpleton at gmail.com> wrote:
>>>>>>>
>>>>>>> 2009/1/12 Tim Valenta <tonightslastsong at gmail.com>:
>>>>>>> >> Still not short enough, so much so I stopped part way through.
>>>>>>> >
>>>>>>> > And for that I apologize.  I do appreciate the straight answer given,
>>>>>>> > though I would appreciate much more the addition of that fact in some
>>>>>>> > point-blank documentation that was easy to find.  I have scoured the
>>>>>>> > mod_python archives by search engine and came out utterly empty
>>>>>>> > handed.  I knew what the problem was, but I could find nothing to
>>>>>>> > remedy the issue.  I haven't encountered anything quite like this in
>>>>>>> > my adventures with PHP, perl, or other similar scripting languages
>>>>>>> > piloted by Apache.  It raises a valid concern to which I ideally
>>>>>>> > should have found answers very quickly.
>>>>>>> >
>>>>>>> > I hope this cures the subsequent issue given in the latter half of my message.
>>>>>>>
>>>>>>> PHP is specifically designed for web applications and it purposely
>>>>>>> throws away all code at the end of each request and thus is reloading
>>>>>>> it all on every request. Have a read of:
>>>>>>>
>>>>>>>  http://blog.ianbicking.org/2008/01/12/what-php-deployment-gets-right/
>>>>>>>
>>>>>>> I would be very surprised if mod_perl auto reloaded code as it is
>>>>>>> similar to Python in that you are applying a non web language to the
>>>>>>> web. Only way perl code would be reload is if you were running them as
>>>>>>> CGI scripts. If you run Python as CGI scripts you will get reloading
>>>>>>> as well, but as with anything CGI, much slower.
>>>>>>>
>>>>>>> BTW, maybe read:
>>>>>>>
>>>>>>>  http://blog.dscpl.com.au/2008/12/using-modwsgi-when-developing-django.html
>>>>>>>
>>>>>>> For WSGI hosted Python web application, this provides auto reloading
>>>>>>> on code changes. You do need to use a real operating system though,
>>>>>>> and not Windows.
>>>>>>>
>>>>>>> Graham
>>>>>>>
>>>>>>> > Tim
>>>>>>> >
>>>>>>> > On Sat, Jan 10, 2009 at 10:23 PM, Graham Dumpleton
>>>>>>> > <graham.dumpleton at gmail.com> wrote:
>>>>>>> >>
>>>>>>> >> 2009/1/11 Tim Valenta <tonightslastsong at gmail.com>:
>>>>>>> >> > Hello all-- I've been experiencing a caching issue from the
>>>>>>> >> > very beginning of my use of mod_python...  It's been at least 2 months now,
>>>>>>> >> > and I keep running into actual issues that prevent me from coding.
>>>>>>> >> > I have a knack for over-explaining, so I'll try to keep this concise yet
>>>>>>> >> > descriptive.
>>>>>>> >>
>>>>>>> >> Still not short enough, so much so I stopped part way through.
>>>>>>> >>
>>>>>>> >> The simple matter of it is that mod_python does not do deep checking
>>>>>>> >> of code for changes, nor does it automatically restart the process
>>>>>>> >> when code is changed. Thus the need to restart Apache when you make
>>>>>>> >> code changes to anything imported from sys.path is expected and normal
>>>>>>> >> behaviour.
>>>>>>> >>
>>>>>>> >> The only time any code is automatically reloaded is the direct code
>>>>>>> >> files imported by mod_python using its own special module importer.
>>>>>>> >> This is documented under 'import_module()' function in:
>>>>>>> >>
>>>>>>> >>  http://www.modpython.org/live/current/doc-html/pyapi-apmeth.html
>>>>>>> >>
>>>>>>> >> Graham
>>>>>>> >>
>>>>>>> >> > I'm developing a site on my local machine, Windows Vista, using Apache 2.2.x
>>>>>>> >> > and mod_python 3.3.1 .  I'm a programmer for a living, and I'm not quick to
>>>>>>> >> > point the finger at the language, *but* (you knew that was coming, eh?) I
>>>>>>> >> > know my code isn't to blame for the issue:
>>>>>>> >> > I write some basic code for an 'index.py' file, using the
>>>>>>> >> > mod_python.publisher handler.  'index.py' includes other modules which I've
>>>>>>> >> > coded from that same location, etc, etc.  Nothing fancy.  Any changes I make
>>>>>>> >> > in the 'index.py' file will be reflected immediately on my local web server.
>>>>>>> >> >  On the other hand, any changes I make to the modules included via import
>>>>>>> >> > from within 'index.py' are completely ignored by the web server.  It took me
>>>>>>> >> > a while to realize that my pages didn't reflect my code.  I also discovered
>>>>>>> >> > that the .pyc files had nothing to do with it.  I finally just restarted the
>>>>>>> >> > apache web service (which in fact runs as a service on my machine), and then
>>>>>>> >> > my code finally gets pushed through to the web server.
>>>>>>> >> > I've been looking around practically every other day for documentation on
>>>>>>> >> > how to make apache/mod_python simply cut it out and stop caching my python
>>>>>>> >> > code, but I've found nothing.  You can imagine the annoyance this presents,
>>>>>>> >> > since I have to restart my web server every single time I make even the
>>>>>>> >> > slightest change to a 'utility.py' file, etc.
>>>>>>> >> > I've been coping with the problem for a while now, but then I've found far
>>>>>>> >> > more annoying issues recently.  To abbreviate the problem into short terms,
>>>>>>> >> > I've got a main module 'MAIN' which imports another module for a class
>>>>>>> >> > 'CLASS'.  CLASS also has a few imports, such as 'backend' stuff for
>>>>>>> >> > interfacing with various databases, etc.  We'll call the 'backend' module
>>>>>>> >> > "BACKEND".  Given the setup, any changes I make to CLASS or BACKEND require
>>>>>>> >> > an apache restart in order to take effect.
>>>>>>> >> > I have a function in CLASS which calls a function from it's imported BACKEND
>>>>>>> >> > module.  I tried adding a parameter to the BACKEND function in question, and
>>>>>>> >> > properly passed said parameter while in CLASS, yet the mod_python debugger
>>>>>>> >> > spits out an error about me having passed 3 arguments, when the BACKEND
>>>>>>> >> > function takes exactly 2.  This is outright false, since my function in
>>>>>>> >> > BACKEND looks like:
>>>>>>> >> > def getUsers(self, req, terms):
>>>>>>> >> >
>>>>>>> >> > and I'm calling it with
>>>>>>> >> > self.backend.getUsers(self.req, search)
>>>>>>> >> >
>>>>>>> >> > In reality, my code dictates that I'm passing 3 (including the implicit
>>>>>>> >> > 'self' argument), and BACKEND's 'getUsers' does in fact take exactly 3
>>>>>>> >> > arguments.  Yet, the debugger is telling me that it takes only 2.
>>>>>>> >> > I was trying to pass it 'req' because I wanted to investigate a little error
>>>>>>> >> > in the code by printing something to the output HTML.  So, my attempt is
>>>>>>> >> > foiled, since somewhere something isn't being updated to what my most
>>>>>>> >> > current code actually says.
>>>>>>> >> > Just to test, I made the 'getUsers' function return immediately with a
>>>>>>> >> > string of gibberish, like 'return "adsfasdfadsfa"'.  this should make my
>>>>>>> >> > other code spin wildly out of control and encounter errors, yet when I
>>>>>>> >> > restart apache and test it... lo and behold, it's completely ignoring my
>>>>>>> >> > goofy 'return' statement.  The 'getUsers' function is still somehow
>>>>>>> >> > returning valid data, as if the 'return' wasn't there at all!
>>>>>>> >> > So then I tried causing actual syntax errors.  The debugger caught this,
>>>>>>> >> > much to my inner joy.  So I tried causing a semantic error instead:
>>>>>>> >> > referencing a non-existent attribute of a non-existent variable:
>>>>>>> >> > madeUpVar.moo = 42
>>>>>>> >> > Syntactically, nothing wrong, but at run time it should most definitely
>>>>>>> >> > encounter a NameError or something equally as realistic.  But I restart
>>>>>>> >> > apache, and... nothing.  The line is completely ignored.
>>>>>>> >> > Which leads me to believe that it's not actually being 'ignored' per se, but
>>>>>>> >> > rather the code being compiled is not the same as the code in play within
>>>>>>> >> > the web server.  When I delete my .pyc files and restart apache and visit
>>>>>>> >> > the URL that triggers my python code, my .py files are in fact being
>>>>>>> >> > recompiled down to their byte code .pyc files.  And clearly the interpreter
>>>>>>> >> > is processing my code, since it flags me on improper syntax.  Yet, no matter
>>>>>>> >> > what kind of syntactically-sound nonsense I put into my code, the changes
>>>>>>> >> > aren't being reflected in my web server.
>>>>>>> >> > These problems come and go, and I've go better explanation than over zealous
>>>>>>> >> > caching.  I imagine that by tomorrow sometime when I start my computer up,
>>>>>>> >> > the problem will have disappeared for the time being.
>>>>>>> >> > I've cursed this computer up and down as I've tried to figure out ANYTHING
>>>>>>> >> > that I can do to alleviate the issue, by to no avail.
>>>>>>> >> > Anybody with counsel to spare my tired brain is welcome to share...
>>>>>>> >> > _______________________________________________
>>>>>>> >> > Mod_python mailing list
>>>>>>> >> > Mod_python at modpython.org
>>>>>>> >> > http://mailman.modpython.org/mailman/listinfo/mod_python
>>>>>>> >> >
>>>>>>> >> >
>>>>>>> >
>>>>>>
>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> Mod_python mailing list
>>>>> Mod_python at modpython.org
>>>>> http://mailman.modpython.org/mailman/listinfo/mod_python
>>>>>
>>>>
>>>
>>
>


More information about the Mod_python mailing list