[mod_python] Caching issues

What version of Python are you using? It is possible that if using
version of Python newer than what that stuff was originally written
against, that it is misbehaving. Have seen one other report of odd
stuff happening. I don't use mod_python any more and don't really have
time to go digging in to why. Really don't understand why people even
want to keep writing stuff to mod_python specific APIs when better
options exist these days for Python web applications.

Graham

2009/1/13 Tim Valenta <tonightslastsong at gmail.com>:
>> You should be able to add multiple directories and so long as a single
>> Python module code file, should be found even when using 'import'.
>
> I tried adding my 'subDir'' to the path, so that it looks like this:
>    PythonOption mod_python.importer.path "['~' , '~/include/']"
> My 'import' keyword attempts still failed when trying to import across
> this 'include' folder into a specific module file residing there.
>
> I simplified the experiment down to the above importer.path, and just
> ./A.py which uses a keyword 'import' to try to grab C.py from this
> 'include' folder.  This had surprisingly still lead to failure to find
> the module.
>
>> 2009/1/13 Tim Valenta <tonightslastsong at gmail.com>:
>>> Thank you for the links-- they prove useful.
>>>
>>> Capitalization shouldn't be an issue here... When I began learning
>>> Unix a good while back, I quickly cast off relying on the OS to
>>> 'figure it out'.  I've been keeping everything case sensitive.
>>>
>>> Backing up to something foundational: I seem to be able to use the
>>> import_module() method without problems now, but practically any use
>>> of standard python 'import'/'from' keywords fails.  The only success I
>>> get is when the desired module is in the same directory as the one
>>> trying to use the 'import' keyword.  Does this stem from the fact that
>>> import_module isn't look for packages, but rather specific files?  Is
>>> it for that reason that I can't use
>>>    from subDir import C
>>> or, the equally broken,
>>>    import subDir.C
>>
>> Packages not supported with mod_python module importer, so these fall
>> through import_module and uses standard Python import mechanism.
>>
>>> Jiggering around with adding more entries in importer.path to my
>>> subdirectories doesn't seem to have any effect.
>>
>> You should be able to add multiple directories and so long as a single
>> Python module code file, should be found even when using 'import'.
>>
>> Graham
>>
>>> If the 'import'
>>> keyword really is bound to it's working directory only, I will likely
>>> in fact have to use import_module() is several places, rather than
>>> just at the head of the request (./A.py)
>>>
>>> Despite my general lack of success, I appreciate the comments you've
>>> made along the way.
>>>
>>> Tim
>>>
>>> On Mon, Jan 12, 2009 at 10:40 PM, Graham Dumpleton
>>> <graham.dumpleton at gmail.com> wrote:
>>>> 2009/1/13 Tim Valenta <tonightslastsong at gmail.com>:
>>>>>> You just need to ensure you set mod_python's own module search path,
>>>>>> ie. mod_python.importer.path
>>>>>
>>>>> And yet, as I've now removed my www directory from my PythonPath, I
>>>>> actually get limited functionality out of the mod_python imports.
>>>>> I've got my apache conf file globally applying the option:
>>>>>
>>>>>        PythonOption mod_python.importer.path "['~']"
>>>>> Also have tried using an absolute path, which is producing the same
>>>>> functionality:
>>>>>        PythonOption mod_python.importer.path "['V:/www/']"
>>>>>
>>>>> My main module ('A') is requested by Apache in my browser, which uses
>>>>> import_module to find and import file B.  File B is in the same
>>>>> directory as A, and they only way it was found was through my apache
>>>>> conf noted above.  This part works just fine, but two cases develop
>>>>> from here:
>>>>>
>>>>> 1) If I use module B to import a third module 'C', I can't seem to
>>>>> make it find module C if it's in a subdirectory.  For instance, doing
>>>>> either 'import subDirectory.moduleC' or doing
>>>>> 'apache.import_module("subDirectory/moduleC")' results in a failure to
>>>>> find the module.  Note that in this case of subdirectories, I'm
>>>>> intentionally avoiding absolute paths to reach my modules.  I only
>>>>> want to get absolute in my Apache conf.  On the flip side, if module C
>>>>> is in the same local directory as A and B, then mod_python finds
>>>>> module C just fine.
>>>>
>>>> Try:
>>>>
>>>>  apache.import_module("./subDirectory/moduleC.py")
>>>>
>>>>> 2 ) If I place C in the same directory as A and B, so that mod_python
>>>>> finds it, I tried using the natural Python 'import' keyword in B to
>>>>> import C.  According to what I understand, this should be fine for
>>>>> auto-reloading, since A imports B via import_module, and so then B can
>>>>> use either importing strategy to get C in the picture because
>>>>> mod_python uses import_module behind the scenes anyway.  In this
>>>>> setup, my caching issue remains, and changes to C are definitely not
>>>>> reflected in apache until a server restart.  Again note that it
>>>>> doesn't matter if I make B use the 'import' keyword or the
>>>>> apache.import_module() method.  Both result in failure to reload.
>>>>>
>>>>> I can't seem to diagnose what's even wrong, since the only reason
>>>>> module A, B, or C is being found in the first place is because of my
>>>>> importer.path variable is set in httpd.conf.  I've been using
>>>>> practically empty files to test this.  Each file contains no more than
>>>>> 5 lines to get the modules imported and then to write out a
>>>>> confirmation to the screen that it worked.
>>>>>
>>>>> Wisdom to offer?
>>>>
>>>> Yes, from memory the module importer isn't properly case insensitive
>>>> on Windows or MacOS X where file systems is case insensitive. Thus,
>>>> make sure your imports match exactly the case of the directories/files
>>>> in the file system.
>>>>
>>>> Also, use:
>>>>
>>>>  from mod_python import apache
>>>>  apache.log_error('__name__ = %s' % repr(__name__))
>>>>  apache.log_error('__file__ = %s' % repr(__file__))
>>>>
>>>> in modules so that information about imported files is logged as
>>>> imported. This will tell you which files are being imported.
>>>>
>>>> With PythonDebug On you should also see messages in Apache error logs
>>>> describing when they are being imported, initially and when changed.
>>>>
>>>> You can also write some code which extracts out of the mod_python
>>>> importer cache the details of everything that has been loaded and the
>>>> relationships. The function to call is:
>>>>
>>>>  from mod_python import apache
>>>>  data = apache.request_modules_graph()
>>>>
>>>> write that to a file and run:
>>>>
>>>>  http://www.graphviz.org/
>>>>  http://www.graphviz.org/Download_windows.php
>>>>
>>>> on it.
>>>>
>>>> Graham
>>>>
>>>>>> Now you do realise you don't need to convert 'import' to
>>>>>> import_module() everywhere?
>>>>>>
>>>>>> The 'import' directive when used inside a file which was already
>>>>>> managed by mod_python calls import_module() internally anyway.
>>>>>>
>>>>>> You just need to ensure you set mod_python's own module search path,
>>>>>> ie. mod_python.importer.path
>>>>>>
>>>>>> Graham
>>>>>>
>>>>>> 2009/1/12 Tim Valenta <tonightslastsong at gmail.com>:
>>>>>> > Okay, I think I'm on the same page as you now-- I've been playing
>>>>>> > around with it and it seems that when I leave my PythonPath to find my
>>>>>> > import modules, I definitely get no auto reloading goodness, while
>>>>>> > when I specify the root path to find them as files, rather than as
>>>>>> > modules (as explained in that documentation you linked to), things
>>>>>> > seem to work as expected.
>>>>>> >
>>>>>> > I wasn't getting any warnings in my apache log, and after putting
>>>>>> > together the details, I shouldn't be expecting any of that behavior
>>>>>> > with two of the same module in memory.  My issue seems to be simply in
>>>>>> > avoiding letting my PythonPath find my code.
>>>>>> >
>>>>>> > Thank you much,
>>>>>> > Tim
>>>>>> >
>>>>>> > On Sun, Jan 11, 2009 at 9:39 PM, Graham Dumpleton
>>>>>> > <graham.dumpleton at gmail.com> wrote:
>>>>>> >> 2009/1/12 Tim Valenta <tonightslastsong at gmail.com>:
>>>>>> >>> Okay, that makes sense.  My PythonPath system var definitely includes
>>>>>> >>> my www directory.
>>>>>> >>
>>>>>> >> That in itself may not be enough to cause problem. You should actually
>>>>>> >> see a warning in Apache error log complaining you have done this. That
>>>>>> >> is, overlapped sys.path with directories that mod_python module
>>>>>> >> importer uses.
>>>>>> >>
>>>>>> >> The real problem with doing this overlapping is that a module which
>>>>>> >> wasn't one managed by mod_python, ie., standard Python module, imports
>>>>>> >> something from the www directory, which also happens to be a handler
>>>>>> >> module, there will actually be too copies in memory and so can give
>>>>>> >> unexpected results.
>>>>>> >>
>>>>>> >> Graham
>>>>>> >>
>>>>>> >>> I had been using a forced path in my apache conf
>>>>>> >>> file to point to '~', for simple reference for the time being.
>>>>>> >>>
>>>>>> >>> I think my problem is simply that the modules I'm trying to import
>>>>>> >>> aren't 'candidates' for reloading.  Other than that, all other modules
>>>>>> >>> should already have been chained together with these
>>>>>> >>> apache.load_module calls.  My test case was pretty simple-- just two
>>>>>> >>> scripts, the first loading the second.
>>>>>> >>>
>>>>>> >>> Tim
>>>>>> >>>
>>>>>> >>> On Sun, Jan 11, 2009 at 8:25 PM, Graham Dumpleton
>>>>>> >>> <graham.dumpleton at gmail.com> wrote:
>>>>>> >>>> 2009/1/12 Tim Valenta <tonightslastsong at gmail.com>:
>>>>>> >>>>> Apologies for more trouble on the caching issue-- I've been adapting
>>>>>> >>>>> some of my scripts over to this apache.load_module function, but I
>>>>>> >>>>> don't think I'm seeing resolution to the issue.  If I've read the
>>>>>> >>>>> documentation properly, it seems that the PythonOption for
>>>>>> >>>>> auto-reloading is set to On by default, so to be clear, I haven't put
>>>>>> >>>>> a line for that in my httpd.conf file.  Are there any other tricks, or
>>>>>> >>>>> have I simply misread the documentation?  I still have to restart the
>>>>>> >>>>> Apache service in order to push the changes.
>>>>>> >>>>
>>>>>> >>>> Reloading does not work for any Python modules/packages installed in
>>>>>> >>>> standard Python module search path. Thus, if you have set PythonPath
>>>>>> >>>> to some directory where you store your modules, or if they are in
>>>>>> >>>> system site-packages directory, they are not candidates for reloading.
>>>>>> >>>>
>>>>>> >>>> Where are the modules you are modifying and expecting to be reloading
>>>>>> >>>> located? Is that location on sys.path, or have you explicitly setup
>>>>>> >>>> mod_python module importer path to tell it where they are?
>>>>>> >>>>
>>>>>> >>>> From within running application, print out __name__ from the module
>>>>>> >>>> you have loaded and are expecting to be reloadable, What is it? Does
>>>>>> >>>> it look like a normal module name or something magic?
>>>>>> >>>>
>>>>>> >>>> How was that module imported? There needs to be a chain from
>>>>>> >>>> mod_python handler for imports right down through to all modules you
>>>>>> >>>> want reloadable. If there aren't, for example, you are using
>>>>>> >>>> import_module() from a normal Python module which isn't a candidate
>>>>>> >>>> for reloading, you will not see the depth checking for reloading
>>>>>> >>>> occurring as you might expect.
>>>>>> >>>>
>>>>>> >>>> Graham
>>>>>> >>>>
>>>>>> >>>>> Tim
>>>>>> >>>>>
>>>>>> >>>>> On Sun, Jan 11, 2009 at 2:32 PM, Tim Valenta <tonightslastsong at gmail.com> wrote:
>>>>>> >>>>>> Despite the jab at Windows, the site will be deployed on a Unix
>>>>>> >>>>>> machine.  I just prefer Windows for ease of use, especially since my
>>>>>> >>>>>> wife has zero experience on a unix terminal or in the dearth of truly
>>>>>> >>>>>> professional software found on the Unix/Linux platform.  So yes, in
>>>>>> >>>>>> the end it will be on a "real" operating system,  But for now I'm
>>>>>> >>>>>> willingly sticking to a more user-friendly environment for the simple
>>>>>> >>>>>> folk.
>>>>>> >>>>>>
>>>>>> >>>>>> Given the characteristics of PHP that allow it to circumvent this
>>>>>> >>>>>> caching stuff, I still think a nice little friendly red-flag reminder
>>>>>> >>>>>> on the main mod_python tutorial should point out this limitation of an
>>>>>> >>>>>> Apache module like this, for those of us like me, who would have gone
>>>>>> >>>>>> many months more before having every accidentally found this manual
>>>>>> >>>>>> import method.
>>>>>> >>>>>>
>>>>>> >>>>>> On Sun, Jan 11, 2009 at 2:23 PM, Graham Dumpleton
>>>>>> >>>>>> <graham.dumpleton at gmail.com> wrote:
>>>>>> >>>>>>>
>>>>>> >>>>>>> 2009/1/12 Tim Valenta <tonightslastsong at gmail.com>:
>>>>>> >>>>>>> >> Still not short enough, so much so I stopped part way through.
>>>>>> >>>>>>> >
>>>>>> >>>>>>> > And for that I apologize.  I do appreciate the straight answer given,
>>>>>> >>>>>>> > though I would appreciate much more the addition of that fact in some
>>>>>> >>>>>>> > point-blank documentation that was easy to find.  I have scoured the
>>>>>> >>>>>>> > mod_python archives by search engine and came out utterly empty
>>>>>> >>>>>>> > handed.  I knew what the problem was, but I could find nothing to
>>>>>> >>>>>>> > remedy the issue.  I haven't encountered anything quite like this in
>>>>>> >>>>>>> > my adventures with PHP, perl, or other similar scripting languages
>>>>>> >>>>>>> > piloted by Apache.  It raises a valid concern to which I ideally
>>>>>> >>>>>>> > should have found answers very quickly.
>>>>>> >>>>>>> >
>>>>>> >>>>>>> > I hope this cures the subsequent issue given in the latter half of my message.
>>>>>> >>>>>>>
>>>>>> >>>>>>> PHP is specifically designed for web applications and it purposely
>>>>>> >>>>>>> throws away all code at the end of each request and thus is reloading
>>>>>> >>>>>>> it all on every request. Have a read of:
>>>>>> >>>>>>>
>>>>>> >>>>>>>  http://blog.ianbicking.org/2008/01/12/what-php-deployment-gets-right/
>>>>>> >>>>>>>
>>>>>> >>>>>>> I would be very surprised if mod_perl auto reloaded code as it is
>>>>>> >>>>>>> similar to Python in that you are applying a non web language to the
>>>>>> >>>>>>> web. Only way perl code would be reload is if you were running them as
>>>>>> >>>>>>> CGI scripts. If you run Python as CGI scripts you will get reloading
>>>>>> >>>>>>> as well, but as with anything CGI, much slower.
>>>>>> >>>>>>>
>>>>>> >>>>>>> BTW, maybe read:
>>>>>> >>>>>>>
>>>>>> >>>>>>>  http://blog.dscpl.com.au/2008/12/using-modwsgi-when-developing-django.html
>>>>>> >>>>>>>
>>>>>> >>>>>>> For WSGI hosted Python web application, this provides auto reloading
>>>>>> >>>>>>> on code changes. You do need to use a real operating system though,
>>>>>> >>>>>>> and not Windows.
>>>>>> >>>>>>>
>>>>>> >>>>>>> Graham
>>>>>> >>>>>>>
>>>>>> >>>>>>> > Tim
>>>>>> >>>>>>> >
>>>>>> >>>>>>> > On Sat, Jan 10, 2009 at 10:23 PM, Graham Dumpleton
>>>>>> >>>>>>> > <graham.dumpleton at gmail.com> wrote:
>>>>>> >>>>>>> >>
>>>>>> >>>>>>> >> 2009/1/11 Tim Valenta <tonightslastsong at gmail.com>:
>>>>>> >>>>>>> >> > Hello all-- I've been experiencing a caching issue from the
>>>>>> >>>>>>> >> > very beginning of my use of mod_python...  It's been at least 2 months now,
>>>>>> >>>>>>> >> > and I keep running into actual issues that prevent me from coding.
>>>>>> >>>>>>> >> > I have a knack for over-explaining, so I'll try to keep this concise yet
>>>>>> >>>>>>> >> > descriptive.
>>>>>> >>>>>>> >>
>>>>>> >>>>>>> >> Still not short enough, so much so I stopped part way through.
>>>>>> >>>>>>> >>
>>>>>> >>>>>>> >> The simple matter of it is that mod_python does not do deep checking
>>>>>> >>>>>>> >> of code for changes, nor does it automatically restart the process
>>>>>> >>>>>>> >> when code is changed. Thus the need to restart Apache when you make
>>>>>> >>>>>>> >> code changes to anything imported from sys.path is expected and normal
>>>>>> >>>>>>> >> behaviour.
>>>>>> >>>>>>> >>
>>>>>> >>>>>>> >> The only time any code is automatically reloaded is the direct code
>>>>>> >>>>>>> >> files imported by mod_python using its own special module importer.
>>>>>> >>>>>>> >> This is documented under 'import_module()' function in:
>>>>>> >>>>>>> >>
>>>>>> >>>>>>> >>  http://www.modpython.org/live/current/doc-html/pyapi-apmeth.html
>>>>>> >>>>>>> >>
>>>>>> >>>>>>> >> Graham
>>>>>> >>>>>>> >>
>>>>>> >>>>>>> >> > I'm developing a site on my local machine, Windows Vista, using Apache 2.2.x
>>>>>> >>>>>>> >> > and mod_python 3.3.1 .  I'm a programmer for a living, and I'm not quick to
>>>>>> >>>>>>> >> > point the finger at the language, *but* (you knew that was coming, eh?) I
>>>>>> >>>>>>> >> > know my code isn't to blame for the issue:
>>>>>> >>>>>>> >> > I write some basic code for an 'index.py' file, using the
>>>>>> >>>>>>> >> > mod_python.publisher handler.  'index.py' includes other modules which I've
>>>>>> >>>>>>> >> > coded from that same location, etc, etc.  Nothing fancy.  Any changes I make
>>>>>> >>>>>>> >> > in the 'index.py' file will be reflected immediately on my local web server.
>>>>>> >>>>>>> >> >  On the other hand, any changes I make to the modules included via import
>>>>>> >>>>>>> >> > from within 'index.py' are completely ignored by the web server.  It took me
>>>>>> >>>>>>> >> > a while to realize that my pages didn't reflect my code.  I also discovered
>>>>>> >>>>>>> >> > that the .pyc files had nothing to do with it.  I finally just restarted the
>>>>>> >>>>>>> >> > apache web service (which in fact runs as a service on my machine), and then
>>>>>> >>>>>>> >> > my code finally gets pushed through to the web server.
>>>>>> >>>>>>> >> > I've been looking around practically every other day for documentation on
>>>>>> >>>>>>> >> > how to make apache/mod_python simply cut it out and stop caching my python
>>>>>> >>>>>>> >> > code, but I've found nothing.  You can imagine the annoyance this presents,
>>>>>> >>>>>>> >> > since I have to restart my web server every single time I make even the
>>>>>> >>>>>>> >> > slightest change to a 'utility.py' file, etc.
>>>>>> >>>>>>> >> > I've been coping with the problem for a while now, but then I've found far
>>>>>> >>>>>>> >> > more annoying issues recently.  To abbreviate the problem into short terms,
>>>>>> >>>>>>> >> > I've got a main module 'MAIN' which imports another module for a class
>>>>>> >>>>>>> >> > 'CLASS'.  CLASS also has a few imports, such as 'backend' stuff for
>>>>>> >>>>>>> >> > interfacing with various databases, etc.  We'll call the 'backend' module
>>>>>> >>>>>>> >> > "BACKEND".  Given the setup, any changes I make to CLASS or BACKEND require
>>>>>> >>>>>>> >> > an apache restart in order to take effect.
>>>>>> >>>>>>> >> > I have a function in CLASS which calls a function from it's imported BACKEND
>>>>>> >>>>>>> >> > module.  I tried adding a parameter to the BACKEND function in question, and
>>>>>> >>>>>>> >> > properly passed said parameter while in CLASS, yet the mod_python debugger
>>>>>> >>>>>>> >> > spits out an error about me having passed 3 arguments, when the BACKEND
>>>>>> >>>>>>> >> > function takes exactly 2.  This is outright false, since my function in
>>>>>> >>>>>>> >> > BACKEND looks like:
>>>>>> >>>>>>> >> > def getUsers(self, req, terms):
>>>>>> >>>>>>> >> >
>>>>>> >>>>>>> >> > and I'm calling it with
>>>>>> >>>>>>> >> > self.backend.getUsers(self.req, search)
>>>>>> >>>>>>> >> >
>>>>>> >>>>>>> >> > In reality, my code dictates that I'm passing 3 (including the implicit
>>>>>> >>>>>>> >> > 'self' argument), and BACKEND's 'getUsers' does in fact take exactly 3
>>>>>> >>>>>>> >> > arguments.  Yet, the debugger is telling me that it takes only 2.
>>>>>> >>>>>>> >> > I was trying to pass it 'req' because I wanted to investigate a little error
>>>>>> >>>>>>> >> > in the code by printing something to the output HTML.  So, my attempt is
>>>>>> >>>>>>> >> > foiled, since somewhere something isn't being updated to what my most
>>>>>> >>>>>>> >> > current code actually says.
>>>>>> >>>>>>> >> > Just to test, I made the 'getUsers' function return immediately with a
>>>>>> >>>>>>> >> > string of gibberish, like 'return "adsfasdfadsfa"'.  this should make my
>>>>>> >>>>>>> >> > other code spin wildly out of control and encounter errors, yet when I
>>>>>> >>>>>>> >> > restart apache and test it... lo and behold, it's completely ignoring my
>>>>>> >>>>>>> >> > goofy 'return' statement.  The 'getUsers' function is still somehow
>>>>>> >>>>>>> >> > returning valid data, as if the 'return' wasn't there at all!
>>>>>> >>>>>>> >> > So then I tried causing actual syntax errors.  The debugger caught this,
>>>>>> >>>>>>> >> > much to my inner joy.  So I tried causing a semantic error instead:
>>>>>> >>>>>>> >> > referencing a non-existent attribute of a non-existent variable:
>>>>>> >>>>>>> >> > madeUpVar.moo = 42
>>>>>> >>>>>>> >> > Syntactically, nothing wrong, but at run time it should most definitely
>>>>>> >>>>>>> >> > encounter a NameError or something equally as realistic.  But I restart
>>>>>> >>>>>>> >> > apache, and... nothing.  The line is completely ignored.
>>>>>> >>>>>>> >> > Which leads me to believe that it's not actually being 'ignored' per se, but
>>>>>> >>>>>>> >> > rather the code being compiled is not the same as the code in play within
>>>>>> >>>>>>> >> > the web server.  When I delete my .pyc files and restart apache and visit
>>>>>> >>>>>>> >> > the URL that triggers my python code, my .py files are in fact being
>>>>>> >>>>>>> >> > recompiled down to their byte code .pyc files.  And clearly the interpreter
>>>>>> >>>>>>> >> > is processing my code, since it flags me on improper syntax.  Yet, no matter
>>>>>> >>>>>>> >> > what kind of syntactically-sound nonsense I put into my code, the changes
>>>>>> >>>>>>> >> > aren't being reflected in my web server.
>>>>>> >>>>>>> >> > These problems come and go, and I've go better explanation than over zealous
>>>>>> >>>>>>> >> > caching.  I imagine that by tomorrow sometime when I start my computer up,
>>>>>> >>>>>>> >> > the problem will have disappeared for the time being.
>>>>>> >>>>>>> >> > I've cursed this computer up and down as I've tried to figure out ANYTHING
>>>>>> >>>>>>> >> > that I can do to alleviate the issue, by to no avail.
>>>>>> >>>>>>> >> > Anybody with counsel to spare my tired brain is welcome to share...
>>>>>> >>>>>>> >> > _______________________________________________
>>>>>> >>>>>>> >> > Mod_python mailing list
>>>>>> >>>>>>> >> > Mod_python at modpython.org
>>>>>> >>>>>>> >> > http://mailman.modpython.org/mailman/listinfo/mod_python
>>>>>> >>>>>>> >> >
>>>>>> >>>>>>> >> >
>>>>>> >>>>>>> >
>>>>>> >>>>>>
>>>>>> >>>>>>
>>>>>> >>>>>>
>>>>>> >>>>> _______________________________________________
>>>>>> >>>>> Mod_python mailing list
>>>>>> >>>>> Mod_python at modpython.org
>>>>>> >>>>> http://mailman.modpython.org/mailman/listinfo/mod_python
>