[mod_python] Re: PyThreadState_Swap(NULL)

On 20/08/2006, at 1:21 PM, Bryan wrote:

> Graham Dumpleton wrote:
>>> For a start, which version of mod_python are you using? There  
>>> were some
>>> important changes in mod_python 3.2.8 to allow third party  
>>> modules for
>>> Python to work properly where those third party modules used the  
>>> simple
>>> GIL API for thread management.
>> Whoops, that should be mod_python 3.2.10. Not 3.2.8.
>> Graham
>
>
> hi graham,
>
> we are using mod_python 3.2.10.  you are right that we are doing  
> something similar to mod_python in that we fully embed python to  
> process requests using our protocol.  the difference is that our  
> main protocol and message dispatcher has nothing to do with apache  
> and can plug into our own proprietary server as well as 3rd party  
> servers. our mod_xxx module is really just an adapter to let our  
> dispatcher plug into apache.  and our python adapter is a plug into  
> our dispatcher along with other language adapters.  this is a  
> snippet of our code and shows you where the failure is.  the  
> failure occurs in the start function and g_py_main_interpreter  
> returned by PyThreadState_Swap(NULL) is NULL.  i wouldn't mind  
> grabbing your latest code from svn and looking at your new API,  
> except that i'm not sure using experimental code at this phase of  
> development process would be wise.  also, if i read what you wrote  
> correctly, i would need to tie our code to use the mod_python  
> framework and i'm not willing to do that since our python adapter  
> has no knowledge of apache.  unless i'm wrong, i believe the only  
> real solution at this point is to use two apache processes. one  
> that includes mod_python which we use for cgi gui apps, and the  
> other with our dispatcher so there would be no conflicts.  this  
> wouldn't be an optimal solution, but it's doable.

I'd have to look over your code and try and get my head around all this
stuff again, but I really think you will be fighting an uphill battle  
to get
something going which doesn't try and work in co-operation with
mod_python some how.

One of the first problems is that mod_python is the one to create the
first interpreter and this first interpreter is special. If a Python  
module
has a C code component and it isn't written in just the right way so as
to work with interpreters other than the first, then it will have  
problems.
Usually this manifests as Python code running in a restricted  
environment
and not able to access various stuff, but other things could also  
happen,
including lock ups and crashes.

The other problem you will have is that Apache on Win32 and also
worker MPM on UNIX is multithreaded and even more work is required
to get that to work properly. I don't see anything in your code that
suggests that it takes multithreading and multiple interpreters into
consideration.

In the short term, using a separate Apache instance as you suggest
will probably be your only quick solution. In the long term, I still  
think
using the proposed API for mod_python is the only way it is going to
work.

You shouldn't have a problem with this if your code is sufficiently  
layered.
That is, the real part of the code which does the work should be written
just like a normal Python extension module which could be loaded into
any command line Python. You would then have two other distinct bits of
code for embedding. Both of these merely setup the interpreter  
environment
and then import your normal Python module and execute it. One of these
bits of code would be for embedding in a standalone application and
the other variant one that works in conjunction with mod_python to allow
embedding when Apache is used and mod_python is also used. If
necessary you can have a variant of Apache module which works when
mod_python is not present.

As I said, if your code is correctly layered and componentised, this  
should
not result in all your code being dependent on mod_python.

Oh, one final possibility that may be possible, especially if your code
were properly layered and your core code is loadable as a Python
module, is simply to import that code Python module from a module
imported in mod_python using PythonImport and trigger it from that.
If necessary, the Apache handlers your have written in mod_xxx could
be written as mod_python handlers instead. Since you don't really
say what mod_xxx does though, I can't really comment on whether
this would be practical.

Graham