[mod_python] Problems under windows: a first diagnostic

[Greg Stein]

On Mon, 5 Jun 2000, [iso-8859-1] Stéphane Bidoul wrote:
>...
> Now, in mod_python, threads are created by Apache, so they should
> somehow be bootstrapped for python. 

Correct.

> In my understanding, mod_python only initializes one thread per
> interpreter: the Apache thread that is happening to create the interpreter
>  (using Py_NewInterpreter(), which in turns calls PyThreadState_New()
> for the interpreter's "main" thread).
> All other apache threads using that same interpreter will not call
> PyThreadState_New()... hence the potential conflict.

ThreadState structures are not specific to a particular (OS) thread. You
can actually use them (not at the same time!) across different threads.
Given the protection of the global lock, it might be okay to even attempt
to use them at the same time.
[ haven't thought through this particular scenario fully yet ]

> To summarize, my understanding of the python doc is that there should
> be one PyThreadState_New() for each (apache thread, python
> interpreter) couple.

Theoretically: yes.

Python's central lock "should" arbitrate the use of the per-interpreter
thread state so that it can be used by multiple Apache threads. Probably
isn't the best idea, though.

Eek. Definitely wrong for the Apache scenario.

The scoop: if python_handler() calls into Python, then returns, then
tstate->frame should be NULL. In this scenario, you could do the
share-tstate thing.

However: somewhere down inside of python_handler's call into Python, it is
quite possible for Python to use Py_BEGIN_ALLOW_THREADS (unlock the global
lock). This will allow another python_handler to grab the tstate and
attempt execution. Let's say the thread execution then flips back to
normal: the original thread's tstate->frame is now *totally* wrong.

[ tstate->frame has a direct correspondence to the C stack; these must
  stay in sync or Bad Things happen ]

> AFAIK, threads are used only in the Windows version of Apache, 
> so the problem should not occur on unix  (athough that will change with Apache 2). 
> Also, if ThreadsPerChild is set to 1, the problem disappears,
> which seems to confirm my diagnostic.
> 
> Again, bear with me if this is totally wrong...
> If this is correct, however, I've no idea (yet) on how to fix it...

Your underlying hypothesis is quite correct. Above is a more detailed
breakdown of "why".

> What do you think? 
> Can a python threading expert confirm or infirm this diagnostic?

That would be me :-)

One other thing: if Apache is going to run threads, then Python *MUST* be
compiled with thread support. Otherwise, things will die very rapidly.
Presuming that you are using the standard Windows distribution, then the
thread support is automatically enabled. I would recommend that mod_python
barf with #error if it detects Apache/Win32 and WITH_THREAD is not
enabled.

Heh. All of this just supports my belief in "free threading" Python even
more... :-)

What is the solution? Each entry to python-handler should construct a
tstate on the fly, call into Python, then destroy the tstate. I used this
design for the Python/COM support in the Win32 distribution. That had to
deal with reentrancy issues, but (luckily) that isn't a problem here.

Presumably, you could also cache/pool tstates and yank them from your
pool on entry to python-handler, returning them on exit. If two threads
came into python-handler (against the same interp!), then additional
tstates would be constructed.

Cheers,
-g

-- 
Greg Stein, http://www.lyra.org/