[mod_python] Concurrence and virtual host interpreter python

On top of what Jim already said ...

On 28/09/2007, Pau Freixes <pfreixes at milnou.net> wrote:
>
> Hi to all, Im new in a list.
>
> Well, this last days I was work for testing and learn how work with multi
> thread and multi interpreter environment in embedding python in C
> applications, and of course mod_python it's a excellent place for
> understand this.

Huh, are you only looking at how things are done at the Python level,
or are you trying to understand how the C code is implemented
underneath?

If you are also interested in the C code level of how it is
implemented, I would recommend looking at mod_wsgi
(http://www.modwsgi.org) instead as its C code is probably easier to
follow and does certain things correctly where as mod_python currently
gets it wrong causing some limitations in what C extension modules can
be used with it.

> Ok, with my lectures and test with mod_python code, i have a some
> questions, somebody can help me for understand this concepts ?
>
> 1. I was try multi concurrence in a virtual host python script with this
> easy code
>
> def handler(req):
>
>     # Request Content Type
>     req.content_type = "text/plain"
>
>     VALUE2 = 1
>     for i in range(1,10):
>         req.write("Value is %d\r\n" % VALUE2)
>         time.sleep(5)
>
>         # change global variable
>         VALUE2 = VALUE2 + 1;
>
>
>     req.write("Hellow World")
>     return apache.OK
>
> And my surprise it's that apache only process the next request when the
> previous request has been served. !!
>
> I don't understand this strange behavior, because mod_python it's made
> for accept concurrent threads in same interpreter, and i test a similar
> situation  in a c code and it's possible do run some threads at same
> time in same interpreter.
>
> time.sleep(5) it's a pretty situation for python core for change the
> context and release GIL and assign to this to other thread. yes ?

This almost suggest you are running prefork MPM with a single process,
or are running prefork MPM in foreground (debugging) mode. In those
cases everything would be serialised.

Suggest you read:

  http://code.google.com/p/modwsgi/wiki/ProcessesAndThreading

This talks about the various Apache MPM modes and what each implies
with respect to use of multiples processes and multithreading.

> 2. And de second question it's in similar situation, when you reuse a
> interpreter for all request to same virtual host, with this code :
>
> import time
>
> VALUE = 1
>
> def handler(req):
>     global VALUE
>
>     # Request Content Type
>     req.content_type = "text/plain"
>
>     for i in range(1,10):
>         req.write("Value is %d\r\n" % VALUE)
>         time.sleep(5)
>
>         # change global variable
>         VALUE = VALUE + 1;
>
>
>     req.write("Hellow World")
>     return apache.OK
>
> VALUE it's a global variable for de index.py, and subsequent request for
> the same url (http://...../index.py) show one incremental value for
> VALUE. For example, this two request produced this two results
>
> http://...../index.py
>
> Value is 1
> Value is 2
> Value is 3
> Value is 4
> Value is 5
> Value is 6
> Value is 7
> Value is 8
> Value is 9
> Hellow World
>
> http://...../index.py
>
> Value is 10
> Value is 11
> Value is 12
> Value is 13
> Value is 14
> Value is 15
> Value is 16
> Value is 17
> Value is 18
> Hellow World
>
> Ok, this situation it's really and i think its very normal because only
> callback to handler it's made with the same interpreter and VALUE is
> not re assigned to value 1, may be ?

Read the document I reference above as it talks about data sharing
issues with the various Apache MPMs.

> 3. And the most important question, why mod_python not use a separate
> interpreters for all request ?

Because creating a new Python interpreter instance for each request
would result in CGI like speeds. Also, creating a new Python
interpreter for each request implies having to destroy the Python
interpreter instance as well. Various C extension modules for Python
are not written properly so as to cope with the destruction of Python
sub interpreters within a process and the module then be loaded again
into a new sub interpreter instance in the same process. The result
typically being that the process will crash. See:

  http://code.google.com/p/modwsgi/issues/detail?id=12&can=1

Graham