[mod_python] how to run multiple subinterpreters from mod_python

Graham Dumpleton grahamd at dscpl.com.au
Tue Jan 23 17:18:14 EST 2007


Suresh Lakshmanan wrote ..
> Hi,
> Is it possible for mod_python to create new subinterpreter when requests
> arrive? I was thinking that mod_python has a configurable pool of python
> interpreters
> and it will delegate a request to an interpreter from the pool.
> 
> But, it looks like this is not the case.

Correct, that is not the case. You are misunderstand what the subinterpreters
are there for.

> Only one subinterpreter is only
> created for a virtual host or directory config according to documentation.

More or less, yes. Each sub interpreter is named and is tied to some part of
the URL namespace. The default is that there will be a sub interpreter for
each virtual host, but this can be overridden by various directives. 

> This is
> creating problems because the other requests have to wait for a long time
> and i get 503 from apache.

The way Python sub interpreters are used, it is highly unlikely that a 503
response is going to be because of them. It may be how you have Apache
configured, but to know you will have to state what platform you are using
and if on UNIX whether you have compiled Apache with the prefork MPM
or worker MPM. You also should post what you Apache configuration is
that controls the number of Apache sub processes and/or threads within
a process.

> It will be useful if i have multiple
> subinterpreters
> to which apache can assign requests arbitrarily. I do not maintain state
> in
> any of the interpreters.

This will not make any difference, what matters is how many Apache child
process are allowed to be created when using prefork MPM, or how many
child processes and/or threads with a process that Apache is allowed to
create when using a multithreaded MPM.

If using prefork MPM and you have restricted the number of Apache child
processes to a small fixed number, then if you get more requests than
that arrive, there will be no processes available to handle the request. If
you are using a multithreaded MPM, then a small number of threads across
all processes will limit how many requests can be handled concurrently.

Creating extra Python subinterpreters only gives you separation of data,
it doesn't provide any additional concurrency abilities. You seem not to
understand this point.

> iow, how do i scale this to a large number of users. how do i introduce
> parallel processing of requests for the same virtual host/directory. at
> the
> moment it looks
> like its single threaded.

If you use prefork, then only one request in each child process can be
handled at a time, but then you should be running with adequate child
processes so that isn't a problem. If using a multithreaded MPM, then there
already can be multiple requests handled at the same time and these
can be handled even within the same Python subinterpreter.

> definitely there should be a way as this is a common problem... can someone
> please point me to the solution.

You probably need to research more about the different MPM models for
Apache. For starters, read:

  http://www.dscpl.com.au/wiki/ModPython/Articles/TheProcessInterpreterModel

You might then see what you can find in documentation about Apache referenced on:

  http://wiki.apache.org/mod_python/

On first guess I would say how you have configured Apache is limiting things.

If you are using a multithreaded MPM, then you might have a third party module
for Python you are using which doesn't release the Python GIL around a long
running operation being performed in C code. Thus, what extra Python modules
are you using?


Graham


More information about the Mod_python mailing list