[mod_python] CGI handler and multithreading: Can i have multiple sys.stdout objects?

Your best bet is the prefork MPM. Remember that the 10M per child is
actually a lot less because pages for executables are shared between
processes on (most) UNIXes, certainly on Solaris they are. I find it hard
to believe that it is slower than regular CGI - it should be at least
several times faster, though not nearly as efficient as using the
publisher or a native handler.

The ultimate solution is not use CGI of course :-)

Grisha

On Thu, 27 Feb 2003, Mike Looijmans wrote:

> mod_python has a nice "cgihandler", that allows me to run out old CGI code in
> mod_python 'mode'. However, I have some problems with that, that some of you
> may know a solution to.
>
> The cgihandler will lock until the request is complete. This is OK, but it
> restrains the number of simultaneous Python requests to just 1. In practice,
> this means that the system gets a lot slower with mod_python, because there
> are a few requests that send a few megabytes of data to the client, and this
> can take a few minutes. The performance hit is unacceptable.
>
> I'm running Apache 2.0.44 in "worker" thread mode on a Solaris 2.6 system.
>
> A workaround is to compile apache with the "pre-fork" system. This however
> yields another problem. The child processes all fire up their own interpreter,
> and the memory load is about 10MB for each child (in 'worker' mode, only one
> interpreter for 20 childs!). This quickly eats up all resources on the
> machine. It also virtually eliminates the possibility to "recycle" the
> database connection and similar objects. The net effect is that this setup is
> also slower than the plain CGI implementation which just starts (runs and
> kills) an interpreter for each CGI request.
>
> I tried removing the thread lock in the cgihandler. All CGI scripts I have
> don't care about the directory they are in (they get all they need from the
> MySQL DB), and all scripts they import are fully re-entrant (or MT safe)
> themselves. This left only one single problem: The "print" command.
>
> Python's print command is being used to send the data to the client. The
> cgihandler replaces sys.stdout and redirects it to the Request object, which
> is ok. But there is only one "sys" module shared by all threads, so all the
> print commands from all threads end up in the same request. In practice, if a
> "long" request is running, and a shorter request interferes, the short one
> will kill sys.stdout and the long one will abort with an exception because
> sys.stdout had suddenly died.
>
> One workaround would be to create a cgihandler 'clone' that passes an "output"
> object to each script (which defaults to sys.stdout for "plain" CGI mode), and
> let the scripts all write() to that object. Drawback is that I will have to
> replace all "print" statements in about 10k+ lines of code. Also, I want the
> scripts to remain CGI compatible for a few more weeks because a few shadow
> systems cannot be upgraded to mod_python yet.
>
> Is there a way that I can let the threads each "print" to their own output,
> for example by having a seperate "sys" module in each thread? (could I delete
> the sys module and re-import it or something similar?)
>
> --
> Mike Looijmans
>
> _______________________________________________
> Mod_python mailing list
> Mod_python at modpython.org
> http://www.modpython.org/mailman/listinfo/mod_python
>