[mod_python] Performance Question

How many Apache processes do you have configured to be started up?

Are you using worker or prefork MPM?

If worker, how many threads in each process?

How fat is your application?

Do you preload/precalculate data when application loaded?

Especially if you are using prefork MPM and only have small number of
initial Apache processes configured, then hitting it with 100
concurrent requests is going to force Apache to start a lot more
processes to handle the requests. If your application is fat, or does
a lot of preloading of data, then starting up those processes will
take time and so initial requests will appear to be slow.

Graham

2008/9/11 Thomas Moyer <tmmoyer at cse.psu.edu>:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> I am having some problems with performance when using mod_python to serve
> some dynamic content.  The request handler gathers information being
> generated by background processes on the system, and then creates a simple
> XML document that is sent to the client. Here is the code I am using with
> some explanation.
>
>    requestedFilePath = "/" + req.filename.split("/")[-1][:-4].replace("_",
> "/").replace("-",".")
>    sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
>    sock.connect(ATTService)
>    Attest = message.recvMsg(sock)[0]
>    sock.close()
>
>    fmt = "%ds" % len(requestedFilePath)
>    msg = struct.pack(fmt, requestedFilePath)
>    sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
>    sock.connect(MHTService)
>    message.sendMsg(sock, msg, fmt)
>    MHTList = message.recvMsg(sock)[0]
>    sock.close()
>
>    req.content_type = 'text/xml'
>    req.write("<ACA>%s%s</ACA>" % (Attest,MHTList))
>    return apache.OK
>
> There are 3 different pieces here.  The first block connects to a daemon
> running on the system that is monitoring the system. I connect to a Unix
> domain socket and receive the most recent information it has.  I then do the
> same thing with another daemon that is gathering information about files
> that are available on the system. Finally I combine these two blocks of
> information and send it to the client.
>
> My problem is, when I start using benchmarking tools like JMeter, and I put
> the system under any sort of load (100 clients for example), the first
> recvMsg call (Attest = message.recvMsg(sock)[0]) takes about 12 seconds to
> complete per client.  I watch top during the run and see that the apache
> processes are completely occupied (usually showing 120+% CPU usage).  The
> system running apache is an 8-core machine with 16GB of RAM so I'm pretty
> sure there isn't a resource problem for either memory or processor.
>
> I'm not really sure where to look next to understand why a simple socket
> recv() call is taking 12 seconds to complete.  The amount of data being
> recv()'d is ~80K of text if that makes a difference.
>
> ~tom
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.8 (Darwin)
>
> iEYEARECAAYFAkjJDI0ACgkQa7Yypxfw9TeOPwCfb29vCZlna6WZL9X/SASxT4Ms
> cxYAnRyDlcMF6gWuSgdT8HHPnq1Urs/H
> =NPk2
> -----END PGP SIGNATURE-----
> _______________________________________________
> Mod_python mailing list
> Mod_python at modpython.org
> http://mailman.modpython.org/mailman/listinfo/mod_python
>