[mod_python] Performance Question

I'm not sure whether this is relevant,
but I vaguely recall that time in send()/recv() show up as CPU time
since it's actually polling behind the scenes, so could it be that
those background processes are causing most of that CPU through
(computation+networking) delay that they cause at that level of
parallelism?

It sounds a little too extreme for it to be that, though. Have you
tested other, similar setups, e.g. doing the same parallelism test
through a command line script outside apache (e.g. by firing off ~100
threads that do the same)?

Regards,
--Bart


On Thu, Sep 11, 2008 at 2:18 PM, Thomas Moyer <tmmoyer at cse.psu.edu> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> I am having some problems with performance when using mod_python to serve
> some dynamic content.  The request handler gathers information being
> generated by background processes on the system, and then creates a simple
> XML document that is sent to the client. Here is the code I am using with
> some explanation.
>
>    requestedFilePath = "/" + req.filename.split("/")[-1][:-4].replace("_",
> "/").replace("-",".")
>    sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
>    sock.connect(ATTService)
>    Attest = message.recvMsg(sock)[0]
>    sock.close()
>
>    fmt = "%ds" % len(requestedFilePath)
>    msg = struct.pack(fmt, requestedFilePath)
>    sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
>    sock.connect(MHTService)
>    message.sendMsg(sock, msg, fmt)
>    MHTList = message.recvMsg(sock)[0]
>    sock.close()
>
>    req.content_type = 'text/xml'
>    req.write("<ACA>%s%s</ACA>" % (Attest,MHTList))
>    return apache.OK
>
> There are 3 different pieces here.  The first block connects to a daemon
> running on the system that is monitoring the system. I connect to a Unix
> domain socket and receive the most recent information it has.  I then do the
> same thing with another daemon that is gathering information about files
> that are available on the system. Finally I combine these two blocks of
> information and send it to the client.
>
> My problem is, when I start using benchmarking tools like JMeter, and I put
> the system under any sort of load (100 clients for example), the first
> recvMsg call (Attest = message.recvMsg(sock)[0]) takes about 12 seconds to
> complete per client.  I watch top during the run and see that the apache
> processes are completely occupied (usually showing 120+% CPU usage).  The
> system running apache is an 8-core machine with 16GB of RAM so I'm pretty
> sure there isn't a resource problem for either memory or processor.
>
> I'm not really sure where to look next to understand why a simple socket
> recv() call is taking 12 seconds to complete.  The amount of data being
> recv()'d is ~80K of text if that makes a difference.
>
> ~tom
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.8 (Darwin)
>
> iEYEARECAAYFAkjJDI0ACgkQa7Yypxfw9TeOPwCfb29vCZlna6WZL9X/SASxT4Ms
> cxYAnRyDlcMF6gWuSgdT8HHPnq1Urs/H
> =NPk2
> -----END PGP SIGNATURE-----
> _______________________________________________
> Mod_python mailing list
> Mod_python at modpython.org
> http://mailman.modpython.org/mailman/listinfo/mod_python
>