[mod_python] Performance Question

Thomas Moyer tmmoyer at cse.psu.edu
Thu Sep 11 09:11:00 EDT 2008


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I've monitored the time spent by the background process to handle one  
request, and the numbers are as expected (i.e. very short times to  
handle a request).  Also monitoring top during the run shows my  
background process as nearly idle during this whole test. Never more  
than 1-2% CPU usage.  The computation in the background process is  
nil.  It merely takes a string of XML it has previously generated and  
sends it to the mod_python module.  Networking should also be nil  
because I am using Unix domain sockets which are local to the system  
and therefor bypass the network stack and simply move bytes around in  
memory.

~tom

On Sep 11, 2008, at 9:06 AM, Bart wrote:

> I'm not sure whether this is relevant,
> but I vaguely recall that time in send()/recv() show up as CPU time
> since it's actually polling behind the scenes, so could it be that
> those background processes are causing most of that CPU through
> (computation+networking) delay that they cause at that level of
> parallelism?
>
> It sounds a little too extreme for it to be that, though. Have you
> tested other, similar setups, e.g. doing the same parallelism test
> through a command line script outside apache (e.g. by firing off ~100
> threads that do the same)?
>
> Regards,
> --Bart
>
>
> On Thu, Sep 11, 2008 at 2:18 PM, Thomas Moyer <tmmoyer at cse.psu.edu>  
> wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> I am having some problems with performance when using mod_python to  
>> serve
>> some dynamic content.  The request handler gathers information being
>> generated by background processes on the system, and then creates a  
>> simple
>> XML document that is sent to the client. Here is the code I am  
>> using with
>> some explanation.
>>
>>   requestedFilePath = "/" + req.filename.split("/")[-1] 
>> [:-4].replace("_",
>> "/").replace("-",".")
>>   sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
>>   sock.connect(ATTService)
>>   Attest = message.recvMsg(sock)[0]
>>   sock.close()
>>
>>   fmt = "%ds" % len(requestedFilePath)
>>   msg = struct.pack(fmt, requestedFilePath)
>>   sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
>>   sock.connect(MHTService)
>>   message.sendMsg(sock, msg, fmt)
>>   MHTList = message.recvMsg(sock)[0]
>>   sock.close()
>>
>>   req.content_type = 'text/xml'
>>   req.write("<ACA>%s%s</ACA>" % (Attest,MHTList))
>>   return apache.OK
>>
>> There are 3 different pieces here.  The first block connects to a  
>> daemon
>> running on the system that is monitoring the system. I connect to a  
>> Unix
>> domain socket and receive the most recent information it has.  I  
>> then do the
>> same thing with another daemon that is gathering information about  
>> files
>> that are available on the system. Finally I combine these two  
>> blocks of
>> information and send it to the client.
>>
>> My problem is, when I start using benchmarking tools like JMeter,  
>> and I put
>> the system under any sort of load (100 clients for example), the  
>> first
>> recvMsg call (Attest = message.recvMsg(sock)[0]) takes about 12  
>> seconds to
>> complete per client.  I watch top during the run and see that the  
>> apache
>> processes are completely occupied (usually showing 120+% CPU  
>> usage).  The
>> system running apache is an 8-core machine with 16GB of RAM so I'm  
>> pretty
>> sure there isn't a resource problem for either memory or processor.
>>
>> I'm not really sure where to look next to understand why a simple  
>> socket
>> recv() call is taking 12 seconds to complete.  The amount of data  
>> being
>> recv()'d is ~80K of text if that makes a difference.
>>
>> ~tom
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v1.4.8 (Darwin)
>>
>> iEYEARECAAYFAkjJDI0ACgkQa7Yypxfw9TeOPwCfb29vCZlna6WZL9X/SASxT4Ms
>> cxYAnRyDlcMF6gWuSgdT8HHPnq1Urs/H
>> =NPk2
>> -----END PGP SIGNATURE-----
>> _______________________________________________
>> Mod_python mailing list
>> Mod_python at modpython.org
>> http://mailman.modpython.org/mailman/listinfo/mod_python
>>
> _______________________________________________
> Mod_python mailing list
> Mod_python at modpython.org
> http://mailman.modpython.org/mailman/listinfo/mod_python


- --
Thomas Moyer
Graduate Student
CSE Department
Penn State University
tmmoyer at cse.psu.edu
http://www.cse.psu.edu/~tmmoyer

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (Darwin)

iEYEARECAAYFAkjJGOQACgkQa7Yypxfw9TcPtACgjXtIoSOMjIRJHw8OQXNfSL0I
2ewAoKG02WbJ8c56Cf61803AERx+2ekv
=qWaY
-----END PGP SIGNATURE-----


More information about the Mod_python mailing list