|
Thomas Moyer
tmmoyer at cse.psu.edu
Thu Sep 11 09:11:00 EDT 2008
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I've monitored the time spent by the background process to handle one request, and the numbers are as expected (i.e. very short times to handle a request). Also monitoring top during the run shows my background process as nearly idle during this whole test. Never more than 1-2% CPU usage. The computation in the background process is nil. It merely takes a string of XML it has previously generated and sends it to the mod_python module. Networking should also be nil because I am using Unix domain sockets which are local to the system and therefor bypass the network stack and simply move bytes around in memory. ~tom On Sep 11, 2008, at 9:06 AM, Bart wrote: > I'm not sure whether this is relevant, > but I vaguely recall that time in send()/recv() show up as CPU time > since it's actually polling behind the scenes, so could it be that > those background processes are causing most of that CPU through > (computation+networking) delay that they cause at that level of > parallelism? > > It sounds a little too extreme for it to be that, though. Have you > tested other, similar setups, e.g. doing the same parallelism test > through a command line script outside apache (e.g. by firing off ~100 > threads that do the same)? > > Regards, > --Bart > > > On Thu, Sep 11, 2008 at 2:18 PM, Thomas Moyer <tmmoyer at cse.psu.edu> > wrote: >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> I am having some problems with performance when using mod_python to >> serve >> some dynamic content. The request handler gathers information being >> generated by background processes on the system, and then creates a >> simple >> XML document that is sent to the client. Here is the code I am >> using with >> some explanation. >> >> requestedFilePath = "/" + req.filename.split("/")[-1] >> [:-4].replace("_", >> "/").replace("-",".") >> sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) >> sock.connect(ATTService) >> Attest = message.recvMsg(sock)[0] >> sock.close() >> >> fmt = "%ds" % len(requestedFilePath) >> msg = struct.pack(fmt, requestedFilePath) >> sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) >> sock.connect(MHTService) >> message.sendMsg(sock, msg, fmt) >> MHTList = message.recvMsg(sock)[0] >> sock.close() >> >> req.content_type = 'text/xml' >> req.write("<ACA>%s%s</ACA>" % (Attest,MHTList)) >> return apache.OK >> >> There are 3 different pieces here. The first block connects to a >> daemon >> running on the system that is monitoring the system. I connect to a >> Unix >> domain socket and receive the most recent information it has. I >> then do the >> same thing with another daemon that is gathering information about >> files >> that are available on the system. Finally I combine these two >> blocks of >> information and send it to the client. >> >> My problem is, when I start using benchmarking tools like JMeter, >> and I put >> the system under any sort of load (100 clients for example), the >> first >> recvMsg call (Attest = message.recvMsg(sock)[0]) takes about 12 >> seconds to >> complete per client. I watch top during the run and see that the >> apache >> processes are completely occupied (usually showing 120+% CPU >> usage). The >> system running apache is an 8-core machine with 16GB of RAM so I'm >> pretty >> sure there isn't a resource problem for either memory or processor. >> >> I'm not really sure where to look next to understand why a simple >> socket >> recv() call is taking 12 seconds to complete. The amount of data >> being >> recv()'d is ~80K of text if that makes a difference. >> >> ~tom >> -----BEGIN PGP SIGNATURE----- >> Version: GnuPG v1.4.8 (Darwin) >> >> iEYEARECAAYFAkjJDI0ACgkQa7Yypxfw9TeOPwCfb29vCZlna6WZL9X/SASxT4Ms >> cxYAnRyDlcMF6gWuSgdT8HHPnq1Urs/H >> =NPk2 >> -----END PGP SIGNATURE----- >> _______________________________________________ >> Mod_python mailing list >> Mod_python at modpython.org >> http://mailman.modpython.org/mailman/listinfo/mod_python >> > _______________________________________________ > Mod_python mailing list > Mod_python at modpython.org > http://mailman.modpython.org/mailman/listinfo/mod_python - -- Thomas Moyer Graduate Student CSE Department Penn State University tmmoyer at cse.psu.edu http://www.cse.psu.edu/~tmmoyer -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iEYEARECAAYFAkjJGOQACgkQa7Yypxfw9TcPtACgjXtIoSOMjIRJHw8OQXNfSL0I 2ewAoKG02WbJ8c56Cf61803AERx+2ekv =qWaY -----END PGP SIGNATURE-----
|