|
Graham Dumpleton
grahamd at dscpl.com.au
Thu Aug 25 16:58:15 EDT 2005
Move the "import" statements outside of the timed area of code.
Some Python module imports could possibly take more than a
trivial amount of time. If the pxtl module import is one such
import, especially since you were using reload, you could have
been incurring this import cost on every time through the code.
Eliminate this possibility and then look at timing each individual
line of code to narrow it down further to the worst offender.
On 25/08/2005, at 4:31 PM, M Chaudhary wrote:
> Even if i remove reload(pxtl) ; there is no improvement.
>
> Graham Dumpleton wrote:
>
>> Why are you doing:
>>
>> reload(pxtl)
>>
>> What is it about "import" that isn't enough?
>>
>> M Chaudhary wrote ..
>>
>>> Hello friends,
>>>
>>> 1) When I perform just file io it takes 0.00026 seconds.
>>> import time
>>> fp = open("xyz.html","r")
>>> s = fp.read()
>>> fp.close()
>>> stime = time.time()
>>> fp = open('xyz.px','w')
>>> fp.write(s)
>>> fp.close()
>>> etime = time.time()
>>> print etime-stime
>>>
>>>
>>> 2) When I perform just pxtl processing it takes 0.064 seconds.
>>>
>>> import time
>>> stime = time.time()
>>> import pxtl
>>> reload(pxtl)
>>> pxtl.processFile('xyz.px')
>>> etime = time.time()
>>> print etime-stime
>>>
>>> 3) When I perform pxtl processing along with file-io it takes 1.33
>>> seconds.
>>> import time
>>> fp = open("xyz.html","r")
>>> s = fp.read()
>>> fp.close()
>>> stime = time.time()
>>> fp = open('xyz.px','w')
>>> fp.write(s)
>>> fp.close()
>>> import pxtl
>>> reload(pxtl)
>>> pxtl.processFile('xyz.px')
>>> etime = time.time()
>>> print etime-stime
>>>
>>> As you can see there is a major difference in 1+2 and 3 .
>>> Can you please tell me what the problem might be and what should i
>>> do ?
>>> I am using PXTL for developing a website along with file io as PXTL
>>> does
>>> not support string processing .
>>> Thanks in Advance.
>>> Regards,
>>> Ajinkya
>>>
>>> Orr, Steve wrote:
>>>
>>>
>>>> Well obviously mod_python is performing well and I doubt anything
>>>> connected with Linux versions would account for that much
>>>> difference so
>>>> the thing to focus on would be tuning dbxml or the code associated
>>>> with
>>>> it. Don't you have a way to run/test chunks of code outside of a web
>>>> environment?
>>>>
>>>> -----Original Message-----
>>>> From: M Chaudhary [mailto:igp-dev at infogridpacific.com] Sent:
>>>> Tuesday, August 23, 2005 12:13 AM
>>>> To: mod_python at modpython.org
>>>> Cc: Orr, Steve; IGP; and at doxdesk.com; xml at sleepycat.com
>>>> Subject: Re: [mod_python] Performance Problem
>>>>
>>>> Hello Orr, Steve ,
>>>> As you told i tested my application using "ab" tool for three
>>>> cases (each for single request and 5 concurrent requests ):-
>>>> and the results were as follows (the following log was generated
>>>> from
>>>>
>>> my
>>>
>>>> mod-python handler itself using time.time() method ) ----
>>>> Attached is the test module . Please check it and view the
>>>> following results.
>>>> #-------------------------------------------------------------------
>>>> ----
>>>> ----------------------------
>>>> # Test 1 -- dbxml :- get a document and write it to browser
>>>> #-------------------------------------------------------------------
>>>> ----
>>>> ----------------------------
>>>> 0.00605797767639 # single request
>>>> 0.0058479309082
>>>> 0.00573110580444
>>>> 0.148849010468
>>>> 0.00549292564392
>>>> 0.00417685508728 # 5 concurrent requests
>>>> 0.00559282302856
>>>> 0.00551891326904
>>>> 0.00600981712341
>>>> 0.00549983978271
>>>> #-------------------------------------------------------------------
>>>> ----
>>>> ----------------------------
>>>> # Test 2 -- pxtl :- pxtl process a File and write it to browser
>>>> #-------------------------------------------------------------------
>>>> ----
>>>> ----------------------------
>>>> 1.65334200859 # single request
>>>> 0.25177192688
>>>> 0.261835098267
>>>> 0.294239997864
>>>> 0.000832080841064
>>>> 0.428733825684 # 5 concurrent requests
>>>> 0.440675020218
>>>> 0.269226074219
>>>> 0.237013816833
>>>> 0.13219499588
>>>> #-------------------------------------------------------------------
>>>> ----
>>>> ----------------------------
>>>> # Test 3 -- pxtl and dbxml :- get a document from dbxml container
>>>> ,pxtl
>>>>
>>>> process it and write it to browser
>>>> #-------------------------------------------------------------------
>>>> ----
>>>> ----------------------------
>>>> 3.01918792725 # single request
>>>> 13.4880371094
>>>> 14.6321718693
>>>> 15.9680738449 # 5 concurrent requests
>>>> 16.1954669952
>>>> 0.010479927063
>>>> 17.400357008
>>>> 12.5239679813
>>>> 11.8724710941
>>>> 10.8379950523
>>>> #-------------------------------------------------------------------
>>>> ----
>>>> ----------------------------
>>>>
>>>> As you can see the first problem is that when i test it for 5
>>>> concurrent
>>>>
>>>> request i get 9 entries in my log file
>>>> each in each case.
>>>> And the second but most annoying problem is for Test 3 ( for both
>>>> pxtl
>>>> and dbxml ) it takes more than 10 seconds
>>>> to satisfy each request .
>>>> Even i tested my application on Upgraded Kernel (2.6.9-11.EL).
>>>> Should i use any other Linux distribution.
>>>> Can you tell me what the problem may be ? And what should i do ?
>>>>
>>>> Regards,
>>>> Ajinkya N
>>>>
>>>>
>>>> Orr, Steve wrote:
>>>>
>>>>
>>>>
>>>>> I can't see that mod_python would be the problem. For a given
>>>>> request,
>>>>> what percentage of the overall response time is involved with
>>>>> database
>>>>> access versus code execution? You need to know that. I'd first try
>>>>> tuning the code before doing any web stuff. Especially tune the
>>>>>
>>>>>
>>>> database
>>>>
>>>>
>>>>> queries and the network access to the database before going
>>>>> through the
>>>>> web server. Then setup database connection pooling so each page
>>>>> request
>>>>> reuses a connection instead of establishing a new connection with
>>>>> each
>>>>> page turn. Can you do some kind of session management to avoid
>>>>>
>>>>>
>>>> repop'ing
>>>>
>>>>
>>>>> "the container" with each page turn?
>>>>>
>>>>> Using ab I did some bench marking a year ago and with an old, cheap
>>>>>
>>>>>
>>>> Dell
>>>>
>>>>
>>>>> 2GHz PC with 512M RAM as the web server I was able to get 750 page
>>>>>
>>>>>
>>>> turns
>>>>
>>>>
>>>>> per second. The database was Oracle and I used the connection
>>>>> pooling
>>>>> feature that comes with cx_Oracle on the OCI. (A persistent
>>>>> connection
>>>>> can be established for each Apache process so this isn't really
>>>>> necessary.) I also reused cursors to avoid the parsing overhead.
>>>>> (Besides doing web development I'm a DBA dweeb.) Without
>>>>> mod_python and
>>>>> connection pooling it got as slow as 11 requests per second. Of
>>>>> course
>>>>> my benchmarking was doing simple stuff but I was satisfied that
>>>>> Apache/mod_python would perform well and I wouldn't have to go
>>>>> down a
>>>>> "Twisted" path. Don't know anything about dbxml but you should
>>>>> address
>>>>> the architecture/code first. If performance is really a big issue
>>>>> then
>>>>> consider load balanced web servers and a good disk array setup on
>>>>> the
>>>>>
>>>>>
>>>> db
>>>>
>>>>
>>>>> server.
>>>>>
>>>>> Steve Orr
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: mod_python-bounces at modpython.org
>>>>> [mailto:mod_python-bounces at modpython.org] On Behalf Of IGP
>>>>> Sent: Wednesday, August 17, 2005 11:01 PM
>>>>> To: xml at sleepycat.com; mod_python at modpython.org
>>>>> Subject: [mod_python] Performance Problem
>>>>>
>>>>> I am accessing dbxml via my mod-python handler.
>>>>> The dbxml manager instance is created , the container is opened
>>>>> and
>>>>>
>>>>>
>>>> the
>>>>
>>>>
>>>>> required operation
>>>>> is performed(querying , geting a document ) for each client
>>>>> request .
>>>>> This is affecting the
>>>>> performance . Is there any way to enhance the performance ? I mean
>>>>> that
>>>>>
>>>>>
>>>>
>>>>
>>>>> the XmlManager instance
>>>>> will be created only once , the container will be opened and the
>>>>> stored
>>>>>
>>>>>
>>>>
>>>>
>>>>> in memory , so that the
>>>>> response time will decrease .
>>>>> I am using following versions of software
>>>>> 1) dbxml - 2.1.8
>>>>> 2) python - 2.3.4
>>>>> 3) mod-python - 1.3.4
>>>>> 4) apache 2.0.52
>>>>> 5) Kernel version-2.6.9-5.0.5-EL
>>>>>
>>>>> Please do reply if you know the solution .
>>>>> Thanks in Advance,
>>>>> Regards,
>>>>> Ajinkya
>>>>>
>>>>> _______________________________________________
>>>>> Mod_python mailing list
>>>>> Mod_python at modpython.org
>>>>> http://mailman.modpython.org/mailman/listinfo/mod_python
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Mod_python mailing list
>>>>> Mod_python at modpython.org
>>>>> http://mailman.modpython.org/mailman/listinfo/mod_python
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>> _______________________________________________
>>>> Mod_python mailing list
>>>> Mod_python at modpython.org
>>>> http://mailman.modpython.org/mailman/listinfo/mod_python
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>> _______________________________________________
>>> Mod_python mailing list
>>> Mod_python at modpython.org
>>> http://mailman.modpython.org/mailman/listinfo/mod_python
>>>
>> _______________________________________________
>> Mod_python mailing list
>> Mod_python at modpython.org
>> http://mailman.modpython.org/mailman/listinfo/mod_python
>>
>>
>>
>>
>>
|