|
SAiello at Jentoo.com
SAiello at Jentoo.com
Sun May 16 23:42:20 EDT 2004
On Sunday 16 May 2004 07:24 pm, Byron Ellacott wrote: > The trouble is that communication between the frontends and backends > probably has a lot of expensive overhead. I'm making a few assumptions > about how your system works here that may be invalid. First up, I'm > assuming that a frontend is connecting to a backend using a TCP > connection. Establishing a TCP connection is a non-trivial operation; > even connecting to the local host can be relatively costly. Doing a DNS > lookup for the IMAP backend's IP address is expensive too, though easy > to avoid. Once a connection's established, you need to present the > backend with your user's credentials, which involves more round trips of > data, and then finally you can do your IMAP operation, which has further > TCP overhead. After that, you need to close the TCP connection, which > is probably done after the user sees their result, but before the Apache > process is available to serve another result. > > Compare this to, for example, storing some cached information in your > session: mod_python is already loading and saving your session via dbm > or shm, so the additional overhead is the marshalling and I/O for the > cached data. Significantly less than talking to an IMAP backend. > > If you're using IMAP connection pools, this section is fairly much > irrelevant, since you're avoiding the high costs of connection setup and > teardown for most requests. However, it's been my experience that > connection pools are difficult to achieve in a forked Apache world. Oh believe me, I know my IMAP connections are rather expensive. Currently I am using Python's imaplib, for all imap queries. You have to understand I am not an advance Python programer. Maybe intermediate programmer, but I am still trying to learn the Pythonic way to do things, in using the language as efficiently as possible. And if my Python code relies on other software (i.e. Apache) trying to understand the dynamics of how that software works, and the interaction between it and Python. So basically my head is swimming. My current goal is to get something up and running with my current knowledge base, running as best as I can design now, learning as I go. Connection Pools seem like a daunting undertaking, it could just be an illusion. Until I understand the basic definition of a Connection Pool, in a programming way. It will seem like a complicate and mysterious thing. Currently, when ever I require IMAP information, a connection needs to be established, user authentication, information request, close IMAP connection, parse the IMAP information into useable lists/distionaries, display web page. In the beginning I really didn't like having to open and close the IMAP connection, seemed like such a waste. So I tried to store the imaplib function into a session var, that was bad. ImapLib can't be pickled due to it using __slots__. I wrote the writer of imapLib, and he said he wasn't even aware it used __slots__, and that he may look at it at some point. So to me my next option was to look at writing my own IMAP library, how hard can it be.. Results, I am not even close to handling tcp sockets. At this point I was flustered, and came to conclusion to design my app as best I can, and after it is working, go back and try to work on the bits I feel can be done better. > mod_python's session data first up, because it will have already tied > your cached data to a particular user, and because a lot of the overhead > of storing/retrieving data is ameliorated with the session work, and > because it's easier to use Grisha's work than to duplicate it. :) That was my thinking to to use the session storage option. It seemes the easiest way to do it. But I am always wary when doing something the easy way. > The trouble I'm having here is that if session locking is working, you > should never encounter a value of sess['REQUESTS'] > 1. The session > should be automatically locking when you first create it, and remaining > locked, as Grisha says, until it's cleaned up when the request > completes. In fact, copy/pasting your code, appending the > "sess['REQUESTS']-=1" and saving the session before returning apache.OK, > I cannot get redirected. I inserted a sleep(10) before the decrement > and return, and hit reload a dozen odd times. The only effect of this > was to make my browser spend two minutes loading the final page. :) > > If I put in "sess.unlock()" right after I create the session, I can get > myself redirected to the error page. > > Any idea how you might be winding up with an unlocked session? What's > your Apache version? mod_python version? What's the request serving > model (worker threads, forked, etc)? I do not use refresh to cause the error, but page links (i.e. like the Next button for the next set of messages). clicking over and over on the A link will cause it. It isn't the browser, I have tried Konqueror, firefox, and IE, they all will get my error page. Below are my system specs. I think Apache is forking, because I am not using threads as a compile option, so that is forked right ? I have to read up on which is better, I tried apache with threads once to see if that was the issue.. still did the same thing. If ya like external access is available to the box, if ya want to take a peek. still rather rough.. so dun laugh Server Specs: Gentoo distribution of GNU/Linux, kernel 2.6.4 Apache 2.0.49 with berkdb, gdbm, & ldap compiled in. mod_python 3.1.3 Sorry for long post, and if I provided to much information, just trying to give ya a feel where I am technical with Python programming and my thought process. Steven
|