[mod_python] Protecting Web apps from to many simultaneous clicks/Hacking

On Mon, 2004-05-17 at 12:42, SAiello at Jentoo.com wrote:
> Connection Pools seem like a daunting undertaking, it could just be an 
> illusion.  Until I understand the basic definition of a Connection Pool, in a 
> programming way. It will seem like a complicate and mysterious thing.

Connection pools in Apache are particularly difficult because it is
rather difficult to safely share open connections across multiple
processes.  In short, this is a complicated and mysterious thing, and I
don't recommend trying it. :)

> Currently, when ever I require IMAP information, a connection needs to be 
> established, user authentication, information request, close IMAP connection, 
> parse the IMAP information into useable lists/distionaries, display web page. 
> In the beginning I really didn't like having to open and close the IMAP 
> connection, seemed like such a waste. So I tried to store the imaplib 
> function into a session var, that was bad. ImapLib can't be pickled due to it 
> using __slots__. I wrote the writer of imapLib, and he said he wasn't even 
> aware it used __slots__, and that he may look at it at some point. So to me 
> my next option was to look at writing my own IMAP library, how hard can it 
> be.. Results, I am not even close to handling tcp sockets. At this point I 
> was flustered, and came to conclusion to design my app as best I can, and 
> after it is working, go back and try to work on the bits I feel can be done 
> better.

You would be unable to pickle ImapLib anyway - pickling takes the state
of an object and serialises it to a data stream, with unpickling being
the reverse.  However, in the case of a TCP connection to an IMAP
server, there is state held by both the OS and the IMAP server about the
established connection which cannot be serialised.

(Imagine: you write a pickled ImapLib object to disk, three days later
when you try to unpickle it, how does the remote IMAP server know that
it's an unpickled object from before?)

One way to solve the problem would be to extend the ImapLib object to
store enough information to reestablish the connection, but this
wouldn't help here - you'd still be doing the expensive connection
establishment when you unpickled the object!

Rather, I would look for places you make the imaplib calls, and see if
you could first check for cached data in your session before doing the
call:

def fetch_index(sess, user, password):
    if sess.has_key('cache') and sess.cache.has_key('index'):
        if sess.cache['index']['expires'] > time.time():
            return sess.cache['index']['data']
    index = imaplib_fetch_index(user, password)
    if not sess.has_key('cache'):
        sess['cache'] = {}
    sess['cache']['index'] = {}
    sess['cache']['index']['expires'] = time.time() + 300
    sess['cache']['index']['data'] = index
    return index

What this does is check if your session has a cache, and if that cache
contains an entry for the imap index, and that the cached entry has not
expired (it only lasts 5 minutes in this case).  If it does, the cached
version is returned.

Otherwise, the old call to fetch the index from imaplib is executed, the
cache is created if it didn't exist, and an entry for the imap index is
added.

> That was my thinking to to use the session storage option. It seemes the 
> easiest way to do it. But I am always wary when doing something the easy way.

I like doing things the easy way.  If it's easy to understand now, it'll
be easy to understand six months later when you need to do maintenance. 
If it's easy to write, it should be easy to spot bugs.  And if it's easy
to do, it'll take you less time than a more complex alternative.

> I do not use refresh to cause the error, but page links (i.e. like the Next 
> button for the next set of messages).  clicking over and over on  the A link 

>From the server end, it's still a bunch of requests coming in hot on
each others' heels.  I used refresh so I didn't have to futz around
making a form that submitted to a mod_python module; I just threw in an
index.py and a PythonHandler index.

> will cause it. It isn't the browser, I have tried Konqueror, firefox, and IE, 
> they all will get my error page. Below are my system specs. I think Apache is 
> forking, because I am not using threads as a compile option, so that is 
> forked right ? I have to read up on which is better, I tried apache with 
> threads once to see if that was the issue..  still did the same thing.

"Which process model is better" is a tricky question, and probably
outside of the immediate scope of this list - and definitely outside the
scope of my experience. :)

> Server Specs:
>   Gentoo distribution of GNU/Linux, kernel 2.6.4
>   Apache 2.0.49 with berkdb, gdbm, & ldap compiled in.
>   mod_python 3.1.3

I'm running 2.0.47 with 3.1.3, but it doesn't sound like this is a
problem with your versions.  I can't shed any light on why your sessions
are apparently not locked; so I don't know if I can help you further. :(

-- 
bje