[mod_python] Session data

Bo Lorentsen <bl at netgroup.dk> wrote:
> > You can use either shared mmap or SysV shared mem (there are
> > python modules for both) to share memory.  I wound up writing
> > a Python shared dictionary (just pickles objects into the
> > filesystem when you set them, and unpickles them when you read
> > them)
> 
> How about performance ? Is'nt it expensive accessing file in a
> dir. for shared data, I mean, you have to reload it if its RW
> data, or is it write ones read many (it sounds like it) ?

I can write about 3000 entries to the dictionary per second.  I
can read about 8000 entries from the dictionary per second.  This
is with SMP-safe locking, but running on a uniprocessor (600 Mhz
PIII).  This is tha naive 30 minute implentation without any C
extensions or anything.

Most of the time is spent pickling, fs access is _really_ fast
on a decent operating system (e.g. Linux, FreeBSD).  If you're
storing really large objects, the pickling may take longer.

If you're storing a lot of items (more than a few thousands) use
the regular db-in-fs options (nested directories with a few 1000
files each or a btree filesystem like ReiserFS).

> > Same idea would work for user sessions, and since it's in the
> > filesystem the kernel will take care of buffering for you and
> > you can use tools like "find" and "rm" to prune the cache.
> > :-)
>
> Hmm, If I like to change user session data offen, I quess I
> would go for the "shared mem" solution.

I tried it when I first implemented the thing , the timing was
similar (within 2-3%).  Which makes sense, because the time to
pickle is dominating anyway.  Bear in mind that the fs cache is
going to keep all your recently accessed files in RAM anyway.

If you do it in the fs, the implementation is much simpler-- you
need to build a balanced tree or hash table in SHM to get good
performance with more than 20-30 items in the database.  And
you're not as portable.  The key for me, though, was that being
able to play with the cache w/ regular user tools and assign
permissions per-item can be extremely useful.

  Sumner