Gregory Trubetskoy
grisha at modpython.org
Wed Nov 15 21:35:42 EST 2000
This is somewhat off-subject, but IMHO the best tool for time-series data storing and charting is RRDtool. It can produce charts for a year of data in a split second. Grisha On Wed, 15 Nov 2000, Graham Swallow wrote: > On Tue, 14 Nov 2000, Alexis Iglauer wrote: > > > My datafiles are time series data, usually for 24 > > hours at a time, so I will have a whole bunch of > > datafiles which make up a contiguous set of data over > > several days/months/years. > > I have a similar situation, but not in apache_mod. The data is > an array of floats (binary portable), one every minute for a week, > over the year. > > Repackaging last weeks data is slow, but you dont do that for > every request. > > I use Array to get it as a string, then I use zlib to get a string, > then I 'escape' the data for MySql (using the standard function) > and write it into a blob field. > > The data comes back with no need for un-escaping (its so that it > goes into an SQL statement). > > Then I call a C function to convert the floats to doubles and > create a BLT/Vector. Its very quick. > > Using GD (or whatever) should not be slow, but a bit of C at the > right place means you move chunks of data around, not iterate > python over thousands of items. > > The 'most recent' file obviously needs special treatment. > You could re-process today every hour (hint: move, start new file, > then process the moved file, if fail - dont delete). > > That still leaves you with a 'double' data source, first SQL, then > the additional file. Alternatively, get the data collector to > post the data ... > > Graham > > _______________________________________________ > Mod_python mailing list > Mod_python at modpython.org > http://www.modpython.org/mailman/listinfo/mod_python >
|