id	summary	reporter	owner	description	type	status	priority	milestone	component	version	resolution	keywords	cc	launchpad_bug
325	flogtool scalability/performance	zooko	warner	"I'm trying to diagnose problems on the current tahoe production grid, and here are some observations:

1.  The {{{logs.pickle}}} file is 2 GB in size, and can't be web-viewed as {{{flogtool web-viewer logs.pickle}}} eventually dies after running out of RAM.

2.  7zipping {{{logs.pickle}}} takes 15 minutes and results in a {{{logs.pickle.7z}}} of 115 MB.  (I chose 7zip because I didn't want to use up too much CPU on the tahoe production grid server, and 7zip is one of the most CPU-efficient among the compressors that also give good ratio on a large file.)

3.  {{{flogtool dump logs.pickle &> logs.dump.txt}}} on my Macbook Pro takes 30 minutes and results in a {{{logs.dump.txt}}} of size 415 MB.

So for even a little-used and small grid like the current tahoe production grid, flogtool has some scalability/performance problems.

By the way, in order to estimate how much redundancy there is in the resulting text log file compared to the pickle, I 7zipped the {{{logs.dump.txt}}}, which resulted in a {{{logs.dump.txt.7z}}} of size 40 MB."	defect	new	major	undecided	operational	0.8.0		logging scalability performance memory