id	summary	reporter	owner	description	type	status	priority	milestone	component	version	resolution	keywords	cc	launchpad_bug
129	high memory usage during GET for large files and slow links	warner	warner	"Load testing revealed that doing a GET of a large file through a slow link
causes the memory footprint of the decoding node to balloon to the size of
the file being downloaded. The cause is simple: decode is outpacing the
download, and we're doing naive twisted.web transport.write for each segment.
This forces the transport to buffer all of the data that we've written and
which the client (in this case a browser on the other end of a DSL line) has
not yet received.

I can think of two possible solutions:

 * make sure that decoding is a producer/consumer process. This means we hold
   off on downloading the shares for a given segment until the consumer (in
   this case the HTTP connection) says they want more (because their buffer
   size has dropped below some value). This changes the control flow in
   download, not coincidentally mirroring a similar change in upload (to
   support offloaded-uploading #116).

 * have the decode process write the data to a temporary file on disk, and
   then pass that off to the web transport to read at its leisure (and
   delete it when finished, using an anonymous filehandle)

Doing producer/consumer probably raises the memory footprint by 1MB for each
active download (holding one segment of plaintext in memory while we wait for
the client to download it, maybe 2MB if we pipeline the next segment's
shares).

The tempfile approach means downloads run full-throttle and then finish,
avoiding the memory overhead, but of course then we have a disk overhead of
the full file size for the duration of the download. In practice, the kernel
will cache these disk files until they get too large, then push them to an
actual disk, with a cache size varying according to whatever else is using
memory.

I'm inclined to implement the producer/consumer thing, but when I think about
it, the kernel is in the best position to make the tradeoff between disk and
memory, so it might be a better approach to simply let it do its job. Client
behavior has an effect too: if people download half of a large file and then
quit and never come back, the tempfile approach means a lot of wasted
fetch/decode effort. On the other hand, the tempfile approach makes it a
!!!lot!!! easier to keep the tempfile around for a couple of hours in case
the client comes back to finish the job. (we'd have to implement
Content-Range: on the GET command, but that might not be all that difficult).
"	defect	closed	critical	0.6.0	code-frontend-web	0.5.1	fixed