id	summary	reporter	owner	description	type	status	priority	milestone	component	version	resolution	keywords	cc	launchpad_bug
798	improve random-access download to retrieve/decrypt less data	warner	davidsarah	"Currently, using a {{{Range:}}} header on an HTTP GET (to fetch just a portion of a file, instead of the whole thing) causes the tahoe client to download the entire file into a tmpfile, then serve out just the portion that was requested. To make this faster, we should only fetch the segments that contain the desired range. Two changes need to happen to make this work:

 * the Downloader must be rewritten, to fetch segments on demand, to (sometimes) cache previously cached segments, and decrypt just the necessary data
 * pycryptopp needs to provide random-access AES processing, so we can decrypt data starting at some point other than the beginning of the file ([http://allmydata.org/trac/pycryptopp/ticket/18 pycryptopp#18])

The new Downloader should have a couple of layers:
 * top layer receives a {{{read(offset, length)}}} request (maybe a even fully-general {{{readv}}})
 * that layer looks at the set of cached segments, tries to satisfy the request from cache
 * if not, submit requests for segment fetches to the lower layer
 * lower layer looks to see what servers it has available, which requests are already in flight
 * it sends out more requests, or prepares new servers (querying for share presence, fetching hash trees, fetching block data)

Similar code will be needed for MDMF mutable files, since those are specified to contain multiple segments, and we'll want random-access for them too.

"	enhancement	closed	major	1.8.0	code-network	1.5.0	fixed	performance pycryptopp random-access download large confidentiality review-needed docs	jeremy@…