Document current crypto and encoding in detail

[davidsarah]

Other than the code, the most comprehensive description of Tahoe's current crypto and erasure encoding that I'm aware of is the Storage Security and Survivability Workshop 2008 paper: ​http://tahoe-lafs.org/~zooko/lafs.pdf. However, that paper does not give the level of detail required for a spec or for a thorough security analysis (for example, it doesn't specify encryption modes or the encoding of inputs to crypto primitives).

This is an obstacle to designing the new crypto, since we don't want to lose features (unless we drop them deliberately) or make mistakes that were avoided in the original design.

[davidsarah]

I guess what this is asking for would be the unwritten doc "#1: Share Format, Encoding Algorithm" described in source:docs/specifications/outline.txt .

[davidsarah]

As an example of the kind of detail I'm looking for, generating a convergent encryption key for an immutable file would be:

key = SHA256d(
        netstring("allmydata_immutable_content_to_key_with_added_secret_v1+"
                  ++ netstring(convergence)
                  ++ netstring("%d,%d,%d" % (k, n, segsize)))
        ++ plaintext)

[warner]

yeah, I've been meaning to write this up for a year and haven't gotten around to it. In general, we've been too dependent upon using code as a specification tool.. as the code gets rearranged (for performance reasons, mostly), it becomes less useful as a form of documentation.

I'm actually looking to build two docs: a text one that extracts the crypto and protocol pieces from the current code, and a diagram one that parallels davidsarah's excellent "Elk Point" proposals. I want to be able to compare the features and complexity of our current encoding format against other proposals, and having similar-format pictures for all of them would help that a lot.