Ticket #778: tests.txt

Sat Oct 17 18:30:13 PDT 2009  Kevan Carstensen <kevan@isnotajoke.com>
  * Alter NoNetworkGrid to allow the creation of readonly servers for testing purposes.

Fri Oct 30 02:19:08 PDT 2009  "Kevan Carstensen" <kevan@isnotajoke.com>
  * Refactor some behavior into a mixin, and add tests for the behavior described in #778

Tue Nov  3 19:36:02 PST 2009  Kevan Carstensen <kevan@isnotajoke.com>
  * Alter tests to use the new form of set_shareholders

Tue Nov  3 19:42:32 PST 2009  Kevan Carstensen <kevan@isnotajoke.com>
  * Minor tweak to an existing test -- make the first server read-write, instead of read-only

Wed Nov  4 03:13:24 PST 2009  Kevan Carstensen <kevan@isnotajoke.com>
  * Add a test for upload.shares_by_server

Wed Nov  4 03:28:49 PST 2009  Kevan Carstensen <kevan@isnotajoke.com>
  * Add more tests for comment:53 in ticket #778

Sun Nov  8 16:37:35 PST 2009  Kevan Carstensen <kevan@isnotajoke.com>
  * Test Tahoe2PeerSelector to make sure that it recognizeses existing shares on readonly servers

Mon Nov 16 11:23:34 PST 2009  Kevan Carstensen <kevan@isnotajoke.com>
  * Re-work 'test_upload.py' to be more readable; add more tests for #778

Sun Nov 22 17:20:08 PST 2009  Kevan Carstensen <kevan@isnotajoke.com>
  * Add tests for the behavior described in #834.

Fri Dec  4 20:34:53 PST 2009  Kevan Carstensen <kevan@isnotajoke.com>
  * Replace "UploadHappinessError" with "UploadUnhappinessError" in tests.

Thu Jan  7 10:13:25 PST 2010  Kevan Carstensen <kevan@isnotajoke.com>
  * Alter various unit tests to work with the new happy behavior

Thu Mar 18 22:06:53 PDT 2010  Kevan Carstensen <kevan@isnotajoke.com>
  * Revisions of the #778 tests, per reviewers' comments
  
  - Fix comments and confusing naming.
  - Add tests for the new error messages suggested by David-Sarah
    and Zooko.
  - Alter existing tests for new error messages.
  - Make sure that the tests continue to work with the trunk.
  - Add a test for a mutual disjointedness assertion that I added to
    upload.servers_of_happiness.
  - Fix the comments to correctly reflect read-onlyness
  - Add a test for an edge case in should_add_server
  - Add an assertion to make sure that share redistribution works as it 
    should
  - Alter tests to work with revised servers_of_happiness semantics
  - Remove tests for should_add_server, since that function no longer exists.
  - Alter tests to know about merge_peers, and to use it before calling 
    servers_of_happiness.
  - Add tests for merge_peers.
  - Add Zooko's puzzles to the tests.
  

New patches:

[Alter NoNetworkGrid to allow the creation of readonly servers for testing purposes.
Kevan Carstensen <kevan@isnotajoke.com>**20091018013013
 Ignore-this: e12cd7c4ddeb65305c5a7e08df57c754
] {
hunk ./src/allmydata/test/no_network.py 219
             c.setServiceParent(self)
             self.clients.append(c)
 
-    def make_server(self, i):
+    def make_server(self, i, readonly=False):
         serverid = hashutil.tagged_hash("serverid", str(i))[:20]
         serverdir = os.path.join(self.basedir, "servers",
                                  idlib.shortnodeid_b2a(serverid))
hunk ./src/allmydata/test/no_network.py 224
         fileutil.make_dirs(serverdir)
-        ss = StorageServer(serverdir, serverid, stats_provider=SimpleStats())
+        ss = StorageServer(serverdir, serverid, stats_provider=SimpleStats(),
+                           readonly_storage=readonly)
         return ss
 
     def add_server(self, i, ss):
}
[Refactor some behavior into a mixin, and add tests for the behavior described in #778
"Kevan Carstensen" <kevan@isnotajoke.com>**20091030091908
 Ignore-this: a6f9797057ca135579b249af3b2b66ac
] {
hunk ./src/allmydata/test/test_upload.py 2
 
-import os
+import os, shutil
 from cStringIO import StringIO
 from twisted.trial import unittest
 from twisted.python.failure import Failure
hunk ./src/allmydata/test/test_upload.py 12
 
 import allmydata # for __full_version__
 from allmydata import uri, monitor, client
-from allmydata.immutable import upload
+from allmydata.immutable import upload, encode
 from allmydata.interfaces import FileTooLargeError, NoSharesError, \
      NotEnoughSharesError
 from allmydata.util.assertutil import precondition
hunk ./src/allmydata/test/test_upload.py 20
 from no_network import GridTestMixin
 from common_util import ShouldFailMixin
 from allmydata.storage_client import StorageFarmBroker
+from allmydata.storage.server import storage_index_to_dir
 
 MiB = 1024*1024
 
hunk ./src/allmydata/test/test_upload.py 91
 class ServerError(Exception):
     pass
 
+class SetDEPMixin:
+    def set_encoding_parameters(self, k, happy, n, max_segsize=1*MiB):
+        p = {"k": k,
+             "happy": happy,
+             "n": n,
+             "max_segment_size": max_segsize,
+             }
+        self.node.DEFAULT_ENCODING_PARAMETERS = p
+
 class FakeStorageServer:
     def __init__(self, mode):
         self.mode = mode
hunk ./src/allmydata/test/test_upload.py 247
     u = upload.FileHandle(fh, convergence=None)
     return uploader.upload(u)
 
-class GoodServer(unittest.TestCase, ShouldFailMixin):
+class GoodServer(unittest.TestCase, ShouldFailMixin, SetDEPMixin):
     def setUp(self):
         self.node = FakeClient(mode="good")
         self.u = upload.Uploader()
hunk ./src/allmydata/test/test_upload.py 254
         self.u.running = True
         self.u.parent = self.node
 
-    def set_encoding_parameters(self, k, happy, n, max_segsize=1*MiB):
-        p = {"k": k,
-             "happy": happy,
-             "n": n,
-             "max_segment_size": max_segsize,
-             }
-        self.node.DEFAULT_ENCODING_PARAMETERS = p
-
     def _check_small(self, newuri, size):
         u = uri.from_string(newuri)
         self.failUnless(isinstance(u, uri.LiteralFileURI))
hunk ./src/allmydata/test/test_upload.py 377
         d.addCallback(self._check_large, SIZE_LARGE)
         return d
 
-class ServerErrors(unittest.TestCase, ShouldFailMixin):
+class ServerErrors(unittest.TestCase, ShouldFailMixin, SetDEPMixin):
     def make_node(self, mode, num_servers=10):
         self.node = FakeClient(mode, num_servers)
         self.u = upload.Uploader()
hunk ./src/allmydata/test/test_upload.py 677
         d.addCallback(_done)
         return d
 
-class EncodingParameters(GridTestMixin, unittest.TestCase):
+class EncodingParameters(GridTestMixin, unittest.TestCase, SetDEPMixin,
+    ShouldFailMixin):
+    def _do_upload_with_broken_servers(self, servers_to_break):
+        """
+        I act like a normal upload, but before I send the results of
+        Tahoe2PeerSelector to the Encoder, I break the first servers_to_break
+        PeerTrackers in the used_peers part of the return result.
+        """
+        assert self.g, "I tried to find a grid at self.g, but failed"
+        broker = self.g.clients[0].storage_broker
+        sh     = self.g.clients[0]._secret_holder
+        data = upload.Data("data" * 10000, convergence="")
+        data.encoding_param_k = 3
+        data.encoding_param_happy = 4
+        data.encoding_param_n = 10
+        uploadable = upload.EncryptAnUploadable(data)
+        encoder = encode.Encoder()
+        encoder.set_encrypted_uploadable(uploadable)
+        status = upload.UploadStatus()
+        selector = upload.Tahoe2PeerSelector("dglev", "test", status)
+        storage_index = encoder.get_param("storage_index")
+        share_size = encoder.get_param("share_size")
+        block_size = encoder.get_param("block_size")
+        num_segments = encoder.get_param("num_segments")
+        d = selector.get_shareholders(broker, sh, storage_index,
+                                      share_size, block_size, num_segments,
+                                      10, 4)
+        def _have_shareholders((used_peers, already_peers)):
+            assert servers_to_break <= len(used_peers)
+            for index in xrange(servers_to_break):
+                server = list(used_peers)[index]
+                for share in server.buckets.keys():
+                    server.buckets[share].abort()
+            buckets = {}
+            for peer in used_peers:
+                buckets.update(peer.buckets)
+            encoder.set_shareholders(buckets)
+            d = encoder.start()
+            return d
+        d.addCallback(_have_shareholders)
+        return d
+
+    def _add_server_with_share(self, server_number, share_number=None,
+                               readonly=False):
+        assert self.g, "I tried to find a grid at self.g, but failed"
+        assert self.shares, "I tried to find shares at self.shares, but failed"
+        ss = self.g.make_server(server_number, readonly)
+        self.g.add_server(server_number, ss)
+        if share_number:
+            # Copy share i from the directory associated with the first 
+            # storage server to the directory associated with this one.
+            old_share_location = self.shares[share_number][2]
+            new_share_location = os.path.join(ss.storedir, "shares")
+            si = uri.from_string(self.uri).get_storage_index()
+            new_share_location = os.path.join(new_share_location,
+                                              storage_index_to_dir(si))
+            if not os.path.exists(new_share_location):
+                os.makedirs(new_share_location)
+            new_share_location = os.path.join(new_share_location,
+                                              str(share_number))
+            shutil.copy(old_share_location, new_share_location)
+            shares = self.find_shares(self.uri)
+            # Make sure that the storage server has the share.
+            self.failUnless((share_number, ss.my_nodeid, new_share_location)
+                            in shares)
+
+    def _setup_and_upload(self):
+        """
+        I set up a NoNetworkGrid with a single server and client,
+        upload a file to it, store its uri in self.uri, and store its
+        sharedata in self.shares.
+        """
+        self.set_up_grid(num_clients=1, num_servers=1)
+        client = self.g.clients[0]
+        client.DEFAULT_ENCODING_PARAMETERS['happy'] = 1
+        data = upload.Data("data" * 10000, convergence="")
+        self.data = data
+        d = client.upload(data)
+        def _store_uri(ur):
+            self.uri = ur.uri
+        d.addCallback(_store_uri)
+        d.addCallback(lambda ign:
+            self.find_shares(self.uri))
+        def _store_shares(shares):
+            self.shares = shares
+        d.addCallback(_store_shares)
+        return d
+
     def test_configure_parameters(self):
         self.basedir = self.mktemp()
         hooks = {0: self._set_up_nodes_extra_config}
hunk ./src/allmydata/test/test_upload.py 784
         d.addCallback(_check)
         return d
 
+    def _setUp(self, ns):
+        # Used by test_happy_semantics and test_prexisting_share_behavior
+        # to set up the grid.
+        self.node = FakeClient(mode="good", num_servers=ns)
+        self.u = upload.Uploader()
+        self.u.running = True
+        self.u.parent = self.node
+
+    def test_happy_semantics(self):
+        self._setUp(2)
+        DATA = upload.Data("kittens" * 10000, convergence="")
+        # These parameters are unsatisfiable with the client that we've made
+        # -- we'll use them to test that the semnatics work correctly.
+        self.set_encoding_parameters(k=3, happy=5, n=10)
+        d = self.shouldFail(NotEnoughSharesError, "test_happy_semantics",
+                            "shares could only be placed on 2 servers "
+                            "(5 were requested)",
+                            self.u.upload, DATA)
+        # Let's reset the client to have 10 servers
+        d.addCallback(lambda ign:
+            self._setUp(10))
+        # These parameters are satisfiable with the client we've made.
+        d.addCallback(lambda ign:
+            self.set_encoding_parameters(k=3, happy=5, n=10))
+        # this should work
+        d.addCallback(lambda ign:
+            self.u.upload(DATA))
+        # Let's reset the client to have 7 servers
+        # (this is less than n, but more than h)
+        d.addCallback(lambda ign:
+            self._setUp(7))
+        # These encoding parameters should still be satisfiable with our 
+        # client setup
+        d.addCallback(lambda ign:
+            self.set_encoding_parameters(k=3, happy=5, n=10))
+        # This, then, should work.
+        d.addCallback(lambda ign:
+            self.u.upload(DATA))
+        return d
+
+    def test_problem_layouts(self):
+        self.basedir = self.mktemp()
+        # This scenario is at 
+        # http://allmydata.org/trac/tahoe/ticket/778#comment:52
+        #
+        # The scenario in comment:52 proposes that we have a layout
+        # like:
+        # server 1: share 1
+        # server 2: share 1
+        # server 3: share 1
+        # server 4: shares 2 - 10
+        # To get access to the shares, we will first upload to one 
+        # server, which will then have shares 1 - 10. We'll then 
+        # add three new servers, configure them to not accept any new
+        # shares, then write share 1 directly into the serverdir of each.
+        # Then each of servers 1 - 3 will report that they have share 1, 
+        # and will not accept any new share, while server 4 will report that
+        # it has shares 2 - 10 and will accept new shares.
+        # We'll then set 'happy' = 4, and see that an upload fails
+        # (as it should)
+        d = self._setup_and_upload()
+        d.addCallback(lambda ign:
+            self._add_server_with_share(1, 0, True))
+        d.addCallback(lambda ign:
+            self._add_server_with_share(2, 0, True))
+        d.addCallback(lambda ign:
+            self._add_server_with_share(3, 0, True))
+        # Remove the first share from server 0.
+        def _remove_share_0():
+            share_location = self.shares[0][2]
+            os.remove(share_location)
+        d.addCallback(lambda ign:
+            _remove_share_0())
+        # Set happy = 4 in the client.
+        def _prepare():
+            client = self.g.clients[0]
+            client.DEFAULT_ENCODING_PARAMETERS['happy'] = 4
+            return client
+        d.addCallback(lambda ign:
+            _prepare())
+        # Uploading data should fail
+        d.addCallback(lambda client:
+            self.shouldFail(NotEnoughSharesError, "test_happy_semantics",
+                            "shares could only be placed on 1 servers "
+                            "(4 were requested)",
+                            client.upload, upload.Data("data" * 10000,
+                                                       convergence="")))
+
+
+        # This scenario is at
+        # http://allmydata.org/trac/tahoe/ticket/778#comment:53
+        #
+        # Set up the grid to have one server
+        def _change_basedir(ign):
+            self.basedir = self.mktemp()
+        d.addCallback(_change_basedir)
+        d.addCallback(lambda ign:
+            self._setup_and_upload())
+        # We want to have a layout like this:
+        # server 1: share 1
+        # server 2: share 2
+        # server 3: share 3
+        # server 4: shares 1 - 10
+        # (this is an expansion of Zooko's example because it is easier
+        #  to code, but it will fail in the same way)
+        # To start, we'll create a server with shares 1-10 of the data 
+        # we're about to upload.
+        # Next, we'll add three new servers to our NoNetworkGrid. We'll add
+        # one share from our initial upload to each of these.
+        # The counterintuitive ordering of the share numbers is to deal with 
+        # the permuting of these servers -- distributing the shares this 
+        # way ensures that the Tahoe2PeerSelector sees them in the order 
+        # described above.
+        d.addCallback(lambda ign:
+            self._add_server_with_share(server_number=1, share_number=2))
+        d.addCallback(lambda ign:
+            self._add_server_with_share(server_number=2, share_number=0))
+        d.addCallback(lambda ign:
+            self._add_server_with_share(server_number=3, share_number=1))
+        # So, we now have the following layout:
+        # server 0: shares 1 - 10
+        # server 1: share 0
+        # server 2: share 1
+        # server 3: share 2
+        # We want to change the 'happy' parameter in the client to 4. 
+        # We then want to feed the upload process a list of peers that
+        # server 0 is at the front of, so we trigger Zooko's scenario.
+        # Ideally, a reupload of our original data should work.
+        def _reset_encoding_parameters(ign):
+            client = self.g.clients[0]
+            client.DEFAULT_ENCODING_PARAMETERS['happy'] = 4
+            return client
+        d.addCallback(_reset_encoding_parameters)
+        # We need this to get around the fact that the old Data 
+        # instance already has a happy parameter set.
+        d.addCallback(lambda client:
+            client.upload(upload.Data("data" * 10000, convergence="")))
+        return d
+
+
+    def test_dropped_servers_in_encoder(self):
+        def _set_basedir(ign=None):
+            self.basedir = self.mktemp()
+        _set_basedir()
+        d = self._setup_and_upload();
+        # Add 5 servers, with one share each from the original
+        # Add a readonly server
+        def _do_server_setup(ign):
+            self._add_server_with_share(1, 1, True)
+            self._add_server_with_share(2)
+            self._add_server_with_share(3)
+            self._add_server_with_share(4)
+            self._add_server_with_share(5)
+        d.addCallback(_do_server_setup)
+        # remove the original server
+        # (necessary to ensure that the Tahoe2PeerSelector will distribute
+        #  all the shares)
+        def _remove_server(ign):
+            server = self.g.servers_by_number[0]
+            self.g.remove_server(server.my_nodeid)
+        d.addCallback(_remove_server)
+        # This should succeed.
+        d.addCallback(lambda ign:
+            self._do_upload_with_broken_servers(1))
+        # Now, do the same thing over again, but drop 2 servers instead
+        # of 1. This should fail.
+        d.addCallback(_set_basedir)
+        d.addCallback(lambda ign:
+            self._setup_and_upload())
+        d.addCallback(_do_server_setup)
+        d.addCallback(_remove_server)
+        d.addCallback(lambda ign:
+            self.shouldFail(NotEnoughSharesError,
+                            "test_dropped_server_in_encoder", "",
+                            self._do_upload_with_broken_servers, 2))
+        return d
+
+
+    def test_servers_with_unique_shares(self):
+        # servers_with_unique_shares expects a dict of 
+        # shnum => peerid as a preexisting shares argument.
+        test1 = {
+                 1 : "server1",
+                 2 : "server2",
+                 3 : "server3",
+                 4 : "server4"
+                }
+        unique_servers = upload.servers_with_unique_shares(test1)
+        self.failUnlessEqual(4, len(unique_servers))
+        for server in ["server1", "server2", "server3", "server4"]:
+            self.failUnlessIn(server, unique_servers)
+        test1[4] = "server1"
+        # Now there should only be 3 unique servers.
+        unique_servers = upload.servers_with_unique_shares(test1)
+        self.failUnlessEqual(3, len(unique_servers))
+        for server in ["server1", "server2", "server3"]:
+            self.failUnlessIn(server, unique_servers)
+        # servers_with_unique_shares expects a set of PeerTracker
+        # instances as a used_peers argument, but only uses the peerid
+        # instance variable to assess uniqueness. So we feed it some fake
+        # PeerTrackers whose only important characteristic is that they 
+        # have peerid set to something.
+        class FakePeerTracker:
+            pass
+        trackers = []
+        for server in ["server5", "server6", "server7", "server8"]:
+            t = FakePeerTracker()
+            t.peerid = server
+            trackers.append(t)
+        # Recall that there are 3 unique servers in test1. Since none of
+        # those overlap with the ones in trackers, we should get 7 back
+        unique_servers = upload.servers_with_unique_shares(test1, set(trackers))
+        self.failUnlessEqual(7, len(unique_servers))
+        expected_servers = ["server" + str(i) for i in xrange(1, 9)]
+        expected_servers.remove("server4")
+        for server in expected_servers:
+            self.failUnlessIn(server, unique_servers)
+        # Now add an overlapping server to trackers.
+        t = FakePeerTracker()
+        t.peerid = "server1"
+        trackers.append(t)
+        unique_servers = upload.servers_with_unique_shares(test1, set(trackers))
+        self.failUnlessEqual(7, len(unique_servers))
+        for server in expected_servers:
+            self.failUnlessIn(server, unique_servers)
+
+
     def _set_up_nodes_extra_config(self, clientdir):
         cfgfn = os.path.join(clientdir, "tahoe.cfg")
         oldcfg = open(cfgfn, "r").read()
}
[Alter tests to use the new form of set_shareholders
Kevan Carstensen <kevan@isnotajoke.com>**20091104033602
 Ignore-this: 3deac11fc831618d11441317463ef830
] {
hunk ./src/allmydata/test/test_encode.py 301
                      (NUM_SEGMENTS-1)*segsize, len(data), NUM_SEGMENTS*segsize)
 
             shareholders = {}
+            servermap = {}
             for shnum in range(NUM_SHARES):
                 peer = FakeBucketReaderWriterProxy()
                 shareholders[shnum] = peer
hunk ./src/allmydata/test/test_encode.py 305
+                servermap[shnum] = str(shnum)
                 all_shareholders.append(peer)
hunk ./src/allmydata/test/test_encode.py 307
-            e.set_shareholders(shareholders)
+            e.set_shareholders(shareholders, servermap)
             return e.start()
         d.addCallback(_ready)
 
merger 0.0 (
hunk ./src/allmydata/test/test_encode.py 462
-            all_peers = []
hunk ./src/allmydata/test/test_encode.py 463
+            servermap = {}
)
hunk ./src/allmydata/test/test_encode.py 467
                 mode = bucket_modes.get(shnum, "good")
                 peer = FakeBucketReaderWriterProxy(mode)
                 shareholders[shnum] = peer
-            e.set_shareholders(shareholders)
+                servermap[shnum] = str(shnum)
+            e.set_shareholders(shareholders, servermap)
             return e.start()
         d.addCallback(_ready)
         def _sent(res):
hunk ./src/allmydata/test/test_upload.py 711
                 for share in server.buckets.keys():
                     server.buckets[share].abort()
             buckets = {}
+            servermap = already_peers.copy()
             for peer in used_peers:
                 buckets.update(peer.buckets)
hunk ./src/allmydata/test/test_upload.py 714
-            encoder.set_shareholders(buckets)
+                for bucket in peer.buckets:
+                    servermap[bucket] = peer.peerid
+            encoder.set_shareholders(buckets, servermap)
             d = encoder.start()
             return d
         d.addCallback(_have_shareholders)
hunk ./src/allmydata/test/test_upload.py 933
         _set_basedir()
         d = self._setup_and_upload();
         # Add 5 servers, with one share each from the original
-        # Add a readonly server
         def _do_server_setup(ign):
             self._add_server_with_share(1, 1, True)
             self._add_server_with_share(2)
}
[Minor tweak to an existing test -- make the first server read-write, instead of read-only
Kevan Carstensen <kevan@isnotajoke.com>**20091104034232
 Ignore-this: a951a46c93f7f58dd44d93d8623b2aee
] hunk ./src/allmydata/test/test_upload.py 934
         d = self._setup_and_upload();
         # Add 5 servers, with one share each from the original
         def _do_server_setup(ign):
-            self._add_server_with_share(1, 1, True)
+            self._add_server_with_share(1, 1)
             self._add_server_with_share(2)
             self._add_server_with_share(3)
             self._add_server_with_share(4)
[Add a test for upload.shares_by_server
Kevan Carstensen <kevan@isnotajoke.com>**20091104111324
 Ignore-this: f9802e82d6982a93e00f92e0b276f018
] hunk ./src/allmydata/test/test_upload.py 1013
             self.failUnlessIn(server, unique_servers)
 
 
+    def test_shares_by_server(self):
+        test = {
+                    1 : "server1",
+                    2 : "server2",
+                    3 : "server3",
+                    4 : "server4"
+               }
+        shares_by_server = upload.shares_by_server(test)
+        self.failUnlessEqual(set([1]), shares_by_server["server1"])
+        self.failUnlessEqual(set([2]), shares_by_server["server2"])
+        self.failUnlessEqual(set([3]), shares_by_server["server3"])
+        self.failUnlessEqual(set([4]), shares_by_server["server4"])
+        test1 = {
+                    1 : "server1",
+                    2 : "server1",
+                    3 : "server1",
+                    4 : "server2",
+                    5 : "server2"
+                }
+        shares_by_server = upload.shares_by_server(test1)
+        self.failUnlessEqual(set([1, 2, 3]), shares_by_server["server1"])
+        self.failUnlessEqual(set([4, 5]), shares_by_server["server2"])
+
+
     def _set_up_nodes_extra_config(self, clientdir):
         cfgfn = os.path.join(clientdir, "tahoe.cfg")
         oldcfg = open(cfgfn, "r").read()
[Add more tests for comment:53 in ticket #778
Kevan Carstensen <kevan@isnotajoke.com>**20091104112849
 Ignore-this: 3bb2edd299a944cc9586e14d5d83ec8c
] {
hunk ./src/allmydata/test/test_upload.py 722
         d.addCallback(_have_shareholders)
         return d
 
-    def _add_server_with_share(self, server_number, share_number=None,
-                               readonly=False):
+    def _add_server(self, server_number, readonly=False):
         assert self.g, "I tried to find a grid at self.g, but failed"
         assert self.shares, "I tried to find shares at self.shares, but failed"
         ss = self.g.make_server(server_number, readonly)
hunk ./src/allmydata/test/test_upload.py 727
         self.g.add_server(server_number, ss)
+
+    def _add_server_with_share(self, server_number, share_number=None,
+                               readonly=False):
+        self._add_server(server_number, readonly)
         if share_number:
hunk ./src/allmydata/test/test_upload.py 732
-            # Copy share i from the directory associated with the first 
-            # storage server to the directory associated with this one.
-            old_share_location = self.shares[share_number][2]
-            new_share_location = os.path.join(ss.storedir, "shares")
-            si = uri.from_string(self.uri).get_storage_index()
-            new_share_location = os.path.join(new_share_location,
-                                              storage_index_to_dir(si))
-            if not os.path.exists(new_share_location):
-                os.makedirs(new_share_location)
-            new_share_location = os.path.join(new_share_location,
-                                              str(share_number))
-            shutil.copy(old_share_location, new_share_location)
-            shares = self.find_shares(self.uri)
-            # Make sure that the storage server has the share.
-            self.failUnless((share_number, ss.my_nodeid, new_share_location)
-                            in shares)
+            self._copy_share_to_server(share_number, server_number)
+
+    def _copy_share_to_server(self, share_number, server_number):
+        ss = self.g.servers_by_number[server_number]
+        # Copy share i from the directory associated with the first 
+        # storage server to the directory associated with this one.
+        assert self.g, "I tried to find a grid at self.g, but failed"
+        assert self.shares, "I tried to find shares at self.shares, but failed"
+        old_share_location = self.shares[share_number][2]
+        new_share_location = os.path.join(ss.storedir, "shares")
+        si = uri.from_string(self.uri).get_storage_index()
+        new_share_location = os.path.join(new_share_location,
+                                          storage_index_to_dir(si))
+        if not os.path.exists(new_share_location):
+            os.makedirs(new_share_location)
+        new_share_location = os.path.join(new_share_location,
+                                          str(share_number))
+        shutil.copy(old_share_location, new_share_location)
+        shares = self.find_shares(self.uri)
+        # Make sure that the storage server has the share.
+        self.failUnless((share_number, ss.my_nodeid, new_share_location)
+                        in shares)
+
 
     def _setup_and_upload(self):
         """
hunk ./src/allmydata/test/test_upload.py 917
         d.addCallback(lambda ign:
             self._add_server_with_share(server_number=3, share_number=1))
         # So, we now have the following layout:
-        # server 0: shares 1 - 10
+        # server 0: shares 0 - 9
         # server 1: share 0
         # server 2: share 1
         # server 3: share 2
hunk ./src/allmydata/test/test_upload.py 934
         # instance already has a happy parameter set.
         d.addCallback(lambda client:
             client.upload(upload.Data("data" * 10000, convergence="")))
+
+
+        # This scenario is basically comment:53, but with the order reversed;
+        # this means that the Tahoe2PeerSelector sees
+        # server 0: shares 1-10
+        # server 1: share 1
+        # server 2: share 2
+        # server 3: share 3
+        d.addCallback(_change_basedir)
+        d.addCallback(lambda ign:
+            self._setup_and_upload())
+        d.addCallback(lambda ign:
+            self._add_server_with_share(server_number=2, share_number=0))
+        d.addCallback(lambda ign:
+            self._add_server_with_share(server_number=3, share_number=1))
+        d.addCallback(lambda ign:
+            self._add_server_with_share(server_number=1, share_number=2))
+        # Copy all of the other shares to server number 2
+        def _copy_shares(ign):
+            for i in xrange(1, 10):
+                self._copy_share_to_server(i, 2)
+        d.addCallback(_copy_shares)
+        # Remove the first server, and add a placeholder with share 0
+        d.addCallback(lambda ign:
+            self.g.remove_server(self.g.servers_by_number[0].my_nodeid))
+        d.addCallback(lambda ign:
+            self._add_server_with_share(server_number=0, share_number=0))
+        # Now try uploading. 
+        d.addCallback(_reset_encoding_parameters)
+        d.addCallback(lambda client:
+            client.upload(upload.Data("data" * 10000, convergence="")))
+        # Try the same thing, but with empty servers after the first one
+        # We want to make sure that Tahoe2PeerSelector will redistribute
+        # shares as necessary, not simply discover an existing layout.
+        d.addCallback(_change_basedir)
+        d.addCallback(lambda ign:
+            self._setup_and_upload())
+        d.addCallback(lambda ign:
+            self._add_server(server_number=2))
+        d.addCallback(lambda ign:
+            self._add_server(server_number=3))
+        d.addCallback(lambda ign:
+            self._add_server(server_number=1))
+        d.addCallback(_copy_shares)
+        d.addCallback(lambda ign:
+            self.g.remove_server(self.g.servers_by_number[0].my_nodeid))
+        d.addCallback(lambda ign:
+            self._add_server(server_number=0))
+        d.addCallback(_reset_encoding_parameters)
+        d.addCallback(lambda client:
+            client.upload(upload.Data("data" * 10000, convergence="")))
+        # Try the following layout
+        # server 0: shares 1-10
+        # server 1: share 1, read-only
+        # server 2: share 2, read-only
+        # server 3: share 3, read-only
+        d.addCallback(_change_basedir)
+        d.addCallback(lambda ign:
+            self._setup_and_upload())
+        d.addCallback(lambda ign:
+            self._add_server_with_share(server_number=2, share_number=0))
+        d.addCallback(lambda ign:
+            self._add_server_with_share(server_number=3, share_number=1,
+                                        readonly=True))
+        d.addCallback(lambda ign:
+            self._add_server_with_share(server_number=1, share_number=2,
+                                        readonly=True))
+        # Copy all of the other shares to server number 2
+        d.addCallback(_copy_shares)
+        # Remove server 0, and add another in its place
+        d.addCallback(lambda ign:
+            self.g.remove_server(self.g.servers_by_number[0].my_nodeid))
+        d.addCallback(lambda ign:
+            self._add_server_with_share(server_number=0, share_number=0,
+                                        readonly=True))
+        d.addCallback(_reset_encoding_parameters)
+        d.addCallback(lambda client:
+            client.upload(upload.Data("data" * 10000, convergence="")))
         return d
 
 
}
[Test Tahoe2PeerSelector to make sure that it recognizeses existing shares on readonly servers
Kevan Carstensen <kevan@isnotajoke.com>**20091109003735
 Ignore-this: 12f9b4cff5752fca7ed32a6ebcff6446
] hunk ./src/allmydata/test/test_upload.py 1125
         self.failUnlessEqual(set([4, 5]), shares_by_server["server2"])
 
 
+    def test_existing_share_detection(self):
+        self.basedir = self.mktemp()
+        d = self._setup_and_upload()
+        # Our final setup should look like this:
+        # server 1: shares 1 - 10, read-only
+        # server 2: empty
+        # server 3: empty
+        # server 4: empty
+        # The purpose of this test is to make sure that the peer selector
+        # knows about the shares on server 1, even though it is read-only.
+        # It used to simply filter these out, which would cause the test
+        # to fail when servers_of_happiness = 4.
+        d.addCallback(lambda ign:
+            self._add_server_with_share(1, 0, True))
+        d.addCallback(lambda ign:
+            self._add_server_with_share(2))
+        d.addCallback(lambda ign:
+            self._add_server_with_share(3))
+        d.addCallback(lambda ign:
+            self._add_server_with_share(4))
+        def _copy_shares(ign):
+            for i in xrange(1, 10):
+                self._copy_share_to_server(i, 1)
+        d.addCallback(_copy_shares)
+        d.addCallback(lambda ign:
+            self.g.remove_server(self.g.servers_by_number[0].my_nodeid))
+        def _prepare_client(ign):
+            client = self.g.clients[0]
+            client.DEFAULT_ENCODING_PARAMETERS['happy'] = 4
+            return client
+        d.addCallback(_prepare_client)
+        d.addCallback(lambda client:
+            client.upload(upload.Data("data" * 10000, convergence="")))
+        return d
+
+
     def _set_up_nodes_extra_config(self, clientdir):
         cfgfn = os.path.join(clientdir, "tahoe.cfg")
         oldcfg = open(cfgfn, "r").read()
[Re-work 'test_upload.py' to be more readable; add more tests for #778
Kevan Carstensen <kevan@isnotajoke.com>**20091116192334
 Ignore-this: 7e8565f92fe51dece5ae28daf442d659
] {
hunk ./src/allmydata/test/test_upload.py 722
         d.addCallback(_have_shareholders)
         return d
 
+
     def _add_server(self, server_number, readonly=False):
         assert self.g, "I tried to find a grid at self.g, but failed"
         assert self.shares, "I tried to find shares at self.shares, but failed"
hunk ./src/allmydata/test/test_upload.py 729
         ss = self.g.make_server(server_number, readonly)
         self.g.add_server(server_number, ss)
 
+
     def _add_server_with_share(self, server_number, share_number=None,
                                readonly=False):
         self._add_server(server_number, readonly)
hunk ./src/allmydata/test/test_upload.py 733
-        if share_number:
+        if share_number is not None:
             self._copy_share_to_server(share_number, server_number)
 
hunk ./src/allmydata/test/test_upload.py 736
+
     def _copy_share_to_server(self, share_number, server_number):
         ss = self.g.servers_by_number[server_number]
         # Copy share i from the directory associated with the first 
hunk ./src/allmydata/test/test_upload.py 752
             os.makedirs(new_share_location)
         new_share_location = os.path.join(new_share_location,
                                           str(share_number))
-        shutil.copy(old_share_location, new_share_location)
+        if old_share_location != new_share_location:
+            shutil.copy(old_share_location, new_share_location)
         shares = self.find_shares(self.uri)
         # Make sure that the storage server has the share.
         self.failUnless((share_number, ss.my_nodeid, new_share_location)
hunk ./src/allmydata/test/test_upload.py 782
         d.addCallback(_store_shares)
         return d
 
+
     def test_configure_parameters(self):
         self.basedir = self.mktemp()
         hooks = {0: self._set_up_nodes_extra_config}
hunk ./src/allmydata/test/test_upload.py 802
         d.addCallback(_check)
         return d
 
+
     def _setUp(self, ns):
         # Used by test_happy_semantics and test_prexisting_share_behavior
         # to set up the grid.
hunk ./src/allmydata/test/test_upload.py 811
         self.u.running = True
         self.u.parent = self.node
 
+
     def test_happy_semantics(self):
         self._setUp(2)
         DATA = upload.Data("kittens" * 10000, convergence="")
hunk ./src/allmydata/test/test_upload.py 844
             self.u.upload(DATA))
         return d
 
-    def test_problem_layouts(self):
-        self.basedir = self.mktemp()
+
+    def test_problem_layout_comment_52(self):
+        def _basedir():
+            self.basedir = self.mktemp()
+        _basedir()
         # This scenario is at 
         # http://allmydata.org/trac/tahoe/ticket/778#comment:52
         #
hunk ./src/allmydata/test/test_upload.py 890
         # Uploading data should fail
         d.addCallback(lambda client:
             self.shouldFail(NotEnoughSharesError, "test_happy_semantics",
-                            "shares could only be placed on 1 servers "
+                            "shares could only be placed on 2 servers "
                             "(4 were requested)",
                             client.upload, upload.Data("data" * 10000,
                                                        convergence="")))
hunk ./src/allmydata/test/test_upload.py 895
 
+        # Do comment:52, but like this:
+        # server 2: empty
+        # server 3: share 0, read-only
+        # server 1: share 0, read-only
+        # server 0: shares 0-9
+        d.addCallback(lambda ign:
+            _basedir())
+        d.addCallback(lambda ign:
+            self._setup_and_upload())
+        d.addCallback(lambda ign:
+            self._add_server_with_share(server_number=2))
+        d.addCallback(lambda ign:
+            self._add_server_with_share(server_number=3, share_number=0,
+                                        readonly=True))
+        d.addCallback(lambda ign:
+            self._add_server_with_share(server_number=1, share_number=0,
+                                        readonly=True))
+        def _prepare2():
+            client = self.g.clients[0]
+            client.DEFAULT_ENCODING_PARAMETERS['happy'] = 3
+            return client
+        d.addCallback(lambda ign:
+            _prepare2())
+        d.addCallback(lambda client:
+            self.shouldFail(NotEnoughSharesError, "test_happy_sematics",
+                            "shares could only be placed on 2 servers "
+                            "(3 were requested)",
+                            client.upload, upload.Data("data" * 10000,
+                                                       convergence="")))
+        return d
+
 
hunk ./src/allmydata/test/test_upload.py 927
+    def test_problem_layout_comment_53(self):
         # This scenario is at
         # http://allmydata.org/trac/tahoe/ticket/778#comment:53
         #
hunk ./src/allmydata/test/test_upload.py 934
         # Set up the grid to have one server
         def _change_basedir(ign):
             self.basedir = self.mktemp()
-        d.addCallback(_change_basedir)
-        d.addCallback(lambda ign:
-            self._setup_and_upload())
-        # We want to have a layout like this:
-        # server 1: share 1
-        # server 2: share 2
-        # server 3: share 3
-        # server 4: shares 1 - 10
-        # (this is an expansion of Zooko's example because it is easier
-        #  to code, but it will fail in the same way)
-        # To start, we'll create a server with shares 1-10 of the data 
-        # we're about to upload.
+        _change_basedir(None)
+        d = self._setup_and_upload()
+        # We start by uploading all of the shares to one server (which has 
+        # already been done above).
         # Next, we'll add three new servers to our NoNetworkGrid. We'll add
         # one share from our initial upload to each of these.
         # The counterintuitive ordering of the share numbers is to deal with 
hunk ./src/allmydata/test/test_upload.py 952
             self._add_server_with_share(server_number=3, share_number=1))
         # So, we now have the following layout:
         # server 0: shares 0 - 9
-        # server 1: share 0
-        # server 2: share 1
-        # server 3: share 2
+        # server 1: share 2
+        # server 2: share 0
+        # server 3: share 1
         # We want to change the 'happy' parameter in the client to 4. 
hunk ./src/allmydata/test/test_upload.py 956
-        # We then want to feed the upload process a list of peers that
-        # server 0 is at the front of, so we trigger Zooko's scenario.
+        # The Tahoe2PeerSelector will see the peers permuted as:
+        # 2, 3, 1, 0
         # Ideally, a reupload of our original data should work.
hunk ./src/allmydata/test/test_upload.py 959
-        def _reset_encoding_parameters(ign):
+        def _reset_encoding_parameters(ign, happy=4):
             client = self.g.clients[0]
hunk ./src/allmydata/test/test_upload.py 961
-            client.DEFAULT_ENCODING_PARAMETERS['happy'] = 4
+            client.DEFAULT_ENCODING_PARAMETERS['happy'] = happy
             return client
         d.addCallback(_reset_encoding_parameters)
hunk ./src/allmydata/test/test_upload.py 964
-        # We need this to get around the fact that the old Data 
-        # instance already has a happy parameter set.
         d.addCallback(lambda client:
             client.upload(upload.Data("data" * 10000, convergence="")))
 
hunk ./src/allmydata/test/test_upload.py 970
 
         # This scenario is basically comment:53, but with the order reversed;
         # this means that the Tahoe2PeerSelector sees
-        # server 0: shares 1-10
-        # server 1: share 1
-        # server 2: share 2
-        # server 3: share 3
+        # server 2: shares 1-10
+        # server 3: share 1
+        # server 1: share 2
+        # server 4: share 3
         d.addCallback(_change_basedir)
         d.addCallback(lambda ign:
             self._setup_and_upload())
hunk ./src/allmydata/test/test_upload.py 992
         d.addCallback(lambda ign:
             self.g.remove_server(self.g.servers_by_number[0].my_nodeid))
         d.addCallback(lambda ign:
-            self._add_server_with_share(server_number=0, share_number=0))
+            self._add_server_with_share(server_number=4, share_number=0))
         # Now try uploading. 
         d.addCallback(_reset_encoding_parameters)
         d.addCallback(lambda client:
hunk ./src/allmydata/test/test_upload.py 1013
         d.addCallback(lambda ign:
             self.g.remove_server(self.g.servers_by_number[0].my_nodeid))
         d.addCallback(lambda ign:
-            self._add_server(server_number=0))
+            self._add_server(server_number=4))
         d.addCallback(_reset_encoding_parameters)
         d.addCallback(lambda client:
             client.upload(upload.Data("data" * 10000, convergence="")))
hunk ./src/allmydata/test/test_upload.py 1017
+        return d
+
+
+    def test_happiness_with_some_readonly_peers(self):
         # Try the following layout
hunk ./src/allmydata/test/test_upload.py 1022
-        # server 0: shares 1-10
-        # server 1: share 1, read-only
-        # server 2: share 2, read-only
-        # server 3: share 3, read-only
-        d.addCallback(_change_basedir)
-        d.addCallback(lambda ign:
-            self._setup_and_upload())
+        # server 2: shares 0-9
+        # server 4: share 0, read-only
+        # server 3: share 1, read-only
+        # server 1: share 2, read-only
+        self.basedir = self.mktemp()
+        d = self._setup_and_upload()
         d.addCallback(lambda ign:
             self._add_server_with_share(server_number=2, share_number=0))
         d.addCallback(lambda ign:
hunk ./src/allmydata/test/test_upload.py 1037
             self._add_server_with_share(server_number=1, share_number=2,
                                         readonly=True))
         # Copy all of the other shares to server number 2
+        def _copy_shares(ign):
+            for i in xrange(1, 10):
+                self._copy_share_to_server(i, 2)
         d.addCallback(_copy_shares)
         # Remove server 0, and add another in its place
         d.addCallback(lambda ign:
hunk ./src/allmydata/test/test_upload.py 1045
             self.g.remove_server(self.g.servers_by_number[0].my_nodeid))
         d.addCallback(lambda ign:
-            self._add_server_with_share(server_number=0, share_number=0,
+            self._add_server_with_share(server_number=4, share_number=0,
                                         readonly=True))
hunk ./src/allmydata/test/test_upload.py 1047
+        def _reset_encoding_parameters(ign, happy=4):
+            client = self.g.clients[0]
+            client.DEFAULT_ENCODING_PARAMETERS['happy'] = happy
+            return client
+        d.addCallback(_reset_encoding_parameters)
+        d.addCallback(lambda client:
+            client.upload(upload.Data("data" * 10000, convergence="")))
+        return d
+
+
+    def test_happiness_with_all_readonly_peers(self):
+        # server 3: share 1, read-only
+        # server 1: share 2, read-only
+        # server 2: shares 0-9, read-only
+        # server 4: share 0, read-only
+        # The idea with this test is to make sure that the survey of
+        # read-only peers doesn't undercount servers of happiness
+        self.basedir = self.mktemp()
+        d = self._setup_and_upload()
+        d.addCallback(lambda ign:
+            self._add_server_with_share(server_number=4, share_number=0,
+                                        readonly=True))
+        d.addCallback(lambda ign:
+            self._add_server_with_share(server_number=3, share_number=1,
+                                        readonly=True))
+        d.addCallback(lambda ign:
+            self._add_server_with_share(server_number=1, share_number=2,
+                                        readonly=True))
+        d.addCallback(lambda ign:
+            self._add_server_with_share(server_number=2, share_number=0,
+                                        readonly=True))
+        def _copy_shares(ign):
+            for i in xrange(1, 10):
+                self._copy_share_to_server(i, 2)
+        d.addCallback(_copy_shares)
+        d.addCallback(lambda ign:
+            self.g.remove_server(self.g.servers_by_number[0].my_nodeid))
+        def _reset_encoding_parameters(ign, happy=4):
+            client = self.g.clients[0]
+            client.DEFAULT_ENCODING_PARAMETERS['happy'] = happy
+            return client
         d.addCallback(_reset_encoding_parameters)
         d.addCallback(lambda client:
             client.upload(upload.Data("data" * 10000, convergence="")))
hunk ./src/allmydata/test/test_upload.py 1099
             self.basedir = self.mktemp()
         _set_basedir()
         d = self._setup_and_upload();
-        # Add 5 servers, with one share each from the original
+        # Add 5 servers
         def _do_server_setup(ign):
hunk ./src/allmydata/test/test_upload.py 1101
-            self._add_server_with_share(1, 1)
+            self._add_server_with_share(1)
             self._add_server_with_share(2)
             self._add_server_with_share(3)
             self._add_server_with_share(4)
hunk ./src/allmydata/test/test_upload.py 1126
         d.addCallback(_remove_server)
         d.addCallback(lambda ign:
             self.shouldFail(NotEnoughSharesError,
-                            "test_dropped_server_in_encoder", "",
+                            "test_dropped_servers_in_encoder",
+                            "lost too many servers during upload "
+                            "(still have 3, want 4)",
+                            self._do_upload_with_broken_servers, 2))
+        # Now do the same thing over again, but make some of the servers
+        # readonly, break some of the ones that aren't, and make sure that
+        # happiness accounting is preserved.
+        d.addCallback(_set_basedir)
+        d.addCallback(lambda ign:
+            self._setup_and_upload())
+        def _do_server_setup_2(ign):
+            self._add_server_with_share(1)
+            self._add_server_with_share(2)
+            self._add_server_with_share(3)
+            self._add_server_with_share(4, 7, readonly=True)
+            self._add_server_with_share(5, 8, readonly=True)
+        d.addCallback(_do_server_setup_2)
+        d.addCallback(_remove_server)
+        d.addCallback(lambda ign:
+            self._do_upload_with_broken_servers(1))
+        d.addCallback(_set_basedir)
+        d.addCallback(lambda ign:
+            self._setup_and_upload())
+        d.addCallback(_do_server_setup_2)
+        d.addCallback(_remove_server)
+        d.addCallback(lambda ign:
+            self.shouldFail(NotEnoughSharesError,
+                            "test_dropped_servers_in_encoder",
+                            "lost too many servers during upload "
+                            "(still have 3, want 4)",
                             self._do_upload_with_broken_servers, 2))
         return d
 
hunk ./src/allmydata/test/test_upload.py 1179
         self.failUnlessEqual(3, len(unique_servers))
         for server in ["server1", "server2", "server3"]:
             self.failUnlessIn(server, unique_servers)
-        # servers_with_unique_shares expects a set of PeerTracker
-        # instances as a used_peers argument, but only uses the peerid
-        # instance variable to assess uniqueness. So we feed it some fake
-        # PeerTrackers whose only important characteristic is that they 
-        # have peerid set to something.
+        # servers_with_unique_shares expects to receive some object with
+        # a peerid attribute. So we make a FakePeerTracker whose only
+        # job is to have a peerid attribute.
         class FakePeerTracker:
             pass
         trackers = []
hunk ./src/allmydata/test/test_upload.py 1185
-        for server in ["server5", "server6", "server7", "server8"]:
+        for (i, server) in [(i, "server%d" % i) for i in xrange(5, 9)]:
             t = FakePeerTracker()
             t.peerid = server
hunk ./src/allmydata/test/test_upload.py 1188
+            t.buckets = [i]
             trackers.append(t)
         # Recall that there are 3 unique servers in test1. Since none of
         # those overlap with the ones in trackers, we should get 7 back
hunk ./src/allmydata/test/test_upload.py 1201
         # Now add an overlapping server to trackers.
         t = FakePeerTracker()
         t.peerid = "server1"
+        t.buckets = [1]
         trackers.append(t)
         unique_servers = upload.servers_with_unique_shares(test1, set(trackers))
         self.failUnlessEqual(7, len(unique_servers))
hunk ./src/allmydata/test/test_upload.py 1207
         for server in expected_servers:
             self.failUnlessIn(server, unique_servers)
+        test = {}
+        unique_servers = upload.servers_with_unique_shares(test)
+        self.failUnlessEqual(0, len(test))
 
 
     def test_shares_by_server(self):
hunk ./src/allmydata/test/test_upload.py 1213
-        test = {
-                    1 : "server1",
-                    2 : "server2",
-                    3 : "server3",
-                    4 : "server4"
-               }
+        test = dict([(i, "server%d" % i) for i in xrange(1, 5)])
         shares_by_server = upload.shares_by_server(test)
         self.failUnlessEqual(set([1]), shares_by_server["server1"])
         self.failUnlessEqual(set([2]), shares_by_server["server2"])
hunk ./src/allmydata/test/test_upload.py 1267
         return d
 
 
+    def test_should_add_server(self):
+        shares = dict([(i, "server%d" % i) for i in xrange(10)])
+        self.failIf(upload.should_add_server(shares, "server1", 4))
+        shares[4] = "server1"
+        self.failUnless(upload.should_add_server(shares, "server4", 4))
+        shares = {}
+        self.failUnless(upload.should_add_server(shares, "server1", 1))
+
+
     def _set_up_nodes_extra_config(self, clientdir):
         cfgfn = os.path.join(clientdir, "tahoe.cfg")
         oldcfg = open(cfgfn, "r").read()
}
[Add tests for the behavior described in #834.
Kevan Carstensen <kevan@isnotajoke.com>**20091123012008
 Ignore-this: d8e0aa0f3f7965ce9b5cea843c6d6f9f
] {
hunk ./src/allmydata/test/test_encode.py 12
 from allmydata.util.assertutil import _assert
 from allmydata.util.consumer import MemoryConsumer
 from allmydata.interfaces import IStorageBucketWriter, IStorageBucketReader, \
-     NotEnoughSharesError, IStorageBroker
+     NotEnoughSharesError, IStorageBroker, UploadHappinessError
 from allmydata.monitor import Monitor
 import common_util as testutil
 
hunk ./src/allmydata/test/test_encode.py 794
         d = self.send_and_recover((4,8,10), bucket_modes=modemap)
         def _done(res):
             self.failUnless(isinstance(res, Failure))
-            self.failUnless(res.check(NotEnoughSharesError), res)
+            self.failUnless(res.check(UploadHappinessError), res)
         d.addBoth(_done)
         return d
 
hunk ./src/allmydata/test/test_encode.py 805
         d = self.send_and_recover((4,8,10), bucket_modes=modemap)
         def _done(res):
             self.failUnless(isinstance(res, Failure))
-            self.failUnless(res.check(NotEnoughSharesError))
+            self.failUnless(res.check(UploadHappinessError))
         d.addBoth(_done)
         return d
hunk ./src/allmydata/test/test_upload.py 13
 import allmydata # for __full_version__
 from allmydata import uri, monitor, client
 from allmydata.immutable import upload, encode
-from allmydata.interfaces import FileTooLargeError, NoSharesError, \
-     NotEnoughSharesError
+from allmydata.interfaces import FileTooLargeError, UploadHappinessError
 from allmydata.util.assertutil import precondition
 from allmydata.util.deferredutil import DeferredListShouldSucceed
 from no_network import GridTestMixin
hunk ./src/allmydata/test/test_upload.py 402
 
     def test_first_error_all(self):
         self.make_node("first-fail")
-        d = self.shouldFail(NoSharesError, "first_error_all",
+        d = self.shouldFail(UploadHappinessError, "first_error_all",
                             "peer selection failed",
                             upload_data, self.u, DATA)
         def _check((f,)):
hunk ./src/allmydata/test/test_upload.py 434
 
     def test_second_error_all(self):
         self.make_node("second-fail")
-        d = self.shouldFail(NotEnoughSharesError, "second_error_all",
+        d = self.shouldFail(UploadHappinessError, "second_error_all",
                             "peer selection failed",
                             upload_data, self.u, DATA)
         def _check((f,)):
hunk ./src/allmydata/test/test_upload.py 452
         self.u.parent = self.node
 
     def _should_fail(self, f):
-        self.failUnless(isinstance(f, Failure) and f.check(NoSharesError), f)
+        self.failUnless(isinstance(f, Failure) and f.check(UploadHappinessError), f)
 
     def test_data_large(self):
         data = DATA
hunk ./src/allmydata/test/test_upload.py 817
         # These parameters are unsatisfiable with the client that we've made
         # -- we'll use them to test that the semnatics work correctly.
         self.set_encoding_parameters(k=3, happy=5, n=10)
-        d = self.shouldFail(NotEnoughSharesError, "test_happy_semantics",
+        d = self.shouldFail(UploadHappinessError, "test_happy_semantics",
                             "shares could only be placed on 2 servers "
                             "(5 were requested)",
                             self.u.upload, DATA)
hunk ./src/allmydata/test/test_upload.py 888
             _prepare())
         # Uploading data should fail
         d.addCallback(lambda client:
-            self.shouldFail(NotEnoughSharesError, "test_happy_semantics",
+            self.shouldFail(UploadHappinessError, "test_happy_semantics",
                             "shares could only be placed on 2 servers "
                             "(4 were requested)",
                             client.upload, upload.Data("data" * 10000,
hunk ./src/allmydata/test/test_upload.py 918
         d.addCallback(lambda ign:
             _prepare2())
         d.addCallback(lambda client:
-            self.shouldFail(NotEnoughSharesError, "test_happy_sematics",
+            self.shouldFail(UploadHappinessError, "test_happy_sematics",
                             "shares could only be placed on 2 servers "
                             "(3 were requested)",
                             client.upload, upload.Data("data" * 10000,
hunk ./src/allmydata/test/test_upload.py 1124
         d.addCallback(_do_server_setup)
         d.addCallback(_remove_server)
         d.addCallback(lambda ign:
-            self.shouldFail(NotEnoughSharesError,
+            self.shouldFail(UploadHappinessError,
                             "test_dropped_servers_in_encoder",
                             "lost too many servers during upload "
                             "(still have 3, want 4)",
hunk ./src/allmydata/test/test_upload.py 1151
         d.addCallback(_do_server_setup_2)
         d.addCallback(_remove_server)
         d.addCallback(lambda ign:
-            self.shouldFail(NotEnoughSharesError,
+            self.shouldFail(UploadHappinessError,
                             "test_dropped_servers_in_encoder",
                             "lost too many servers during upload "
                             "(still have 3, want 4)",
hunk ./src/allmydata/test/test_upload.py 1275
         self.failUnless(upload.should_add_server(shares, "server1", 1))
 
 
+    def test_exception_messages_during_peer_selection(self):
+        # server 1: readonly, no shares
+        # server 2: readonly, no shares
+        # server 3: readonly, no shares
+        # server 4: readonly, no shares
+        # server 5: readonly, no shares
+        # This will fail, but we want to make sure that the log messages
+        # are informative about why it has failed.
+        self.basedir = self.mktemp()
+        d = self._setup_and_upload()
+        d.addCallback(lambda ign:
+            self._add_server_with_share(server_number=1, readonly=True))
+        d.addCallback(lambda ign:
+            self._add_server_with_share(server_number=2, readonly=True))
+        d.addCallback(lambda ign:
+            self._add_server_with_share(server_number=3, readonly=True))
+        d.addCallback(lambda ign:
+            self._add_server_with_share(server_number=4, readonly=True))
+        d.addCallback(lambda ign:
+            self._add_server_with_share(server_number=5, readonly=True))
+        d.addCallback(lambda ign:
+            self.g.remove_server(self.g.servers_by_number[0].my_nodeid))
+        def _reset_encoding_parameters(ign):
+            client = self.g.clients[0]
+            client.DEFAULT_ENCODING_PARAMETERS['happy'] = 4
+            return client
+        d.addCallback(_reset_encoding_parameters)
+        d.addCallback(lambda client:
+            self.shouldFail(UploadHappinessError, "test_selection_exceptions",
+                            "peer selection failed for <Tahoe2PeerSelector "
+                            "for upload dglev>: placed 0 shares out of 10 "
+                            "total (10 homeless), want to place on 4 servers,"
+                            " sent 5 queries to 5 peers, 0 queries placed "
+                            "some shares, 5 placed none "
+                            "(of which 5 placed none due to the server being "
+                            "full and 0 placed none due to an error)",
+                            client.upload,
+                            upload.Data("data" * 10000, convergence="")))
+
+
+        # server 1: readonly, no shares
+        # server 2: broken, no shares
+        # server 3: readonly, no shares
+        # server 4: readonly, no shares
+        # server 5: readonly, no shares
+        def _reset(ign):
+            self.basedir = self.mktemp()
+        d.addCallback(_reset)
+        d.addCallback(lambda ign:
+            self._setup_and_upload())
+        d.addCallback(lambda ign:
+            self._add_server_with_share(server_number=1, readonly=True))
+        d.addCallback(lambda ign:
+            self._add_server_with_share(server_number=2))
+        def _break_server_2(ign):
+            server = self.g.servers_by_number[2].my_nodeid
+            # We have to break the server in servers_by_id, 
+            # because the ones in servers_by_number isn't wrapped,
+            # and doesn't look at its broken attribute
+            self.g.servers_by_id[server].broken = True
+        d.addCallback(_break_server_2)
+        d.addCallback(lambda ign:
+            self._add_server_with_share(server_number=3, readonly=True))
+        d.addCallback(lambda ign:
+            self._add_server_with_share(server_number=4, readonly=True))
+        d.addCallback(lambda ign:
+            self._add_server_with_share(server_number=5, readonly=True))
+        d.addCallback(lambda ign:
+            self.g.remove_server(self.g.servers_by_number[0].my_nodeid))
+        def _reset_encoding_parameters(ign):
+            client = self.g.clients[0]
+            client.DEFAULT_ENCODING_PARAMETERS['happy'] = 4
+            return client
+        d.addCallback(_reset_encoding_parameters)
+        d.addCallback(lambda client:
+            self.shouldFail(UploadHappinessError, "test_selection_exceptions",
+                            "peer selection failed for <Tahoe2PeerSelector "
+                            "for upload dglev>: placed 0 shares out of 10 "
+                            "total (10 homeless), want to place on 4 servers,"
+                            " sent 5 queries to 5 peers, 0 queries placed "
+                            "some shares, 5 placed none "
+                            "(of which 4 placed none due to the server being "
+                            "full and 1 placed none due to an error)",
+                            client.upload,
+                            upload.Data("data" * 10000, convergence="")))
+        return d
+
+
     def _set_up_nodes_extra_config(self, clientdir):
         cfgfn = os.path.join(clientdir, "tahoe.cfg")
         oldcfg = open(cfgfn, "r").read()
}
[Replace "UploadHappinessError" with "UploadUnhappinessError" in tests.
Kevan Carstensen <kevan@isnotajoke.com>**20091205043453
 Ignore-this: 83f4bc50c697d21b5f4e2a4cd91862ca
] {
replace ./src/allmydata/test/test_encode.py [A-Za-z_0-9] UploadHappinessError UploadUnhappinessError
replace ./src/allmydata/test/test_upload.py [A-Za-z_0-9] UploadHappinessError UploadUnhappinessError
}
[Alter various unit tests to work with the new happy behavior
Kevan Carstensen <kevan@isnotajoke.com>**20100107181325
 Ignore-this: 132032bbf865e63a079f869b663be34a
] {
hunk ./src/allmydata/test/common.py 941
             # We need multiple segments to test crypttext hash trees that are
             # non-trivial (i.e. they have more than just one hash in them).
             cl0.DEFAULT_ENCODING_PARAMETERS['max_segment_size'] = 12
+            # Tests that need to test servers of happiness using this should
+            # set their own value for happy -- the default (7) breaks stuff.
+            cl0.DEFAULT_ENCODING_PARAMETERS['happy'] = 1
             d2 = cl0.upload(immutable.upload.Data(TEST_DATA, convergence=""))
             def _after_upload(u):
                 filecap = u.uri
hunk ./src/allmydata/test/test_checker.py 283
         self.basedir = "checker/AddLease/875"
         self.set_up_grid(num_servers=1)
         c0 = self.g.clients[0]
+        c0.DEFAULT_ENCODING_PARAMETERS['happy'] = 1
         self.uris = {}
         DATA = "data" * 100
         d = c0.upload(Data(DATA, convergence=""))
hunk ./src/allmydata/test/test_system.py 93
         d = self.set_up_nodes()
         def _check_connections(res):
             for c in self.clients:
+                c.DEFAULT_ENCODING_PARAMETERS['happy'] = 5
                 all_peerids = c.get_storage_broker().get_all_serverids()
                 self.failUnlessEqual(len(all_peerids), self.numclients)
                 sb = c.storage_broker
hunk ./src/allmydata/test/test_system.py 205
                                                       add_to_sparent=True))
         def _added(extra_node):
             self.extra_node = extra_node
+            self.extra_node.DEFAULT_ENCODING_PARAMETERS['happy'] = 5
         d.addCallback(_added)
 
         HELPER_DATA = "Data that needs help to upload" * 1000
hunk ./src/allmydata/test/test_system.py 705
         self.basedir = "system/SystemTest/test_filesystem"
         self.data = LARGE_DATA
         d = self.set_up_nodes(use_stats_gatherer=True)
+        def _new_happy_semantics(ign):
+            for c in self.clients:
+                c.DEFAULT_ENCODING_PARAMETERS['happy'] = 1
+        d.addCallback(_new_happy_semantics)
         d.addCallback(self._test_introweb)
         d.addCallback(self.log, "starting publish")
         d.addCallback(self._do_publish1)
hunk ./src/allmydata/test/test_system.py 1129
         d.addCallback(self.failUnlessEqual, "new.txt contents")
         # and again with something large enough to use multiple segments,
         # and hopefully trigger pauseProducing too
+        def _new_happy_semantics(ign):
+            for c in self.clients:
+                # these get reset somewhere? Whatever.
+                c.DEFAULT_ENCODING_PARAMETERS['happy'] = 1
+        d.addCallback(_new_happy_semantics)
         d.addCallback(lambda res: self.PUT(public + "/subdir3/big.txt",
                                            "big" * 500000)) # 1.5MB
         d.addCallback(lambda res: self.GET(public + "/subdir3/big.txt"))
hunk ./src/allmydata/test/test_upload.py 178
 
 class FakeClient:
     DEFAULT_ENCODING_PARAMETERS = {"k":25,
-                                   "happy": 75,
+                                   "happy": 25,
                                    "n": 100,
                                    "max_segment_size": 1*MiB,
                                    }
hunk ./src/allmydata/test/test_upload.py 316
         data = self.get_data(SIZE_LARGE)
         segsize = int(SIZE_LARGE / 2.5)
         # we want 3 segments, since that's not a power of two
-        self.set_encoding_parameters(25, 75, 100, segsize)
+        self.set_encoding_parameters(25, 25, 100, segsize)
         d = upload_data(self.u, data)
         d.addCallback(extract_uri)
         d.addCallback(self._check_large, SIZE_LARGE)
hunk ./src/allmydata/test/test_upload.py 395
     def test_first_error(self):
         mode = dict([(0,"good")] + [(i,"first-fail") for i in range(1,10)])
         self.make_node(mode)
+        self.set_encoding_parameters(k=25, happy=1, n=50)
         d = upload_data(self.u, DATA)
         d.addCallback(extract_uri)
         d.addCallback(self._check_large, SIZE_LARGE)
hunk ./src/allmydata/test/test_upload.py 513
 
         self.make_client()
         data = self.get_data(SIZE_LARGE)
-        self.set_encoding_parameters(50, 75, 100)
+        # if there are 50 peers, then happy needs to be <= 50
+        self.set_encoding_parameters(50, 50, 100)
         d = upload_data(self.u, data)
         d.addCallback(extract_uri)
         d.addCallback(self._check_large, SIZE_LARGE)
hunk ./src/allmydata/test/test_upload.py 560
 
         self.make_client()
         data = self.get_data(SIZE_LARGE)
-        self.set_encoding_parameters(100, 150, 200)
+        # if there are 50 peers, then happy should be no more than 50 if
+        # we want this to work.
+        self.set_encoding_parameters(100, 50, 200)
         d = upload_data(self.u, data)
         d.addCallback(extract_uri)
         d.addCallback(self._check_large, SIZE_LARGE)
hunk ./src/allmydata/test/test_upload.py 580
 
         self.make_client(3)
         data = self.get_data(SIZE_LARGE)
-        self.set_encoding_parameters(3, 5, 10)
+        self.set_encoding_parameters(3, 3, 10)
         d = upload_data(self.u, data)
         d.addCallback(extract_uri)
         d.addCallback(self._check_large, SIZE_LARGE)
hunk ./src/allmydata/test/test_web.py 4073
         self.basedir = "web/Grid/exceptions"
         self.set_up_grid(num_clients=1, num_servers=2)
         c0 = self.g.clients[0]
+        c0.DEFAULT_ENCODING_PARAMETERS['happy'] = 2
         self.fileurls = {}
         DATA = "data" * 100
         d = c0.create_dirnode()
}
[Revisions of the #778 tests, per reviewers' comments
Kevan Carstensen <kevan@isnotajoke.com>**20100319050653
 Ignore-this: 617307cec6bde9427211354e0e58734d
 
 - Fix comments and confusing naming.
 - Add tests for the new error messages suggested by David-Sarah
   and Zooko.
 - Alter existing tests for new error messages.
 - Make sure that the tests continue to work with the trunk.
 - Add a test for a mutual disjointedness assertion that I added to
   upload.servers_of_happiness.
 - Fix the comments to correctly reflect read-onlyness
 - Add a test for an edge case in should_add_server
 - Add an assertion to make sure that share redistribution works as it 
   should
 - Alter tests to work with revised servers_of_happiness semantics
 - Remove tests for should_add_server, since that function no longer exists.
 - Alter tests to know about merge_peers, and to use it before calling 
   servers_of_happiness.
 - Add tests for merge_peers.
 - Add Zooko's puzzles to the tests.
 
] {
hunk ./src/allmydata/test/test_encode.py 28
 class FakeBucketReaderWriterProxy:
     implements(IStorageBucketWriter, IStorageBucketReader)
     # these are used for both reading and writing
-    def __init__(self, mode="good"):
+    def __init__(self, mode="good", peerid="peer"):
         self.mode = mode
         self.blocks = {}
         self.plaintext_hashes = []
hunk ./src/allmydata/test/test_encode.py 36
         self.block_hashes = None
         self.share_hashes = None
         self.closed = False
+        self.peerid = peerid
 
     def get_peerid(self):
hunk ./src/allmydata/test/test_encode.py 39
-        return "peerid"
+        return self.peerid
 
     def _start(self):
         if self.mode == "lost-early":
hunk ./src/allmydata/test/test_encode.py 306
             for shnum in range(NUM_SHARES):
                 peer = FakeBucketReaderWriterProxy()
                 shareholders[shnum] = peer
-                servermap[shnum] = str(shnum)
+                servermap.setdefault(shnum, set()).add(peer.get_peerid())
                 all_shareholders.append(peer)
             e.set_shareholders(shareholders, servermap)
             return e.start()
hunk ./src/allmydata/test/test_encode.py 463
         def _ready(res):
             k,happy,n = e.get_param("share_counts")
             assert n == NUM_SHARES # else we'll be completely confused
-            all_peers = []
+            servermap = {}
             for shnum in range(NUM_SHARES):
                 mode = bucket_modes.get(shnum, "good")
hunk ./src/allmydata/test/test_encode.py 466
-                peer = FakeBucketReaderWriterProxy(mode)
+                peer = FakeBucketReaderWriterProxy(mode, "peer%d" % shnum)
                 shareholders[shnum] = peer
hunk ./src/allmydata/test/test_encode.py 468
-                servermap[shnum] = str(shnum)
+                servermap.setdefault(shnum, set()).add(peer.get_peerid())
             e.set_shareholders(shareholders, servermap)
             return e.start()
         d.addCallback(_ready)
hunk ./src/allmydata/test/test_upload.py 16
 from allmydata.interfaces import FileTooLargeError, UploadUnhappinessError
 from allmydata.util.assertutil import precondition
 from allmydata.util.deferredutil import DeferredListShouldSucceed
+from allmydata.util.happinessutil import servers_of_happiness, \
+                                         shares_by_server, merge_peers
 from no_network import GridTestMixin
 from common_util import ShouldFailMixin
 from allmydata.storage_client import StorageFarmBroker
hunk ./src/allmydata/test/test_upload.py 708
         num_segments = encoder.get_param("num_segments")
         d = selector.get_shareholders(broker, sh, storage_index,
                                       share_size, block_size, num_segments,
-                                      10, 4)
+                                      10, 3, 4)
         def _have_shareholders((used_peers, already_peers)):
             assert servers_to_break <= len(used_peers)
             for index in xrange(servers_to_break):
hunk ./src/allmydata/test/test_upload.py 720
             for peer in used_peers:
                 buckets.update(peer.buckets)
                 for bucket in peer.buckets:
-                    servermap[bucket] = peer.peerid
+                    servermap.setdefault(bucket, set()).add(peer.peerid)
             encoder.set_shareholders(buckets, servermap)
             d = encoder.start()
             return d
hunk ./src/allmydata/test/test_upload.py 764
         self.failUnless((share_number, ss.my_nodeid, new_share_location)
                         in shares)
 
+    def _setup_grid(self):
+        """
+        I set up a NoNetworkGrid with a single server and client.
+        """
+        self.set_up_grid(num_clients=1, num_servers=1)
 
     def _setup_and_upload(self):
         """
hunk ./src/allmydata/test/test_upload.py 776
         upload a file to it, store its uri in self.uri, and store its
         sharedata in self.shares.
         """
-        self.set_up_grid(num_clients=1, num_servers=1)
+        self._setup_grid()
         client = self.g.clients[0]
         client.DEFAULT_ENCODING_PARAMETERS['happy'] = 1
         data = upload.Data("data" * 10000, convergence="")
hunk ./src/allmydata/test/test_upload.py 814
 
 
     def _setUp(self, ns):
-        # Used by test_happy_semantics and test_prexisting_share_behavior
+        # Used by test_happy_semantics and test_preexisting_share_behavior
         # to set up the grid.
         self.node = FakeClient(mode="good", num_servers=ns)
         self.u = upload.Uploader()
hunk ./src/allmydata/test/test_upload.py 825
     def test_happy_semantics(self):
         self._setUp(2)
         DATA = upload.Data("kittens" * 10000, convergence="")
-        # These parameters are unsatisfiable with the client that we've made
-        # -- we'll use them to test that the semnatics work correctly.
+        # These parameters are unsatisfiable with only 2 servers.
         self.set_encoding_parameters(k=3, happy=5, n=10)
         d = self.shouldFail(UploadUnhappinessError, "test_happy_semantics",
hunk ./src/allmydata/test/test_upload.py 828
-                            "shares could only be placed on 2 servers "
-                            "(5 were requested)",
+                            "shares could only be placed or found on 2 "
+                            "server(s). We were asked to place shares on "
+                            "at least 5 server(s) such that any 3 of them "
+                            "have enough shares to recover the file",
                             self.u.upload, DATA)
         # Let's reset the client to have 10 servers
         d.addCallback(lambda ign:
hunk ./src/allmydata/test/test_upload.py 836
             self._setUp(10))
-        # These parameters are satisfiable with the client we've made.
+        # These parameters are satisfiable with 10 servers.
         d.addCallback(lambda ign:
             self.set_encoding_parameters(k=3, happy=5, n=10))
hunk ./src/allmydata/test/test_upload.py 839
-        # this should work
         d.addCallback(lambda ign:
             self.u.upload(DATA))
         # Let's reset the client to have 7 servers
hunk ./src/allmydata/test/test_upload.py 845
         # (this is less than n, but more than h)
         d.addCallback(lambda ign:
             self._setUp(7))
-        # These encoding parameters should still be satisfiable with our 
-        # client setup
+        # These parameters are satisfiable with 7 servers.
         d.addCallback(lambda ign:
             self.set_encoding_parameters(k=3, happy=5, n=10))
hunk ./src/allmydata/test/test_upload.py 848
-        # This, then, should work.
         d.addCallback(lambda ign:
             self.u.upload(DATA))
         return d
hunk ./src/allmydata/test/test_upload.py 862
         #
         # The scenario in comment:52 proposes that we have a layout
         # like:
-        # server 1: share 1
-        # server 2: share 1
-        # server 3: share 1
-        # server 4: shares 2 - 10
+        # server 0: shares 1 - 9
+        # server 1: share 0, read-only
+        # server 2: share 0, read-only
+        # server 3: share 0, read-only
         # To get access to the shares, we will first upload to one 
hunk ./src/allmydata/test/test_upload.py 867
-        # server, which will then have shares 1 - 10. We'll then 
+        # server, which will then have shares 0 - 9. We'll then 
         # add three new servers, configure them to not accept any new
hunk ./src/allmydata/test/test_upload.py 869
-        # shares, then write share 1 directly into the serverdir of each.
-        # Then each of servers 1 - 3 will report that they have share 1, 
-        # and will not accept any new share, while server 4 will report that
-        # it has shares 2 - 10 and will accept new shares.
+        # shares, then write share 0 directly into the serverdir of each,
+        # and then remove share 0 from server 0 in the same way.
+        # Then each of servers 1 - 3 will report that they have share 0, 
+        # and will not accept any new share, while server 0 will report that
+        # it has shares 1 - 9 and will accept new shares.
         # We'll then set 'happy' = 4, and see that an upload fails
         # (as it should)
         d = self._setup_and_upload()
hunk ./src/allmydata/test/test_upload.py 878
         d.addCallback(lambda ign:
-            self._add_server_with_share(1, 0, True))
+            self._add_server_with_share(server_number=1, share_number=0,
+                                        readonly=True))
         d.addCallback(lambda ign:
hunk ./src/allmydata/test/test_upload.py 881
-            self._add_server_with_share(2, 0, True))
+            self._add_server_with_share(server_number=2, share_number=0,
+                                        readonly=True))
         d.addCallback(lambda ign:
hunk ./src/allmydata/test/test_upload.py 884
-            self._add_server_with_share(3, 0, True))
+            self._add_server_with_share(server_number=3, share_number=0,
+                                        readonly=True))
         # Remove the first share from server 0.
hunk ./src/allmydata/test/test_upload.py 887
-        def _remove_share_0():
+        def _remove_share_0_from_server_0():
             share_location = self.shares[0][2]
             os.remove(share_location)
         d.addCallback(lambda ign:
hunk ./src/allmydata/test/test_upload.py 891
-            _remove_share_0())
+            _remove_share_0_from_server_0())
         # Set happy = 4 in the client.
         def _prepare():
             client = self.g.clients[0]
hunk ./src/allmydata/test/test_upload.py 901
             _prepare())
         # Uploading data should fail
         d.addCallback(lambda client:
-            self.shouldFail(UploadUnhappinessError, "test_happy_semantics",
-                            "shares could only be placed on 2 servers "
-                            "(4 were requested)",
+            self.shouldFail(UploadUnhappinessError,
+                            "test_problem_layout_comment_52_test_1",
+                            "shares could be placed or found on 4 server(s), "
+                            "but they are not spread out evenly enough to "
+                            "ensure that any 3 of these servers would have "
+                            "enough shares to recover the file. "
+                            "We were asked to place shares on at "
+                            "least 4 servers such that any 3 of them have "
+                            "enough shares to recover the file",
                             client.upload, upload.Data("data" * 10000,
                                                        convergence="")))
 
hunk ./src/allmydata/test/test_upload.py 932
                                         readonly=True))
         def _prepare2():
             client = self.g.clients[0]
-            client.DEFAULT_ENCODING_PARAMETERS['happy'] = 3
+            client.DEFAULT_ENCODING_PARAMETERS['happy'] = 4
             return client
         d.addCallback(lambda ign:
             _prepare2())
hunk ./src/allmydata/test/test_upload.py 937
         d.addCallback(lambda client:
-            self.shouldFail(UploadUnhappinessError, "test_happy_sematics",
-                            "shares could only be placed on 2 servers "
-                            "(3 were requested)",
+            self.shouldFail(UploadUnhappinessError,
+                            "test_problem_layout_comment_52_test_2",
+                            "shares could only be placed on 3 server(s) such "
+                            "that any 3 of them have enough shares to recover "
+                            "the file, but we were asked to use at least 4 "
+                            "such servers.",
                             client.upload, upload.Data("data" * 10000,
                                                        convergence="")))
         return d
hunk ./src/allmydata/test/test_upload.py 956
         def _change_basedir(ign):
             self.basedir = self.mktemp()
         _change_basedir(None)
-        d = self._setup_and_upload()
-        # We start by uploading all of the shares to one server (which has 
-        # already been done above).
+        # We start by uploading all of the shares to one server.
         # Next, we'll add three new servers to our NoNetworkGrid. We'll add
         # one share from our initial upload to each of these.
         # The counterintuitive ordering of the share numbers is to deal with 
hunk ./src/allmydata/test/test_upload.py 962
         # the permuting of these servers -- distributing the shares this 
         # way ensures that the Tahoe2PeerSelector sees them in the order 
-        # described above.
+        # described below.
+        d = self._setup_and_upload()
         d.addCallback(lambda ign:
             self._add_server_with_share(server_number=1, share_number=2))
         d.addCallback(lambda ign:
hunk ./src/allmydata/test/test_upload.py 975
         # server 1: share 2
         # server 2: share 0
         # server 3: share 1
-        # We want to change the 'happy' parameter in the client to 4. 
+        # We change the 'happy' parameter in the client to 4. 
         # The Tahoe2PeerSelector will see the peers permuted as:
         # 2, 3, 1, 0
         # Ideally, a reupload of our original data should work.
hunk ./src/allmydata/test/test_upload.py 988
             client.upload(upload.Data("data" * 10000, convergence="")))
 
 
-        # This scenario is basically comment:53, but with the order reversed;
-        # this means that the Tahoe2PeerSelector sees
-        # server 2: shares 1-10
-        # server 3: share 1
-        # server 1: share 2
-        # server 4: share 3
+        # This scenario is basically comment:53, but changed so that the 
+        # Tahoe2PeerSelector sees the server with all of the shares before
+        # any of the other servers.
+        # The layout is:
+        # server 2: shares 0 - 9
+        # server 3: share 0 
+        # server 1: share 1 
+        # server 4: share 2
+        # The Tahoe2PeerSelector sees the peers permuted as:
+        # 2, 3, 1, 4
+        # Note that server 0 has been replaced by server 4; this makes it 
+        # easier to ensure that the last server seen by Tahoe2PeerSelector
+        # has only one share. 
         d.addCallback(_change_basedir)
         d.addCallback(lambda ign:
             self._setup_and_upload())
hunk ./src/allmydata/test/test_upload.py 1012
             self._add_server_with_share(server_number=1, share_number=2))
         # Copy all of the other shares to server number 2
         def _copy_shares(ign):
-            for i in xrange(1, 10):
+            for i in xrange(0, 10):
                 self._copy_share_to_server(i, 2)
         d.addCallback(_copy_shares)
         # Remove the first server, and add a placeholder with share 0
hunk ./src/allmydata/test/test_upload.py 1024
         d.addCallback(_reset_encoding_parameters)
         d.addCallback(lambda client:
             client.upload(upload.Data("data" * 10000, convergence="")))
+
+
         # Try the same thing, but with empty servers after the first one
         # We want to make sure that Tahoe2PeerSelector will redistribute
         # shares as necessary, not simply discover an existing layout.
hunk ./src/allmydata/test/test_upload.py 1029
+        # The layout is:
+        # server 2: shares 0 - 9
+        # server 3: empty
+        # server 1: empty
+        # server 4: empty
         d.addCallback(_change_basedir)
         d.addCallback(lambda ign:
             self._setup_and_upload())
hunk ./src/allmydata/test/test_upload.py 1043
             self._add_server(server_number=3))
         d.addCallback(lambda ign:
             self._add_server(server_number=1))
+        d.addCallback(lambda ign:
+            self._add_server(server_number=4))
         d.addCallback(_copy_shares)
         d.addCallback(lambda ign:
             self.g.remove_server(self.g.servers_by_number[0].my_nodeid))
hunk ./src/allmydata/test/test_upload.py 1048
-        d.addCallback(lambda ign:
-            self._add_server(server_number=4))
         d.addCallback(_reset_encoding_parameters)
         d.addCallback(lambda client:
             client.upload(upload.Data("data" * 10000, convergence="")))
hunk ./src/allmydata/test/test_upload.py 1051
+        # Make sure that only as many shares as necessary to satisfy
+        # servers of happiness were pushed.
+        d.addCallback(lambda results:
+            self.failUnlessEqual(results.pushed_shares, 3))
         return d
 
 
hunk ./src/allmydata/test/test_upload.py 1133
 
 
     def test_dropped_servers_in_encoder(self):
+        # The Encoder does its own "servers_of_happiness" check if it 
+        # happens to lose a bucket during an upload (it assumes that 
+        # the layout presented to it satisfies "servers_of_happiness"
+        # until a failure occurs)
+        # 
+        # This test simulates an upload where servers break after peer
+        # selection, but before they are written to.
         def _set_basedir(ign=None):
             self.basedir = self.mktemp()
         _set_basedir()
hunk ./src/allmydata/test/test_upload.py 1146
         d = self._setup_and_upload();
         # Add 5 servers
         def _do_server_setup(ign):
-            self._add_server_with_share(1)
-            self._add_server_with_share(2)
-            self._add_server_with_share(3)
-            self._add_server_with_share(4)
-            self._add_server_with_share(5)
+            self._add_server_with_share(server_number=1)
+            self._add_server_with_share(server_number=2)
+            self._add_server_with_share(server_number=3)
+            self._add_server_with_share(server_number=4)
+            self._add_server_with_share(server_number=5)
         d.addCallback(_do_server_setup)
         # remove the original server
         # (necessary to ensure that the Tahoe2PeerSelector will distribute
hunk ./src/allmydata/test/test_upload.py 1159
             server = self.g.servers_by_number[0]
             self.g.remove_server(server.my_nodeid)
         d.addCallback(_remove_server)
-        # This should succeed.
+        # This should succeed; we still have 4 servers, and the 
+        # happiness of the upload is 4.
         d.addCallback(lambda ign:
             self._do_upload_with_broken_servers(1))
         # Now, do the same thing over again, but drop 2 servers instead
hunk ./src/allmydata/test/test_upload.py 1164
-        # of 1. This should fail.
+        # of 1. This should fail, because servers_of_happiness is 4 and 
+        # we can't satisfy that.
         d.addCallback(_set_basedir)
         d.addCallback(lambda ign:
             self._setup_and_upload())
hunk ./src/allmydata/test/test_upload.py 1175
             self.shouldFail(UploadUnhappinessError,
                             "test_dropped_servers_in_encoder",
                             "lost too many servers during upload "
-                            "(still have 3, want 4)",
+                            "(happiness is now 3, but we wanted 4)",
                             self._do_upload_with_broken_servers, 2))
         # Now do the same thing over again, but make some of the servers
         # readonly, break some of the ones that aren't, and make sure that
hunk ./src/allmydata/test/test_upload.py 1202
             self.shouldFail(UploadUnhappinessError,
                             "test_dropped_servers_in_encoder",
                             "lost too many servers during upload "
-                            "(still have 3, want 4)",
+                            "(happiness is now 3, but we wanted 4)",
                             self._do_upload_with_broken_servers, 2))
         return d
 
hunk ./src/allmydata/test/test_upload.py 1207
 
-    def test_servers_with_unique_shares(self):
-        # servers_with_unique_shares expects a dict of 
-        # shnum => peerid as a preexisting shares argument.
+    def test_merge_peers(self):
+        # merge_peers merges a list of used_peers and a dict of 
+        # shareid -> peerid mappings.
+        shares = {
+                    1 : set(["server1"]),
+                    2 : set(["server2"]),
+                    3 : set(["server3"]),
+                    4 : set(["server4", "server5"]),
+                    5 : set(["server1", "server2"]),
+                 }
+        # if not provided with a used_peers argument, it should just
+        # return the first argument unchanged.
+        self.failUnlessEqual(shares, merge_peers(shares, set([])))
+        class FakePeerTracker:
+            pass
+        trackers = []
+        for (i, server) in [(i, "server%d" % i) for i in xrange(5, 9)]:
+            t = FakePeerTracker()
+            t.peerid = server
+            t.buckets = [i]
+            trackers.append(t)
+        expected = {
+                    1 : set(["server1"]),
+                    2 : set(["server2"]),
+                    3 : set(["server3"]),
+                    4 : set(["server4", "server5"]),
+                    5 : set(["server1", "server2", "server5"]),
+                    6 : set(["server6"]),
+                    7 : set(["server7"]),
+                    8 : set(["server8"]),
+                   }
+        self.failUnlessEqual(expected, merge_peers(shares, set(trackers)))
+        shares2 = {}
+        expected = {
+                    5 : set(["server5"]),
+                    6 : set(["server6"]),
+                    7 : set(["server7"]),
+                    8 : set(["server8"]),
+                   }
+        self.failUnlessEqual(expected, merge_peers(shares2, set(trackers)))
+        shares3 = {}
+        trackers = []
+        expected = {}
+        for (i, server) in [(i, "server%d" % i) for i in xrange(10)]:
+            shares3[i] = set([server])
+            t = FakePeerTracker()
+            t.peerid = server
+            t.buckets = [i]
+            trackers.append(t)
+            expected[i] = set([server])
+        self.failUnlessEqual(expected, merge_peers(shares3, set(trackers)))
+
+
+    def test_servers_of_happiness_utility_function(self):
+        # These tests are concerned with the servers_of_happiness()
+        # utility function, and its underlying matching algorithm. Other
+        # aspects of the servers_of_happiness behavior are tested
+        # elsehwere These tests exist to ensure that
+        # servers_of_happiness doesn't under or overcount the happiness
+        # value for given inputs.
+
+        # servers_of_happiness expects a dict of 
+        # shnum => set(peerids) as a preexisting shares argument.
         test1 = {
hunk ./src/allmydata/test/test_upload.py 1271
-                 1 : "server1",
-                 2 : "server2",
-                 3 : "server3",
-                 4 : "server4"
+                 1 : set(["server1"]),
+                 2 : set(["server2"]),
+                 3 : set(["server3"]),
+                 4 : set(["server4"])
                 }
hunk ./src/allmydata/test/test_upload.py 1276
-        unique_servers = upload.servers_with_unique_shares(test1)
-        self.failUnlessEqual(4, len(unique_servers))
-        for server in ["server1", "server2", "server3", "server4"]:
-            self.failUnlessIn(server, unique_servers)
-        test1[4] = "server1"
-        # Now there should only be 3 unique servers.
-        unique_servers = upload.servers_with_unique_shares(test1)
-        self.failUnlessEqual(3, len(unique_servers))
-        for server in ["server1", "server2", "server3"]:
-            self.failUnlessIn(server, unique_servers)
-        # servers_with_unique_shares expects to receive some object with
-        # a peerid attribute. So we make a FakePeerTracker whose only
-        # job is to have a peerid attribute.
+        happy = servers_of_happiness(test1)
+        self.failUnlessEqual(4, happy)
+        test1[4] = set(["server1"])
+        # We've added a duplicate server, so now servers_of_happiness
+        # should be 3 instead of 4.
+        happy = servers_of_happiness(test1)
+        self.failUnlessEqual(3, happy)
+        # The second argument of merge_peers should be a set of 
+        # objects with peerid and buckets as attributes. In actual use, 
+        # these will be PeerTracker instances, but for testing it is fine 
+        # to make a FakePeerTracker whose job is to hold those instance
+        # variables to test that part.
         class FakePeerTracker:
             pass
         trackers = []
hunk ./src/allmydata/test/test_upload.py 1296
             t.peerid = server
             t.buckets = [i]
             trackers.append(t)
-        # Recall that there are 3 unique servers in test1. Since none of
-        # those overlap with the ones in trackers, we should get 7 back
-        unique_servers = upload.servers_with_unique_shares(test1, set(trackers))
-        self.failUnlessEqual(7, len(unique_servers))
-        expected_servers = ["server" + str(i) for i in xrange(1, 9)]
-        expected_servers.remove("server4")
-        for server in expected_servers:
-            self.failUnlessIn(server, unique_servers)
-        # Now add an overlapping server to trackers.
+        # Recall that test1 is a server layout with servers_of_happiness
+        # = 3.  Since there isn't any overlap between the shnum ->
+        # set([peerid]) correspondences in test1 and those in trackers,
+        # the result here should be 7.
+        test2 = merge_peers(test1, set(trackers))
+        happy = servers_of_happiness(test2)
+        self.failUnlessEqual(7, happy)
+        # Now add an overlapping server to trackers. This is redundant,
+        # so it should not cause the previously reported happiness value
+        # to change.
         t = FakePeerTracker()
         t.peerid = "server1"
         t.buckets = [1]
hunk ./src/allmydata/test/test_upload.py 1310
         trackers.append(t)
-        unique_servers = upload.servers_with_unique_shares(test1, set(trackers))
-        self.failUnlessEqual(7, len(unique_servers))
-        for server in expected_servers:
-            self.failUnlessIn(server, unique_servers)
+        test2 = merge_peers(test1, set(trackers))
+        happy = servers_of_happiness(test2)
+        self.failUnlessEqual(7, happy)
         test = {}
hunk ./src/allmydata/test/test_upload.py 1314
-        unique_servers = upload.servers_with_unique_shares(test)
-        self.failUnlessEqual(0, len(test))
+        happy = servers_of_happiness(test)
+        self.failUnlessEqual(0, happy)
+        # Test a more substantial overlap between the trackers and the 
+        # existing assignments.
+        test = {
+            1 : set(['server1']),
+            2 : set(['server2']),
+            3 : set(['server3']),
+            4 : set(['server4']),
+        }
+        trackers = []
+        t = FakePeerTracker()
+        t.peerid = 'server5'
+        t.buckets = [4]
+        trackers.append(t)
+        t = FakePeerTracker()
+        t.peerid = 'server6'
+        t.buckets = [3, 5]
+        trackers.append(t)
+        # The value returned by servers_of_happiness is the size 
+        # of a maximum matching in the bipartite graph that
+        # servers_of_happiness() makes between peerids and share
+        # numbers. It should find something like this:
+        # (server 1, share 1)
+        # (server 2, share 2)
+        # (server 3, share 3)
+        # (server 5, share 4)
+        # (server 6, share 5)
+        # 
+        # and, since there are 5 edges in this matching, it should
+        # return 5.
+        test2 = merge_peers(test, set(trackers))
+        happy = servers_of_happiness(test2)
+        self.failUnlessEqual(5, happy)
+        # Zooko's first puzzle: 
+        # (from http://allmydata.org/trac/tahoe-lafs/ticket/778#comment:156)
+        #
+        # server 1: shares 0, 1
+        # server 2: shares 1, 2
+        # server 3: share 2
+        #
+        # This should yield happiness of 3.
+        test = {
+            0 : set(['server1']),
+            1 : set(['server1', 'server2']),
+            2 : set(['server2', 'server3']),
+        }
+        self.failUnlessEqual(3, servers_of_happiness(test))
+        # Zooko's second puzzle:        
+        # (from http://allmydata.org/trac/tahoe-lafs/ticket/778#comment:158)
+        # 
+        # server 1: shares 0, 1
+        # server 2: share 1
+        # 
+        # This should yield happiness of 2.
+        test = {
+            0 : set(['server1']),
+            1 : set(['server1', 'server2']),
+        }
+        self.failUnlessEqual(2, servers_of_happiness(test))
 
 
     def test_shares_by_server(self):
hunk ./src/allmydata/test/test_upload.py 1377
-        test = dict([(i, "server%d" % i) for i in xrange(1, 5)])
-        shares_by_server = upload.shares_by_server(test)
-        self.failUnlessEqual(set([1]), shares_by_server["server1"])
-        self.failUnlessEqual(set([2]), shares_by_server["server2"])
-        self.failUnlessEqual(set([3]), shares_by_server["server3"])
-        self.failUnlessEqual(set([4]), shares_by_server["server4"])
+        test = dict([(i, set(["server%d" % i])) for i in xrange(1, 5)])
+        sbs = shares_by_server(test)
+        self.failUnlessEqual(set([1]), sbs["server1"])
+        self.failUnlessEqual(set([2]), sbs["server2"])
+        self.failUnlessEqual(set([3]), sbs["server3"])
+        self.failUnlessEqual(set([4]), sbs["server4"])
         test1 = {
hunk ./src/allmydata/test/test_upload.py 1384
-                    1 : "server1",
-                    2 : "server1",
-                    3 : "server1",
-                    4 : "server2",
-                    5 : "server2"
+                    1 : set(["server1"]),
+                    2 : set(["server1"]),
+                    3 : set(["server1"]),
+                    4 : set(["server2"]),
+                    5 : set(["server2"])
                 }
hunk ./src/allmydata/test/test_upload.py 1390
-        shares_by_server = upload.shares_by_server(test1)
-        self.failUnlessEqual(set([1, 2, 3]), shares_by_server["server1"])
-        self.failUnlessEqual(set([4, 5]), shares_by_server["server2"])
+        sbs = shares_by_server(test1)
+        self.failUnlessEqual(set([1, 2, 3]), sbs["server1"])
+        self.failUnlessEqual(set([4, 5]), sbs["server2"])
+        # This should fail unless the peerid part of the mapping is a set
+        test2 = {1: "server1"}
+        self.shouldFail(AssertionError,
+                       "test_shares_by_server",
+                       "",
+                       shares_by_server, test2)
 
 
     def test_existing_share_detection(self):
hunk ./src/allmydata/test/test_upload.py 1405
         self.basedir = self.mktemp()
         d = self._setup_and_upload()
         # Our final setup should look like this:
-        # server 1: shares 1 - 10, read-only
+        # server 1: shares 0 - 9, read-only
         # server 2: empty
         # server 3: empty
         # server 4: empty
hunk ./src/allmydata/test/test_upload.py 1437
         return d
 
 
-    def test_should_add_server(self):
-        shares = dict([(i, "server%d" % i) for i in xrange(10)])
-        self.failIf(upload.should_add_server(shares, "server1", 4))
-        shares[4] = "server1"
-        self.failUnless(upload.should_add_server(shares, "server4", 4))
-        shares = {}
-        self.failUnless(upload.should_add_server(shares, "server1", 1))
-
-
     def test_exception_messages_during_peer_selection(self):
hunk ./src/allmydata/test/test_upload.py 1438
-        # server 1: readonly, no shares
-        # server 2: readonly, no shares
-        # server 3: readonly, no shares
-        # server 4: readonly, no shares
-        # server 5: readonly, no shares
+        # server 1: read-only, no shares
+        # server 2: read-only, no shares
+        # server 3: read-only, no shares
+        # server 4: read-only, no shares
+        # server 5: read-only, no shares
         # This will fail, but we want to make sure that the log messages
         # are informative about why it has failed.
         self.basedir = self.mktemp()
hunk ./src/allmydata/test/test_upload.py 1468
             self.shouldFail(UploadUnhappinessError, "test_selection_exceptions",
                             "peer selection failed for <Tahoe2PeerSelector "
                             "for upload dglev>: placed 0 shares out of 10 "
-                            "total (10 homeless), want to place on 4 servers,"
-                            " sent 5 queries to 5 peers, 0 queries placed "
+                            "total (10 homeless), want to place shares on at "
+                            "least 4 servers such that any 3 of them have "
+                            "enough shares to recover the file, "
+                            "sent 5 queries to 5 peers, 0 queries placed "
                             "some shares, 5 placed none "
                             "(of which 5 placed none due to the server being "
                             "full and 0 placed none due to an error)",
hunk ./src/allmydata/test/test_upload.py 1479
                             upload.Data("data" * 10000, convergence="")))
 
 
-        # server 1: readonly, no shares
+        # server 1: read-only, no shares
         # server 2: broken, no shares
hunk ./src/allmydata/test/test_upload.py 1481
-        # server 3: readonly, no shares
-        # server 4: readonly, no shares
-        # server 5: readonly, no shares
+        # server 3: read-only, no shares
+        # server 4: read-only, no shares
+        # server 5: read-only, no shares
         def _reset(ign):
             self.basedir = self.mktemp()
         d.addCallback(_reset)
hunk ./src/allmydata/test/test_upload.py 1496
         def _break_server_2(ign):
             server = self.g.servers_by_number[2].my_nodeid
             # We have to break the server in servers_by_id, 
-            # because the ones in servers_by_number isn't wrapped,
-            # and doesn't look at its broken attribute
+            # because the one in servers_by_number isn't wrapped,
+            # and doesn't look at its broken attribute when answering
+            # queries.
             self.g.servers_by_id[server].broken = True
         d.addCallback(_break_server_2)
         d.addCallback(lambda ign:
hunk ./src/allmydata/test/test_upload.py 1509
             self._add_server_with_share(server_number=5, readonly=True))
         d.addCallback(lambda ign:
             self.g.remove_server(self.g.servers_by_number[0].my_nodeid))
-        def _reset_encoding_parameters(ign):
+        def _reset_encoding_parameters(ign, happy=4):
             client = self.g.clients[0]
hunk ./src/allmydata/test/test_upload.py 1511
-            client.DEFAULT_ENCODING_PARAMETERS['happy'] = 4
+            client.DEFAULT_ENCODING_PARAMETERS['happy'] = happy
             return client
         d.addCallback(_reset_encoding_parameters)
         d.addCallback(lambda client:
hunk ./src/allmydata/test/test_upload.py 1518
             self.shouldFail(UploadUnhappinessError, "test_selection_exceptions",
                             "peer selection failed for <Tahoe2PeerSelector "
                             "for upload dglev>: placed 0 shares out of 10 "
-                            "total (10 homeless), want to place on 4 servers,"
-                            " sent 5 queries to 5 peers, 0 queries placed "
+                            "total (10 homeless), want to place shares on at "
+                            "least 4 servers such that any 3 of them have "
+                            "enough shares to recover the file, "
+                            "sent 5 queries to 5 peers, 0 queries placed "
                             "some shares, 5 placed none "
                             "(of which 4 placed none due to the server being "
                             "full and 1 placed none due to an error)",
hunk ./src/allmydata/test/test_upload.py 1527
                             client.upload,
                             upload.Data("data" * 10000, convergence="")))
+        # server 0, server 1 = empty, accepting shares
+        # This should place all of the shares, but still fail with happy=4.
+        # We want to make sure that the exception message is worded correctly.
+        d.addCallback(_reset)
+        d.addCallback(lambda ign:
+            self._setup_grid())
+        d.addCallback(lambda ign:
+            self._add_server_with_share(server_number=1))
+        d.addCallback(_reset_encoding_parameters)
+        d.addCallback(lambda client:
+            self.shouldFail(UploadUnhappinessError, "test_selection_exceptions",
+                            "shares could only be placed or found on 2 "
+                            "server(s). We were asked to place shares on at "
+                            "least 4 server(s) such that any 3 of them have "
+                            "enough shares to recover the file.",
+                            client.upload, upload.Data("data" * 10000,
+                                                       convergence="")))
+        # servers 0 - 4 = empty, accepting shares
+        # This too should place all the shares, and this too should fail,
+        # but since the effective happiness is more than the k encoding
+        # parameter, it should trigger a different error message than the one
+        # above.
+        d.addCallback(_reset)
+        d.addCallback(lambda ign:
+            self._setup_grid())
+        d.addCallback(lambda ign:
+            self._add_server_with_share(server_number=1))
+        d.addCallback(lambda ign:
+            self._add_server_with_share(server_number=2))
+        d.addCallback(lambda ign:
+            self._add_server_with_share(server_number=3))
+        d.addCallback(lambda ign:
+            self._add_server_with_share(server_number=4))
+        d.addCallback(_reset_encoding_parameters, happy=7)
+        d.addCallback(lambda client:
+            self.shouldFail(UploadUnhappinessError, "test_selection_exceptions",
+                            "shares could only be placed on 5 server(s) such "
+                            "that any 3 of them have enough shares to recover "
+                            "the file, but we were asked to use at least 7 "
+                            "such servers.",
+                            client.upload, upload.Data("data" * 10000,
+                                                       convergence="")))
+        # server 0: shares 0 - 9
+        # server 1: share 0, read-only
+        # server 2: share 0, read-only
+        # server 3: share 0, read-only
+        # This should place all of the shares, but fail with happy=4.
+        # Since the number of servers with shares is more than the number
+        # necessary to reconstitute the file, this will trigger a different
+        # error message than either of those above. 
+        d.addCallback(_reset)
+        d.addCallback(lambda ign:
+            self._setup_and_upload())
+        d.addCallback(lambda ign:
+            self._add_server_with_share(server_number=1, share_number=0,
+                                        readonly=True))
+        d.addCallback(lambda ign:
+            self._add_server_with_share(server_number=2, share_number=0,
+                                        readonly=True))
+        d.addCallback(lambda ign:
+            self._add_server_with_share(server_number=3, share_number=0,
+                                        readonly=True))
+        d.addCallback(_reset_encoding_parameters, happy=7)
+        d.addCallback(lambda client:
+            self.shouldFail(UploadUnhappinessError, "test_selection_exceptions",
+                            "shares could be placed or found on 4 server(s), "
+                            "but they are not spread out evenly enough to "
+                            "ensure that any 3 of these servers would have "
+                            "enough shares to recover the file. We were asked "
+                            "to place shares on at least 7 servers such that "
+                            "any 3 of them have enough shares to recover the "
+                            "file",
+                            client.upload, upload.Data("data" * 10000,
+                                                       convergence="")))
         return d
 
 
}

Context:

[setup: add licensing declaration for setuptools (noticed by the FSF compliance folks)
zooko@zooko.com**20100309184415
 Ignore-this: 2dfa7d812d65fec7c72ddbf0de609ccb
] 
[setup: fix error in licensing declaration from Shawn Willden, as noted by the FSF compliance division
zooko@zooko.com**20100309163736
 Ignore-this: c0623d27e469799d86cabf67921a13f8
] 
[CREDITS to Jacob Appelbaum
zooko@zooko.com**20100304015616
 Ignore-this: 70db493abbc23968fcc8db93f386ea54
] 
[desert-island-build-with-proper-versions
jacob@appelbaum.net**20100304013858] 
[docs: a few small edits to try to guide newcomers through the docs
zooko@zooko.com**20100303231902
 Ignore-this: a6aab44f5bf5ad97ea73e6976bc4042d
 These edits were suggested by my watching over Jake Appelbaum's shoulder as he completely ignored/skipped/missed install.html and also as he decided that debian.txt wouldn't help him with basic installation. Then I threw in a few docs edits that have been sitting around in my sandbox asking to be committed for months.
] 
[TAG allmydata-tahoe-1.6.1
david-sarah@jacaranda.org**20100228062314
 Ignore-this: eb5f03ada8ea953ee7780e7fe068539
] 
[Change install.html to reference 1.6.1 instead of 1.6.0
david-sarah@jacaranda.org**20100228061941
 Ignore-this: 4738440e66a12dcf2cadf968fba5337
] 
[docs: fix the asymptotic network performance of mutable file download in performance.txt, rename the howto-make-a-release file
zooko@zooko.com**20100228061439
 Ignore-this: c983b2fa7864f717ec17fb556f8a95d2
] 
[Change code that gives a base32 SI or an empty string to be more straightforward. (#948)
david-sarah@jacaranda.org**20100227065551
 Ignore-this: ba2b0eb430635fcfb09faeca5046ed21
] 
[Additional test for DIR2-LIT directories in test_web.py, fixed version (#948)
david-sarah@jacaranda.org**20100225041824
 Ignore-this: 86d710f438439f27aa372b84411af011
] 
[Updates to NEWS for 1.6.1
david-sarah@jacaranda.org**20100224081542
 Ignore-this: ae1ca1892d7013bcb5f54f201459632
] 
[Additional fixes for DIR2-LIT More Info page and deep-check/manifest operations (#948)
david-sarah@jacaranda.org**20100224080220
 Ignore-this: 3b431b712f380b5476231ebd99648a7f
] 
[directories: add DIR2-LIT directories to test_deepcheck.py (#948)
david-sarah@jacaranda.org**20100224075433
 Ignore-this: ed1dcbe45870f5efae0ebbcdff677a4b
] 
[dirnode: add tests of literal dirnodes (current and fix for #948)
david-sarah@jacaranda.org**20100224043345
 Ignore-this: f18cd17d72ed2495a646fa6c3af42aa1
] 
[Additional fix for abbrev_si, with test
david-sarah@jacaranda.org**20100222033652
 Ignore-this: 7dc1c7031cd395fb4ec0a5aa96e69a10
] 
[Additions to test_web.py for #948
david-sarah@jacaranda.org**20100222025352
 Ignore-this: b99be703923efc75db75894a05e6a527
] 
[Change direct accesses to an_uri.storage_index to calls to .get_storage_index() (fixes #948)
david-sarah@jacaranda.org**20100222024504
 Ignore-this: 91f6fccb5fd9456aa0e02d312f902928
] 
[Tweak to 'tahoe ls --help' output (#837)
david-sarah@jacaranda.org**20100224030231
 Ignore-this: 9c86ff8ee1f2c9b8a4f6e205a58905f
] 
[Test behaviour of 'tahoe ls' for unknown objects (#837)
david-sarah@jacaranda.org**20100224025913
 Ignore-this: b999f6239796a90cadb41e8650aa3782
] 
[Improve behaviour of 'tahoe ls' for unknown objects, addressing kevan's comments
david-sarah@jacaranda.org**20100220061313
 Ignore-this: 6205025c477f1c999473a4ae67e1c83
] 
[docs: update relnotes.txt for v1.6.1
zooko@zooko.com**20100224065755
 Ignore-this: 6d078e94425462ac8d074e3e7c82da28
] 
[docs: NEWS and relnotes-short.txt and CREDITS for v1.6.1
zooko@zooko.com**20100224065231
 Ignore-this: 41c056ae48c639e5a934d4c1983bc118
] 
[misc/coverage.el: improve filename matching
Brian Warner <warner@lothar.com>**20100224044757
 Ignore-this: 8d9fb1d2a71e01370da006a2fef04346
] 
[test_util.py: improve coverage of util.time_format
Brian Warner <warner@lothar.com>**20100224044637
 Ignore-this: bd93495132fe73a9c117d35c1a4e2d72
] 
[docs/performance.txt: split out CPU from network, expand on mutable costs
Brian Warner <warner@lothar.com>**20100224043813
 Ignore-this: 4779e78ca0eed1dcbd1652e6287219f1
] 
[docs/FTP: the Twisted patch (t3462) has landed, will be in the next release
Brian Warner <warner@lothar.com>**20100223210402
 Ignore-this: ddc5c8da8c95d8c19380d8c7ecbaf18
] 
[Change OphandleTable to use a deterministic clock, so we can test it
Kevan Carstensen <kevan@isnotajoke.com>**20100220210713
 Ignore-this: a7437f4eda359bdfa243bd534f23bf52
 
 To test the changes for #577, we need a deterministic way to simulate
 the passage of long periods of time. twisted.internet.task.Clock seems,
 from my Googling, to be the way to go for this functionality. I changed
 a few things so that OphandleTable would use twisted.internet.task.Clock
 when testing:
 
   * WebishServer.__init___ now takes an optional 'clock' parameter,
   * which it passes to the root.Root instance it creates.
   * root.Root.__init__ now takes an optional 'clock' parameter, which it
     passes to the OphandleTable.__init__ method.
   * OphandleTable.__init__ now takes an optional 'clock' parameter. If
     it is provided, and it isn't None, its callLater method will be used
     to schedule ophandle expirations (as opposed to using
     reactor.callLater, which is what OphandleTable does normally).
   * The WebMixin object in test_web.py now sets a self.clock parameter,
     which is a twisted.internet.task.Clock that it feeds to the 
     WebishServer it creates. 
 
 Tests using the WebMixin can control the passage of time in
 OphandleTable by accessing self.clock. 
] 
[Add tests for the ophandle expiration behavior in #577
Kevan Carstensen <kevan@isnotajoke.com>**20100221010455
 Ignore-this: 87a435108999c24920354b58fd78353f
] 
[Update docs/frontends/webapi.txt to reflect the new expiration times in #577
Kevan Carstensen <kevan@isnotajoke.com>**20100221010716
 Ignore-this: cefee2ba800c285ae4148fe2dff39a3b
] 
[Increase ophandle expiration times, per #577
Kevan Carstensen <kevan@isnotajoke.com>**20100221010512
 Ignore-this: 247f61fe8855a0c76fef3777a957f495
] 
[More cleanups to test_cli using new utilities for reading and writing files.
david-sarah@jacaranda.org**20100206013855
 Ignore-this: 9fd2294406b346bfe9144fff6a61f789
] 
[Fix race conditions and missing callback in allmydata.test.test_cli.Cp.test_copy_using_filecap, add utilities for one-liner reading and writing of files, and fix cases in test_cli where files were not being closed after writing.
david-sarah@jacaranda.org**20100206013727
 Ignore-this: 49da6c33190d526a4ae84c472f04d5f4
] 
[setup: comment-out the dependency on pycrypto, see #953
zooko@zooko.com**20100215050844
 Ignore-this: 2751120921ff35b8189d8fcd896da149
] 
[Add tests for #939
Kevan Carstensen <kevan@isnotajoke.com>**20100212062137
 Ignore-this: 5459e8c64ba76cca70aa720e68549637
] 
[Alter CLI utilities to handle nonexistent aliases better
Kevan Carstensen <kevan@isnotajoke.com>**20100211024318
 Ignore-this: e698ea4a57f5fe27c24336581ca0cf65
] 
[web/storage.py: display total-seen on the last-complete-cycle line. For #940.
Brian Warner <warner@lothar.com>**20100208002010
 Ignore-this: c0ed860f3e9628d3171d2b055d96c5aa
] 
[adding pycrypto to the auto dependencies
secorp@allmydata.com**20100206054314
 Ignore-this: b873fc00a6a5b001d30d479e6053cf2f
] 
[docs running.html - "tahoe run ." does not work with the current installation, replaced with "tahoe start ."
secorp@allmydata.com**20100206165320
 Ignore-this: fdb2dcb0e417d303cd43b1951a4f8c03
] 
[code coverage: replace figleaf with coverage.py, should work on py2.6 now.
Brian Warner <warner@lothar.com>**20100203165421
 Ignore-this: 46ab590360be6a385cb4fc4e68b6b42c
 
 It still lacks the right HTML report (the builtin report is very pretty, but
 lacks the "lines uncovered" numbers that I want), and the half-finished
 delta-from-last-run measurements.
] 
[More comprehensive changes and ticket references for NEWS
david-sarah@jacaranda.org**20100202061256
 Ignore-this: 696cf0106e8a7fd388afc5b55fba8a1b
] 
[docs: install.html: link into Python 2.5.5 download page
zooko@zooko.com**20100202065852
 Ignore-this: 1a9471b8175b7de5741d8445a7ede29d
] 
[TAG allmydata-tahoe-1.6.0
zooko@zooko.com**20100202061125
 Ignore-this: dee6ade7ac1452cf5d1d9c69a8146d84
] 
[docs: install.html: recommend Python 2.5 (because I can build extension modules for it with mingw), architecture.txt: point out that our Proof of Retrievability feature is client-side-only
zooko@zooko.com**20100202053842
 Ignore-this: e33fd413a91771c77b17d7de0f215bea
] 
[architecture.txt: remove trailing whitespace, wrap lines: no content changes
Brian Warner <warner@lothar.com>**20100202055304
 Ignore-this: 1662f37d1162858ac2619db27bcc411f
] 
[docs: a couple of small edits to release notes (thanks Peter)
zooko@zooko.com**20100202054832
 Ignore-this: 1d0963c43ff19c92775b124c49c8a88a
] 
[docs: CREDITS: where due
zooko@zooko.com**20100202053831
 Ignore-this: 11646dd603ac715ae8277a4bb9562215
] 
[docs: a few small edits to performance.txt and README
zooko@zooko.com**20100202052750
 Ignore-this: bf8b1b7438e8fb6da09eec9713c78533
] 
[docs: a few edits to architecture.txt, most significantly highlighting "future work" to avoid confusing it with the current version, and adding a "future work" about a random-sampling Proof of Retrievability verifier
zooko@zooko.com**20100202045117
 Ignore-this: 81122b3042ea9ee6bc12e795c2386d59
] 
[docs: a few edits and updates to relnotes.txt, relnotes-short.txt, and NEWS in preparation for v1.6.0
zooko@zooko.com**20100202043222
 Ignore-this: d90c644fa61d78e33cbdf0be428bb07a
] 
[Document leakage of cap URLs via phishing filters in known_issues.txt
david-sarah@jacaranda.org**20100202015238
 Ignore-this: 78e668dbca77c0e3a73e10c0b74cf024
] 
[docs: updates to relnotes.txt, NEWS, architecture, historical_known_issues, install.html, etc.
zooko@zooko.com**20100201181809
 Ignore-this: f4fc924652af746862c8ee4d9ba97bf6
] 
[immutable: downloader accepts notifications of buckets even if those notifications arrive after he has begun downloading shares.
zooko@zooko.com**20100201061610
 Ignore-this: 5b09709f27603a3157eba7ba70028955
 This can be useful if one of the ones that he has already begun downloading fails. See #287 for discussion. This fixes part of #287 which part was a regression caused by #928, namely this fixes fail-over in case a share is corrupted (or the server returns an error or disconnects). This does not fix the related issue mentioned in #287 if a server hangs and doesn't reply to requests for blocks.
 
] 
[tests: don't require tahoe to run with no noise if we are using an old twisted that emits DeprecationWarnings
zooko@zooko.com**20100201052323
 Ignore-this: 69668c772cce612a0c6936a2195ebd2a
] 
[Use if instead of assert to check for twisted ftp patch
david-sarah@jacaranda.org**20100127015529
 Ignore-this: 66959d946bd1a835ece6f074e75086b2
] 
[tests: stop being surprised that Nevow no longer prints out warnings when it tries to find its static files
zooko@zooko.com**20100201041144
 Ignore-this: 77b4ac383165d98dfe2a9008ce794742
 Unless we are using a sufficiently new version of Nevow, in which case if it prints out warnings then this is a hard test failure. :-)
] 
[cli: suppress DeprecationWarnings emitted from importing nevow and twisted. Fixes #859
david-sarah@jacaranda.org**20100201004429
 Ignore-this: 22d7216921cd5f04381c0194ed501bbe
] 
[Fill in 'docs/performance.txt' with some performance information
Kevan Carstensen <kevan@isnotajoke.com>**20100202005914
 Ignore-this: c66b255b2bd2e7e11f5707b25e7b38be
] 
[Improvements to test_unknownnode to cover invalid cap URIs with known prefixes
david-sarah@jacaranda.org**20100130063908
 Ignore-this: e1a298942c21207473e418ea5efd6276
] 
[Fix invalid trailing commas in JSON example
david-sarah@jacaranda.org**20100129201742
 Ignore-this: d99e0a8ead4fafabf39a1daf11ec450b
] 
[Improvements to test_hung_server, and fix for status updates in download.py
david-sarah@jacaranda.org**20100130064303
 Ignore-this: dd889c643afdcf0f86d55855aafda6ad
] 
[immutable: fix bug in tests, change line-endings to unix style, add comment
zooko@zooko.com**20100129184237
 Ignore-this: f6bd875fe974c55c881e05eddf8d3436
] 
[New tests for #928
david-sarah@jacaranda.org**20100129123845
 Ignore-this: 5c520f40141f0d9c000ffb05a4698995
] 
[immutable: download from the first servers which provide at least K buckets instead of waiting for all servers to reply
zooko@zooko.com**20100127233417
 Ignore-this: c855355a40d96827e1d0c469a8d8ab3f
 This should put an end to the phenomenon I've been seeing that a single hung server can cause all downloads on a grid to hang.  Also it should speed up all downloads by (a) not-waiting for responses to queries that it doesn't need, and (b) downloading shares from the servers which answered the initial query the fastest.
 Also, do not count how many buckets you've gotten when deciding whether the download has enough shares or not -- instead count how many buckets to *unique* shares that you've gotten.  This appears to improve a slightly weird behavior in the current download code in which receiving >= K different buckets all to the same sharenumber would make it think it had enough to download the file when in fact it hadn't.
 This patch needs tests before it is actually ready for trunk.
] 
[Eliminate 'foo if test else bar' syntax that isn't supported by Python 2.4
david-sarah@jacaranda.org**20100129035210
 Ignore-this: 70eafd487b4b6299beedd63b4a54a0c
] 
[Fix example JSON in webapi.txt that cannot occur in practice
david-sarah@jacaranda.org**20100129032742
 Ignore-this: 361a1ba663d77169aeef93caef870097
] 
[Add mutable field to t=json output for unknown nodes, when mutability is known
david-sarah@jacaranda.org**20100129031424
 Ignore-this: 1516d63559bdfeb6355485dff0f5c04e
] 
[Show -IMM and -RO suffixes for types of immutable and read-only unknown nodes in directory listings
david-sarah@jacaranda.org**20100128220800
 Ignore-this: dc5c17c0a566398f88e4303c41321e66
] 
[Fix inaccurate comment in test_mutant_dirnodes_are_omitted
david-sarah@jacaranda.org**20100128202456
 Ignore-this: 9fa17ed7feac9e4d084f1b2338c76fca
] 
[docs: update relnotes.txt for Tahoe-LAFS v1.6
zooko@zooko.com**20100128171257
 Ignore-this: 920df92152aead69ef861b9b2e8ff218
] 
[Address comments by Kevan on 833 and add test for stripping spaces
david-sarah@jacaranda.org**20100127230642
 Ignore-this: de36aeaf4afb3ba05dbeb49a5e9a6b26
] 
[Miscellaneous documentation, test, and code formatting tweaks.
david-sarah@jacaranda.org**20100127070309
 Ignore-this: 84ca7e4bb7c64221ae2c61144ef5edef
] 
[Prevent mutable objects from being retrieved from an immutable directory, and associated forward-compatibility improvements.
david-sarah@jacaranda.org**20100127064430
 Ignore-this: 5ef6a3554cf6bef0bf0712cc7d6c0252
] 
[test_runner: cleanup, refactor common code into a non-executable method
Brian Warner <warner@lothar.com>**20100127224040
 Ignore-this: 4cb4aada87777771f688edfd8129ffca
 
 Having both test_node() and test_client() (one of which calls the other) felt
 confusing to me, so I changed it to have test_node(), test_client(), and a
 common do_create() helper method.
] 
[scripts/runner.py: simplify David-Sarah's clever grouped-commands usage trick
Brian Warner <warner@lothar.com>**20100127223758
 Ignore-this: 70877ebf06ae59f32960b0aa4ce1d1ae
] 
[tahoe backup: skip all symlinks, with warning. Fixes #850, addresses #641.
Brian Warner <warner@lothar.com>**20100127223517
 Ignore-this: ab5cf05158d32a575ca8efc0f650033f
] 
[NEWS: update with all recent user-visible changes
Brian Warner <warner@lothar.com>**20100127222209
 Ignore-this: 277d24568018bf4f3fb7736fda64eceb
] 
["tahoe backup": fix --exclude-vcs docs to include Git
Brian Warner <warner@lothar.com>**20100127201044
 Ignore-this: 756a58dde21bdc65aa62b81803605b5
] 
[docs: fix references to --no-storage, explanation of [storage] section
Brian Warner <warner@lothar.com>**20100127200956
 Ignore-this: f4be1763a585e1ac6299a4f1b94a59e0
] 
[docs: further CREDITS level-ups for Nils, Kevan, David-Sarah
zooko@zooko.com**20100126170021
 Ignore-this: 1e513e85cf7b7abf57f056e6d7544b38
] 
[Patch to accept t=set-children as well as t=set_children
david-sarah@jacaranda.org**20100124030020
 Ignore-this: 2c061f12af817cdf77feeeb64098ec3a
] 
[Fix boodlegrid use of set_children
david-sarah@jacaranda.org**20100126063414
 Ignore-this: 3aa2d4836f76303b2bacecd09611f999
] 
[ftpd: clearer error message if Twisted needs a patch (by Nils Durner)
zooko@zooko.com**20100126143411
 Ignore-this: 440e6831ae6da5135c1edd081c93871f
] 
[Add 'docs/performance.txt', which (for the moment) describes mutable file performance issues
Kevan Carstensen <kevan@isnotajoke.com>**20100115204500
 Ignore-this: ade4e500217db2509aee35aacc8c5dbf
] 
[docs: more CREDITS for François, Kevan, and David-Sarah
zooko@zooko.com**20100126132133
 Ignore-this: f37d4977c13066fcac088ba98a31b02e
] 
[tahoe_backup.py: display warnings on errors instead of stopping the whole backup. Fix #729.
francois@ctrlaltdel.ch**20100120094249
 Ignore-this: 7006ea4b0910b6d29af6ab4a3997a8f9
 
 This patch displays a warning to the user in two cases:
   
   1. When special files like symlinks, fifos, devices, etc. are found in the
      local source.
   
   2. If files or directories are not readables by the user running the 'tahoe
      backup' command.
 
 In verbose mode, the number of skipped files and directories is printed at the
 end of the backup.
 
 Exit status returned by 'tahoe backup':
 
   - 0 everything went fine
   - 1 the backup failed
   - 2 files were skipped during the backup
 
] 
[Warn about test failures due to setting FLOG* env vars
david-sarah@jacaranda.org**20100124220629
 Ignore-this: 1c25247ca0f0840390a1b7259a9f4a3c
] 
[Message saying that we couldn't find bin/tahoe should say where we looked
david-sarah@jacaranda.org**20100116204556
 Ignore-this: 1068576fd59ea470f1e19196315d1bb
] 
[Change running.html to describe 'tahoe run'
david-sarah@jacaranda.org**20100112044409
 Ignore-this: 23ad0114643ce31b56e19bb14e011e4f
] 
[cli: merge the better version of David-Sarah's split-usage-and-help patch with the earlier version that I mistakenly committed
zooko@zooko.com**20100126044559
 Ignore-this: 284d188e13b7901013cbb650168e6447
] 
[Split tahoe --help options into groups.
david-sarah@jacaranda.org**20100112043935
 Ignore-this: 610f9c41b00e6863e3cd047379733e3a
] 
[cli: split usage strings into groups (patch by David-Sarah Hopwood)
zooko@zooko.com**20100126043921
 Ignore-this: 51928d266a7292b873f87f7d53c9a01e
] 
[Add create-node CLI command, and make create-client equivalent to create-node --no-storage (fixes #760)
david-sarah@jacaranda.org**20100116052055
 Ignore-this: 47d08b18c69738685e13ff365738d5a
] 
[Remove replace= parameter to mkdir-immutable and mkdir-with-children
david-sarah@jacaranda.org**20100124224325
 Ignore-this: 25207bcc946c0c43d9528718e76ba7b
] 
[contrib/fuse/runtests.py: Fix #888, configure settings in tahoe.cfg and don't treat warnings as failure
francois@ctrlaltdel.ch**20100109123010
 Ignore-this: 2590d44044acd7dfa3690c416cae945c
 
 Fix a few bitrotten pieces in the FUSE test script.  It now configures tahoe
 node settings by editing tahoe.cfg which is the new supported method.
 
 It alos tolerate warnings issued by the mount command, the cause of these
 warnings is the same as in #876 (contrib/fuse/runtests.py doesn't tolerate
 deprecations warnings).
 
] 
[Fix webapi t=mkdir with multpart/form-data, as on the Welcome page. Closes #919.
Brian Warner <warner@lothar.com>**20100121065052
 Ignore-this: 1f20ea0a0f1f6d6c1e8e14f193a92c87
] 
[tahoe_add_alias.py: minor refactoring
Brian Warner <warner@lothar.com>**20100115064220
 Ignore-this: 29910e81ad11209c9e493d65fd2dab9b
] 
[test_dirnode.py: reduce scope of a Client instance, suggested by Kevan.
Brian Warner <warner@lothar.com>**20100115062713
 Ignore-this: b35efd9e6027e43de6c6f509bfb4ccaa
] 
[test_provisioning: STAN is not always a list. Fix by David-Sarah Hopwood.
Brian Warner <warner@lothar.com>**20100115014632
 Ignore-this: 9989de7f1e00907706d2b63153138219
] 
[web/directory.py mkdir-immutable: hush pyflakes, add TODO for #903 behavior
Brian Warner <warner@lothar.com>**20100114222804
 Ignore-this: 717cd3b9a1c8aeee76938c9641db7356
] 
[hush pyflakes-0.4.0 warnings: slightly less-trivial fixes. Closes #900.
Brian Warner <warner@lothar.com>**20100114221719
 Ignore-this: f774f4637e256ad55502659413a811a8
 
 This includes one fix (in test_web) which was testing the wrong thing.
] 
[hush pyflakes-0.4.0 warnings: remove trivial unused variables. For #900.
Brian Warner <warner@lothar.com>**20100114221529
 Ignore-this: e96106c8f1a99fbf93306fbfe9a294cf
] 
[tahoe add-alias/create-alias: don't corrupt non-newline-terminated alias
Brian Warner <warner@lothar.com>**20100114210246
 Ignore-this: 9c994792e53a85159d708760a9b1b000
 file. Closes #741.
] 
[change docs and --help to use "grid" instead of "virtual drive": closes #892.
Brian Warner <warner@lothar.com>**20100114201119
 Ignore-this: a20d4a4dcc4de4e3b404ff72d40fc29b
 
 Thanks to David-Sarah Hopwood for the patch.
] 
[backupdb.txt: fix ST_CTIME reference
Brian Warner <warner@lothar.com>**20100114194052
 Ignore-this: 5a189c7a1181b07dd87f0a08ea31b6d3
] 
[client.py: fix/update comments on KeyGenerator
Brian Warner <warner@lothar.com>**20100113004226
 Ignore-this: 2208adbb3fd6a911c9f44e814583cabd
] 
[Clean up log.err calls, for one of the issues in #889.
Brian Warner <warner@lothar.com>**20100112013343
 Ignore-this: f58455ce15f1fda647c5fb25d234d2db
 
 allmydata.util.log.err() either takes a Failure as the first positional
 argument, or takes no positional arguments and must be invoked in an
 exception handler. Fixed its signature to match both foolscap.logging.log.err
 and twisted.python.log.err . Included a brief unit test.
] 
[tidy up DeadReferenceError handling, ignore them in add_lease calls
Brian Warner <warner@lothar.com>**20100112000723
 Ignore-this: 72f1444e826fd0b9db6d318f89603c38
 
 Stop checking separately for ConnectionDone/ConnectionLost, since those have
 been folded into DeadReferenceError since foolscap-0.3.1 . Write
 rrefutil.trap_deadref() in terms of rrefutil.trap_and_discard() to improve
 code coverage.
] 
[NEWS: improve "tahoe backup" notes, mention first-backup-after-upgrade duration
Brian Warner <warner@lothar.com>**20100111190132
 Ignore-this: 10347c590b3375964579ba6c2b0edb4f
 
 Thanks to Francois Deppierraz for the suggestion.
] 
[test_repairer: add (commented-out) test_each_byte, to see exactly what the
Brian Warner <warner@lothar.com>**20100110203552
 Ignore-this: 8e84277d5304752edeff052b97821815
 Verifier misses
 
 The results (described in #819) match our expectations: it misses corruption
 in unused share fields and in most container fields (which are only visible
 to the storage server, not the client). 1265 bytes of a 2753 byte
 share (hosting a 56-byte file with an artifically small segment size) are
 unused, mostly in the unused tail of the overallocated UEB space (765 bytes),
 and the allocated-but-unwritten plaintext_hash_tree (480 bytes).
] 
[repairer: fix some wrong offsets in the randomized verifier tests, debugged by Brian
zooko@zooko.com**20100110203721
 Ignore-this: 20604a609db8706555578612c1c12feb
 fixes #819
] 
[test_repairer: fix colliding basedir names, which caused test inconsistencies
Brian Warner <warner@lothar.com>**20100110084619
 Ignore-this: b1d56dd27e6ab99a7730f74ba10abd23
] 
[repairer: add deterministic test for #819, mark as TODO
zooko@zooko.com**20100110013619
 Ignore-this: 4cb8bb30b25246de58ed2b96fa447d68
] 
[contrib/fuse/runtests.py: Tolerate the tahoe CLI returning deprecation warnings
francois@ctrlaltdel.ch**20100109175946
 Ignore-this: 419c354d9f2f6eaec03deb9b83752aee
 
 Depending on the versions of external libraries such as Twisted of Foolscap,
 the tahoe CLI can display deprecation warnings on stdout.  The tests should
 not interpret those warnings as a failure if the node is in fact correctly
 started.
   
 See http://allmydata.org/trac/tahoe/ticket/859 for an example of deprecation
 warnings.
 
 fixes #876
] 
[contrib: fix fuse_impl_c to use new Python API
zooko@zooko.com**20100109174956
 Ignore-this: 51ca1ec7c2a92a0862e9b99e52542179
 original patch by Thomas Delaet, fixed by François, reviewed by Brian, committed by me
] 
[docs: CREDITS: add David-Sarah to the CREDITS file
zooko@zooko.com**20100109060435
 Ignore-this: 896062396ad85f9d2d4806762632f25a
] 
[mutable/publish: don't loop() right away upon DeadReferenceError. Closes #877
Brian Warner <warner@lothar.com>**20100102220841
 Ignore-this: b200e707b3f13aa8251981362b8a3e61
 
 The bug was that a disconnected server could cause us to re-enter the initial
 loop() call, sending multiple queries to a single server, provoking an
 incorrect UCWE. To fix it, stall the loop() with an eventual.fireEventually()
] 
[immutable/checker.py: oops, forgot some imports. Also hush pyflakes.
Brian Warner <warner@lothar.com>**20091229233909
 Ignore-this: 4d61bd3f8113015a4773fd4768176e51
] 
[mutable repair: return successful=False when numshares<k (thus repair fails),
Brian Warner <warner@lothar.com>**20091229233746
 Ignore-this: d881c3275ff8c8bee42f6a80ca48441e
 instead of weird errors. Closes #874 and #786.
 
 Previously, if the file had 0 shares, this would raise TypeError as it tried
 to call download_version(None). If the file had some shares but fewer than
 'k', it would incorrectly raise MustForceRepairError.
 
 Added get_successful() to the IRepairResults API, to give repair() a place to
 report non-code-bug problems like this.
] 
[node.py/interfaces.py: minor docs fixes
Brian Warner <warner@lothar.com>**20091229230409
 Ignore-this: c86ad6342ef0f95d50639b4f99cd4ddf
] 
[NEWS: fix 1.4.1 announcement w.r.t. add-lease behavior in older releases
Brian Warner <warner@lothar.com>**20091229230310
 Ignore-this: bbbbb9c961f3bbcc6e5dbe0b1594822
] 
[checker: don't let failures in add-lease affect checker results. Closes #875.
Brian Warner <warner@lothar.com>**20091229230108
 Ignore-this: ef1a367b93e4d01298c2b1e6ca59c492
 
 Mutable servermap updates and the immutable checker, when run with
 add_lease=True, send both the do-you-have-block and add-lease commands in
 parallel, to avoid an extra round trip time. Many older servers have problems
 with add-lease and raise various exceptions, which don't generally matter.
 The client-side code was catching+ignoring some of them, but unrecognized
 exceptions were passed through to the DYHB code, concealing the DYHB results
 from the checker, making it think the server had no shares.
 
 The fix is to separate the code paths. Both commands are sent at the same
 time, but the errback path from add-lease is handled separately. Known
 exceptions are ignored, the others (both unknown-remote and all-local) are
 logged (log.WEIRD, which will trigger an Incident), but neither will affect
 the DYHB results.
 
 The add-lease message is sent first, and we know that the server handles them
 synchronously. So when the checker is done, we can be sure that all the
 add-lease messages have been retired. This makes life easier for unit tests.
] 
[test_cli: verify fix for "tahoe get" not creating empty file on error (#121)
Brian Warner <warner@lothar.com>**20091227235444
 Ignore-this: 6444d52413b68eb7c11bc3dfdc69c55f
] 
[addendum to "Fix 'tahoe ls' on files (#771)"
Brian Warner <warner@lothar.com>**20091227232149
 Ignore-this: 6dd5e25f8072a3153ba200b7fdd49491
 
 tahoe_ls.py: tolerate missing metadata
 web/filenode.py: minor cleanups
 test_cli.py: test 'tahoe ls FILECAP'
] 
[Fix 'tahoe ls' on files (#771). Patch adapted from Kevan Carstensen.
Brian Warner <warner@lothar.com>**20091227225443
 Ignore-this: 8bf8c7b1cd14ea4b0ebd453434f4fe07
 
 web/filenode.py: also serve edge metadata when using t=json on a
                  DIRCAP/childname object.
 tahoe_ls.py: list file objects as if we were listing one-entry directories.
              Show edge metadata if we have it, which will be true when doing
              'tahoe ls DIRCAP/filename' and false when doing 'tahoe ls
              FILECAP'
] 
[tahoe_get: don't create the output file on error. Closes #121.
Brian Warner <warner@lothar.com>**20091227220404
 Ignore-this: 58d5e793a77ec6e87d9394ade074b926
] 
[webapi: don't accept zero-length childnames during traversal. Closes #358, #676.
Brian Warner <warner@lothar.com>**20091227201043
 Ignore-this: a9119dec89e1c7741f2289b0cad6497b
 
 This forbids operations that would implicitly create a directory with a
 zero-length (empty string) name, like what you'd get if you did "tahoe put
 local /oops/blah" (#358) or "POST /uri/CAP//?t=mkdir" (#676). The error
 message is fairly friendly too.
 
 Also added code to "tahoe put" to catch this error beforehand and suggest the
 correct syntax (i.e. without the leading slash).
] 
[CLI: send 'Accept:' header to ask for text/plain tracebacks. Closes #646.
Brian Warner <warner@lothar.com>**20091227195828
 Ignore-this: 44c258d4d4c7dac0ed58adb22f73331
 
 The webapi has been looking for an Accept header since 1.4.0, but it treats a
 missing header as equal to */* (to honor RFC2616). This change finally
 modifies our CLI tools to ask for "text/plain, application/octet-stream",
 which seems roughly correct (we either want a plain-text traceback or error
 message, or an uninterpreted chunk of binary data to save to disk). Some day
 we'll figure out how JSON fits into this scheme.
] 
[Makefile: upload-tarballs: switch from xfer-client to flappclient, closes #350
Brian Warner <warner@lothar.com>**20091227163703
 Ignore-this: 3beeecdf2ad9c2438ab57f0e33dcb357
 
 I've also set up a new flappserver on source@allmydata.org to receive the
 tarballs. We still need to replace the gutsy buildslave (which is where the
 tarballs used to be generated+uploaded) and give it the new FURL.
] 
[misc/ringsim.py: make it deterministic, more detail about grid-is-full behavior
Brian Warner <warner@lothar.com>**20091227024832
 Ignore-this: a691cc763fb2e98a4ce1767c36e8e73f
] 
[misc/ringsim.py: tool to discuss #302
Brian Warner <warner@lothar.com>**20091226060339
 Ignore-this: fc171369b8f0d97afeeb8213e29d10ed
] 
[docs: fix helper.txt to describe new config style
zooko@zooko.com**20091224223522
 Ignore-this: 102e7692dc414a4b466307f7d78601fe
] 
[docs/stats.txt: add TOC, notes about controlling gatherer's listening port
Brian Warner <warner@lothar.com>**20091224202133
 Ignore-this: 8eef63b0e18db5aa8249c2eafde02c05
 
 Thanks to Jody Harris for the suggestions.
] 
[Add docs/stats.py, explaining Tahoe stats, the gatherer, and the munin plugins.
Brian Warner <warner@lothar.com>**20091223052400
 Ignore-this: 7c9eeb6e5644eceda98b59a67730ccd5
] 
[more #859: avoid deprecation warning for unit tests too, hush pyflakes
Brian Warner <warner@lothar.com>**20091215000147
 Ignore-this: 193622e24d31077da825a11ed2325fd3
 
 * factor maybe-import-sha logic into util.hashutil
] 
[use hashlib module if available, thus avoiding a DeprecationWarning for importing the old sha module; fixes #859
zooko@zooko.com**20091214212703
 Ignore-this: 8d0f230a4bf8581dbc1b07389d76029c
] 
[docs: reflow architecture.txt to 78-char lines
zooko@zooko.com**20091208232943
 Ignore-this: 88f55166415f15192e39407815141f77
] 
[docs: update the about.html a little
zooko@zooko.com**20091208212737
 Ignore-this: 3fe2d9653c6de0727d3e82bd70f2a8ed
] 
[docs: remove obsolete doc file "codemap.txt"
zooko@zooko.com**20091113163033
 Ignore-this: 16bc21a1835546e71d1b344c06c61ebb
 I started to update this to reflect the current codebase, but then I thought (a) nobody seemed to notice that it hasn't been updated since December 2007, and (b) it will just bit-rot again, so I'm removing it.
] 
[mutable/retrieve.py: stop reaching into private MutableFileNode attributes
Brian Warner <warner@lothar.com>**20091208172921
 Ignore-this: 61e548798c1105aed66a792bf26ceef7
] 
[mutable/servermap.py: stop reaching into private MutableFileNode attributes
Brian Warner <warner@lothar.com>**20091208172608
 Ignore-this: b40a6b62f623f9285ad96fda139c2ef2
] 
[mutable/servermap.py: oops, query N+e servers in MODE_WRITE, not k+e
Brian Warner <warner@lothar.com>**20091208171156
 Ignore-this: 3497f4ab70dae906759007c3cfa43bc
 
 under normal conditions, this wouldn't cause any problems, but if the shares
 are really sparse (perhaps because new servers were added), then
 file-modifies might stop looking too early and leave old shares in place
] 
[control.py: fix speedtest: use download_best_version (not read) on mutable nodes
Brian Warner <warner@lothar.com>**20091207060512
 Ignore-this: 7125eabfe74837e05f9291dd6414f917
] 
[FTP-and-SFTP.txt: fix ssh-keygen pointer
Brian Warner <warner@lothar.com>**20091207052803
 Ignore-this: bc2a70ee8c58ec314e79c1262ccb22f7
] 
[setup: ignore _darcs in the "test-clean" test and make the "clean" step remove all .egg's in the root dir
zooko@zooko.com**20091206184835
 Ignore-this: 6066bd160f0db36d7bf60aba405558d2
] 
[remove MutableFileNode.download(), prefer download_best_version() instead
Brian Warner <warner@lothar.com>**20091201225438
 Ignore-this: 5733eb373a902063e09fd52cc858dec0
] 
[Simplify immutable download API: use just filenode.read(consumer, offset, size)
Brian Warner <warner@lothar.com>**20091201225330
 Ignore-this: bdedfb488ac23738bf52ae6d4ab3a3fb
 
 * remove Downloader.download_to_data/download_to_filename/download_to_filehandle
 * remove download.Data/FileName/FileHandle targets
 * remove filenode.download/download_to_data/download_to_filename methods
 * leave Downloader.download (the whole Downloader will go away eventually)
 * add util.consumer.MemoryConsumer/download_to_data, for convenience
   (this is mostly used by unit tests, but it gets used by enough non-test
    code to warrant putting it in allmydata.util)
 * update tests
 * removes about 180 lines of code. Yay negative code days!
 
 Overall plan is to rewrite immutable/download.py and leave filenode.read() as
 the sole read-side API.
] 
[server.py: undo my bogus 'correction' of David-Sarah's comment fix
Brian Warner <warner@lothar.com>**20091201024607
 Ignore-this: ff4bb58f6a9e045b900ac3a89d6f506a
 
 and move it to a better line
] 
[Implement more coherent behavior when copying with dircaps/filecaps (closes #761). Patch by Kevan Carstensen.
"Brian Warner <warner@lothar.com>"**20091130211009] 
[storage.py: update comment
"Brian Warner <warner@lothar.com>"**20091130195913] 
[storage server: detect disk space usage on Windows too (fixes #637)
david-sarah@jacaranda.org**20091121055644
 Ignore-this: 20fb30498174ce997befac7701fab056
] 
[make status of finished operations consistently "Finished"
david-sarah@jacaranda.org**20091121061543
 Ignore-this: 97d483e8536ccfc2934549ceff7055a3
] 
[NEWS: update with all user-visible changes since the last release
Brian Warner <warner@lothar.com>**20091127224217
 Ignore-this: 741da6cd928e939fb6d21a61ea3daf0b
] 
[update "tahoe backup" docs, and webapi.txt's mkdir-with-children
Brian Warner <warner@lothar.com>**20091127055900
 Ignore-this: defac1fb9a2335b0af3ef9dbbcc67b7e
] 
[Add dirnodes to backupdb and "tahoe backup", closes #606.
Brian Warner <warner@lothar.com>**20091126234257
 Ignore-this: fa88796fcad1763c6a2bf81f56103223
 
 * backups now share dirnodes with any previous backup, in any location,
   so renames and moves are handled very efficiently
 * "tahoe backup" no longer bothers reading the previous snapshot
 * if you switch grids, you should delete ~/.tahoe/private/backupdb.sqlite,
   to force new uploads of all files and directories
] 
[webapi: fix t=check for DIR2-LIT (i.e. empty immutable directories)
Brian Warner <warner@lothar.com>**20091126232731
 Ignore-this: 8513c890525c69c1eca0e80d53a231f8
] 
[PipelineError: fix str() on python2.4 . Closes #842.
Brian Warner <warner@lothar.com>**20091124212512
 Ignore-this: e62c92ea9ede2ab7d11fe63f43b9c942
] 
[test_uri.py: s/NewDirnode/Dirnode/ , now that they aren't "new" anymore
Brian Warner <warner@lothar.com>**20091120075553
 Ignore-this: 61c8ef5e45a9d966873a610d8349b830
] 
[interface name cleanups: IFileNode, IImmutableFileNode, IMutableFileNode
Brian Warner <warner@lothar.com>**20091120075255
 Ignore-this: e3d193c229e2463e1d0b0c92306de27f
 
 The proper hierarchy is:
  IFilesystemNode
  +IFileNode
  ++IMutableFileNode
  ++IImmutableFileNode
  +IDirectoryNode
 
 Also expand test_client.py (NodeMaker) to hit all IFilesystemNode types.
] 
[class name cleanups: s/FileNode/ImmutableFileNode/
Brian Warner <warner@lothar.com>**20091120072239
 Ignore-this: 4b3218f2d0e585c62827e14ad8ed8ac1
 
 also fix test/bench_dirnode.py for recent dirnode changes
] 
[Use DIR-IMM and t=mkdir-immutable for "tahoe backup", for #828
Brian Warner <warner@lothar.com>**20091118192813
 Ignore-this: a4720529c9bc6bc8b22a3d3265925491
] 
[web/directory.py: use "DIR-IMM" to describe immutable directories, not DIR-RO
Brian Warner <warner@lothar.com>**20091118191832
 Ignore-this: aceafd6ab4bf1cc0c2a719ef7319ac03
] 
[web/info.py: hush pyflakes
Brian Warner <warner@lothar.com>**20091118191736
 Ignore-this: edc5f128a2b8095fb20686a75747c8
] 
[make get_size/get_current_size consistent for all IFilesystemNode classes
Brian Warner <warner@lothar.com>**20091118191624
 Ignore-this: bd3449cf96e4827abaaf962672c1665a
 
 * stop caching most_recent_size in dirnode, rely upon backing filenode for it
 * start caching most_recent_size in MutableFileNode
 * return None when you don't know, not "?"
 * only render None as "?" in the web "more info" page
 * add get_size/get_current_size to UnknownNode
] 
[ImmutableDirectoryURIVerifier: fix verifycap handling
Brian Warner <warner@lothar.com>**20091118164238
 Ignore-this: 6bba5c717b54352262eabca6e805d590
] 
[Add t=mkdir-immutable to the webapi. Closes #607.
Brian Warner <warner@lothar.com>**20091118070900
 Ignore-this: 311e5fab9a5f28b9e8a28d3d08f3c0d
 
 * change t=mkdir-with-children to not use multipart/form encoding. Instead,
   the request body is all JSON. t=mkdir-immutable uses this format too.
 * make nodemaker.create_immutable_dirnode() get convergence from SecretHolder,
   but let callers override it
 * raise NotDeepImmutableError instead of using assert()
 * add mutable= argument to DirectoryNode.create_subdirectory(), default True
] 
[move convergence secret into SecretHolder, next to lease secret
Brian Warner <warner@lothar.com>**20091118015444
 Ignore-this: 312f85978a339f2d04deb5bcb8f511bc
] 
[nodemaker: implement immutable directories (internal interface), for #607
Brian Warner <warner@lothar.com>**20091112002233
 Ignore-this: d09fccf41813fdf7e0db177ed9e5e130
 
 * nodemaker.create_from_cap() now handles DIR2-CHK and DIR2-LIT
 * client.create_immutable_dirnode() is used to create them
 * no webapi yet
] 
[stop using IURI()/etc as an adapter
Brian Warner <warner@lothar.com>**20091111224542
 Ignore-this: 9611da7ea6a4696de2a3b8c08776e6e0
] 
[clean up uri-vs-cap terminology, emphasize cap instances instead of URI strings
Brian Warner <warner@lothar.com>**20091111222619
 Ignore-this: 93626385f6e7f039ada71f54feefe267
 
  * "cap" means a python instance which encapsulates a filecap/dircap (uri.py)
  * "uri" means a string with a "URI:" prefix
  * FileNode instances are created with (and retain) a cap instance, and
    generate uri strings on demand
  * .get_cap/get_readcap/get_verifycap/get_repaircap return cap instances
  * .get_uri/get_readonly_uri return uri strings
 
 * add filenode.download_to_filename() for control.py, should find a better way
 * use MutableFileNode.init_from_cap, not .init_from_uri
 * directory URI instances: use get_filenode_cap, not get_filenode_uri
 * update/cleanup bench_dirnode.py to match, add Makefile target to run it
] 
[add parser for immutable directory caps: DIR2-CHK, DIR2-LIT, DIR2-CHK-Verifier
Brian Warner <warner@lothar.com>**20091104181351
 Ignore-this: 854398cc7a75bada57fa97c367b67518
] 
[wui: s/TahoeLAFS/Tahoe-LAFS/
zooko@zooko.com**20091029035050
 Ignore-this: 901e64cd862e492ed3132bd298583c26
] 
[tests: bump up the timeout on test_repairer to see if 120 seconds was too short for François's ARM box to do the test even when it was doing it right.
zooko@zooko.com**20091027224800
 Ignore-this: 95e93dc2e018b9948253c2045d506f56
] 
[dirnode.pack_children(): add deep_immutable= argument
Brian Warner <warner@lothar.com>**20091026162809
 Ignore-this: d5a2371e47662c4bc6eff273e8181b00
 
 This will be used by DIR2:CHK to enforce the deep-immutability requirement.
] 
[webapi: use t=mkdir-with-children instead of a children= arg to t=mkdir .
Brian Warner <warner@lothar.com>**20091026011321
 Ignore-this: 769cab30b6ab50db95000b6c5a524916
 
 This is safer: in the earlier API, an old webapi server would silently ignore
 the initial children, and clients trying to set them would have to fetch the
 newly-created directory to discover the incompatibility. In the new API,
 clients using t=mkdir-with-children against an old webapi server will get a
 clear error.
] 
[nodemaker.create_new_mutable_directory: pack_children() in initial_contents=
Brian Warner <warner@lothar.com>**20091020005118
 Ignore-this: bd43c4eefe06fd32b7492bcb0a55d07e
 instead of creating an empty file and then adding the children later.
 
 This should speed up mkdir(initial_children) considerably, removing two
 roundtrips and an entire read-modify-write cycle, probably bringing it down
 to a single roundtrip. A quick test (against the volunteergrid) suggests a
 30% speedup.
 
 test_dirnode: add new tests to enforce the restrictions that interfaces.py
 claims for create_new_mutable_directory(): no UnknownNodes, metadata dicts
] 
[test_dirnode.py: add tests of initial_children= args to client.create_dirnode
Brian Warner <warner@lothar.com>**20091017194159
 Ignore-this: 2e2da28323a4d5d815466387914abc1b
 and nodemaker.create_new_mutable_directory
] 
[update many dirnode interfaces to accept dict-of-nodes instead of dict-of-caps
Brian Warner <warner@lothar.com>**20091017192829
 Ignore-this: b35472285143862a856bf4b361d692f0
 
 interfaces.py: define INodeMaker, document argument values, change
                create_new_mutable_directory() to take dict-of-nodes. Change
                dirnode.set_nodes() and dirnode.create_subdirectory() too.
 nodemaker.py: use INodeMaker, update create_new_mutable_directory()
 client.py: have create_dirnode() delegate initial_children= to nodemaker
 dirnode.py (Adder): take dict-of-nodes instead of list-of-nodes, which
                     updates set_nodes() and create_subdirectory()
 web/common.py (convert_initial_children_json): create dict-of-nodes
 web/directory.py: same
 web/unlinked.py: same
 test_dirnode.py: update tests to match
] 
[dirnode.py: move pack_children() out to a function, for eventual use by others
Brian Warner <warner@lothar.com>**20091017180707
 Ignore-this: 6a823fb61f2c180fd38d6742d3196a7a
] 
[move dirnode.CachingDict to dictutil.AuxValueDict, generalize method names,
Brian Warner <warner@lothar.com>**20091017180005
 Ignore-this: b086933cf429df0fcea16a308d2640dd
 improve tests. Let dirnode _pack_children accept either dict or AuxValueDict.
] 
[test/common.py: update FakeMutableFileNode to new contents= callable scheme
Brian Warner <warner@lothar.com>**20091013052154
 Ignore-this: 62f00a76454a2190d1c8641c5993632f
] 
[The initial_children= argument to nodemaker.create_new_mutable_directory is
Brian Warner <warner@lothar.com>**20091013031922
 Ignore-this: 72e45317c21f9eb9ec3bd79bd4311f48
 now enabled.
] 
[client.create_mutable_file(contents=) now accepts a callable, which is
Brian Warner <warner@lothar.com>**20091013031232
 Ignore-this: 3c89d2f50c1e652b83f20bd3f4f27c4b
 invoked with the new MutableFileNode and is supposed to return the initial
 contents. This can be used by e.g. a new dirnode which needs the filenode's
 writekey to encrypt its initial children.
 
 create_mutable_file() still accepts a bytestring too, or None for an empty
 file.
] 
[webapi: t=mkdir now accepts initial children, using the same JSON that t=json
Brian Warner <warner@lothar.com>**20091013023444
 Ignore-this: 574a46ed46af4251abf8c9580fd31ef7
 emits.
 
 client.create_dirnode(initial_children=) now works.
] 
[replace dirnode.create_empty_directory() with create_subdirectory(), which
Brian Warner <warner@lothar.com>**20091013021520
 Ignore-this: 6b57cb51bcfcc6058d0df569fdc8a9cf
 takes an initial_children= argument
] 
[dirnode.set_children: change return value: fire with self instead of None
Brian Warner <warner@lothar.com>**20091013015026
 Ignore-this: f1d14e67e084e4b2a4e25fa849b0e753
] 
[dirnode.set_nodes: change return value: fire with self instead of None
Brian Warner <warner@lothar.com>**20091013014546
 Ignore-this: b75b3829fb53f7399693f1c1a39aacae
] 
[dirnode.set_children: take a dict, not a list
Brian Warner <warner@lothar.com>**20091013002440
 Ignore-this: 540ce72ce2727ee053afaae1ff124e21
] 
[dirnode.set_uri/set_children: change signature to take writecap+readcap
Brian Warner <warner@lothar.com>**20091012235126
 Ignore-this: 5df617b2d379a51c79148a857e6026b1
 instead of a single cap. The webapi t=set_children call benefits too.
] 
[replace Client.create_empty_dirnode() with create_dirnode(), in anticipation
Brian Warner <warner@lothar.com>**20091012224506
 Ignore-this: cbdaa4266ecb3c6496ffceab4f95709d
 of adding initial_children= argument.
 
 Includes stubbed-out initial_children= support.
] 
[test_web.py: use a less-fake client, making test harness smaller
Brian Warner <warner@lothar.com>**20091012222808
 Ignore-this: 29e95147f8c94282885c65b411d100bb
] 
[webapi.txt: document t=set_children, other small edits
Brian Warner <warner@lothar.com>**20091009200446
 Ignore-this: 4d7e76b04a7b8eaa0a981879f778ea5d
] 
[Verifier: check the full cryptext-hash tree on each share. Removed .todos
Brian Warner <warner@lothar.com>**20091005221849
 Ignore-this: 6fb039c5584812017d91725e687323a5
 from the last few test_repairer tests that were waiting on this.
] 
[Verifier: check the full block-hash-tree on each share
Brian Warner <warner@lothar.com>**20091005214844
 Ignore-this: 3f7ccf6d253f32340f1bf1da27803eee
 
 Removed the .todo from two test_repairer tests that check this. The only
 remaining .todos are on the three crypttext-hash-tree tests.
] 
[Verifier: check the full share-hash chain on each share
Brian Warner <warner@lothar.com>**20091005213443
 Ignore-this: 3d30111904158bec06a4eac22fd39d17
 
 Removed the .todo from two test_repairer tests that check this.
] 
[test_repairer: rename Verifier test cases to be more precise and less verbose
Brian Warner <warner@lothar.com>**20091005201115
 Ignore-this: 64be7094e33338c7c2aea9387e138771
] 
[immutable/checker.py: rearrange code a little bit, make it easier to follow
Brian Warner <warner@lothar.com>**20091005200252
 Ignore-this: 91cc303fab66faf717433a709f785fb5
] 
[test/common.py: wrap docstrings to 80cols so I can read them more easily
Brian Warner <warner@lothar.com>**20091005200143
 Ignore-this: b180a3a0235cbe309c87bd5e873cbbb3
] 
[immutable/download.py: wrap to 80cols, no functional changes
Brian Warner <warner@lothar.com>**20091005192542
 Ignore-this: 6b05fe3dc6d78832323e708b9e6a1fe
] 
[CHK-hashes.svg: cross out plaintext hashes, since we don't include
Brian Warner <warner@lothar.com>**20091005010803
 Ignore-this: bea2e953b65ec7359363aa20de8cb603
 them (until we finish #453)
] 
[docs: a few licensing clarifications requested by Ubuntu
zooko@zooko.com**20090927033226
 Ignore-this: 749fc8c9aeb6dc643669854a3e81baa7
] 
[setup: remove binary WinFUSE modules
zooko@zooko.com**20090924211436
 Ignore-this: 8aefc571d2ae22b9405fc650f2c2062
 I would prefer to have just source code, or indications of what 3rd-party packages are required, under revision control, and have the build process generate o
 r acquire the binaries as needed.  Also, having these in our release tarballs is interfering with getting Tahoe-LAFS uploaded into Ubuntu Karmic.  (Technicall
 y, they would accept binary modules as long as they came with the accompanying source so that they could satisfy their obligations under GPL2+ and TGPPL1+, bu
 t it is easier for now to remove the binaries from the source tree.)
 In this case, the binaries are from the tahoe-w32-client project: http://allmydata.org/trac/tahoe-w32-client , from which you can also get the source.
] 
[setup: remove binary _fusemodule.so 's
zooko@zooko.com**20090924211130
 Ignore-this: 74487bbe27d280762ac5dd5f51e24186
 I would prefer to have just source code, or indications of what 3rd-party packages are required, under revision control, and have the build process generate or acquire the binaries as needed.  Also, having these in our release tarballs is interfering with getting Tahoe-LAFS uploaded into Ubuntu Karmic.  (Technically, they would accept binary modules as long as they came with the accompanying source so that they could satisfy their obligations under GPL2+ and TGPPL1+, but it is easier for now to remove the binaries from the source tree.)
 In this case, these modules come from the MacFUSE project: http://code.google.com/p/macfuse/
] 
[doc: add a copy of LGPL2 for documentation purposes for ubuntu
zooko@zooko.com**20090924054218
 Ignore-this: 6a073b48678a7c84dc4fbcef9292ab5b
] 
[setup: remove a convenience copy of figleaf, to ease inclusion into Ubuntu Karmic Koala
zooko@zooko.com**20090924053215
 Ignore-this: a0b0c990d6e2ee65c53a24391365ac8d
 We need to carefully document the licence of figleaf in order to get Tahoe-LAFS into Ubuntu Karmic Koala.  However, figleaf isn't really a part of Tahoe-LAFS per se -- this is just a "convenience copy" of a development tool.  The quickest way to make Tahoe-LAFS acceptable for Karmic then, is to remove figleaf from the Tahoe-LAFS tarball itself.  People who want to run figleaf on Tahoe-LAFS (as everyone should want) can install figleaf themselves.  I haven't tested this -- there may be incompatibilities between upstream figleaf and the copy that we had here...
] 
[setup: shebang for misc/build-deb.py to fail quickly
zooko@zooko.com**20090819135626
 Ignore-this: 5a1b893234d2d0bb7b7346e84b0a6b4d
 Without this patch, when I ran "chmod +x ./misc/build-deb.py && ./misc/build-deb.py" then it hung indefinitely.  (I wonder what it was doing.)
] 
[docs: Shawn Willden grants permission for his contributions under GPL2+|TGPPL1+
zooko@zooko.com**20090921164651
 Ignore-this: ef1912010d07ff2ffd9678e7abfd0d57
] 
[docs: Csaba Henk granted permission to license fuse.py under the same terms as Tahoe-LAFS itself
zooko@zooko.com**20090921154659
 Ignore-this: c61ba48dcb7206a89a57ca18a0450c53
] 
[setup: mark setup.py as having utf-8 encoding in it
zooko@zooko.com**20090920180343
 Ignore-this: 9d3850733700a44ba7291e9c5e36bb91
] 
[doc: licensing cleanups
zooko@zooko.com**20090920171631
 Ignore-this: 7654f2854bf3c13e6f4d4597633a6630
 Use nice utf-8 © instead of "(c)". Remove licensing statements on utility modules that have been assigned to allmydata.com by their original authors. (Nattraverso was not assigned to allmydata.com -- it was LGPL'ed -- but I checked and src/allmydata/util/iputil.py was completely rewritten and doesn't contain any line of code from nattraverso.)  Add notes to misc/debian/copyright about licensing on files that aren't just allmydata.com-licensed.
] 
[build-deb.py: run darcsver early, otherwise we get the wrong version later on
Brian Warner <warner@lothar.com>**20090918033620
 Ignore-this: 6635c5b85e84f8aed0d8390490c5392a
] 
[new approach for debian packaging, sharing pieces across distributions. Still experimental, still only works for sid.
warner@lothar.com**20090818190527
 Ignore-this: a75eb63db9106b3269badbfcdd7f5ce1
] 
[new experimental deb-packaging rules. Only works for sid so far.
Brian Warner <warner@lothar.com>**20090818014052
 Ignore-this: 3a26ad188668098f8f3cc10a7c0c2f27
] 
[setup.py: read _version.py and pass to setup(version=), so more commands work
Brian Warner <warner@lothar.com>**20090818010057
 Ignore-this: b290eb50216938e19f72db211f82147e
 like "setup.py --version" and "setup.py --fullname"
] 
[test/check_speed.py: fix shbang line
Brian Warner <warner@lothar.com>**20090818005948
 Ignore-this: 7f3a37caf349c4c4de704d0feb561f8d
] 
[setup: remove bundled version of darcsver-1.2.1
zooko@zooko.com**20090816233432
 Ignore-this: 5357f26d2803db2d39159125dddb963a
 That version of darcsver emits a scary error message when the darcs executable or the _darcs subdirectory is not found.
 This error is hidden (unless the --loud option is passed) in darcsver >= 1.3.1.
 Fixes #788.
] 
[de-Service-ify Helper, pass in storage_broker and secret_holder directly.
Brian Warner <warner@lothar.com>**20090815201737
 Ignore-this: 86b8ac0f90f77a1036cd604dd1304d8b
 This makes it more obvious that the Helper currently generates leases with
 the Helper's own secrets, rather than getting values from the client, which
 is arguably a bug that will likely be resolved with the Accounting project.
] 
[immutable.Downloader: pass StorageBroker to constructor, stop being a Service
Brian Warner <warner@lothar.com>**20090815192543
 Ignore-this: af5ab12dbf75377640a670c689838479
 child of the client, access with client.downloader instead of
 client.getServiceNamed("downloader"). The single "Downloader" instance is
 scheduled for demolition anyways, to be replaced by individual
 filenode.download calls.
] 
[tests: double the timeout on test_runner.RunNode.test_introducer since feisty hit a timeout
zooko@zooko.com**20090815160512
 Ignore-this: ca7358bce4bdabe8eea75dedc39c0e67
 I'm not sure if this is an actual timing issue (feisty is running on an overloaded VM if I recall correctly), or it there is a deeper bug.
] 
[stop making History be a Service, it wasn't necessary
Brian Warner <warner@lothar.com>**20090815114415
 Ignore-this: b60449231557f1934a751c7effa93cfe
] 
[Overhaul IFilesystemNode handling, to simplify tests and use POLA internally.
Brian Warner <warner@lothar.com>**20090815112846
 Ignore-this: 1db1b9c149a60a310228aba04c5c8e5f
 
 * stop using IURI as an adapter
 * pass cap strings around instead of URI instances
 * move filenode/dirnode creation duties from Client to new NodeMaker class
 * move other Client duties to KeyGenerator, SecretHolder, History classes
 * stop passing Client reference to dirnode/filenode constructors
   - pass less-powerful references instead, like StorageBroker or Uploader
 * always create DirectoryNodes by wrapping a filenode (mutable for now)
 * remove some specialized mock classes from unit tests
 
 Detailed list of changes (done one at a time, then merged together)
 
 always pass a string to create_node_from_uri(), not an IURI instance
 always pass a string to IFilesystemNode constructors, not an IURI instance
 stop using IURI() as an adapter, switch on cap prefix in create_node_from_uri()
 client.py: move SecretHolder code out to a separate class
 test_web.py: hush pyflakes
 client.py: move NodeMaker functionality out into a separate object
 LiteralFileNode: stop storing a Client reference
 immutable Checker: remove Client reference, it only needs a SecretHolder
 immutable Upload: remove Client reference, leave SecretHolder and StorageBroker
 immutable Repairer: replace Client reference with StorageBroker and SecretHolder
 immutable FileNode: remove Client reference
 mutable.Publish: stop passing Client
 mutable.ServermapUpdater: get StorageBroker in constructor, not by peeking into Client reference
 MutableChecker: reference StorageBroker and History directly, not through Client
 mutable.FileNode: removed unused indirection to checker classes
 mutable.FileNode: remove Client reference
 client.py: move RSA key generation into a separate class, so it can be passed to the nodemaker
 move create_mutable_file() into NodeMaker
 test_dirnode.py: stop using FakeClient mockups, use NoNetworkGrid instead. This simplifies the code, but takes longer to run (17s instead of 6s). This should come down later when other cleanups make it possible to use simpler (non-RSA) fake mutable files for dirnode tests.
 test_mutable.py: clean up basedir names
 client.py: move create_empty_dirnode() into NodeMaker
 dirnode.py: get rid of DirectoryNode.create
 remove DirectoryNode.init_from_uri, refactor NodeMaker for customization, simplify test_web's mock Client to match
 stop passing Client to DirectoryNode, make DirectoryNode.create_with_mutablefile the normal DirectoryNode constructor, start removing client from NodeMaker
 remove Client from NodeMaker
 move helper status into History, pass History to web.Status instead of Client
 test_mutable.py: fix minor typo
] 
[docs: edits for docs/running.html from Sam Mason
zooko@zooko.com**20090809201416
 Ignore-this: 2207e80449943ebd4ed50cea57c43143
] 
[docs: install.html: instruct Debian users to use this document and not to go find the DownloadDebianPackages page, ignore the warning at the top of it, and try it
zooko@zooko.com**20090804123840
 Ignore-this: 49da654f19d377ffc5a1eff0c820e026
 http://allmydata.org/pipermail/tahoe-dev/2009-August/002507.html
] 
[docs: relnotes.txt: reflow to 63 chars wide because google groups and some web forms seem to wrap to that
zooko@zooko.com**20090802135016
 Ignore-this: 53b1493a0491bc30fb2935fad283caeb
] 
[docs: about.html: fix English usage noticed by Amber
zooko@zooko.com**20090802050533
 Ignore-this: 89965c4650f9bd100a615c401181a956
] 
[docs: fix mis-spelled word in about.html
zooko@zooko.com**20090802050320
 Ignore-this: fdfd0397bc7cef9edfde425dddeb67e5
] 
[TAG allmydata-tahoe-1.5.0
zooko@zooko.com**20090802031303
 Ignore-this: 94e5558e7225c39a86aae666ea00f166
] 
Patch bundle hash:
f3394b6a4ddd6b67b78458401b5f84474f17f5be