Tue Jun 16 20:23:25 MDT 2009 Shawn Willden * Fix missing size for JSON of mutable files New patches: [Fix missing size for JSON of mutable files Shawn Willden **20090617022325 Ignore-this: ed4f17772853593f225b494aeb87f44b ] { hunk ./src/allmydata/web/filenode.py 181 d.addCallback(lambda dn: FileDownloader(dn, filename)) return d if t == "json": - return FileJSONMetadata(ctx, self.node) + if self.node.is_mutable(): + d = defer.maybeDeferred(self.node.get_size_of_best_version) + d.addCallback(lambda sz: FileJSONMetadata(ctx, self.node, sz)) + return d + else: + return FileJSONMetadata(ctx, self.node) if t == "info": return MoreInfo(self.node) if t == "uri": hunk ./src/allmydata/web/filenode.py 417 return req.deferred -def FileJSONMetadata(ctx, filenode): +def FileJSONMetadata(ctx, filenode, best_size = None): if filenode.is_readonly(): rw_uri = None ro_uri = filenode.get_uri() hunk ./src/allmydata/web/filenode.py 425 rw_uri = filenode.get_uri() ro_uri = filenode.get_readonly_uri() data = ("filenode", {}) - data[1]['size'] = filenode.get_size() + if best_size: + data[1]['size'] = best_size + else: + data[1]['size'] = filenode.get_size() if ro_uri: data[1]['ro_uri'] = ro_uri if rw_uri: } Context: [tests: bump up a timeout that expired on Zandr's box zooko@zooko.com**20090613195703 Ignore-this: 853000f43665396b3734a99f72bd472d ] [util: hooray! A clean implementation of this simple utility! Black Dew pointed out that the inverse of time.gmtime() is hidden in the "calendar" module. zooko@zooko.com**20090613160112 Ignore-this: 8359cc48ca1b8e2793e8b2afe2050cf4 ] [util: Brian's horrible hack to figure out how much localtime and utctime differ. Now we'll see if it works on Windows. zooko@zooko.com**20090612204556 Ignore-this: 8c36431da4707da76472956c7750ecbd ] [util: oops, time.tzset() doesn't work on Windows -- hopefully the new "London" unit test passes on Windows when we skip tzset() on platforms that don't have it zooko@zooko.com**20090612000920 Ignore-this: 7d314b8334cfa3f65f9635e3d3eb727e ] [setup: edit install.html to warn Windows users away from Python v2.6 zooko@zooko.com**20090611225506 Ignore-this: 89ad63eab49ede883ef92f2de5b5fc54 ] [util: fix time_format.iso_utc_time_to_seconds() so that it works even in London zooko@zooko.com**20090611221129 Ignore-this: 14dbb9840587797f848226fdb3645c08 ] [setup: run the same "make quicktest" on Windows as on non-Windows zooko@zooko.com**20090611193214 Ignore-this: d0c78377d3892373ec3d91e9e98bd8bd I checked and it behaves about as well on Windows as the previous version did. ] [tests: significantly increase timeouts that triggered on Zandr's ARM box zooko@zooko.com**20090610161043 Ignore-this: 2b3c556c5166a8267b4b15664d3aadfb ] [test: multiple by 10 or so all timeouts that Zandr's ARM box just overran zooko@zooko.com**20090610125639 Ignore-this: bf62f063ab46814fd78de55a5fbc9d84 ] [tests: bump up timeout on a test that timed out on draco zooko@zooko.com**20090610044628 Ignore-this: f598b98cbae44dc947937c6ca54c10cb ] [tests: raise the timeout for test_cli since Zandr's ARM machine totally burst through the old one zooko@zooko.com**20090609210509] [test_cli.Backup: increase timeout massively, it takes 1200s on zandr's ARM linkstation warner@lothar.com**20090609052801] [tests: double the timeouts on some tests which time-out on Francois's box zooko@zooko.com**20090609021753 Ignore-this: b2727b04402f24a9b9123d2f84068106 ] [tests: bump up timeouts so that the tests can finish before timeout on Francois's little arm box zooko@zooko.com**20090608225557 Ignore-this: fb83698338b2f12546cd3e1dcb896d34 ] [tests: increase timeouts on some other tests that timed-out on Francois's arm box zooko@zooko.com**20090605143437 Ignore-this: 2903cc20d914fc074c8d7a6c47740ba6 ] [tests: bump up the timeout on a bunch of tests that took longer than the default timeout (120s) on François Lenny-armv5tel zooko@zooko.com**20090605031444 Ignore-this: 84d67849b1f8edc88bf7001e31b5f7f3 ] [backup: remove the --no-backupdb command, the handling of "can't import sqlite", and the related tests, and change an error message to more correctly indicate failure to load the database from disk rather than failure to import sqlite module zooko@zooko.com**20090604173131 Ignore-this: 8200a9fdfc49243c280ecd1d0c44fa19 Fixes #728. ] [setup: require pysqlite >= v2.0.5. if we are running on Python < 2.5 zooko@zooko.com**20090604154548 Ignore-this: cf04f46079821df209d01dad2e24b40b ] [setup: add pysqlite and sqlite to get_package_versions() zooko@zooko.com**20090604153728 Ignore-this: a1dea7fabeab2b08fb0d8d462facdb4d ] [more refactoring: move get_all_serverids() and get_nickname_for_serverid() from Client to storage_broker warner@lothar.com**20090602030750] [more storage_broker refactoring: downloader gets a broker instead of a client, warner@lothar.com**20090602022511 use Client.get_storage_broker() accessor instead of direct attribute access. ] [test_runner.py: remove test_client_no_noise: the issue in question is warner@lothar.com**20090601225007 ticketed in http://divmod.org/trac/ticket/2830 and doesn't need a Tahoe-side change, plus this test fails on win32 for unrelated reasons (and test_client is the place to think about the win32 issue). ] [remove plaintext-hashing code from the helper interface, to close #722 warner@lothar.com**20090601224916 and deny the Helper the ability to mount a partial-information-guessing attack. This will probably break compatibility between new clients and very old (pre-1.0) helpers. ] [start to factor server-connection-management into a distinct 'StorageServerFarmBroker' object, separate from the client and the introducer. This is the starting point for #467: static server selection warner@lothar.com**20090601210604] [mutable: catch and display first error, so code bugs which break all servers get displayed better warner@lothar.com**20090601210407] [misc/run-with-pythonpath.py: exec() the child (on unix), to remove the intermediate process warner@lothar.com**20090601210137] [docs: small edit to about.html zooko@zooko.com**20090528233422 Ignore-this: 1cfbb1f8426ed6d63b2d3952e4464ddc ] [docs: add links to Tahoe-LAFS for Paranoids and Tahoe-LAFS for Corporates in about.html zooko@zooko.com**20090528232717 Ignore-this: 7b70baa700d6b6f6e9ceec4132efe5 ] [docs: edit about.html and include network-and-reliance-topology.png (loaded from http://allmydata.org ) zooko@zooko.com**20090527150916 Ignore-this: 44adc61cde8ced8be2f0a7dfc7d95dad ] [docs: a few more edits to network-and-reliance-topology.svg zooko@zooko.com**20090527150458 Ignore-this: 2eac8c33fe71be25ff809b399c6193c1 ] [docs: update network-and-reliance-topology.svg for beauty and clarity zooko@zooko.com**20090527031123 Ignore-this: 5510914849771900ac29b4312470d84 ] [docs: update NEWS, relnotes.txt, CREDITS to mention WUI Style zooko@zooko.com**20090526233654 Ignore-this: 72d16ec833bc4a22af23d29ea1d5ff8b ] [Modify markup of Tahoe web pages to be more amenable to styling; some minor changes of wording. Kevin Reid **20090526232545 Ignore-this: 8845937f0df6c7ddc07abe3211428a6f ] [Tweak wording in directory page: not-read-only is "modifiable", mention creating a directory _in this directory_. Kevin Reid **20090526232414 Ignore-this: f006ec52ba2051802e025a60bcface56 ] [Comment on duplication of code/markup found during styling project. Kevin Reid **20090503203442 Ignore-this: a4b7f9f0ab57d2c03be9ba761be8d854 ] [Add CSS styles to spiff up the Tahoe WUI's appearance, particularly the welcome page and directories. Kevin Reid **20090503203142 Ignore-this: 5c50af241c1a958b5180ef2b6a49f626 ] [Link all Tahoe web pages to the /tahoe_css stylesheet which already exists. Kevin Reid **20090503202533 Ignore-this: 2ea8d14d3168b9502cf39d5ea3f2f2a8 ] [Fix broken link from Provisioning to Reliability page. Kevin Reid **20090501191050 Ignore-this: 56dc1a5e659b70cc02dc4df7b5d518cd ] [docs: network-and-reliance-topology.svg: nicer server icons, mv out of the "specifications" subdir zooko@zooko.com**20090526165842 Ignore-this: 8f47ab3a0ab782c1f0d46e10bcaebe5b ] [docs: update network-and-reliance-topology.svg zooko@zooko.com**20090526163105 Ignore-this: 2b864b4ed8743d4a15dfbb7eff3fa561 ] [accounting-overview.txt: more edits warner@lothar.com**20090523190359] [accounting-overview.txt: small edits warner@lothar.com**20090523184011] [_auto_deps.py: require foolscap-0.4.1, which adds an important fix for py2.4 warner@lothar.com**20090523011103] [immutable/encode.py: tolerate immediate _remove_shareholder by copying the warner@lothar.com**20090522184424 landlord list before iterating over it. This can probably only happen in unit tests, but cleaning it up makes certain test failures easier to analyze. ] [switch to using RemoteException instead of 'wrapped' RemoteReferences. Should fix #653, the rref-EQ problem warner@lothar.com**20090522004632] [switch all foolscap imports to use foolscap.api or foolscap.logging warner@lothar.com**20090522003823] [_auto_deps.py: bump our foolscap dependency to 0.4.0, since I'm about to start using its new features warner@lothar.com**20090522002100] [test_runner.py: fix minor typo warner@lothar.com**20090520033620] [setup: fix bug (wrong import) in error message, as noticed by pyflakes zooko@zooko.com**20090519195642 Ignore-this: f1b9f8c00b46c1b5f2f20e5fc424f341 ] [setup: fix trivial bug in recent patch to test base64.py at startup zooko@zooko.com**20090519195129 Ignore-this: f6be038f74b53ca69e7109fe34adfbc ] [setup: make Tahoe exit at startup with a useful error message if the base64.py module is buggy (fixes part of #710) zooko@zooko.com**20090519194555 Ignore-this: aa4d398235ddca8d417d61c9688e154 ] [test_introducer.py: add a test for the python2.4.0/2.4.1 bug in base64.b32decode warner@lothar.com**20090519034101] [immutable WriteBucketProxy: use pipeline to speed up uploads by overlapping roundtrips, for #392 warner@lothar.com**20090518234422] [util/pipeline.py: new utility class to manage size-limited work pipelines, for #392 warner@lothar.com**20090518234326] [docs: add a diagram that I'm about to show to the Boulder Linux Users Group: network-and-reliance-topology.svg zooko@zooko.com**20090514232059 Ignore-this: 2420c0a7c254c9f0f2349d9130490d33 ] [tests: mark test_runner as coded in utf-8 instead of ascii zooko@zooko.com**20090507223151 Ignore-this: ccf1ba9e5a9b53602701a36f9fdb545e ] [tests: raise timeout on test_runner.RunNode.test_introducer from 120s to 240s, since it hit the 120s time-out on François Lenny-armv5tel zooko@zooko.com**20090507215012 Ignore-this: ba18fe6832ba255d4971e8f623ed7da5 ] [setup: fix comment in setup.py zooko@zooko.com**20090507215003 Ignore-this: c46ef664630d52733138ef7fbc551c1c ] [docs: how_to_make_a_tahoe_release.txt: a couple of small edits zooko@zooko.com**20090507214932 Ignore-this: ae92aa835ad369f4b9e6e49d681957a3 ] [.darcs-boringfile: also ignore .gitignore warner@allmydata.com**20090415210550 Ignore-this: d29db314a1e506f6240859559436b4c3 ] [.darcs-boringfile: ignore .git, I'm starting to play around with it warner@allmydata.com**20090415205929 Ignore-this: 89234453516483c9586cd6e1351e88b5 ] [fix quicktest: stop using setuptools, add misc/run-with-pythonpath.py, to make it run faster warner@lothar.com**20090414201400] [TAG allmydata-tahoe-1.4.1 zooko@zooko.com**20090414025636 Ignore-this: de78fc32364c83e9f4e26b5abcfdea4a ] [docs: inaugurate Tahoe-1.4.1, since I left out a handful of patches from the Tahoe-1.4.0 release zooko@zooko.com**20090414025430 Ignore-this: 12d5ff7dc842668bdf318c5539272089 ] [docs: add "darcs pull" to how_to_make_a_tahoe_release.txt, and renumber zooko@zooko.com**20090414024342 Ignore-this: d54d33e5f7e170eea12b6d6a16d0fc87 ] [TAG allmydata-tahoe-1.4.0 zooko@zooko.com**20090414022015 Ignore-this: 6e19652fa9775baf6e97fddf6c0313f2 ] [docs: edit relnotes.txt and promote Tahoe from filesystem to cloud storage thingie zooko@zooko.com**20090414021913 Ignore-this: 78cc79078c234d0467f6290dcae456b9 ] [docs: add note about pywin32 to install.html zooko@zooko.com**20090413185210 Ignore-this: d386abfccfdc1015b8f1216d95b4792f ] [trivial: fix comment zooko@zooko.com**20090413174138 Ignore-this: d45a9786c44793bc830dab1d8c9dd57c ] [docs: a few edits/updates about dirnodes zooko@zooko.com**20090413160837 Ignore-this: 107fc1796e6c7f5b68b7e2517cce516a ] [docs: change install.html to point to the 1.4.0 release zip file instead of 1.3.0 zooko@zooko.com**20090413160649 Ignore-this: ec6b177d6689894b9842a012da04e5dc ] [docs: edit install.html regarding versions of Python zooko@zooko.com**20090413160612 Ignore-this: 1de165ad7645be32ef671ece3fcae9ea ] [docs: some small edits to install.html zooko@zooko.com**20090413160414 Ignore-this: 1e7142ea444fef61c684c089407d675 ] [NEWS: mention that the new 'reliability' page requires numpy warner@lothar.com**20090413110446] [docs: update relnotes.txt, NEWS for Tahoe-1.4.0 release! zooko@zooko.com**20090413041405 Ignore-this: d2eacb26b359a020956ee14b63d95dc5 ] [tests: remove obsolete test (it tests functionality that is long gone) which occasionally ERRORs now that we have more metadata (since [20090411225205-92b7f-7adfb89cb4db4ac7d28427934dea3d2c108f6476]) zooko@zooko.com**20090413023621 Ignore-this: b9f1b1225015f59ffd7e0ee1633e4098 ] [trivial: remove trailing whitespace and unused import zooko@zooko.com**20090412021742 Ignore-this: e249172dd0ef51ee034819bc4c62cd9d ] [dirnode: add 'tahoe'/'linkcrtime' and 'tahoe'/'linkmotime' to take the place of what 'mtime'/'ctime' originally did, and make the 'tahoe' subdict be unwritable through the set_children API zooko@zooko.com**20090411225205 Ignore-this: b48b0812f353891c62f371bedb3e9880 Also add extensive documentation in docs/frontends/webapi.txt about the behaviors of these values. See ticket #628. ] [docs: webapi.txt edits to explain a few things better, adjust indentation, editing zooko@zooko.com**20090411224828 Ignore-this: 3a8a5559c7f0c8a585a4c0b5a2c80451 ] [rollback [20090226150237-b2345-1e916a746a7f4627b050f02f0e442fae5caf69d4] for 1.4.0 release; #645 zooko@zooko.com**20090411181906 Ignore-this: 15aa9ce6d1d49e9447f32e233d136bab ] [setup: update comments about what versions we require of our dependencies zooko@zooko.com**20090410174345 Ignore-this: 7ec194dea9d1a60aa584aebe4ee28064 ] [NEWS: finish formatting new items. Still needs proofreading and updating the release name+date warner@lothar.com**20090410015655] [#622: disable 'Repair' button on check-results page until we make it work correctly warner@lothar.com**20090409235959] [web: make sure that PUT /uri?mutable=false really means immutable, fixes #675 warner@lothar.com**20090408021340] [NEWS: format some (but not all) items warner@lothar.com**20090407211514] [wui: edit some of the human-readable parts of the wui such as button labels zooko@zooko.com**20090407185459 Ignore-this: 145722f4627271ea1d43107a0c7ce0e1 (The word "parent" suggests that you can go up a directory hierarchy -- perhaps that word is vestigial.) ] [wui: fix bug in which empty directory is marked as "unreadable", add test, remove exclamation point zooko@zooko.com**20090407182834 Ignore-this: 2623a7ecd9c7c46b3c984fbaddf43ad0 ] [NEWS: add unformatted list of code changes since previous release warner@lothar.com**20090407021155] [docs: modify how-to-relase notes a tiny bit warner@lothar.com**20090407021135] [hashtree.py: another micro-optimization, expand on a comment warner@lothar.com**20090404043527] [hashtree: fix tests of depth_of zooko@zooko.com**20090403235614 Ignore-this: 70c96e407fe732fe75335c15df5f8200 ] [trivial: source code metadata zooko@zooko.com**20090403233315 Ignore-this: 23858d0320b9b7ba0e6d2fe4adeccea8 ] [trivial: whitespace zooko@zooko.com**20090403233302 Ignore-this: fed1c44b08c699cfcaf9d2d50353ec10 ] [hashtree: a couple of tiny improvements zooko@zooko.com**20090403233037 Ignore-this: a74764aa40b36f7056ac4321af628714 * calculate depth-first with math instead of traversing the actual tree * don't mark a node with a red dot if you instead compare it with an extant hash value (tiny optimization) * edit a comment about checking the root node ] [leases, time_format: modify time stamping in lease description zooko@zooko.com**20090403225904 Ignore-this: c20b18d117a8199cdec08d8b5991789e * emit lease expiry date in ISO-8601'ish format as well as Brian's format * rename iso_utc_time_to_localseconds() to iso_utc_time_to_seconds() * add iso_utc_date() * simplify the body of iso_utc_time_to_seconds() ] [hashtree: fix O(N**2) behavior, to improve fatal alacrity problems in a 10GB file (#670). Also improve docstring. warner@lothar.com**20090331202127 Ignore-this: a04f72ed2b783fc880932fc5c482182b ] [docs: setup: Norm Hardy suggested that it would be easier if users realized that they already had Python (especially true for Mac users) zooko@zooko.com**20090325035459 Ignore-this: e1bb76a1be4d6d541090d8d9e7e73db9 ] [expirer: include crawler progress in the JSON status output warner@allmydata.com**20090324205137 Ignore-this: f65ccc0a57ed7c5d9565131350d87d2d ] [docs/proposed: new Accounting overview, discuss in #666 warner@allmydata.com**20090324015752 Ignore-this: 7379396536e85194b1c1bec21288adc7 ] [docs/proposed: move old accounting docs out of the way warner@allmydata.com**20090324015457 Ignore-this: 196fe85ef3592f7ef8e58a101eed0494 ] [move GC docs out of proposed/, since it's all implemented now. Add reference to configuration.txt . Add expire.*= suggestions to tahoe.cfg . warner@allmydata.com**20090323230820 Ignore-this: ce0ec977f49c4a3642cf3a7ea7789dc4 ] [storage: use constant-time comparison for write-enablers and lease-secrets warner@lothar.com**20090323032128 Ignore-this: efbbb15242b3f306c32498aae7fd75d ] [hashutil: add constant-time comparison function, to avoid timing attacks when python's short-circuiting data-dependent == operator is used to, say, check a write-enabler warner@lothar.com**20090323032055 Ignore-this: c5c1f5e529ab1b352c949f3e0d9abf20 ] [WUI: fix display of empty directories, it threw an exception before warner@allmydata.com**20090320235809 Ignore-this: e598bb806d75411d202ba90fc251ad2b ] [storage webstatus: insert spaces when we're configured to expire multiple sharetypes warner@allmydata.com**20090320224450 Ignore-this: 18f44396384c8598d984eaa11dc92cf7 ] [expirer: tolerate empty buckets, refactor bucketsize-counting code a bit, don't increment -mutable/-immutable counters unless we actually know the sharetype warner@allmydata.com**20090320191816 Ignore-this: 16827fe12b874a0205b0a7fdb6b7f960 ] [storage: improve wording of status message warner@allmydata.com**20090319184837 Ignore-this: 66684623d50c89fa02915665dbbfec98 ] [parse_date: insist that it returns an int warner@allmydata.com**20090319184443 Ignore-this: a1f5ae0c16a151d218c06fc4411f03c1 ] [storage status: report expiration-cutoff-date like 19-Mar-2009 (as opposed to the tahoe.cfg input format of 2009-03-19), for redundancy: someone who gets the month and day switched will have a better chance to spot the problem in the storage-status output if it's in a different format warner@allmydata.com**20090319180756 Ignore-this: df2c3778260c6389d32f4b47b004ab35 ] [expirer: clean up constructor args, add tahoe.cfg controls, use cutoff_date instead of date_cutoff warner@allmydata.com**20090319010009 Ignore-this: 2b6aaa6d5e6ff9fd417f32978b443fd2 ] [util/time_format: new routine to parse dates like 2009-03-18, switch expirer to use it. I'd prefer to use 18-Mar-2009, but it is surprisingly non-trivial to build a parser that will take UTC dates instead of local dates warner@allmydata.com**20090319005814 Ignore-this: 4f08b301c780a78084fa02a09b5dda3 ] [expirer: change setup, config options, in preparation for adding tahoe.cfg controls warner@allmydata.com**20090319002138 Ignore-this: b23a53e97f2a9fb7a005e9fe40e83fac ] [expirer: track mutable-vs-immutable sharecounts and sizes, report them on the web status page for comparison warner@allmydata.com**20090318202504 Ignore-this: 87e809bf5dedef3f0bc8f4a7b90e42d2 ] [add utility function to parse durations, for lease-expiration config warner@lothar.com**20090317070117 Ignore-this: 2f4fa4352c887b4767969f0ea7931644 ] [expirer: add mode to expire only-mutable or only-immutable shares warner@lothar.com**20090317065118 Ignore-this: b0b25427e3b1516bdfe293528b8e4a4e ] [tahoe_ls.py: add comment about error cases to improve warner@lothar.com**20090317051206 Ignore-this: 7c678b92a55b2f7c6f0d96fb6ece74ed ] [GC: add date-cutoff -based expiration, add proposed docs warner@lothar.com**20090317051041 Ignore-this: a5c0ecbcc2666eb04f2daa67331d1948 ] [test_deepcheck: remove the 10s timeout: our dapper buildslave requires 30s, and the reduced timeout was only there because this tests fails by timeout rather than explicitly warner@lothar.com**20090316005134 Ignore-this: 4ce081283c26719be513b5ce8381a81a ] [tahoe cp -r: add --caps-only flag, to write filecaps into local files instead of actual file contents. Used only for debugging and as a quick tree-comparison tool. warner@lothar.com**20090315231958 Ignore-this: 8ecdf2b08601ae9e9fec5885bf640262 ] [dirnode deep_traverse: insert a turn break (fireEventually) at least once every 100 files, otherwise a CHK followed by more than 158 LITs can overflow the stack, sort of like #237. warner@allmydata.com**20090313233135 Ignore-this: 39b78faa947ed9461f2d120f6843e59f ] [consolidate: remove pointless 'else' after for loop warner@allmydata.com**20090313082751 Ignore-this: 517f41316f7ae40b92b9eef8269a4b69 ] [consolidate: add eta, flush stdout warner@allmydata.com**20090313082451 Ignore-this: 845f63adccc32557c6864ae6120ba836 ] [consolidate: tolerate unicode dirnames warner@allmydata.com**20090313065402 Ignore-this: 7e65703ed3d12d4bd5ec14b693e5f61f ] [dirnode.py: when doing deep-traverse, walk each directory in alphabetical order, to make things like 'manifest' more predictable warner@allmydata.com**20090313065046 Ignore-this: 9a80055a93a6b11853d4e8202bacec14 ] [consolidator: add progress to scan-old-directory passes warner@allmydata.com**20090313054728 Ignore-this: adc67a34f4f19fd58c5bc76301b3df36 ] [consolidator: fix cycle detection to not trigger on merely shared directories, add snapshot counter to progress warner@allmydata.com**20090313042229 Ignore-this: eba2cf9f1b1364b8e4c5ae4fa030a99f ] [consolidator: re-use more directories, add total directories seen-vs-used counts warner@allmydata.com**20090313034801 Ignore-this: 6e743d2940a9fe129cee31008c894d70 ] [tahoe_backup.py: tolerate more time formats warner@allmydata.com**20090313011600 Ignore-this: ca74f56f0dce7d19810c5a7a75bc623c ] [consolidator: add more verbose traversal of directories warner@allmydata.com**20090312232900 Ignore-this: 8ff0e17c6174566832a566a111032db4 ] [consolidate: create multiple numbered backups of the original Archives directory, not just the first time warner@allmydata.com**20090312230427 Ignore-this: e4985f76969b584d099b050781aa561c ] [add 'tahoe debug consolidate' command, to merge directories created by repeated 'tahoe cp -r' or the allmydata win32 backup tool, into the form that would have been created by 'tahoe backup'. warner@allmydata.com**20090312205606 Ignore-this: 66569ca2190aa7b0f9199bcf09dcb27e ] [tahoe_cp.py: return 0 for success, instead of None warner@allmydata.com**20090312205345 Ignore-this: 808f6d8617f8c4e7fde455e6c7639fab ] [setup: specify in the debian/control files that tahoe is compatible with Python 2.6 zooko@zooko.com**20090311225902 Ignore-this: d0793013e4c868d92793d932ef92a62d ] [setup: update the debian/copyright text to reflect the current licences zooko@zooko.com**20090311152952 Ignore-this: 806a95b1b79d6bb20507db5c7201af45 ] [expirer: fix prediction math, thanks to Zandr for the catch warner@allmydata.com**20090309204217 Ignore-this: e20f784483ea1a15e35627fe95506272 ] [storage.expirer: oops, fix upgrade-handler code warner@lothar.com**20090309035516 Ignore-this: af8bc0ea1c7f305b33c06f4a9bf58de3 ] [storage.expirer: handle upgrades better warner@lothar.com**20090309034220 Ignore-this: dd654305b94bc3209426a3e008d57363 ] [storage.expirer: exercise the last missing line of webstatus code warner@lothar.com**20090309033828 Ignore-this: fc4aa34734cae32eec1db623ca0b145b ] [expirer: tolerate corrupt shares, add them to the state and history for future examination warner@lothar.com**20090309030840 Ignore-this: 5ae7e68471ed700cc68beb408e0f303 ] [storage/immutable: raise a specific error upon seeing a bad version number, instead of using assert. Also wrap to 80cols. warner@lothar.com**20090309030732 Ignore-this: 5331d9680ffceff029fbbbcdece7f282 ] [storage: add test coverage for BucketReader.__repr__ warner@lothar.com**20090309030527 Ignore-this: 60ce4e8c5200cc9885a3057f914e78b4 ] [storage/mutable: raise a specific error upon seeing bad magic, instead of using assert warner@lothar.com**20090309020201 Ignore-this: 8daa77362902f5d6ef793e9602a1383b ] [nodeadmin: node stops itself if a hotline file hasn't been touched in 120 seconds now, instead of in 60 seconds zooko@zooko.com**20090308025039 Ignore-this: a9cfc605d3fcad912bd2d0b35d5143ec A test failed on draco (MacPPC) because it took 67.1 seconds to get around to running the test, and the node had already stopped itself when the hotline file was 60 seconds old. ] [expirer: make web display a bit more consistent warner@lothar.com**20090307221442 Ignore-this: 7fec9d9bffc0bddeb51c1baa8e7ea020 ] [web/storage.py: tolerate unknown-future displays, I'm not sure why LeaseCrawler.test_unpredictable_future didn't catch this warner@lothar.com**20090307220243 Ignore-this: 3d4e5baa8cc6d5d26edcea29fda8593d ] [tahoe_cp.py: improve error reporting slightly: don't json-interpret HTTP errors, pass through tahoe webapi error messages warner@lothar.com**20090307114051 Ignore-this: a8beccb67adb082a92509d439b27d68e ] [test_cli.py: assert that 'ls' on an unrecoverable file now gives a better error message warner@lothar.com**20090307110815 Ignore-this: 18e9758e4d0ca7faeaf5bd481c2d72ff ] [test_web: test behavior of broken-dirnode GET, both html and json warner@lothar.com**20090307105707 Ignore-this: c0e5b45eee28959f899efa1bd189d6bd ] [web: when a dirnode can't be read, emit a regular HTML page but with the child-table and upload-forms replaced with an apologetic message. Make sure to include the 'get info' links so the user can do a filecheck warner@lothar.com**20090307105601 Ignore-this: f949d6bd58c0c2fd60fd5fa730115f3f ] [web/common: split out exception-to-explanation+code mapping to a separate humanize_failure() function, so it can be used by other code. Add explanation for mutable UnrecoverableFileError. warner@lothar.com**20090307105408 Ignore-this: 92f326804ba73bb446c5df5992b7d72e ] [test_cli: use explicit (and stable) testdirs, instead of using self.mktemp warner@lothar.com**20090307090428 Ignore-this: 7c58d159e4f33d01635c3445d9e591f9 ] [test_storage: solaris doesn't appear to give the same block count as other platforms, so don't assert as much about 'diskbytes' recovered warner@lothar.com**20090307084518 Ignore-this: 55b35c094ce78c50c8ede42062c5ea13 ] [tahoe_cp.py: don't be verbose unless --verbose is passed: default is one line of success warner@lothar.com**20090307084421 Ignore-this: f75bb818081e0c2993378154775a69d1 ] [setup.cfg: use allmydata.org as a mirror for some of our necessary dependencies warner@allmydata.com**20090307045753 Ignore-this: ce174de4ad2dd96f184eadf4d62ea779 ] [storage: add a lease-checker-and-expirer crawler, plus web status page. warner@allmydata.com**20090307044517 Ignore-this: 4355224f89b959c6f1a256a7e6c88c3b This walks slowly through all shares, examining their leases, deciding which are still valid and which have expired. Once enabled, it will then remove the expired leases, and delete shares which no longer have any valid leases. Note that there is not yet a tahoe.cfg option to enable lease-deletion: the current code is read-only. A subsequent patch will add a tahoe.cfg knob to control this, as well as docs. Some other minor items included in this patch: tahoe debug dump-share has a new --leases-only flag storage sharefile/leaseinfo code is cleaned up storage web status page (/storage) has more info, more tests coverage space-left measurement on OS-X should be more accurate (it was off by 2048x) (use stat .f_frsize instead of f_bsize) ] [setup: comment-out the use of the directory full of dependency packages on the test grid -- the test grid is too unreliable! :-( zooko@zooko.com**20090307044332 Ignore-this: c09f2e5f0b383b4227df4b07c267cb05 ] [setup: create a "make deb-lenny-head" target zooko@zooko.com**20090306191057 Ignore-this: 4b2ff187a3d08dcfe9318980ca92f097 I made this patch by copying [20090305220021-92b7f-89d987c7d05306b5cb03a64f2956a652c10a7296] and changing the name from "intrepid" to "lenny". I haven't tested it. ] [trivial: remove unused import detected by pyflakes, and remove trailing whitespace zooko@zooko.com**20090305223204 Ignore-this: 991001a50da1a357a519c3cb880d7ee1 ] [setup: enable build of .debs for Ubuntu Intrepid, thanks to DarKNesS_WolF zooko@zooko.com**20090305220021 Ignore-this: 88dbb3f72c2446b7734ac437189b67df ] [trivial: use more specific function for ascii-encoding storage index zooko@zooko.com**20090222175751 Ignore-this: c1a78a71cf04d7dbd80a71106f9b21a1 ] [doc: update how_to_make_a_tahoe_release.txt zooko@zooko.com**20090222175739 Ignore-this: 6a2e1592741b362bc170167a9cadc0b ] [test_cli: validate non-HTML error response of 'tahoe get' on an unrecoverable file warner@lothar.com**20090304041146] [web: full patch for HTML-vs-plaintext traceback renderings, improve test coverage of exception rendering warner@lothar.com**20090304035630] [web/common.py: use 'Accept:' header to control HTML-vs-text/plain traceback renderings warner@lothar.com**20090304035457] [test/common: rearrange shouldHTTPError warner@lothar.com**20090304014059] [web: move plural() to common.py warner@lothar.com**20090304014019] [add more information to NotEnoughSharesError, split out new exceptions for no-servers and no-source-of-ueb-hash warner@lothar.com**20090304013715] [test_web: move shouldHTTPError into common.WebErrorMixin warner@lothar.com**20090303225620] [test_web: clean up shouldHTTPError methods warner@lothar.com**20090303224604] [immutable checker add-lease: catch remote IndexError here too warner@allmydata.com**20090227071724 Ignore-this: 94ee6064ce26409381c6451cd50fdade ] [servermap add-lease: fix the code that's supposed to catch remote IndexErrors, I forgot that they present as ServerFailures instead. This should stop the deluge of Incidents that occur when you do add-lease against 1.3.0 servers warner@allmydata.com**20090227070426 Ignore-this: 3d7bc87d9587b51d44b27317d9eded23 ] [rrefutil: add check_remote utility function warner@allmydata.com**20090227065957 Ignore-this: d859d0fae87b7e84ad3c2893350ed519 ] [rrefutil: add trap_remote utility and friends warner@allmydata.com**20090227065524 Ignore-this: a594050cdd9bcca073d8029819dbc35 ] [web: improve layout of storage status with a table warner@allmydata.com**20090227015838 Ignore-this: 5c12f5e296ad1cf224f85e307813eb09 ] [crawler: add ETA to get_progress() warner@allmydata.com**20090227014248 Ignore-this: 27ae76c0530b83323209be70df3d8a4b ] [Fix for bug #645, correct path handling logic so that it works from sshfs Alberto Berti **20090226150237 Ignore-this: e9c1b2d48ebf4ba68100d76e54154a78 ] [wui: s/Provisioning/Reliability/ ; suggested by Terrell zooko@zooko.com**20090226030904 Ignore-this: f9559702e760dc3c57e2fc36f087d91e ] [startstop_node.py: when restarting, tolerate a stale pid too warner@allmydata.com**20090226015245 Ignore-this: 59197d3f400417be64fa0f91a64fae12 ] [web: fix the ERROR: line to work the same in python2.4 and 2.5 warner@lothar.com**20090225074621] [test_cli/test_web: fix spurious test failure on solaris (maybe python2.4?) due to variations in the way that exceptions are stringified warner@allmydata.com**20090225060128 Ignore-this: 829fb7d67fd199babdde1443b64a5381 ] [CLI: modify 'tahoe manifest' and 'tahoe deep-check' to report ERROR: properly. For #590. warner@allmydata.com**20090225054415 Ignore-this: 99162f894fdd24112a869e14848c3dea ] [webapi: modify streaming deep-manifest/deep-checker to emit an ERROR: line if they encounter an unrecoverable+untraversable directory. For #590. warner@allmydata.com**20090225051335 Ignore-this: e6bc49368fb0e7ada7cff477fd63ffe6 ] [scripts/common: fix alias handling on windows again, emit slightly nicer error message in response to an unknown alias warner@allmydata.com**20090225042136 Ignore-this: 76df800d131aed6701b5c7408105b134 ] [#165: make 'tahoe restart --force' the default behavior: warn but do not stop if restart is used on something that wasn't a running node, and always try to start it afterwards. This is particularly important for #315 (restart -m), because otherwise a single not-already-running node will prevent all nodes from being restarted, resulting in longer downtime than necessary warner@allmydata.com**20090225034213 Ignore-this: 26d8b0eea0279ca793faf23c5e948d90 ] [test_deepcheck: switch deep-check tests to use no-network too. This cuts the runtime down by about 50% warner@allmydata.com**20090225030457 Ignore-this: b3a98ed18c5752c9016c047e95d42b ] [test_deepcheck: convert MutableChecker to no-network GridTest warner@allmydata.com**20090225020010 Ignore-this: eccba7fda129330b642886271a61a573 ] [tests/no_network: move GET into the GridTestMixin class warner@allmydata.com**20090225003300 Ignore-this: 7779ad38c2d687ae328ba3cb6164a7a4 ] [common_web.py: oops, add .fields to the other FakeRequest warner@allmydata.com**20090225000459 Ignore-this: 7144269b5e083553ee2c3e7afea00604 ] [test_cli: exercise the recent tolerate-'c:\dir\file.txt' fix in scripts/common, recorded in a separate match to make it easier to merge the fix to prod warner@allmydata.com**20090224235620 Ignore-this: 2cae196dd4ccb578b2abae085376e0d7 ] [scripts/common: on windows, tolerate paths like 'c:\dir\file.txt', by treating single-letter aliases on windows/cygwin as non-aliases warner@allmydata.com**20090224235522 Ignore-this: 96d37644b7f81ac768ff4a1d1915eb46 ] [test/common_web.py: add a .fields attribute to our FakeRequest, since we support versions of Nevow that are old enough to not do it themselves warner@allmydata.com**20090224232050 Ignore-this: 239aaed746128e54e886762782541a1f ] [Two small fixes on documentation for cli backup command. Alberto Berti **20090224223634 Ignore-this: 5634a6dadad6e4e43a112de7fe5c74c ] [test_web: add (disabled) test to see what happens when deep-check encounters an unrecoverable directory. We still need code changes to improve this behavior. warner@lothar.com**20090224214017 Ignore-this: e839f1b0ec40f53fedcd809c2a30d5f9 ] [Add elapsed timestamp to cli backup command final summary. Alberto Berti **20090224171425 Ignore-this: 9a042d11f95ee9f6858a5096d513c0bc ] [Added documentation for '--exclude' and friends cli backup command. Alberto Berti **20090224153049 Ignore-this: bbc791fa56e38535bb82cc3077ffde90 ] [test_repairer: change to use faster no_network.GridTestMixin, split Verifier tests into separate cases, refactor judgement funcs into shared methods warner@lothar.com**20090224041506 Ignore-this: 584ce72d6276da5edc00562793d4ee53 ] [immutable/checker.py: trap ShareVersionIncompatible too. Also, use f.check warner@lothar.com**20090224041405 Ignore-this: b667e8d3192116293babcacdeed42898 instead of examining the value returned by f.trap, because the latter appears to squash exception types down into their base classes (i.e. since ShareVersionIncompatible is a subclass of LayoutInvalid, f.trap(Failure(ShareVersionIncompatible)) == LayoutInvalid). All this resulted in 'incompatible' shares being misclassified as 'corrupt'. ] [immutable/layout.py: wrap to 80 cols, no functional changes warner@lothar.com**20090224005837 Ignore-this: 40019480180ec34141506a28d7711608 ] [test_repairer: change Repairer to use much-faster no_network.GridTestMixin. As a side-effect, fix what I think was a bug: some of the assert-minimal-effort-expended checks were mixing write counts and allocate counts warner@lothar.com**20090223234227 Ignore-this: d58bd0a909f9939775730cda4a858cae ] [test/no_network.py: add a basic stats provider warner@lothar.com**20090223233937 Ignore-this: c9f3cc4eed99cfc36f68938ceff4162c ] [tests: stop using setUpClass/tearDownClass, since they've been deprecated in Twisted-8.2.0 warner@lothar.com**20090223204312 Ignore-this: 24c6592141cf64103530c024f93a5b88 ] [test_checker: improve test coverage for checker results warner@lothar.com**20090223201943 Ignore-this: 83e173602f0f4c811a7a9893d85385df ] [Fixed tests again so they will pass on windows. Alberto Berti **20090223003502 Ignore-this: 80d5074e7153642a2fa2a77958bfb50d ] [misc/*: remove RuntimeError too warner@lothar.com**20090222233401 Ignore-this: b76f8a184f75bb28eb9d8002f957936a ] [scripts: stop using RuntimeError, for #639 warner@lothar.com**20090222233106 Ignore-this: 686a424442670fffbd4d1816c284a601 ] [mutable/publish: stop using RuntimeError, for #639 warner@lothar.com**20090222233056 Ignore-this: 2a80a661c7850d97357caddad48c6e9d ] [remove more RuntimeError from unit tests, for #639 warner@lothar.com**20090222232855 Ignore-this: 1a1c3e1457f3f29ba7101fe406ee5f43 ] [stop using RuntimeError in unit tests, for #639 warner@lothar.com**20090222232722 Ignore-this: 475ce0c0dcd7a1f5ed83ef460312efea ] [ftpd/sftpd: stop using RuntimeError, for #639 warner@lothar.com**20090222232426 Ignore-this: 97001362c4ba9e94b2e254e229b79987 ] [Added tests for the cse when listdir is an iterator Alberto Berti **20090222224356 Ignore-this: 218fb2aba02c28b4b1e5324bdb5adeaa ] [Fixed tests so that they pass also on buildbots. Alberto Berti **20090222224311 Ignore-this: fcb91cd6acf028382411d23d380a4576 ] [Use failUnlessEqual instead of failUnless(a == b) Alberto Berti **20090222224214 Ignore-this: 8f9144632e3ac9acb4726fb48a083bf4 ] [Better implementation of filtering algorithm. Alberto Berti **20090222224049 Ignore-this: 67a8bd2f99bcc87ca2443bef13370a87 ] [Removed '.hgrags' from vcs excludes Alberto Berti **20090222223946 Ignore-this: 3e94c22fc9d85f380ee11fb8bdb4d1e9 ] [docs: CREDITS to Alberto Berti zooko@zooko.com**20090222193314 Ignore-this: 74d370ada3234cce9e58aec15d739f71 ] [Added tests for the --exclude* options of backup command. Alberto Berti **20090222165106 Ignore-this: f1b931cf2e7929ce47b737c022bca707 ] [Added --exclude, --exclude-from and --exclude-vcs options to backup command. Alberto Berti **20090222170829 Ignore-this: 4912890229cd54a2f61f14f06bc4afcc It is still impossible to specify absolute exclusion path, only relative. I must check with tar or rsync how they allow them to be specified. ] [Raise a more explanatory exception for errors encountered during backup processing. Alberto Berti **20090222170252 Ignore-this: f6b8ffe2a903ba07a2c1c59130dac1e4 ] [Added tests for the fixed alias related command's synopsis Alberto Berti **20090222163732 Ignore-this: 4432b4e88e990ba53a5b3fe0f12db2ac ] [Add missing synopsis and descriptions for alias commands. Alberto Berti **20090221003106 Ignore-this: 8aedd03d36d92d912102c7f29e4ca697 ] [docs: move many specification-like documents into specifications/ warner@lothar.com**20090222054054 Ignore-this: a4110cc478198c0611205aba1ccf54f4 ] [test_web.py: increase test coverage of web.status.plural() warner@lothar.com**20090222000116 Ignore-this: 3138c9d5d2410d8e1121e9b2ed694169 ] [crawler: fix performance problems: only save state once per timeslice (not after every bucket), don't start the crawler until 5 minutes after node startup warner@lothar.com**20090221205649 Ignore-this: e6551569982bd31d19779ff15c2d6f58 ] [test_system: oops, don't assume that all files in storage/ are in a deep storage/shares/prefix/si/shnum path, since now the crawler pickle has a short path warner@lothar.com**20090221061710 Ignore-this: fde76d0e5cae853014d1bb18b5f17dae ] [crawler: tolerate low-resolution system clocks (i.e. windows) warner@lothar.com**20090221061533 Ignore-this: 57286a3abcaf44f6d1a78c3c1ad547a5 ] [BucketCountingCrawler: store just the count, not cycle+count, since it's too easy to make usage mistakes otherwise warner@lothar.com**20090221035831 Ignore-this: 573b6f651af74380cdd64059fbbdda4b ] [test_storage: startService the server, as is now the standard practice warner@lothar.com**20090221035755 Ignore-this: 3999889bd628fe4039bbcf1b29160453 ] [crawler: load state from the pickle in init, rather than waiting until startService, so get_state() can be called early warner@lothar.com**20090221035720 Ignore-this: ecd128a5f4364c0daf4b72d791340b66 ] [BucketCountingCrawler: rename status and state keys to use 'bucket' instead of 'share', because the former is more accurate warner@lothar.com**20090221034606 Ignore-this: cf819f63fac9506c878d6c9715ce35b7 ] [storage: also report space-free-for-root and space-free-for-nonroot, since that helps users understand the space-left-for-tahoe number better warner@lothar.com**20090221032856 Ignore-this: 9fdf0475f758acd98b73026677170b45 ] [storage: add bucket-counting share crawler, add its output (number of files+directories maintained by a storage server) and status to the webapi /storage page warner@lothar.com**20090221030408 Ignore-this: 28761c5e076648026bc5f518506db65c ] [storage: move si_b2a/si_a2b/storage_index_to_dir out of server.py and into common.py warner@lothar.com**20090221030309 Ignore-this: 645056428ab797f0b542831c82bf192a ] [crawler: add get_progress, clean up get_state warner@lothar.com**20090221002743 Ignore-this: 9bea69f154c75b31a53425a8ea67789b ] [web/storage: make sure we can handle platforms without os.statvfs too warner@lothar.com**20090220220353 Ignore-this: 79d4cb8482a8543b9759dc949c86c587 ] [crawler: provide for one-shot crawlers, which stop after their first full cycle, for share-upgraders and database-populaters warner@lothar.com**20090220211911 Ignore-this: fcdf72c5ffcafa374d376388be6fa5c5 ] [web: add Storage status page, improve tests warner@lothar.com**20090220202926 Ignore-this: e34d5270dcf0237fe72f573f717c7a4 ] [storage: include reserved_space in stats warner@lothar.com**20090220202920 Ignore-this: b5b480fe0abad0148ecad0c1fb47ecae ] [web/check_results: sort share identifiers in the sharemap display warner@lothar.com**20090220182922 Ignore-this: 5c7bfcee3e15c7082c3653eb8a460960 ] [webapi: pass client through constructor arguments, remove IClient, should make it easier to test web renderers in isolation warner@lothar.com**20090220181554 Ignore-this: e7848cd1bee8faf2ce7aaf040b9bf8e3 ] [test/no_network: do startService on the storage servers, make it easier to customize the storage servers warner@lothar.com**20090220022254 Ignore-this: e62f328721c007e4c5ee023a6efdf66d ] [crawler: modify API to support upcoming bucket-counting crawler warner@lothar.com**20090220013142 Ignore-this: 808f8382837b13082f8b245db2ebee06 ] [test_backupdb: make the not-a-database file larger, since the older sqlite-2.3.2 on OS-X is easily fooled warner@lothar.com**20090220000409 Ignore-this: 694d2ca5053bb96e91670765d0cedf2e ] [web/reliability: add parameter descriptions, adapted from a patch from Terrell Russell. warner@lothar.com**20090219222918 Ignore-this: 835f5ab01e1aff31b2ff9febb9a51f3 ] [test_crawler: hush pyflakes warner@lothar.com**20090219202340 Ignore-this: 765d22c9c9682cc86c5205dc130500af ] [test_crawler: disable the percentage-of-cpu-used test, since it is too unreliable on our slow buildslaves. But leave the code in place for developers to run by hand. warner@lothar.com**20090219201654 Ignore-this: ff7cf5cfa79c6f2ef0cf959495dd989a ] [reliability.py: fix the numpy conversion, it was completely broken. Thanks to Terrell Russell for the help. warner@lothar.com**20090219195515 Ignore-this: f2b1eb65855111b338e1487feee1bbcf ] [reliability: switch to NumPy, since Numeric is deprecated warner@lothar.com**20090219074435 Ignore-this: f588a68e9bcd3b0bc3653570882b6fd5 ] [setup.py: fix pyflakes complaints warner@lothar.com**20090219073643 Ignore-this: a314e5456b0a796bc9f70232a119ec68 ] [move show-tool-versions out of setup.py and into a separate script in misc/ , since setuptools is trying to build and install a bunch of stuff first warner@lothar.com**20090219073558 Ignore-this: 9e56bc43026379212e6b6671ed6a1fd4 ] [test_crawler: don't require >=1 cycle on cygwin warner@lothar.com**20090219065818 Ignore-this: b8d2d40f26aeb30a7622479840a04635 ] [setup.py: add show_tool_versions command, for the benefit of a new buildbot step warner@lothar.com**20090219062436 Ignore-this: 21d761c76a033e481831584bedc60c86 ] [setup.py: wrap to 80 cols, no functional changes warner@lothar.com**20090219055751 Ignore-this: d29e57c6ee555f2ee435667b7e13e60b ] [crawler: use fileutil.move_info_place in preference to our own version warner@lothar.com**20090219051342 Ignore-this: ee4e46f3de965610503ba36b28184db9 ] [fileutil: add move_into_place(), to perform the standard unix trick of atomically replacing a file, with a fallback for windows warner@lothar.com**20090219051310 Ignore-this: c1d35e8ca88fcb223ea194513611c511 ] [crawler: fix problems on windows and our slow cygwin slave warner@lothar.com**20090219042431 Ignore-this: 8019cb0da79ba00c536183a6f57b4cab ] [#633: first version of a rate-limited interruptable share-crawler warner@lothar.com**20090219034633 Ignore-this: 5d2d30c743e3b096a8e775d5a9b33601 ] [change StorageServer to take nodeid in the constructor, instead of assigning it later, since it's cleaner and because the original problem (Tubs not being ready until later) went away warner@lothar.com**20090218222301 Ignore-this: 740d582f20c93bebf60e21d9a446d3d2 ] [test_system: split off checker tests to test_deepcheck.py, this file is too big warner@lothar.com**20090218214234 Ignore-this: 82bf8db81dfbc98224bbf694054a8761 ] [break storage.py into smaller pieces in storage/*.py . No behavioral changes. warner@lothar.com**20090218204655 Ignore-this: 312d408d1cacc5a764d791b53ebf8f91 ] [immutable/layout: minor change to repr name warner@lothar.com**20090218204648 Ignore-this: c8781ef15b7dea63b39236a1899b86ce ] [docs: add lease-tradeoffs diagram warner@lothar.com**20090218204137 Ignore-this: c22a589ad465dac846da834c30dc4083 ] [interfaces.py: allow add/renew/cancel-lease to return Any, so that 1.3.1 clients (the first to use these calls) can tolerate future storage servers which might return something other than None warner@lothar.com**20090218192903 Ignore-this: dcbb704a05416ecc66d90fb486c3d75b ] [docs/debian.txt: minor edit warner@lothar.com**20090218032212 Ignore-this: 64ff1fb163ffca4bcfd920254f1cf866 ] [add --add-lease to 'tahoe check', 'tahoe deep-check', and webapi. warner@lothar.com**20090218013243 Ignore-this: 176b2006cef5041adcb592ee83e084dd ] [change RIStorageServer.remote_add_lease to exit silently in case of no-such-bucket, instead of raising IndexError, because that makes the upcoming --add-lease feature faster and less noisy warner@lothar.com**20090218013053 Ignore-this: 6fdfcea2c832178f1ce72ab0ff510f3a ] [CLI #590: convert 'tahoe deep-check' to streaming form, improve display, add tests warner@lothar.com**20090217231511 Ignore-this: 6d88eb94b1c877eacc8c5ca7d0aac776 ] [interfaces.py: document behavior of add_lease/renew_lease/cancel_lease, before I change it warner@lothar.com**20090217194809 Ignore-this: 703c6712926b8edb19d55d790b65a400 ] [test_backupdb: improve error messages if the test fails warner@lothar.com**20090217170838 Ignore-this: ef657e87c66e4304d3e0aca9831b84c ] [webapi #590: add streaming deep-check. Still need a CLI tool to use it. warner@lothar.com**20090217053553 Ignore-this: a0edd3d2a531c48a64d8397f7e4b208c ] [test_web.Grid: change the CHECK() function to make it easier to test t= values with hyphens in them warner@lothar.com**20090217050034 Ignore-this: 410c08735347c2057df52f6716520228 ] [test_web: improve checker-results coverage with a no-network -based test, enhance no-network harness to assist, fix some bugs in web/check_results.py that were exposed warner@lothar.com**20090217041242 Ignore-this: fe54bb66a9ae073c002a7af51cd1e18 ] [web: fix handling of reliability page when Numeric is not available warner@lothar.com**20090217015658 Ignore-this: 9d329182f1b2e5f812e5e7eb5f4cf2ed ] [test/no_network: update comments with setup timing: no_network takes 50ms, SystemTestMixin takes 2s (on my laptop) warner@lothar.com**20090217000643 Ignore-this: cc778fa3219775b25057bfc9491f8f34 ] [test_upload: rewrite in terms of no-network GridTestMixin, improve no_network.py as necessary warner@lothar.com**20090216234457 Ignore-this: 80a341d5aa3036d24de98e267499d70d ] [test_download: rewrite in terms of no-network GridTestMixin, improve no_network.py as necessary warner@lothar.com**20090216233658 Ignore-this: ec2febafd2403830519120fb3f3ca04e ] [test_dirnode.py: convert Deleter to new no-network gridtest warner@lothar.com**20090216232348 Ignore-this: 8041739442ec4db726675e48f9775ae9 ] [test_cli.py: modify to use the new 'no-network' gridtest instead of SystemTestMixin, which speeds it up from 73s to 43s on my system warner@lothar.com**20090216232005 Ignore-this: ec6d010c9182aa72049d1fb894cf890e ] [tests: fix no_network framework to work with upload/download and checker warner@lothar.com**20090216231947 Ignore-this: 74b4dbd66b8384ae7c7544969fe4f744 ] [client.py: improve docstring warner@lothar.com**20090216231532 Ignore-this: bbaa9e3f63fdb0048e3125c4681b2d1f ] [test_cli: add test coverage for help strings warner@lothar.com**20090216210833 Ignore-this: d2020849107f687448e159a19d0e5dab ] [test/no_network: new test harness, like system-test but doesn't use the network so it's faster warner@lothar.com**20090216205844 Ignore-this: 31678f7bdef30b0216fd657fc6145534 ] [interfaces.py: minor docstring edit warner@lothar.com**20090216205816 Ignore-this: cec3855070197f7920b370f95e8b07bd ] [setup: if you sdist_dsc (to produce the input files for dpkg-buildpackage) then run darcsver first zooko@zooko.com**20090216201558 Ignore-this: b85be51b3d4a9a19a3366e690f1063e2 ] [doc: a few edits to docs made after the 1.3.0 release zooko@zooko.com**20090216201539 Ignore-this: dbff3b929d88134d862f1dffd1ef068a ] [test_cli: improve test coverage slightly warner@lothar.com**20090216030451 Ignore-this: e01ccc6a6fb44aaa4fb14fe8669e2065 ] [test_util: get almost full test coverage of dictutil, starting with the original pyutil tests as a base. The remaining three uncovered lines involve funny cases of ValueOrderedDict that I can't figure out how to get at warner@lothar.com**20090216023210 Ignore-this: dc1f0c6d8c003c0ade38bc8f8516b04d ] [provisioning/reliability: add tests, hush pyflakes, remove dead code, fix web links warner@lothar.com**20090215222451 Ignore-this: 7854df3e0130d9388f06efd4c797262f ] [util/statistics: add tests, fix mean_repair_cost warner@lothar.com**20090215222326 Ignore-this: c576eabc74c23b170702018fc3c122d9 ] [test_repairer: hush pyflakes warner@lothar.com**20090215222310 Ignore-this: 875eb52e86077cda77efd02da77f8cfa ] [lossmodel.lyx: move draft paper into docs/proposed/, since it's unfinished warner@lothar.com**20090215221905 Ignore-this: 7f7ee204e47fd66932759c94deefe68 ] [build a 'reliability' web page, with a simulation of file decay and repair over time warner@lothar.com**20090213234234 Ignore-this: 9e9623eaac7b0637bbd0071f082bd345 ] [docs: relnotes-short.txt zooko@zooko.com**20090215163510 Ignore-this: 683649bb13499bbe0e5cea2e1716ff59 linkedin.com imposed a strict limit on the number of characters I could post. This forced me to prune and prune and edit and edit until relnotes.txt was a quarter of its former size. Here's the short version. ] [TAG allmydata-tahoe-1.3.0 zooko@zooko.com**20090214000556 Ignore-this: aa6c9a31a14a58ad2298cb7b08d3ea70 ] [docs: a few last-minute edits to the docs for 1.3.0 (also this patch will accompany the tag and conveniently trigger the buildbots to build a 1.3.0 version) zooko@zooko.com**20090214000500 Ignore-this: 879c9b10f0e5b9ed0031236e0714ddfa ] [setup: remove attempt to automatically satisfy dependency on pywin32 zooko@zooko.com**20090213234939 Ignore-this: ac02d54a956f7cc58bd3c0802764005f ] [NEWS: mention #625 (deep-repair breaks when it tries to repair a read-only directory) warner@lothar.com**20090213225534 Ignore-this: a3b6b3dd483f8a4a459dfc064760326f ] [test_dirnode: #625 run deep-check on a readonly dirnode too warner@lothar.com**20090213205337 Ignore-this: 2a18d33a7cc99c9959b7182e37b35077 ] [docs: CREDITS zooko@zooko.com**20090213201245 Ignore-this: 5d3101e680739e6cdacb4351b518ae33 ] [figleaf_htmlizer: fix order of summary counters warner@lothar.com**20090213155753 Ignore-this: aa6f0cd37e79857482967496252088da ] [Makefile: minor, add .PHONY declarations warner@lothar.com**20090213072341 Ignore-this: 7666385dc250014198454707ee82cd1c ] [figleaf_htmlizer: oops, re-ignore files that aren't under root, like code in auto-built eggs warner@lothar.com**20090213060022 Ignore-this: b4e67d5e025198d01e373728d1a74b11 ] [setup: relnotes.txt mention the iPhone app and CIFS/SMB (tahoe-w32-client) zooko@zooko.com**20090213044121 Ignore-this: 2c9b8720579c4c146e4416c5a02c77a5 ] [setup: make the "full version string" be "allmydata-tahoe/1.3.0" instead of "allmydata-tahoe-1.3.0" and the UserAgent string of the cli be "allmydata-tahoe/1.3.0 (tahoe-client)" zooko@zooko.com**20090213043738 Ignore-this: f41a0472abc9910ad180831963d740e8 This is webbish. Thanks to kpreid for suggesting it. ] [docs: relnotes.txt final (!?) update for 1.3.0! zooko@zooko.com**20090213042814 Ignore-this: 7a959eba00115474ff048cd84ecab495 ] [docs: known_issues.txt: my version of #615, remove "issue numbers", edits, move tahoe-1.1.0 issues to historical zooko@zooko.com**20090213041621 Ignore-this: 58dee952a3139791ba0fe03f03fcf8bb ] [docs: CREDITS zooko@zooko.com**20090213034228 Ignore-this: d6bb651d657ed8967ca1dfb23afbd00e ] [Makefile: add figleaf-delta-output, to render figleaf coverage differences with a previous run warner@lothar.com**20090212211829 Ignore-this: 9c78861c4abed3d6c2d013ff40d9b4fb ] [figleaf_htmlizer: emit stats to stdout, so buildbot can see it warner@lothar.com**20090212211020 Ignore-this: bf5b503717580561c2fdb6c619159c9 ] [figleaf_htmlizer: render changes in coverage relative to a previous test run using --old-coverage warner@lothar.com**20090212210412 Ignore-this: 6c2bfc21a0671f1a07e19d3f9fd17157 ] [immutable repairer: fix DownUpConnector so that it satisfies short reads the were requested after the last write and before the close zooko@zooko.com**20090212230447 Ignore-this: 2856bb9582d9c38a82fd9aad7fd10b57 This is probably the cause of the very rare "loss of progress" bug. This is tested by unit tests. A recent patch changed this to errback instead of losing progress, and now this patch is changing it again to return a short read instead of errbacking. Returning a short read is what the uploader (in encode.py) is expecting, when it is reading the last block of the ciphertext, which might be shorter than the other blocks. ] [immutable repairer: add an assertion that a certain value in this tricky function is always what I think it is zooko@zooko.com**20090212223132 Ignore-this: 65b6920d67d7289f0dfc38c8e42f7c82 ] [immutable repairer: add tests of how it handles short reads zooko@zooko.com**20090212224853 Ignore-this: bca74cfa2b1f27d733540fccd1c29839 ] [nodeadmin: node stops itself if a hotline file hasn't been touched in 60 seconds now, instead of in 40 seconds zooko@zooko.com**20090212212131 Ignore-this: a6f44b9dfd9f405d4fe9ef66a6d293da A test failed on draco (MacPPC) because it took 49 seconds to get around to running the test, and the node had already stopped itself when the hotline file was 40 seconds old. ] [immutable repairer: errback any pending readers of DownUpConnectorwhen it runs out of bytes, and test that fact zooko@zooko.com**20090212021129 Ignore-this: efd1fac753dad541fe5a0f232bcbf161 ] [figleaf_htmlizer: more rearranging, behavior should still be unchanged warner@allmydata.com**20090212020515 Ignore-this: 8fb7dd723b2d26bdcf1d57685b113bb8 ] [figleaf_htmlizer: break it up into subfunctions, behavior should still be unchanged warner@allmydata.com**20090212015607 Ignore-this: 9644ec95b0f9950722c13ad3c9654e22 ] [figleaf_htmlizer: rewrite in class form, behavior should be the same as before warner@allmydata.com**20090212014050 Ignore-this: d00b838c0af1ea2d71be246bbcf36c53 ] [figleaf_htmlizer: rewrite with twisted.python.usage, remove logging: should behave the same as before warner@allmydata.com**20090212011643 Ignore-this: 86f961e369625e9ab5188e817b083fe4 ] [figleaf_htmlizer: expand tabs, fix to 4-space indents. No functional changes. warner@allmydata.com**20090212010542 Ignore-this: 5c9aa3704eacf1b42a985ade8e43dd1f ] [Makefile: include the figleaf pickle in the uploaded coverage data, for later differential analysis warner@allmydata.com**20090212000913 Ignore-this: e31002f4a7d55d8a2ca1131a1066c066 ] [Makefile: fix test-clean, by ignoring _appname.py and removing src/allmydata_tahoe.egg-info warner@allmydata.com**20090212000434 Ignore-this: 17e4513eb0c844f434b863d40319dfdd ] [setup: from the perspective of darcs, src/allmydata/_appname.py is a very boring file, indeed zooko@zooko.com**20090211233108 Ignore-this: d22a1d76879744fa2a4dd6ea218511ba ] [versioning: include an "appname" in the application version string in the versioning protocol, and make that appname be controlled by setup.py zooko@zooko.com**20090211231816 Ignore-this: 6a0e62492bd271cdaaf696c4a35bc919 It is currently hardcoded in setup.py to be 'allmydata-tahoe'. Ticket #556 is to make it configurable by a runtime command-line argument to setup.py: "--appname=foo", but I suddenly wondered if we really wanted that and at the same time realized that we don't need that for tahoe-1.3.0 release, so this patch just hardcodes it in setup.py. setup.py inspects a file named 'src/allmydata/_appname.py' and assert that it contains the string "__appname__ = 'allmydata-tahoe'", and creates it if it isn't already present. src/allmydata/__init__.py import _appname and reads __appname__ from it. The rest of the Python code imports allmydata and inspects "allmydata.__appname__", although actually every use it uses "allmydata.__full_version__" instead, where "allmydata.__full_version__" is created in src/allmydata/__init__.py to be: __full_version__ = __appname + '-' + str(__version__). All the code that emits an "application version string" when describing what version of a protocol it supports (introducer server, storage server, upload helper), or when describing itself in general (introducer client), usese allmydata.__full_version__. This fixes ticket #556 at least well enough for tahoe-1.3.0 release. ] [setup: add new darcsver-1.2.1 which includes Brian's and Nils Durner's patch to ignore non-ascii chars in XML output zooko@zooko.com**20090211215253 Ignore-this: d0319ec509f6562a45c9a555cf37d3cc ] [backupdb: cosmetic: capitalize the no-pysqlite instructions properly. Thanks to Terrell Russell for the catch. warner@allmydata.com**20090211212830 Ignore-this: a17b34a12bbe96ad5b531ef5d293471e ] [test_repairer: disable repair-from-corruption tests until other things are improved well enough to make it useful warner@allmydata.com**20090211210159 Ignore-this: bf2c780be028f0f5556f1aed04cc29b9 ] [NEWS: explain limitations of the new repairer warner@allmydata.com**20090211204352 Ignore-this: 455822aeeae12b999fb28ba8a9265a51 ] [setup: removed bundled darcsver-1.2.0 zooko@zooko.com**20090211201904 Ignore-this: a826819eb8665e6d00e30a929f1f9831 (I'm about to add a new bundled darcsver-1.2.1, but I want to see what the buildbots will do when there is no bundled darcsver present.) ] [trivial: whitespace zooko@zooko.com**20090211171935 Ignore-this: 38b042422093adc2e176e448051f19f ] [docs/known_issues: mention #615 javascript-vs-frames, for zooko to improve/rewrite warner@allmydata.com**20090211201453 Ignore-this: b4805670c3700d90db39fb008b0f2c92 ] [storage: make add-lease work, change default ownernum=1 since 0 is reserved to mean 'no lease here' warner@allmydata.com**20090211053938 Ignore-this: 9437ec1529c34351f2725df37e0859c ] [#620: storage: allow mutable shares to be deleted, with a writev where new_length=0 warner@allmydata.com**20090211053756 Ignore-this: 81f79e0d72f7572bdc1c9f01bb91620 ] [test_cli.Backup: insert some stalls to make sure two successive backups get distinct timestamps, avoiding intermittent failures warner@allmydata.com**20090211023709 Ignore-this: a146380e9adf94c2d301b59e4d23f02 ] [#619: make 'tahoe backup' complain and refuse to run if sqlite is unavailable and --no-backupdb is not passed warner@allmydata.com**20090211004910 Ignore-this: 46ca8be19011b0ec11e7a10cb2556386 ] [NEWS: point out that 'tahoe backup' requires a 1.3.0-or-later client node warner@allmydata.com**20090210210025 Ignore-this: 26e3e87dd9c762e59d6e0eb9f1acfb8d ] [tests: increase the default timeout for SystemTestMixin -based tests to 300 seconds, since our slower buildslaves sometimes take longer than the default 120s warner@allmydata.com**20090210204412 Ignore-this: 781ec7b1060fa8c75dc12401b07ee358 ] [immutable: repairer: add a simple test to exercise the "leftover" code path, fix the bug (and rename the variable "leftover" to "extra") zooko@zooko.com**20090210181245 Ignore-this: 8426b3cd55ff38e390c4d1d5c0b87e9d ] [docs: not-quite-final version of relnotes.txt for tahoe-1.3.0 zooko@zooko.com**20090210170227 Ignore-this: 64e11f3619d537eae28f4d33977bd7ab ] [docs: small edit to about.html zooko@zooko.com**20090210170219 Ignore-this: fa79838f4cdac17c09b6c3332e8a68b5 ] [docs: NEWS: move the most exciting items to the top, break them out of less exciting categories, update a couple of stale bits, and a touch of editing zooko@zooko.com**20090210084843 Ignore-this: 181b31e3338b14b70ee9bc068ed2e4fb ] [docs: edit about.html zooko@zooko.com**20090210080102 Ignore-this: d96f9b21f88d4c7a552f9ed3db5c6af4 ] [immutable: tighten preconditions -- you can write empty strings or read zero bytes, and add the first simple unit test of DownUpConnector zooko@zooko.com**20090210065647 Ignore-this: 81aad112bb240b23f92e38f06b4ae140 ] [immutable: tests: the real WRITE_LEEWAY is 35 (it was a mistake to move it from 10 to 35 earlier -- I had seen a failure in which it took 35 times as many writes as I thought were optimal, but I misread and thought it took only 20 times as many) zooko@zooko.com**20090210055348 Ignore-this: e81c34d31fe2e3fd641a284a300352cc ] [immutable: defensive programming: assert that the encrypted readable gave you no more than the number of bytes you asked for zooko@zooko.com**20090210054605 Ignore-this: 72cac6a6ccdd9533af4bc02b64598e6d (There is a bug in the current DownUpConnector which can cause it to give more bytes than you asked for on one request, and then less on the next, effectively shifting some of the bytes to an earlier request, but I think this bug never gets triggered in practice.) ] [docs: suggest Python 2.5 -- Python 2.6 is not as well tested yet zooko@zooko.com**20090210054421 Ignore-this: 3ef6988c693330d4937b4d8e1a996c39 ] [immutable: tests: sigh, raise, again the limit of how many extra writes you can do and still pass this test zooko@zooko.com**20090210020931 Ignore-this: 91faf5d6919ca27f8212efc8d19b04c5 Obviously requiring the code under test to perform within some limit isn't very meaningful if we raise the limit whenever the test goes outside of it. But I still don't want to remove the test code which measures how many writes (and, elsewhere, how many reads) a client does in order to fulfill these duties. Let this number -- now 20 -- stand as an approximation of the inefficiency of our code divided by my mental model of how many operations are actually optimal for these duties. ] [immutable: tests: assert that verifier gives a clean bill of health after corruption and repair (the previous patch mistakenly did this only after deletion and repair), and also test whether deleting seven other shares and then downloading works. Also count the number of shares stored in the local filesystem. zooko@zooko.com**20090210020841 Ignore-this: ac803d0599f336c308fe74a2582e6aa4 ] [immutable: test: add a test after attempting to repair from corruption: does a full verify run give the file a clean bill of health? If not, the you haven't successfully repaired it. zooko@zooko.com**20090210010149 Ignore-this: 43faea747e7afccaae230d50c067adc6 This will make the repairer tests more consistent -- less accidentally passing due to getting lucky. ] [immutable: tests: put shares back to their pristine condition in between each test of corrupting-and-repairing them zooko@zooko.com**20090210002956 Ignore-this: 45de680a6ac69b1845c0c74534913dec This is important, because if the repairer doesn't completely repair all kinds of corruption (as the current one doesn't), then the successive tests get messed up by assuming that the shares were uncorrupted when the test first set about to corrupt them. ] [upload: add a think-of-the-compatibility note to UploadResults warner@allmydata.com**20090209205004 Ignore-this: 9ae325ac7281ab67ce45bccb8af672a ] [helper #609: uploading client should ignore old helper's UploadResults, which were in a different format warner@allmydata.com**20090209204543 Ignore-this: 4868a60d01fa528077d8cf6efa9db35d ] [test_runner: skip all spawnProcess-using tests on cygwin, since spawnProcess just hangs forever warner@lothar.com**20090209083400 Ignore-this: e8c2c85650b61cf084cb8a8852118b86 ] [test_runner.py: revert the CreateNode section to using runner() inline, rather than spawning a process, to get more precise coverage warner@lothar.com**20090209082617 Ignore-this: e7f0ae5c9a2c2d8157289ef39fda371 ] [storage #596: announce 'tolerates-immutable-read-overrun' to the version announcement, to indicate that a read() on an immutable share where offset+length is beyond the end of the file will return a truncated string instead of raising an exception warner@lothar.com**20090209015602 Ignore-this: cbd07102909449da55067184a63fc0d1 ] [test_upload: add test of maximum-immutable-share-size, to complete the last item of #538 warner@lothar.com**20090209014127 Ignore-this: 943b9b11812ad784ec824db7a8a7aff9 ] [docs/specifications: add an outline of the spec documents we'd like to have some day warner@lothar.com**20090208234748 Ignore-this: b591ad0361810e5aae37cba07fdfbd43 ] [test_cli.Backup: capture stderr when sqlite is unavailable warner@lothar.com**20090207211440 Ignore-this: 4978b1a149e32bd31e66d5bc4269bc55 ] [docs: add mac/README.txt to explain the first few basic facts about what the files are in this directory zooko@zooko.com**20090207223321 Ignore-this: d89a55b55cab69d2171529f62fcfbc11 ] [test_repairer: wrap comments to 80cols, my laptop does not have a wide screen. No functional changes. warner@lothar.com**20090207200626 Ignore-this: f539c156f3b79cfe49c7cf0fa788994e ] [immutable/checker: wrap comments to 80cols, my laptop does not have a wide screen. No functional changes. warner@lothar.com**20090207200439 Ignore-this: ad8f03eb17b217987268f76d15fa5655 ] [test/common.py: in share-layout-reading code, use '>L' consistently, since '>l' doesn't specify the signedness and the windows tests appear to be failing with an endianness-like problem (version==0x01000000). Also use binary mode when editing sharefiles warner@lothar.com**20090207193817 Ignore-this: dec2c7208b2fe7a56e74f61a8dc304f6 ] [test_cli: increase timeout on test_backup, since our dapper buildslave is really slow warner@lothar.com**20090206081753] [backupdb.py: catch OperationalError on duplicate-insert too, since pysqlite2 on dapper raises it instead of IntegrityError warner@lothar.com**20090206073401] [test_backupdb.py: reset the check-timers after one step, otherwise a slow host can false-fail warner@lothar.com**20090206073305] [NEWS: announce the #598 'tahoe backup' command warner@lothar.com**20090206054416] [docs/CLI: document 'tahoe backup' warner@allmydata.com**20090206041445 Ignore-this: 60dade71212f2a65d3c0aaca7fb8ba00 ] [test_cli.backup: oops, fix test to work even when sqlite is unavailable warner@allmydata.com**20090206041042 Ignore-this: 8a550049c8eb27c34ab2440263e07593 ] [#598: add cli+backupdb tests, improve user display, update docs, move docs out of proposed/ warner@allmydata.com**20090206040701 Ignore-this: 7a795db5573247471c6a268fb0aa23c0 ] [#598: add backupdb to 'tahoe backup' command, enable it by default warner@allmydata.com**20090206015640 Ignore-this: 4e6a158d97549c55dbc49f6d69be8c44 ] [add sqlite-based backupdb, for #598 and others (including 'tahoe cp'). Not enabled yet. warner@allmydata.com**20090206001756 Ignore-this: 36d9a56b257e481091fd1a105318cc25 ] [setup: require new bundled setuptools-0.6c12dev zooko@zooko.com**20090205152923 Ignore-this: 516bbb2195a20493fa72a0ca11a2361c ] [setup: bundle setuptools-0.6c12dev (our own toothpick of setuptools) this version completes my patch to fix http://bugs.python.org/setuptools/issue54 , which is necessary for tahoe to build with --prefix=support without doing a lot of PYTHONPATH gymnastics around the call to setup.py zooko@zooko.com**20090205152818 Ignore-this: da7b1587ee91180c4a1a56f217311de3 ] [setup: remove old bundled setuptools-0.6c11dev (our own toothpick of setuptools) zooko@zooko.com**20090205152758 Ignore-this: 21140fb5a62bf855d9ffa8af8ff4cdac ] [setup: add a case to execute "python .../twistd.py" if "twistd" is not found zooko@zooko.com**20090205000620 Ignore-this: 6d472905042d14707bdf5d591d76248e ] [doc: specify Python >= 2.4.2 zooko@zooko.com**20090204213840 Ignore-this: 108c60b69fdb1d0fcb95810703ce415a ] [setup: merge recent patches that change the set of bundled tools in misc/dependencies/ zooko@zooko.com**20090204193222 Ignore-this: 4e5df8a8b5da51d122022dabb40f4b62 ] [setup: bundle new setuptools_trial zooko@zooko.com**20090204192454 Ignore-this: a213fca1fd09ba2c3999dc948dafba98 ] [setup: remove old bundled setuptools_trial zooko@zooko.com**20090204192442 Ignore-this: 18ed85a4dae721d3a25c67c9f5672802 ] [setup: bundle new setuptools_darcs zooko@zooko.com**20090204192429 Ignore-this: 232482cdc411fb200e37f77ca8914760 ] [setup: remove old bundled setuptools_darcs zooko@zooko.com**20090204192418 Ignore-this: 2af1cca4a5ed087120b5373a63600792 ] [roll back the upgrade of the bundled setuptools_trial from 0.5 to 0.5.1 zooko@zooko.com**20090204073836 Ignore-this: a939f9bd569b6db315dc62561edec56f Upgrading setuptools trial in that way leads to a very tricky problem in setuptools's handling of recursive installation of the build-time dependencies of build-time dependencies... rolling back: Tue Feb 3 22:17:18 MST 2009 zooko@zooko.com * setup: bundle new setuptools_trial-0.5.1 A ./misc/dependencies/setuptools_trial-0.5.1.tar Tue Feb 3 22:17:32 MST 2009 zooko@zooko.com * setup: remove old bundled setuptools_trial R ./misc/dependencies/setuptools_trial-0.5.tar ] [setup: remove a couple of horrible work-arounds in setup.py now that we rely on our own toothpick of setuptools which fixes those issues zooko@zooko.com**20090204052405 Ignore-this: a11bc74a5cb879db42c4fe468c375577 also specify that we need our latest revision (11dev) of our toothpick of setuptools also *always* setup_require setuptools_darcs at module import time. Formerly we added setup_require setuptools_darcs only if the PKG-INFO file were not already created. There is some weird, irreproducible bug to do with setuptool_darcs, and I guess that the fact that whether it is required or not depends on that autogenerated file might have something to do with it. Anyway, this is simpler. ] [setup: remove old bundled setuptools_trial zooko@zooko.com**20090204051732 Ignore-this: 3e753f59c15f5787097be1ce21985c83 ] [setup: bundle new setuptools_trial-0.5.1 zooko@zooko.com**20090204051718 Ignore-this: a28f2b48075a9383984da37c4f200121 ] [setup: bundle new setuptools_darcs-1.2.5 zooko@zooko.com**20090204051704 Ignore-this: 7f868649d64c1e4b42a56fab3e32ea13 ] [setup: remove old bundled setuptools_darcs zooko@zooko.com**20090204051640 Ignore-this: f52325aca56c86573f16890c2c4fa4d7 ] [setup: remove bundled darcsver in gzipped format zooko@zooko.com**20090204051628 Ignore-this: d3272f89c0c3275fca83df84f221fe45 ] [setup: bundle darcsver in ungzipped format zooko@zooko.com**20090204051617 Ignore-this: a3a17bcaba2476577c9695911c630408 ] [docs: mention issues using flogtool on Windows zooko@zooko.com**20090204033410 Ignore-this: 6122bcb82eea32d2a936a59d77233743 ] [webapi: add verifycap (spelled 'verify_url') to the t=json output on files and directories. Closes #559. warner@allmydata.com**20090204012248 Ignore-this: 7da755304f6708b4973e4a7c1bcf8a43 ] [setup: stop trying to add specific dirs to site-dirs to work-around setuptools #17 zooko@zooko.com**20090203042352] [setup: require and automatically use setuptools-0.6c11dev (our own toothpick of setuptools) which is bundled zooko@zooko.com**20090203042323] [setup: remove old bundled setuptools-0.6c10dev (our own toothpick of setuptools) zooko@zooko.com**20090203042304] [setup: bundle setuptools-0.6c11dev (our own toothpick of setuptools) zooko@zooko.com**20090203041950 Hopefully this one fixes the issue with easy_install not searching the sys.path for packages that were requested to be installed, (setuptools #17), thus allowing us to do away with the "--site-dirs=" kludges, which are currently breaking some of our buildbots. ] [CLI: tahoe_check: stop escaping the JSON output when using --raw warner@allmydata.com**20090203031232 Ignore-this: c8a450a6e28731c352c0414b59861502 ] [#598: first cut of 'tahoe backup' command: no backupdb, but yes shared-unchanged-directories and Archives/TIMESTAMP and Latest/ warner@allmydata.com**20090203030902 Ignore-this: 650df5631523b63dd138978b8f3aa372 ] [CLI: move node-url -should-end-in-slash code into a common place, so other tools can rely upon it later warner@allmydata.com**20090203030856 Ignore-this: dd94e011d74bcc371f36011a34230649 ] [tahoe_ls: CLI command should return rc=0, not None warner@allmydata.com**20090203030720 Ignore-this: 18993c782cf84edc01e4accb6f8bf31a ] [setup: add site-dirs for Debian and for some (?) Mac OS X zooko@zooko.com**20090203000955 Ignore-this: ebd8b925dbe51bae390f8e8ddb429209 ] [misc/dependencies: update setuptools_trial from 0.4 to 0.5 warner@allmydata.com**20090202233657 Ignore-this: 9e338f824287552100ac92d78d19fe19 ] [.darcs-boringfile: ignore bin/tahoe and bin/tahoe-script.py, since both are now generated warner@allmydata.com**20090202230333 Ignore-this: 92aa108838182076e71f0270bdba5887 ] [Makefile: fix 'clean' target to remove bin/tahoe and bin/tahoe-script.py, since both are now generated warner@allmydata.com**20090202230307 Ignore-this: b01f539cd85dbcbba26616224cad0146 ] [storage: disable test_large_share again: my linux laptop has less than 4 GiB free zooko@zooko.com**20090131041649 Ignore-this: da931a572a88699371057f932106b375 ] [web/directory: add a link from readwrite directories to a read-only version, and fix the 'SI=xxx' header to actually use the storage index, not the writekey warner@allmydata.com**20090131013205 Ignore-this: a1f21f81e6dbf88e591085efd1a57740 ] [uri: add abbrev_si() method, which returns the abbreviated storage index warner@allmydata.com**20090131013110 Ignore-this: bb3d9483570dbe0dc9ecdc1f31d8d79f ] [setup: make sure you use darcsver whenever you are going to run trial zooko@zooko.com**20090130203819 Ignore-this: 2bd632d7540020c0dd893d337163d396 This fixes the bug Brian had where he ran "python ./setup.py trial" and the allmydata-tahoe version number came out as 0.0.0. ] [setup: require setuptools_trial >= 0.5, and delegate to it the job of deciding which Twisted reactor to use for the current platform zooko@zooko.com**20090130043133 Ignore-this: 9e354184d6c989ddf16c7e16a3295ef2 ] [Makefile: use 'setup.py test' for test/quicktest targets (instead of Brian Warner **20090130102536 Ignore-this: da07fae8417bc54a610786bc57959c5e 'setup.py trial'). 'setup.py trial' clobbers the tahoe .egg's PKG-INFO "Version:" field (resetting it to 0.0.0), possibly because it isn't invoking the darcsver subcommand that 'setup.py test' does before it runs the 'trial' subcommand. This slows down quicktest by another couple of seconds (see #591) and adds more noise to its output, but without this change, 'make test' and 'make quicktest' fail on test_runner (which spawns bin/tahoe as a subprocess, and with a mangled Version:, the setuptools-based entry point script refuses to recognize our source tree as a usable version of Tahoe). ] [Makefile: remove the obsolete stats-gatherer-run target warner@allmydata.com**20090130010131 Ignore-this: c6f278f8f8fafcd23fb942aa681c842e ] [setup: remove the "build three times in a row" kludge now that #229 is fixed, and spell build "build" instead of "build_tahoe" zooko@zooko.com**20090129195952 Ignore-this: e50ec860284fbc13d37cdc50bc09390a ] [setup: require darcsver >= 1.2.0 and rely exclusively on darcsver to set the version string zooko@zooko.com**20090129185640 Ignore-this: b7ed63526015c0769812f3c6f2342b8c ] [setup: add new bundled darcsver-1.2.0 zooko@zooko.com**20090129185115 Ignore-this: 2e58040639d589205ad79d96d95c076f ] [setup: remove old bundled darcsver-1.1.8 zooko@zooko.com**20090129185102 Ignore-this: f4c735ebb71ec4bf9e4a10bcbe959573 ] [setup: add doc explaining why we set zip_safe=False zooko@zooko.com**20090129174802 Ignore-this: 7dd345f33bd10b9a362980bfb904ca60 ] [setup: setup.cfg aliases get expanded only once, so put everything you want to happen during the "test" step in the alias zooko@zooko.com**20090129165815 Ignore-this: fcf87cf9592410a7b9a5150f2ccd3a13 ] [setup: invoke darcsver whenever doing an sdist zooko@zooko.com**20090129165125 Ignore-this: 88b9bf4fae0303250ada810d351b566 ] [setup: more verbose assertion failure in test_runner zooko@zooko.com**20090129164906 Ignore-this: 2d657efb77e1bc562e628e51c8fb6236 ] [setup: subclass setuptools.Command instead of distutils Command zooko@zooko.com**20090129130058 Ignore-this: 8ef3cbf8cd11a5f3451854aa3298c826 There's almost no difference between them, but let's be consistent try to use the setuptools API as it was intended. ] [setup: temporarily comment-out the horrible kludge to work-around setuptools #17, while I figure out how to solve it better zooko@zooko.com**20090129130000 Ignore-this: 9e42491cfa8042c90a6a96081c00a053 ] [setup: always create a support dir and populate it with a site-packages and add same to the PYTHONPATH, just in case someone is going to do "build", "develop", or "test" or something else which triggers a build zooko@zooko.com**20090129045608 Ignore-this: d3740cd285f1d30a111536863a8e8457 I think there must be a much better solution for this -- probably to fix setuptools #54 and ship our own fork of setuptools and rely on it. ] [setup: if any of "build", "develop", or "test" appear in the sys.argv then that means we'll be doing a develop, so add the workarounds for setuptools #17 in any case zooko@zooko.com**20090129045534 Ignore-this: 38645dfadf3ba7b42370e795b7c90214 I think there must be a much better solution for this -- probably to fix setuptools #17 and ship our own fork of setuptools and rely on it. ] [setup: add metadata indicating compatibility with python 2.6 zooko@zooko.com**20090129002628 Ignore-this: ca9af281354ed62f654d6b15267c8d6b ] [setup: a new improved way to create tahoe executables zooko@zooko.com**20090129000716 Ignore-this: d1b038ab8efb949125d7592ebcceccba Create the 'tahoe-script.py' file under the 'bin' directory. The 'tahoe-script.py' file is exactly the same as the 'tahoe-script.template' script except that the shebang line is rewritten to use our sys.executable for the interpreter. On Windows, create a tahoe.exe will execute it. On non-Windows, make a symlink to it from 'tahoe'. The tahoe.exe will be copied from the setuptools egg's cli.exe and this will work from a zip-safe and non-zip-safe setuptools egg. ] [storage: enable the test of a share larger than 2 GiB; this will probably be too expensive on Mac OS X, but I think it won't be on all the other platforms ; I will probably set it to SkipTest if the sys.platform is Mac after seeing the results of this buildbot run zooko@zooko.com**20090128223312 Ignore-this: 1b08b73b8f9ec4b5f629b734c556f2ed ] [setup: bundle darcsver-1.1.8 zooko@zooko.com**20090128200326 Ignore-this: e2923c03860a45f4a49d79bd51acba77 ] [setup: remove old bundled dependency darcsver-1.1.5 zooko@zooko.com**20090128182012 Ignore-this: db20eeded096231e9d1da397d595c39d ] [CLI: fix examples in tahoe put --help warner@allmydata.com**20090127213909 Ignore-this: 1fe319f70c3791482bb381c06d4a066b ] [trivial: remove unused imports noticed by pyflakes zooko@zooko.com**20090127211148 Ignore-this: aee8bae8aa6f641fe15c5fe947d92f77 ] [setup: fix test_runner to invoke bin/tahoe.exe instead of bin/tahoe if on Windows (this is what happens when a user invokes bin/tahoe on Windows) zooko@zooko.com**20090127203717 Ignore-this: e8795f2d3c70e871839d893bdfec0186 ] [setup: fix test_runner to assert that lines which are output to stderr must end with a punctuation mark (apparently re.search("x$", "x\r\n") does not match. :-() zooko@zooko.com**20090127203505 Ignore-this: d323b364d7cafb4c5847c0cc8346702e ] [setup: fix test_runner.RunNode.test_baddir -- it was left incomplete and broken by a recent patch zooko@zooko.com**20090127203245 Ignore-this: fee5c807945f45bce4919106771914f3 ] [setup: hack the sys.argv to set poll reactor if "python ./setup.py test" in addition to if "python ./setup.py trial"; remove another hack which has been moved setup.cfg; remove setup_requires Twisted since now we actually setup_require setuptools_trial and it install_requires Twisted. zooko@zooko.com**20090127044046 Ignore-this: 38c5afd730024cca63bc84f8ab7100f4 ] [setup: go ahead and check for noise in test_client_no_noise zooko@zooko.com**20090126234616 Ignore-this: dff1a3511fdfad1a61fe73e4277c8981 ] [setup: always run "build" before running "test" zooko@zooko.com**20090126233240 Ignore-this: 8cf76347ba24f02023f1690a470569df ] [setup: add a test for a warning message from importing nevow, marked as TODO, comment-out the assertion of no-noise inside other test_runner tests zooko@zooko.com**20090126233046 Ignore-this: 6b1554ed9268988fd65b8e8aac75ed4e ] [setup: always run build_tahoe before running tests zooko@zooko.com**20090126233024 Ignore-this: da31145fa86a61a307dc5dcc4debc0bb ] [diskwatcher: cache growth results (which take 30s to generate) for 3 minutes, to help munin, which invokes it 6 times in a row every 5 minutes warner@lothar.com**20090125230639] [make streaming-manifest stop doing work after the HTTP connection is dropped warner@allmydata.com**20090124013908] [setup: sys.exit() raises a SystemExit exception on Python 2.4 on Solaris -- fix it so that bin/tahoe doesn't interpret this as its brother having been non-executable zooko@zooko.com**20090124004911 Ignore-this: 117db7036306bbe8b5008d18ba23cb9b ] [setup: find a "bin/tahoe" executable to test based on allmydata.__file__ instead of based on the CWD zooko@zooko.com**20090124003437 Ignore-this: 25282068ce695c12a2b1f23c6fd2b205 This means that the tests still work if you are executing them from a CWD other than the src dir -- *if* the "bin/tahoe" is found at os.path.dirname(os.path.dirname(os.path.dirname(allmydata.__file__))). If no file is found at that location, then just skip the tests of executing the "tahoe" executable, because we don't want to accidentally run those tests against an executable from a different version of tahoe. ] [rollback the #591 fix, since it breaks quicktest entirely warner@allmydata.com**20090123232812] [#509: remove non-streaming 'tahoe manifest' CLI form warner@allmydata.com**20090123230002] [CLI.txt: improve docs on 'tahoe manifest' to cover --verify-cap, --repair-cap, and streaming JSON warner@allmydata.com**20090123225939] [#509 CLI: add 'tahoe manifest --stream' warner@allmydata.com**20090123223321] [#509: test_system.py: add test for streamed-manifest warner@allmydata.com**20090123223247] [test_system: rearrange DeepCheckWebGood to make it easier to add CLI tests warner@allmydata.com**20090123221306] [setup: use "trial" executable instead of the setuptools_trial plugin for "make quicktest" zooko@zooko.com**20090123225830 Ignore-this: f70fb8873e0ac94f0c55e57f0f826334 This is to fix #591 ("make quicktest" could be quicker and less noisy). This means that "make quicktest" won't work unless the user has manually installed Twisted already such that the "trial" executable is on their PATH and the Twisted package is on their PYTHONPATH. This bypasses the behavior of setuptools_trial which builds and checks package dependencies before executing the tests. ] [CLI: remove the '-v' alias for --version-and-path, to free up '-v' for more traditional uses (like --verbose) warner@allmydata.com**20090123212150] [#590: add webish t=stream-manifest warner@allmydata.com**20090123040136] [dirnode: add get_repair_cap() warner@allmydata.com**20090123034449] [dirnode.deep_traverse: fix docstring to describe the correct return value warner@allmydata.com**20090123033950] [filenode: add get_repair_cap(), which uses the read-write filecap for immutable files, and the verifycap for immutable files warner@allmydata.com**20090123033836] [setup: make the bin/tahoe executable exit with the appropriate exit code zooko@zooko.com**20090122215405 Ignore-this: 98fc14e578be45b7c1b527fc7ace73b1 ] [setup: add test that the tests are testing the right source code zooko@zooko.com**20090122215240 Ignore-this: f56c1bc525924154042fefa5d30b04f6 This is a test of #145, and I think that now the code passes this test. ] [trivial: removed unused imports noticed by pyflakes zooko@zooko.com**20090122215213 Ignore-this: a83aa1f27b31a72aed07caa866abfafe ] [setup: change test_runner to invoke "bin/tahoe" in a subprocess instead of executing runner.runner() zooko@zooko.com**20090122213818 Ignore-this: f7ef67adf1b9508617c9a7d305191627 This is necessary because loading allmydata code now depends on PYTHONPATH manipulation which is done in the "bin/tahoe" script. Unfortunately it makes test_runner slower since it launches and waits for many subprocesses. ] [setup: fix "tahoe start" to work on Windows even when a Tahoe base dir hasn't been configured in the Windows registry zooko@zooko.com**20090121184720 Ignore-this: ba147a8f75e8aa9cdc3ee0a56dbf7413 ] [docs: trivial naming change zooko@zooko.com**20090121025042 Ignore-this: a9c2fe6119c43683c6f88662e60de306 ] [rollback the feature of making "ambient upload authority" configurable zooko@zooko.com**20090121024735 Ignore-this: 3fcea1b8179e6278adc360414b527b8b This reverses some, but not all, of the changes that were committed in the following set of patches. rolling back: Sun Jan 18 09:54:30 MST 2009 toby.murray * add 'web.ambient_upload_authority' as a paramater to tahoe.cfg M ./src/allmydata/client.py -1 +3 M ./src/allmydata/test/common.py -7 +9 A ./src/allmydata/test/test_ambient_upload_authority.py M ./src/allmydata/web/root.py +12 M ./src/allmydata/webish.py -1 +4 Sun Jan 18 09:56:08 MST 2009 zooko@zooko.com * trivial: whitespace I ran emacs's "M-x whitespace-cleanup" on the files that Toby's recent patch had touched that had trailing whitespace on some lines. M ./src/allmydata/test/test_ambient_upload_authority.py -9 +8 M ./src/allmydata/web/root.py -2 +1 M ./src/allmydata/webish.py -2 +1 Mon Jan 19 14:16:19 MST 2009 zooko@zooko.com * trivial: remove unused import noticed by pyflakes M ./src/allmydata/test/test_ambient_upload_authority.py -1 Mon Jan 19 21:38:35 MST 2009 toby.murray * doc: describe web.ambient_upload_authority M ./docs/configuration.txt +14 M ./docs/frontends/webapi.txt +11 Mon Jan 19 21:38:57 MST 2009 zooko@zooko.com * doc: add Toby Murray to the CREDITS M ./CREDITS +4 ] [setup: add new bundled setuptools_trial-0.4 zooko@zooko.com**20090120234012 Ignore-this: f260255f657af2792beac9314fdc9ab4 ] [setup: remove old bundled setuptools_trial-0.2 zooko@zooko.com**20090120233918 Ignore-this: 6fce4d5106357981871ba0b8c7fb464e ] [setup: require darcsver always, and not just when we see the string "darcsver" in sys.argv, because the new aliases hack means that the string might not appear in sys.argv zooko@zooko.com**20090120184229 Ignore-this: beecca08d9f59704be7ef1aeca6fd779 ] [setup: fix test_system to require tahoe under its package==distribution name "allmydata-tahoe" instead of its module name "allmydata" zooko@zooko.com**20090120183809 Ignore-this: 51e925c03dbcb9ccf1d7ee34c91c6e72 ] [setup: use setup.cfg aliases to map "setup.py test" to "setup.py trial" and "setup.py build" to "setup.py darcsver --count-all-patches build_tahoe" zooko@zooko.com**20090120183723 Ignore-this: f390676787f4d521c17fbe96fb2cd2a6 Thanks to dpeterson for the suggestion. ] [doc: add Toby Murray to the CREDITS zooko@zooko.com**20090120043857 Ignore-this: eedb7e9d47ddee5cbe189b55251d0859 ] [doc: describe web.ambient_upload_authority toby.murray**20090120043835 Ignore-this: cc1920b2c5d4d587af84c4d251ad0e4b ] [trivial: remove unused import noticed by pyflakes zooko@zooko.com**20090119211619 Ignore-this: 4999f513a5c8d73ed8f79c2b012fea6b ] [setup: simplify install.html a tad zooko@zooko.com**20090119210447 Ignore-this: 529b2f225b3d98ed3bc99a4962e781ee ] [setup: refactor versions-and-paths and use pkg_resources to find them zooko@zooko.com**20090119210435 Ignore-this: b368d8ede7531f1d79ee3c2c1a2cc116 Using pkg_resources is probably better if it works -- zope.interface doesn't have a __version__ attribute that we can query, but pkg_resources knows zope.interface's version number, for one thing. This code falls back to the old way -- looking at the __version__ attributes and __file__ attributes -- if the pkg_resources way doesn't answer. Note that this patch also changes the capitalization of "Nevow", "Twisted", and "pyOpenSSL", and the spelling of "allmydata-tahoe". These changes are not frivolous: they are reflecting the fact that we are naming Python packages (technically called Python "distributions") instead of Python modules (technically and confusingly called Python "packages") here. The package ("distribution") is named "allmydata-tahoe". The module ("package") is named "allmydata". ] [setup: undo (for the second time) the use of the --multi-version feature zooko@zooko.com**20090119205352 Ignore-this: 80bdbb488c3e4fb042bcd968a9bc120a When this feature is turned on, then setuptools doesn't create easy-install.pth, setuptools.pth, or site.py in the target site-packages dir. I don't know why not and we should probably open a ticket on the setuptools tracker and/or hack setuptools to create those files anyway. But for now (for the Tahoe-1.3.0 release), we're going to leave --multi-version mode off and require users to manually uninstall any packages which are too old and thus conflict with our newer dependencies. ] [trivial: whitespace zooko@zooko.com**20090118165815 Ignore-this: 6f97042f221da3ad931c6b545edc6a30 Ran "M-x whitespace-cleanup" on files that Toby's recent patch touched, even though they didn't have trailing whitespace. ] [trivial: whitespace zooko@zooko.com**20090118165608 Ignore-this: 8539e7e73e43f459f7b82e84dface95c I ran emacs's "M-x whitespace-cleanup" on the files that Toby's recent patch had touched that had trailing whitespace on some lines. ] [trivial: whitespace zooko@zooko.com**20090118165458 Ignore-this: dedfb44829ea73f0e39f582f16112506 ] [add 'web.ambient_upload_authority' as a paramater to tahoe.cfg toby.murray**20090118165430 Ignore-this: 2c6ed484009c03fe9db1bb6eb67500ff ] [doc: add Larry Hosken to CREDITS zooko@zooko.com**20090117164943 Ignore-this: f2433a296ab2485872d22538bd0f64b2 ] [run build_tahoe command with trial commmand cgalvan@mail.utexas.edu**20090117000047] [adding multi-version support cgalvan@mail.utexas.edu**20090116230326] [setup: addition of setuptools_trial egg to the base dir is boring from the point of view of revision control zooko@zooko.com**20090116200554 Ignore-this: 63dd48c06356f638ca85c5e4a25457b1 ] [prevent --site-dirs from being passed to the 'install' command cgalvan@mail.utexas.edu**20090116195732] [add option to show version and path to the tahoe executable cgalvan@mail.utexas.edu**20090116184751] [More lossmodel work, on repair. Shawn Willden **20090116025648] [setup: put back configuration of the PYTHONPATH which is necessary to build the Windows packages zooko@zooko.com**20090115023751 Ignore-this: 936b4d58babb84d1766695ad8c61ced7 ] [node.py: use NODEDIR/tmp for the 'tempfile' module's temporary directory, so webapi upload tempfiles are put there instead of /tmp . You can set it to something else by setting [node]tempdir in tahoe.cfg warner@allmydata.com**20090115020015] [web/operations: undo the disable-ophandle-expiration change that inadvertently got included in zooko's recent 'rename wapi.txt to webapi.txt' patch, causing test failures warner@allmydata.com**20090115011459] [Loss model work (temp1) Shawn Willden **20090115030058] [cli: tests: skip symlink test if there is no os.symlink zooko@zooko.com**20090115001010 Ignore-this: 4987fea4fe070c2dd5ff75401fbf89e1 ] [webapi.txt: explain POST /uri/?t=upload, thanks to Toby Murray for the catch warner@allmydata.com**20090115000803] [mutable: move recent operation history management code (MutableWatcher) into history.py, have History provide stats warner@allmydata.com**20090114233620] [download: tiny cleanup of history code warner@allmydata.com**20090114224151] [upload: move upload history into History object warner@allmydata.com**20090114224106] [immutable/download.py move recent-downloads history out of Downloader and into a separate class. upload/etc will follow soon. warner@allmydata.com**20090114221424] [docs: rename wapi.txt to webapi.txt zooko@zooko.com**20090114195348 Ignore-this: 419685f2807714bab4069fbaa3a02c1c Because Brian argues that the file contains a description of the wui as well as of the wapi, and because the name "webapi.txt" might be more obvious to the untrained eye. ] [setup: configure setup.cfg to point setuptools at the index.html page instead of straight at the tahoe directory listing zooko@zooko.com**20090114185023 Ignore-this: 71e06cbbd91397f3e12109e48d3f8a81 This is necessary, because if setuptools looks at the directory listing, then it follows the link named "More Info" in attempt to download the file, and this fails. ] [setup: fix previous patch to set reactor to poll reactor on linux or cygwin zooko@zooko.com**20090114164022 Ignore-this: 9e37242ca3cfd47c6f69ee2438ec743b ] [setup: use poll reactor for trial if on linux2 or cygwin zooko@zooko.com**20090114151546 Ignore-this: f6c4b45745527811c9f72448d74649f5 ] [docs: rename frontends/webapi.txt to frontends/wapi.txt zooko@zooko.com**20090114025143 Ignore-this: c35acd8dc7c1106ca31104b6db43e5ad rename CLI.txt to frontends/CLI.txt change a few mentions of "webapi" to "wapi" fixes #582 ] [Statistics module Shawn Willden **20090114021235 Added a statistics module for calculating various facets of share survival statistics. ] [upload: use WriteBucketProxy_v2 when uploading a large file (with shares larger than 4GiB). This finally closes #346. I think we can now handle immutable files up to 48EiB. warner@allmydata.com**20090113021442] [deep-check-and-repair: improve results and their HTML representation warner@allmydata.com**20090113005619] [test_repairer.py: hush pyflakes: remove duplicate/shadowed function name, by using the earlier definition (which is identical) warner@allmydata.com**20090112214509] [hush pyflakes by removing unused imports warner@allmydata.com**20090112214120] [immutable repairer zooko@zooko.com**20090112170022 Ignore-this: f17cb07b15a554b31fc5203cf4f64d81 This implements an immutable repairer by marrying a CiphertextDownloader to a CHKUploader. It extends the IDownloadTarget interface so that the downloader can provide some metadata that the uploader requires. The processing is incremental -- it uploads the first segments before it finishes downloading the whole file. This is necessary so that you can repair large files without running out of RAM or using a temporary file on the repairer. It requires only a verifycap, not a readcap. That is: it doesn't need or use the decryption key, only the integrity check codes. There are several tests marked TODO and several instances of XXX in the source code. I intend to open tickets to document further improvements to functionality and testing, but the current version is probably good enough for Tahoe-1.3.0. ] [util: dictutil: add DictOfSets.union(key, values) and DictOfSets.update(otherdictofsets) zooko@zooko.com**20090112165539 Ignore-this: 84fb8a2793238b077a7a71aa03ae9d2 ] [setup: update doc in setup.cfg zooko@zooko.com**20090111151319 Ignore-this: 296bfa1b9dbdac876f2d2c8e4e2b1294 ] [setup: Point setuptools at a directory on the allmydata.org test grid to find dependencies. zooko@zooko.com**20090111151126 Ignore-this: f5b6a8f5ce3ba08fea2573b5c582aba8 Don't include an unrouteable IP address in find_links (fixes #574). ] [immutable: separate tests of immutable upload/download from tests of immutable checking/repair zooko@zooko.com**20090110210739 Ignore-this: 9e668609d797ec86a618ed52602c111d ] [trivial: minor changes to in-line comments -- mark plaintext-hash-tree as obsolete zooko@zooko.com**20090110205601 Ignore-this: df286154e1acde469f28e9bd00bb1068 ] [immutable: make the web display of upload results more human-friendly, like they were before my recent change to the meaning of the "sharemap" zooko@zooko.com**20090110200209 Ignore-this: 527d067334f982cb2d3e185f72272f60 ] [immutable: fix edit-o in interfaces.py documentation introduced in recent patch zooko@zooko.com**20090110185408 Ignore-this: f255d09aa96907c402583fc182379391 ] [immutable: redefine the "sharemap" member of the upload results to be a map from shnum to set of serverids zooko@zooko.com**20090110174623 Ignore-this: 10300a2333605bc26c4ee9c7ab7dae10 It used to be a map from shnum to a string saying "placed this share on XYZ server". The new definition is more in keeping with the "sharemap" object that results from immutable file checking and repair, and it is more useful to the repairer, which is a consumer of immutable upload results. ] [naming: finish renaming "CheckerResults" to "CheckResults" zooko@zooko.com**20090110000052 Ignore-this: b01bd1d066d56eff3a6322e0c3a9fbdc ] [storage.py : replace 4294967295 with 2**32-1: python does constant folding, I measured this statement as taking 50ns, versus the 400ns for the call to min(), or the 9us required for the 'assert not os.path.exists' syscall warner@allmydata.com**20090110015222] [storage.py: announce a maximum-immutable-share-size based upon a 'df' of the disk. Fixes #569, and this should be the last requirement for #346 (remove 12GiB filesize limit) warner@allmydata.com**20090110013736] [set bin/tahoe executable permissions and leave build_tahoe in sys.argv cgalvan@mail.utexas.edu**20090109210640] [setup: merge relaxation of the version of setuptools that we require at runtime with an indentation change zooko@zooko.com**20090109190949 Ignore-this: eb396c71563b9917c8a485efc5bebb36 ] [setup: remove custom Trial class inside our setup.py and use the setuptools_trial plugin zooko@zooko.com**20081205232207 Ignore-this: e0f68169e8ac1b5a54b796e8905c7b80 ] [setup: we require pywin32 if building on Windows (plus some formatting and comment fixes) zooko@zooko.com**20081205231911 Ignore-this: c1d1966cfe458a6380bfd5dce09010ff ] [fix bin/tahoe executable for Windows cgalvan@mail.utexas.edu**20090109184222] [use subprocess.call instead of os.execve in bin/tahoe cgalvan@mail.utexas.edu**20090109180300] [setup: attempt to remove the custom setuptools-ish logic in setup.py -- the result works on my Windows box but doesn't yield a working ./bin/tahoe on Windows, and hasn't been tested yet on other platforms zooko@zooko.com**20081205233054 Ignore-this: 843e7514870d7a4e708646acaa7c9699 ] [setup: integrate the bundled setuptools_trial plugin with Chris Galvan's patch to use that plugin zooko@zooko.com**20081201174804 Ignore-this: 5d03e936cf45f67a39f993704024788c ] [use_setuptools_trial.patch cgalvan@mail.utexas.edu**20081121205759] [setup: bundle setuptools_trial in misc/dependencies/ zooko@zooko.com**20081201174438 Ignore-this: f13a4a1af648f9ab9b3b3438cf94053f ] [test_helper: hush pyflakes by avoiding use of 'uri' as a variable, since it shadows an import of the same name warner@allmydata.com**20090109025941] [immutable/checker: include a summary (with 'Healthy' or 'Not Healthy' and a count of shares) in the checker results warner@allmydata.com**20090109020145] [webapi/deep-manifest t=JSON: don't return the (large) manifest/SI/verifycap lists unless the operation has completed, to avoid the considerable CPU+memory cost of creating the JSON (for 330k dirnodes, it could take two minutes to generate 275MB of JSON). They must be paid eventually, but not on every poll warner@allmydata.com**20090109015932] [dirnode deep-traversal: remove use of Limiter, stick with strict depth-first-traversal, to reduce memory usage during very large (300k+ dirnode) traversals warner@allmydata.com**20090109014116] [immutable: add a monitor API to CiphertextDownloader with which to tell it to stop its work zooko@zooko.com**20090108204215 Ignore-this: f96fc150fa68fc2cec46c943171a5d48 ] [naming: Rename a few things which I touched or changed in the recent patch to download-without-decrypting. zooko@zooko.com**20090108181307 Ignore-this: 495ce8d8854c5db5a09b35b856809fba Rename "downloadable" to "target". Rename "u" to "v" in FileDownloader.__init__(). Rename "_uri" to "_verifycap" in FileDownloader. Rename "_downloadable" to "_target" in FileDownloader. Rename "FileDownloader" to "CiphertextDownloader". ] [immutable: refactor download to do only download-and-decode, not decryption zooko@zooko.com**20090108175349 Ignore-this: 1e4f26f6390a67aa5714650017c4dca1 FileDownloader takes a verify cap and produces ciphertext, instead of taking a read cap and producing plaintext. FileDownloader does all integrity checking including the mandatory ciphertext hash tree and the optional ciphertext flat hash, rather than expecting its target to do some of that checking. Rename immutable.download.Output to immutable.download.DecryptingOutput. An instance of DecryptingOutput can be passed to FileDownloader to use as the latter's target. Text pushed to the DecryptingOutput is decrypted and then pushed to *its* target. DecryptingOutput satisfies the IConsumer interface, and if its target also satisfies IConsumer, then it forwards and pause/unpause signals to its producer (which is the FileDownloader). This patch also changes some logging code to use the new logging mixin class. Check integrity of a segment and decrypt the segment one block-sized buffer at a time instead of copying the buffers together into one segment-sized buffer (reduces peak memory usage, I think, and is probably a tad faster/less CPU, depending on your encoding parameters). Refactor FileDownloader so that processing of segments and of tail-segment share as much code is possible. FileDownloader and FileNode take caps as instances of URI (Python objects), not as strings. ] [trivial: tiny changes to test code zooko@zooko.com**20090108172048 Ignore-this: b1a434cd40a87c3d027fef4ce609d25c ] [immutable: Make more parts of download use logging mixins and know what their "parent msg id" is. zooko@zooko.com**20090108172530 Ignore-this: a4296b5f9b75933d644fd222e1fba079 ] [trivial: M-x whitespace-cleanup on src/immutable/download.py zooko@zooko.com**20090108164901 Ignore-this: bb62daf511e41a69860be657cde8df04 ] [immutable: ValidatedExtendedURIProxy computes and stores the tail data size as a convenience to its caller. zooko@zooko.com**20090108164139 Ignore-this: 75c561d73b17418775faafa60fbbd45b The "tail data size" is how many of the bytes of the tail segment are data (as opposed to padding). ] [immutable: define a new interface IImmutableFileURI and declare that CHKFileURI and LiteralFileURI provide it zooko@zooko.com**20090107182451 Ignore-this: 12c256a0d20655cd73739d45fff0d4d8 ] [util: log: allow empty msgs (because downloader is using the "format" alternative with no "msg" argument) zooko@zooko.com**20090107175411 Ignore-this: 832c333bf027a30a2fcf96e462297ac5 ] ['tahoe cp -r', upon encountering a dangling symlink, would assert out. Larry Hosken **20090108055114 Ignore-this: 46e75845339faa69ffb3addb7ce74f28 This was somewhat sad; the assertion didn't say what path caused the error, what went wrong. So... silently skip over things that are neither dirs nor files. ] [immutable: fix error in validation of ciphertext hash tree and add test for that code zooko@zooko.com**20090108054012 Ignore-this: 3241ce66373ebc514ae6e6f086f6daa2 pyflakes pointed out to me that I had committed some code that is untested, since it uses an undefined name. This patch exercises that code -- the validation of the ciphertext hash tree -- by corrupting some of the share files in a very specific way, and also fixes the bug. ] [immutable: do not catch arbitrary exceptions/failures from the attempt to get a crypttext hash tree -- catch only ServerFailure, IntegrityCheckReject, LayoutInvalid, ShareVersionIncompatible, and DeadReferenceError zooko@zooko.com**20090108042551 Ignore-this: 35f208af1b9f8603df25ed69047360d1 Once again I inserted a bug into the code, and once again it was hidden by something catching arbitrary exception/failure and assuming that it means the server failed to provide valid data. ] [download: make sure you really get all the crypttext hashes zooko@zooko.com**20090108022638 Ignore-this: c1d5ebb048e81f706b9098e26876e040 We were not making sure that we really got all the crypttext hashes during download. If a server were to return less than the complete set of crypttext hashes, then our subsequent attempt to verify the correctness of the ciphertext would fail. (And it wouldn't be obvious without very careful debugging why it had failed.) This patch makes it so that you keep trying to get ciphertext hashes until you have a full set or you run out of servers to ask. ] [util: deferredutil: undo my recent patch to use our own implementation of gatherResults zooko@zooko.com**20090107170005 Ignore-this: c8c5421b47ab5a83c7ced8b08add80e8 It seems to cause lots of failures on some builders. ] [util: deferredutil: implement our own gatherResults instead of using Twisted's zooko@zooko.com**20090107163207 Ignore-this: aa676b2b6cfb73bbca15827cb7c0a43e Because we want to maintain backwards compatibility to Twisted 2.4.0. ] [trivial: M-x whitespace-cleanup zooko@zooko.com**20090107162528 Ignore-this: 69bef4518477ca875785f0e0b8ab0000 ] [util: deferredutil: add basic test for deferredutil.gatherResults zooko@zooko.com**20090107141342 Ignore-this: ad457053c8ee3a04921fdcdb639c03d Also I checked and Twisted 2.4.0 supports .subFailure and the other parts of the API that we require. ] [trivial: fix redefinition of name "log" in imports (pyflakes) zooko@zooko.com**20090107040829 Ignore-this: cdcf7ff84082323ebc022b186127e678 ] [immutable: refactor uploader to do just encoding-and-uploading, not encryption zooko@zooko.com**20090107034822 Ignore-this: 681f3ad6827a93f1431d6e3f818840a9 This makes Uploader take an EncryptedUploadable object instead of an Uploadable object. I also changed it to return a verify cap instead of a tuple of the bits of data that one finds in a verify cap. This will facilitate hooking together an Uploader and a Downloader to make a Repairer. Also move offloaded.py into src/allmydata/immutable/. ] [trivial: whitespace and docstring tidyups zooko@zooko.com**20090107034104 Ignore-this: 34db3eec599efbb2088a87333abfb797 ] [storage.py: explain what this large and hard-to-recognize 4294967295 number is warner@allmydata.com**20090106195721] [rename "checker results" to "check results", because it is more parallel to "check-and-repair results" zooko@zooko.com**20090106193703 Ignore-this: d310e3d7f42a76df68536650c996aa49 ] [immutable: tests: verifier doesn't always catch corrupted share hashes zooko@zooko.com**20090106190449 Ignore-this: a9be83b8e2350ae9af808476015fe0e4 Maybe it already got one of the corrupted hashes from a different server and it doesn't double-check that the hash from every server is correct. Or another problem. But in any case I'm marking this as TODO because an even better (more picky) verifier is less urgent than repairer. ] [immutable: fix the writing of share data size into share file in case the share file is used by a < v1.3.0 storage server zooko@zooko.com**20090106182404 Ignore-this: 7d6025aba05fe8140bb712e71e89f1ba Brian noticed that the constant was wrong, and in fixing that I noticed that we should be saturating instead of modding. This code would never matter unless a server downgraded or a share migrated from Tahoe >= v1.3.0 to Tahoe < v1.3.0. Even in that case, this bug would never matter unless the share size were exactly 4,294,967,296 bytes long. Brian, for good reason, wanted this to be spelled "2**32" instead of "4294967296", but I couldn't stand to see a couple of more Python bytecodes interpreted in the middle of a core, frequent operation on the server like immutable share creation. ] [trivial: whitespace cleanup zooko@zooko.com**20090106172058 Ignore-this: 50ee40d42cc8d8f39d2f8ed15f6790d4 ] [util: base32: require str-not-unicode inputs -- effectively rolls back [3306] and [3307] zooko@zooko.com**20090106164122 Ignore-this: 1030c2d37e636d194c99ec99707ae86f ] [trivial: fix a bunch of pyflakes complaints zooko@zooko.com**20090106140054 Ignore-this: 9a515a237248a148bcf8db68f70566d4 ] [cli: make startstop_node wait 40 seconds instead of 20 for a process to go away after we signalled it to go away, before emitting a warning zooko@zooko.com**20090106135106 Ignore-this: 2da4b794b6a7e2e2ad6904cce61b0f10 Because the unit tests on the VirtualZooko? buildslave failed when it took 31 seconds for a process to go away. Perhaps getting warning message after only 5 seconds instead of 40 seconds is desirable, and we should change the unit tests and set this back to 5, but I don't know exactly how to change the unit tests. Perhaps match this particular warning message about the shutdown taking a while and allow the code under test to pass if the only stderr that it emits is this warning. ] [immutable: new checker and verifier zooko@zooko.com**20090106002818 Ignore-this: 65441f8fdf0db8bcedeeb3fcbbd07d12 New checker and verifier use the new download class. They are robust against various sorts of failures or corruption. They return detailed results explaining what they learned about your immutable files. Some grotesque sorts of corruption are not properly handled yet, and those ones are marked as TODO or commented-out in the unit tests. There is also a repairer module in this patch with the beginnings of a repairer in it. That repairer is mostly just the interface to the outside world -- the core operation of actually reconstructing the missing data blocks and uploading them is not in there yet. This patch also refactors the unit tests in test_immutable so that the handling of each kind of corruption is reported as passing or failing separately, can be separately TODO'ified, etc. The unit tests are also improved in various ways to require more of the code under test or to stop requiring unreasonable things of it. :-) ] [trivial: fix inline comment in test code zooko@zooko.com**20090105235342 Ignore-this: b3d79b9644052e6402b2f7d0125f678a ] [immutable: handle another form of share corruption with LayoutInvalid exception instead of AssertionError zooko@zooko.com**20090105234645 Ignore-this: fee5f6572efca5435ef54ed32552ca9d ] [trivial: remove unused import (pyflakes) zooko@zooko.com**20090105233120 Ignore-this: 47b6989ffa5b3a5733e45e8feb507959 ] [immutable: skip the test of large files, because that is too hard on the host if it doesn't efficiently handle sparse files zooko@zooko.com**20090105230727 Ignore-this: 7d35a6cdb1ea6be2adf0e6dacefe01a7 ] [immutable: raise a LayoutInvalid exception instead of an AssertionError if the share is corrupted so that the sharehashtree is the wrong size zooko@zooko.com**20090105200114 Ignore-this: b63d028c44dcd04ef424d6460b46e349 ] [immutable: stop reading past the end of the sharefile in the process of optimizing download -- Tahoe storage servers < 1.3.0 return an error if you read past the end of the share file zooko@zooko.com**20090105194057 Ignore-this: 365e1f199235a55c0354ba6cb2b05a04 ] [immutable: tidy up the notification of waiters for ReadBucketProxy zooko@zooko.com**20090105193522 Ignore-this: 6b93478dae3d627b9d3cbdd254afbe7e ] [immutable: refactor downloader to be more reusable for checker/verifier/repairer (and better) zooko@zooko.com**20090105155145 Ignore-this: 29a22b1eb4cb530d4b69c12aa0d00740 The code for validating the share hash tree and the block hash tree has been rewritten to make sure it handles all cases, to share metadata about the file (such as the share hash tree, block hash trees, and UEB) among different share downloads, and not to require hashes to be stored on the server unnecessarily, such as the roots of the block hash trees (not needed since they are also the leaves of the share hash tree), and the root of the share hash tree (not needed since it is also included in the UEB). It also passes the latest tests including handling corrupted shares well. ValidatedReadBucketProxy takes a share_hash_tree argument to its constructor, which is a reference to a share hash tree shared by all ValidatedReadBucketProxies for that immutable file download. ValidatedReadBucketProxy requires the block_size and share_size to be provided in its constructor, and it then uses those to compute the offsets and lengths of blocks when it needs them, instead of reading those values out of the share. The user of ValidatedReadBucketProxy therefore has to have first used a ValidatedExtendedURIProxy to compute those two values from the validated contents of the URI. This is pleasingly simplifies safety analysis: the client knows which span of bytes corresponds to a given block from the validated URI data, rather than from the unvalidated data stored on the storage server. It also simplifies unit testing of verifier/repairer, because now it doesn't care about the contents of the "share size" and "block size" fields in the share. It does not relieve the need for share data v2 layout, because we still need to store and retrieve the offsets of the fields which come after the share data, therefore we still need to use share data v2 with its 8-byte fields if we want to store share data larger than about 2^32. Specify which subset of the block hashes and share hashes you need while downloading a particular share. In the future this will hopefully be used to fetch only a subset, for network efficiency, but currently all of them are fetched, regardless of which subset you specify. ReadBucketProxy hides the question of whether it has "started" or not (sent a request to the server to get metadata) from its user. Download is optimized to do as few roundtrips and as few requests as possible, hopefully speeding up download a bit. ] [util: add gatherResults which is a deferred-list-like thing that doesn't wrap failures in a FirstError zooko@zooko.com**20090104165202 Ignore-this: a284fb8ab8a00a39416a67dc5d9a451e ] [immutable: fix think-o in previous patch which caused all reads to return "", and also optimize by not opening the file when the answer is going to be "" zooko@zooko.com**20090103200245 Ignore-this: 8ac4d0b0399cd74e8a424ffbcf3d9eb9 ] [immutable: when storage server reads from immutable share, don't try to read past the end of the file (Python allocates space according to the amount of data requested, so if there is corruption and that number is huge it will do a huge memory allocation) zooko@zooko.com**20090103192222 Ignore-this: e533a65d74437676d5116369fd7c663b ] [immutable: mark a failing download test as "todo", because I think it is revealing a limitation of the current downloader's handling of corrupted shares zooko@zooko.com**20090103190003 Ignore-this: 1d429912dda92d986e2ee366d73e088c ] [docs: update install.html to recommend Python v2 instead of Python v2.5.2 zooko@zooko.com**20090103183100 Ignore-this: 5dbea379c59e0d9be817cdd9c8393d65 ] [trivial: remove unused import (pyflakes) zooko@zooko.com**20090103182215 Ignore-this: 4a29a14fa4580460a5e61fa0aa88b9b2 ] [merge_install.patch cgalvan@mail.utexas.edu**20090102164434 Ignore-this: aa6d4c05d583a0724eb218fef04c3940 ] [setup: new install doc -- doesn't require GNU make or a C++ compiler any more! zooko@zooko.com**20081201180933 Ignore-this: 753e8d1e6f32e2ddcd7a082050114725 ] [immutable: fix test for truncated reads of URI extension block size zooko@zooko.com**20090103174427 Ignore-this: d9ff9dfff88b4cc7aa6751ce2e9088a6 ] [immutable: further loosen the performance-regression test to allow up to 45 reads zooko@zooko.com**20090103174109 Ignore-this: 614f7dba9c0d310a220e74e45441f07 This does raise the question of if there is any point to this test, since I apparently don't know what the answer *should* be, and whenever one of the buildbots fails then I redefine success. But, I'm about to commit a bunch of patches to implement checker, verifier, and repairer as well as to refactor downloader, and I would really like to know if these patches *increase* the number of reads required even higher than it currently is. ] [trivial: another place where I accidentally committed a note-to-self about the lease fields in the server-side share file zooko@zooko.com**20090103172941 Ignore-this: c23c7095ffccdf5aa033ed434b50582b ] [immutable: fix detection of truncated shares to take into account the fieldsize -- either 4 or 8 zooko@zooko.com**20090103005745 Ignore-this: 710184bd90f73dc18f3899d90ec6e972 ] [immutable: raise LayoutInvalid instead of struct.error when a share is truncated zooko@zooko.com**20090103004806 Ignore-this: 346c779045f79725965a0f2d3eea41f9 To fix this error from the Windows buildslave: [ERROR]: allmydata.test.test_immutable.Test.test_download_from_only_3_remaining_shares Traceback (most recent call last): File "C:\Documents and Settings\buildslave\windows-native-tahoe\windows\build\src\allmydata\immutable\download.py", line 135, in _bad raise NotEnoughSharesError("ran out of peers, last error was %s" % (f,)) allmydata.interfaces.NotEnoughSharesError: ran out of peers, last error was [Failure instance: Traceback: : unpack requires a string argument of length 4 c:\documents and settings\buildslave\windows-native-tahoe\windows\build\support\lib\site-packages\foolscap-0.3.2-py2.5.egg\foolscap\call.py:667:_done c:\documents and settings\buildslave\windows-native-tahoe\windows\build\support\lib\site-packages\foolscap-0.3.2-py2.5.egg\foolscap\call.py:53:complete c:\Python25\lib\site-packages\twisted\internet\defer.py:239:callback c:\Python25\lib\site-packages\twisted\internet\defer.py:304:_startRunCallbacks --- --- c:\Python25\lib\site-packages\twisted\internet\defer.py:317:_runCallbacks C:\Documents and Settings\buildslave\windows-native-tahoe\windows\build\src\allmydata\immutable\layout.py:374:_got_length C:\Python25\lib\struct.py:87:unpack ] =============================================================================== ] [immutable: whoops, it actually takes up to 39 reads sometimes to download a corrupted file zooko@zooko.com**20090102234302 Ignore-this: ef009d179eb4f84a56559017b633d819 ] [immutable: add more detailed tests of download, including testing the count of how many reads different sorts of downloads take zooko@zooko.com**20090102225459 Ignore-this: d248eb3982fdb05b43329142a723f5a1 ] [trivial: a few improvements to in-line doc and code, and renaming of test/test_immutable_checker.py to test/test_immutable.py zooko@zooko.com**20090102224941 Ignore-this: 27b97a06c3edad1821f43876b4350f3 That file currently tests checker and verifier and repairer, and will soon also test downloader. ] [immutable: fix name change from BadOrMissingShareHash to BadOrMissingHash zooko@zooko.com**20090102192709 Ignore-this: 3f22ca1ee045beabb11559512ba130f4 One of the instances of the name accidentally didn't get changed, and pyflakes noticed. The new downloader/checker/verifier/repairer unit tests would also have noticed, but those tests haven't been rolled into a patch and applied to this repo yet... ] [trivial: remove unused import -- thanks, pyflakes zooko@zooko.com**20090102192128 Ignore-this: d99c7349ba6f8db971e31cf8789883d5 ] [immutable: download.py: Raise the appropriate type of exception to indicate the cause of failure, e.g. BadOrMissingHash, ServerFailure, IntegrityCheckReject (which is a supertype of BadOrMissingHash). This helps users (such as verifier/repairer) catch certain classes of reasons for "why did this download not work". The tests of verifier/repairer test this code and rely on this code. zooko@zooko.com**20090102185858 Ignore-this: 377bf621bbb6e360a98fd287bb1593f1 ] [immutable: ReadBucketProxy defines classes of exception: LayoutInvalid and its two subtypes RidiculouslyLargeURIExtensionBlock and ShareVersionIncompatible. This helps users (such as verifier/repairer) catch certain classes of reasons for "why did this download not work". This code gets exercised by the verifier/repairer unit tests, which corrupt the shares on disk in order to trigger problems like these. zooko@zooko.com**20090102181554 Ignore-this: 2288262a59ee695f524859ed4b0b39d5 ] [immutable: ValidatedExtendedURIProxy computes and stores block_size and share_size for the convenience of its users zooko@zooko.com**20090102174317 Ignore-this: 2bab64048fffc05dc6592d617aeb412f ] [remove_sumo_install.patch cgalvan@mail.utexas.edu**20090102162347 Ignore-this: f328570b1da1ccfbaebc770d40748046 ] [doc: remove notes to self that I accidentally included in a recent patch zooko@zooko.com**20090102041457 Ignore-this: d0039512dbde09811fdec48a2e00dc4 ] [docs: remove caveat about Nevow incompatibility with Python 2.6 since the latest version of Nevow has fixed it zooko@zooko.com**20090102034135 Ignore-this: 4cb2ceb41f53e07dab0f623e01044edc ] [immutable: make the test of large files more likely to work by requesting to allocate space for only one huge share, not three zooko@zooko.com**20081231215942 Ignore-this: d7073de4764506550e184f8fdc670962 ] [trivial: "M-x whitespace-cleanup", and also remove an unused variable zooko@zooko.com**20081231214233 Ignore-this: 54c33c205aa88de8655e4232d07f083e ] [immutable: storage servers accept any size shares now zooko@zooko.com**20081231214226 Ignore-this: 28669d591dddaff69088cba4483da61a Nathan Wilcox observed that the storage server can rely on the size of the share file combined with the count of leases to unambiguously identify the location of the leases. This means that it can hold any size share data, even though the field nominally used to hold the size of the share data is only 32 bits wide. With this patch, the storage server still writes the "size of the share data" field (just in case the server gets downgraded to an earlier version which requires that field, or the share file gets moved to another server which is of an earlier vintage), but it doesn't use it. Also, with this patch, the server no longer rejects requests to write shares which are >= 2^32 bytes in size, and it no longer rejects attempts to read such shares. This fixes http://allmydata.org/trac/tahoe/ticket/346 (increase share-size field to 8 bytes, remove 12GiB filesize limit), although there remains open a question of how clients know that a given server can handle large shares (by using the new versioning scheme, probably). Note that share size is also limited by another factor -- how big of a file we can store on the local filesystem on the server. Currently allmydata.com typically uses ext3 and I think we typically have block size = 4 KiB, which means that the largest file is about 2 TiB. Also, the hard drives themselves are only 1 TB, so the largest share is definitely slightly less than 1 TB, which means (when K == 3), the largest file is less than 3 TB. This patch also refactors the creation of new sharefiles so that only a single fopen() is used. This patch also helps with the unit-testing of repairer, since formerly it was unclear what repairer should expect to find if the "share data size" field was corrupted (some corruptions would have no effect, others would cause failure to download). Now it is clear that repairer is not required to notice if this field is corrupted since it has no effect on download. :-) ] [trivial: "M-x whitespace-cleanup" on immutable/layout.py zooko@zooko.com**20081231210702 Ignore-this: 8be47d01cf40d1f81aeb0011a0a0caa ] [trivial: remove unused import -- thanks, pyflakes zooko@zooko.com**20081231212556 Ignore-this: a70cd39a7d633bde2bb5275dfd4d3781 ] [rrefutil: generically wrap any errback from callRemote() in a ServerFailure instance zooko@zooko.com**20081231202830 Ignore-this: c949eaf8589ed4c3c232f17808fdce6a This facilitates client code to easily catch ServerFailures without also catching exceptions arising from client-side code. See also: http://foolscap.lothar.com/trac/ticket/105 # make it easy to distinguish server-side failures/exceptions from client-side ] [immutable: more detailed tests for checker/verifier/repairer zooko@zooko.com**20081231201838 Ignore-this: dd16beef604b0917f4493bc4ef35ab74 There are a lot of different ways that a share could be corrupted, or that attempting to download it might fail. These tests attempt to exercise many of those ways and require the checker/verifier/repairer to handle each kind of failure well. ] [docs: add note about non-ascii chars in cli to NEWS zooko@zooko.com**20081230081728 Ignore-this: c6f45a1d944af3c77942a4bf740ee24c ] [cli: make startstop_node wait 20 seconds instead of 5 for a process to go away after we signalled it to go away zooko@zooko.com**20081230072022 Ignore-this: 3b0d47649e32b01ff55a506245c674c6 Because the unit tests on the VirtualZooko buildslave failed when it took 16 seconds for a process to go away. Perhaps getting notification after only 5 seconds instead of 20 seconds is desirable, and we should change the unit tests and set this back to 5, but I don't know exactly how to change the unit tests. Perhaps match this particular warning message about the shutdown taking a while and allow the code under test to pass if the only stderr that it emits is this warning. ] [docs: editing changes and updated news in known_issues.txt zooko@zooko.com**20081230070116 Ignore-this: e5dddc4446e3335a6c4eee7472e0670e ] [docs: split historical/historical_known_issues.txt out of known_issues.txt zooko@zooko.com**20081230065226 Ignore-this: 9b6d0d679294110deeb0ea18b4ad7ac8 All issues which are relevant to users of v1.1, v1.2, or v1.3 go in known_issues.txt. All issues which are relevant to users of v1.0 go in historical/historical_known_issues.txt. ] [doc: sundry amendments to docs and in-line code comments zooko@zooko.com**20081228225954 Ignore-this: a38057b9bf0f00afeea1c468b2237c36 ] [doc: add mention of "tahoe create-alias" in the security-warning section of CLI.txt zooko@zooko.com**20081224211646 Ignore-this: 6bb0ab3af59a79e05ebccb800d9a12b0 ] [doc: trivial: remove trailing whitespace zooko@zooko.com**20081224211634 Ignore-this: 6ff234bc7632c3ae4d4f2be2198bb97d ] [doc: warn that unicode might not work, in CLI.txt zooko@zooko.com**20081224211618 Ignore-this: 89355b53aab40af1d45a3746bb90ed10 ] [doc: use the term "filesystem" rather than "virtual drive" in CLI.txt zooko@zooko.com**20081224211614 Ignore-this: c9541955201671c1a3a8c6ca7be4e7d ] [cli: mark unicode filenames as unsupported -- see #534 for details zooko@zooko.com**20081224192802 Ignore-this: b209ccbd838f633ec201e2e97156847c ] [cli: undo the effects of [http://allmydata.org/trac/tahoe/changeset/20081222235453-92b7f-f841e18afb94e1fd95e6dafb799a3d876dd85c69] zooko@zooko.com**20081224155317 Ignore-this: d34ee20d89221357e32872d721d7685f We're just going to mark unicode in the cli as unsupported for tahoe-lafs-1.3.0. Unicode filenames on the command-line do actually work for some platforms and probably only if the platform encoding is utf-8, but I'm not sure, and in any case for it to be marked as "supported" it would have to work on all platforms, be thoroughly tested, and also we would have to understand why it worked. :-) ] [test: extend timeout on the hotline file that prevents the client from stopping itself zooko@zooko.com**20081222030629 Ignore-this: 391f48caef9d6ad558e540ded56a8075 The 20-second timeout was apparently tripped on my Powerbook G4 "draco". ] [cli: decode all cli arguments, assuming that they are utf-8 encoded zooko@zooko.com**20081222235453 Ignore-this: d92b4d146e1dc9848c6a4b6aaaa3d1e9 Also encode all args to urllib as utf-8 because urllib doesn't handle unicode objects. I'm not sure if it is appropriate to *assume* utf-8 encoding of cli args. Perhaps the Right thing to do is to detect the platform encoding. Any ideas? This patch is mostly due to François Deppierraz. ] [util/base32: the identity trans table needn't have any contents -- we are using string.translate solely to delete known chars zooko@zooko.com**20081222234808 Ignore-this: 8fe03ec6571726f44425fc5905387b78 ] [util/base32: allow unicode inputs to a2b() or could_be_base32_encoded(), and encode them with utf-8 before processing them zooko@zooko.com**20081222234713 Ignore-this: e1eb4caed2f78b2fef0df4bbf8bb26f7 ] [util/base32: loosen the precondition forbidding unicode and requiring str -- now it requires either unicode or str zooko@zooko.com**20081222222237 Ignore-this: 3481d644bdc5345facbc199d33653f37 Hopefully this will make it so that tests pass with François Deppierraz's patch to fix the tahoe cli's handling of unicode argument. ] [immutable: don't catch all exception when downloading, catch only DeadReferenceError and IntegrityCheckReject zooko@zooko.com**20081221234135 Ignore-this: 1abe05c3a5910378abc3920961f19aee ] [immutable: invent download.BadOrMissingHashError which is raised if either hashtree.BadHashError, hashtree.NotEnoughHashesError, and which is a subclass of IntegrityCheckReject zooko@zooko.com**20081221234130 Ignore-this: 1b04d7e9402ebfb2cd4c7648eb16af84 ] [dirnode: don't check MAC on entries in dirnodes zooko@zooko.com**20081221233518 Ignore-this: efacb56d18259219c910cf5c84b17340 In an ancient version of directories, we needed a MAC on each entry. In modern times, the entire dirnode comes with a digital signature, so the MAC on each entry is redundant. With this patch, we no longer check those MACs when reading directories, but we still produce them so that older readers will accept directories that we write. ] [immutable, checker, and tests: improve docstrings, assertions, tests zooko@zooko.com**20081221210752 Ignore-this: 403ed5ca120d085d582cd5695d8371f No functional changes, but remove unused code, improve or fix docstrings, etc. ] [cli: if response code from wapi server is not 200 then stop instead of proceeding zooko@zooko.com**20081220134918 Ignore-this: 907481c941fc5696630b9c118137fb52 Also, include the data that failed to json parse in an exception raised by the json parser. ] [immutable: when downloading an immutable file, use primary shares if they are available zooko@zooko.com**20081220131456 Ignore-this: f7b8b76fd7df032673ab072384eaa989 Primary shares require no erasure decoding so the more primary shares you have, the less CPU is used. ] [trivial: remove unused import (thanks, pyflakes) zooko@zooko.com**20081219194629 Ignore-this: 96e15d6de43dd1204a8933171f194189 ] [try to tidy up uri-as-string vs. uri-as-object zooko@zooko.com**20081219143924 Ignore-this: 4280727007c29f5b3e9273a34519893f I get confused about whether a given argument or return value is a uri-as-string or uri-as-object. This patch adds a lot of assertions that it is one or the other, and also changes CheckerResults to take objects not strings. In the future, I hope that we generally use Python objects except when importing into or exporting from the Python interpreter e.g. over the wire, the UI, or a stored file. ] [immutable: remove the last bits of code (only test code or unused code) which did something with plaintext hashes or plaintext hash trees zooko@zooko.com**20081219141807 Ignore-this: d10d26b279794383f27fa59ec4a50219 ] [immutable: use new logging mixins to simplify logging zooko@zooko.com**20081217000450 Ignore-this: 7d942905d1ea8f34753dbb997e1857f3 ] [immutable: refactor ReadBucketProxy a little zooko@zooko.com**20081216235325 Ignore-this: b3733257769eff3b3e9625bd04643fd6 ] [debug: pass empty optional arguments to ReadBucketProxy zooko@zooko.com**20081216235145 Ignore-this: 7132cdc6a52767fbbcca03b242a16982 because those arguments are about to become non-optional (for other code than test/debug code) ] [uri: generalize regexp that recognizes tahoe URLs to work for any host and port zooko@zooko.com**20081216234930 Ignore-this: 4a7716b8034c8e5ed9698a99f1ec5cb4 ] [util: logging: refactor some common logging behavior into mixins zooko@zooko.com**20081216233807 Ignore-this: d91408bc984d1cf1fae30134f6cddb13 ] [pyutil: assertutil: copy in simplified assertutil from pyutil zooko@zooko.com**20081216233745 Ignore-this: cd4a33186c8c134104f07018ab448583 ] [pyutil: assertutil: simplify handling of exception during formatting of precondition message, and reduce dependency to just the Python Standard Library's logging module zooko@zooko.com**20081210131057 Ignore-this: 4a7f1aa5b9f7ac60067347db9cdf5f28 ] [client: add get_servers() zooko@zooko.com**20081208230400 Ignore-this: 1b9b3ff483849563342f467c39fdd15d ] [mutable publish: if we are surprised by shares that match what we would have written anyways, don't be surprised. This should fix one of the two #546 problems, in which we re-use a server and forget that we already sent them a share. warner@allmydata.com**20081210044449] [NEWS: updated to most recent user-visible changes, including the 8123-to-3456 change warner@allmydata.com**20081209231146] [immutable: remove unused code to produce plaintext hashes zooko@zooko.com**20081209224546 Ignore-this: 1ff9b6fa48e0617fea809998a0e3b6e ] [finish renaming 'subshare' to 'block' in immutable/encode.py and in docs/ zooko@zooko.com**20081209223318 Ignore-this: 3d1b519f740c3d1030cb733f76fdae61 ] [introducer: fix bug in recent simplification caught by Brian's sharp code-reviewing eye zooko@zooko.com**20081208231634 Ignore-this: 29854954577018d658be49142177edf2 ] [introducer: simplify get_permuted_peers() implementation and add get_peers() zooko@zooko.com**20081208225725 Ignore-this: 8299c0dc187521f34187e54c72e57dc9 ] [webapi.txt: minor edits warner@allmydata.com**20081208213256] [rename "get_verifier()" to "get_verify_cap()" zooko@zooko.com**20081208184411 Ignore-this: 3ea4d7a78c802b23f628a37cc643c11a ] [setup: try depending on setuptools >= 0.6c6 instead of >= 0.6c7 at run-time, to be able to use the setuptools that came with Ubuntu Gutsy zooko@zooko.com**20081208174725 Ignore-this: 1cfefa8891f627c7ed46f1ff127eeee9 ] [setup: loosen requirement on simplejson to >= 1.4 zooko@zooko.com**20081208143537 Ignore-this: 2e4bec12f047f3f525caa6f234b58784 That's the version of simplejson that comes with ubuntu feisty, and the one that we've required for most of our history. Currently the Ubuntu dapper buildslave fails (see issue #534), and setting the simplejson requirement to be >= 2.0 would fix that failure, but I don't understand why. ] [setup: require simplejson >= 1.7.1 zooko@zooko.com**20081208043412 Ignore-this: ab0e8ba82f0d10bc650bc80732bf3d0e That's the version that comes with gutsy, and we don't really understand why increasing the required version number helped with issue #553. ] [mutable: merge renaming with test patches zooko@zooko.com**20081207144519 Ignore-this: a922a8b231090fb35b9ef84d99e9dba3 ] [mutable: rename mutable/node.py to mutable/filenode.py and mutable/repair.py to mutable/repairer.py zooko@zooko.com**20081207142008 Ignore-this: ecee635b01a21e6f866a11bb349712a3 To be more consistent with the immutable layout that I am working on. ] [web/directory.py: really really fix #553. Unfortunately it's tricky to simulate the behavior of a brower's relative-url handling in a unit test. warner@allmydata.com**20081206051412] [filenode.py: Fix partial HTTP Range header handling according to RFC2616 francois@ctrlaltdel.ch**20081118134135 Tahoe webapi was failing on HTTP request containing a partial Range header. This change allows movies players like mplayer to seek in movie files stored in tahoe. Associated tests for GET and HEAD methods are also included ] [mutable.modify(): after UCWE, publish even if the second invocation of the modifier didn't modify anything. For #551. warner@allmydata.com**20081206044923] [dirnode.py: dirnode.delete which hits UCWE should not fail with NoSuchChildError. Fixes #550. warner@allmydata.com**20081206040837] [MutableFileNode.modify: pass first_time= and servermap= to the modifier callback warner@allmydata.com**20081206040710] [misc/cpu-watcher.tac: tolerate disk-full errors when writing the pickle, and pickle corruption from earlier disk-full errors warner@allmydata.com**20081205215412] [web: fix more info links again zooko@zooko.com**20081205213939 Ignore-this: d51cf2c6393b5799dc615952680cd079 Really, *really* closes #553. ] [web: fix moreinfo link zooko@zooko.com**20081205212939 Ignore-this: 89913601a159437a2c151dd3652e6a94 ] [web: "More Info" link describes the same file that the "file" link points to, rather than to the file under the same name in this directory zooko@zooko.com**20081205210502 Ignore-this: 5017754e11749b376c7fa66d1acb2a58 It's a subtle but real difference. Fixes #553 -- "More Info" link should point to a file/dir, not a dir+childname . ] [minor: fix unused imports -- thanks, pyflakes zooko@zooko.com**20081205190723 Ignore-this: 799f6a16360ac1aee8f6e0eb35a28a88 ] [download: refactor handling of URI Extension Block and crypttext hash tree, simplify things zooko@zooko.com**20081205141754 Ignore-this: 51b9952ea2406b0eea60e8d72654fd99 Refactor into a class the logic of asking each server in turn until one of them gives an answer that validates. It is called ValidatedThingObtainer. Refactor the downloading and verification of the URI Extension Block into a class named ValidatedExtendedURIProxy. The new logic of validating UEBs is minimalist: it doesn't require the UEB to contain any unncessary information, but of course it still accepts such information for backwards compatibility (so that this new download code is able to download files uploaded with old, and for that matter with current, upload code). The new logic of validating UEBs follows the practice of doing all validation up front. This practice advises one to isolate the validation of incoming data into one place, so that all of the rest of the code can assume only valid data. If any redundant information is present in the UEB+URI, the new code cross-checks and asserts that it is all fully consistent. This closes some issues where the uploader could have uploaded inconsistent redundant data, which would probably have caused the old downloader to simply reject that download after getting a Python exception, but perhaps could have caused greater harm to the old downloader. I removed the notion of selecting an erasure codec from codec.py based on the string that was passed in the UEB. Currently "crs" is the only such string that works, so "_assert(codec_name == 'crs')" is simpler and more explicit. This is also in keeping with the "validate up front" strategy -- now if someone sets a different string than "crs" in their UEB, the downloader will reject the download in the "validate this UEB" function instead of in a separate "select the codec instance" function. I removed the code to check plaintext hashes and plaintext Merkle Trees. Uploaders do not produce this information any more (since it potentially exposes confidential information about the file), and the unit tests for it were disabled. The downloader before this patch would check that plaintext hash or plaintext merkle tree if they were present, but not complain if they were absent. The new downloader in this patch complains if they are present and doesn't check them. (We might in the future re-introduce such hashes over the plaintext, but encrypt the hashes which are stored in the UEB to preserve confidentiality. This would be a double- check on the correctness of our own source code -- the current Merkle Tree over the ciphertext is already sufficient to guarantee the integrity of the download unless there is a bug in our Merkle Tree or AES implementation.) This patch increases the lines-of-code count by 8 (from 17,770 to 17,778), and reduces the uncovered-by-tests lines-of-code count by 24 (from 1408 to 1384). Those numbers would be more meaningful if we omitted src/allmydata/util/ from the test-coverage statistics. ] [test_web: add get_permuted_peers, to unbreak recent checker_results change warner@allmydata.com**20081205081210] [web checker_results: include a table of servers in permuted order, so you can see the places where new servers have been inserted warner@allmydata.com**20081205080309] [test_system.py: assert less about the stats we get, since shares (and thus allocate() calls) are distributed randomly warner@allmydata.com**20081204232704] [stats: don't return booleans: it violates the schema. Add a test. warner@lothar.com**20081204210124] [test_system.py: don't ask the stats-gatherer to poll: it tolerates failures, so it isn't really giving us enough test coverage. Removing the call will make it more clear that we need to improve the tests later warner@lothar.com**20081204210053] [confwiz.py - removing hardcoded version number secorp@allmydata.com**20081203023831] [CLI: check for pre-existing aliases in 'tahoe create-alias' and 'tahoe add-alias' warner@lothar.com**20081203022022] [test_cli: pass rc out of do_cli() too warner@lothar.com**20081203020828] [setup: one more address to send release announcements to zooko@zooko.com**20081203015040 Ignore-this: 87cb7a9c3a1810ff0c87908548027ac5 ] [setup: another note about the process of making a tahoe release: mail to duplicity-talk@nongnu.org zooko@zooko.com**20081203014414 Ignore-this: 77ffd6f7412cdc3283c1450cfde9fdf1 ] [test_storage.py: more windows-vs-readonly-storage fixes warner@lothar.com**20081203014102] [docs/webapi.txt: update helper section to discuss tahoe.cfg warner@lothar.com**20081203010726] [docs/webapi.txt: update to discuss tahoe.cfg, not BASEDIR/webport warner@lothar.com**20081203010612] [storage.py: oops, fix windows again, readonly_storage wasn't getting picked up properly warner@lothar.com**20081203010317] [test_download.py: remove extra base32 import warner@lothar.com**20081203003126] [test_download: test both mutable and immutable pre-generated shares warner@lothar.com**20081203003007] [test_download.py: added 'known-answer-tests', to make sure current code can download a file that was created by earlier code warner@lothar.com**20081203002208] [docs/configuration.txt: fix minor typo warner@lothar.com**20081202215101] [storage.py: unbreak readonly_storage=True on windows warner@allmydata.com**20081202014946] [#542 'tahoe create-key-generator': fix the .tac file this creates to be compatible with modern code, add a test warner@allmydata.com**20081201234721] [storage.py: fix minor typo in comment warner@lothar.com**20081201232540] [storage: replace sizelimit with reserved_space, make the stats 'disk_avail' number incorporate this reservation warner@lothar.com**20081201232421] [util/abbreviate: add abbreviated-size parser warner@lothar.com**20081201232412] [wui/wapi: change the default port number from 8123 to 3456 to avoid conflict with TorButton zooko@zooko.com**20081125235737 Ignore-this: 47ea30bafd5917a7e1dbc88aa0190f8e See ticket #536 for details. ] [setup: move the requirement on simplejson from setup.py to _auto_deps.py, and loosen it from >= 2.0.5 to > 1.8.1 zooko@zooko.com**20081125203751 Ignore-this: 4403781ef878547ee09e7e010eb5b49a We'll see if this fixes the tests on all of our current buildslaves, and if it does then I'll be happy to leave it at "> 1.8.1" for now, even though I don't know exactly what versions of simplejson changed exactly what behavior that interacts with exactly what environment. See http://allmydata.org/trac/tahoe/ticket/534 for uncertainties. ] [setup.py: Require simplejson version >= 2.0.5 francois@ctrlaltdel.ch**20081125171727] [mutable publish: reinstate the foolscap-reference-token-bug workaround, both for the original reasons and because of an apparent new foolscap bug that's triggered by reference tokens. See #541 for details. warner@allmydata.com**20081125202735] [setup: fix missing import -- thanks, pyflakes zooko@zooko.com**20081125155528 Ignore-this: 1fc042da2882b7b2f71cde93eb234a47 ] [setup: correctly detect Arch Linux in platform description zooko@zooko.com**20081125155118 Ignore-this: 37a7648f190679d3e973270a73133189 ] [dirnode manifest: add verifycaps, both to internal API and to webapi. This will give the manual-GC tools more to work with, so they can estimate how much space will be freed. warner@allmydata.com**20081124204046] [control.py: use get_buckets() instead of get_version() to measure ping time, because the latter changed recently warner@lothar.com**20081123051323] [upload: when using a Helper, insist that it provide protocols/helper/v1 . Related to #538. warner@allmydata.com**20081122022932] [upload: don't use servers which can't support the share size we need. This ought to avoid #439 problems. Some day we'll have a storage server which advertises support for a larger share size. No tests yet. warner@allmydata.com**20081122022812] [#538: fetch version and attach to the rref. Make IntroducerClient demand v1 support. warner@allmydata.com**20081122020727] [#538: add remote_get_version() to four main Referenceable objects: Introducer Service, Storage Server, Helper, CHK Upload Helper. Remove unused storage-server get_versions(). warner@allmydata.com**20081121234352] [setup: turn off --multi-version until I can figure out why it breaks test_runner zooko@zooko.com**20081121043645 Ignore-this: 36bf5db4122e6bc4e12588d9717a1e32 ] [setup: require setuptools >= 0.6c7 to run zooko@zooko.com**20081121043611 Ignore-this: e92e07c7e8edbaadcd44db7e8f4a028 ] [setup: use "setup.py develop --multi-version" so that if there is a too-old version of a dependency installed this doesn't prevent Tahoe's "develop" and run-in-place from working zooko@zooko.com**20081120201545 Ignore-this: 898f21fc1b16ae39c292fdd1ef42c446 ] [setup: we require setuptools > 0.6a9 in order to parse requirements that have a dot in them such as "zope.interface" zooko@zooko.com**20081120151503 Ignore-this: a6304de8f1f44defc50438d72a13e58f In the near future we might start actually relying on setuptools's pkg_resources's "require()" function to make modules importable, so we can't just skip zope.interface. ] [test_dirnode: add an explainError call warner@allmydata.com**20081119220212] [manifest: add storage-index strings to the json results warner@allmydata.com**20081119220027] [manifest: include stats in results. webapi is unchanged. warner@allmydata.com**20081119210347] [misc/spacetime/diskwatcher.tac: remove dead code warner@allmydata.com**20081119200552] [mutable: respect the new tahoe.cfg 'shares.needed' and 'shares.total' settings warner@allmydata.com**20081119200501] [oops, update tests to match 'tahoe stats' change warner@allmydata.com**20081119023259] [cli: tahoe stats: abbreviate total sizes too warner@allmydata.com**20081119022816] [cli: 'tahoe stats': add abbreviated size to the histogram. Not sure this actually improves things. warner@allmydata.com**20081119021736] [util/abbreviate: little utility to abbreviate seconds and bytes warner@allmydata.com**20081119021142] [cli: add 'tahoe check' and 'tahoe deep-check' commands, with primitive reporting code warner@allmydata.com**20081119011210] [cli: factor out slow-http-operation to a separate module warner@allmydata.com**20081119011113] [cli: tahoe stats/manifest: change --verbose to --raw, since I want -v for --verify for check/deep-check/repair warner@allmydata.com**20081119003608] [test_system: make 'where' strings more helpful, to track down test failures better warner@allmydata.com**20081119002950] [webapi: add 'summary' string to checker results JSON warner@allmydata.com**20081119002826] [munin/tahoe_disktotal: add a 'disk used' line, since it will always be less than disktotal warner@allmydata.com**20081118214431] [munin/tahoe_introstats: add line for distinct-storage-hosts (which counts machines instead of nodes) warner@allmydata.com**20081118213238] [webapi: introducer stats: add 'announcement_distinct_hosts' to the t=json form, to show how many distinct hosts are providing e.g. storage services warner@allmydata.com**20081118213015] ['tahoe create-key-generator': fix help text warner@allmydata.com**20081118074758] [#330: convert stats-gatherer into a .tac file service, add 'tahoe create-stats-gatherer' warner@allmydata.com**20081118074620] [munin/tahoe_diskused: new plugin to show total disk space used across the grid warner@allmydata.com**20081118072525] [munin/tahoe_disktotal: new plugin to show total disk space (used and unused) in the grid warner@allmydata.com**20081118065101] [tahoe.cfg: add controls for k and N (and shares-of-happiness) warner@allmydata.com**20081118062944] [cli: add tests for 'tahoe stats --verbose' warner@allmydata.com**20081118041114] [cli: add --verbose to 'tahoe manifest', to show the raw JSON data warner@allmydata.com**20081118040219] [diskwatcher: record total-space (the size of the disk as reported by df) in the db, report it to HTTP clients. This will involve a 50-item-per-second upgrade process when it is first used on old data warner@allmydata.com**20081118034516] [dirnode manifest/stats: process more than one LIT file per tree; we were accidentally ignoring all but the first warner@allmydata.com**20081115045049] [limiter.py: fix stack blowout by inserting an eventual-send between _done and maybe_start_task. This was causing failures during a 'tahoe manifest' of a large set of directories warner@allmydata.com**20081115031144] [New credit file entry francois@ctrlaltdel.ch**20081114140548] [test_cli.py: Ensure that we can read our uploaded files back francois@ctrlaltdel.ch**20081114134458] [test_cli.py: use str objects instead of unicode ones francois@ctrlaltdel.ch**20081114134137 This will hopefully fix failing tests with LC_ALL=C ] [CLI: add 'tahoe stats', to run start-deep-stats and print the results warner@allmydata.com**20081114014350] [test_system.py: fix new 'tahoe manifest' tests to not break on windows, by providing --node-directory instead of --node-url warner@allmydata.com**20081113212748] [test for bug #534, unicode filenames francois@ctrlaltdel.ch**20081113111951 This test assure that uploading a file whose name contains unicode character doesn't prevent further uploads in the same directory. ] [Fix an filename encoding issue with "tahoe cp" francois@ctrlaltdel.ch**20081111200803] [web/info.py: use 128-bit ophandles instead of 64-bit warner@allmydata.com**20081113021842] [CLI: add 'tahoe manifest', which takes a directory and returns a list of things you can reach from it warner@allmydata.com**20081113021725] [create_node.py: also remove now-unused import of pkg_resources warner@allmydata.com**20081113004716] [tahoe.cfg: add tub.location, to override the location hints we include in our FURL. This replaces advertised_ip_addresses, which doesn't remain useful enough to retain it. Helps with #517 (Tor). warner@allmydata.com**20081113004458] [setup: remove pkg_resources.require() from create_node.py and add it to runner.py zooko@zooko.com**20081112212503 Ignore-this: 763324202456a59b833b14eb4027171 Brian correctly points out that the latter is an entry point. ] [docs: fix cutnpasto in source:docs/logging.txt zooko@zooko.com**19700105140422 Ignore-this: de0f9ceb8e0ca4c158492ad2f9a6ba6f ] [tests: fix comment zooko@zooko.com**19700105101055 Ignore-this: fabedea917895568b1fca75a480111b9 ] [tests: add tahoe_cp to the list of scripts that we don't actually have tests for yet zooko@zooko.com**19700105100058 Ignore-this: ac89583992fb1b48d9a4680344569d91 ] [setup: the .tac files created by create_node.py call pkg_resources.require() so that they can load tahoe and twisted packages which were installed with setuptools multi-version mode zooko@zooko.com**19700101235005 Ignore-this: e1db03f86e0407a91087d8ada6b477fd Also the create_node.py script itself uses pkg_resources.require() for the same reason. ] [web/info: don't let an unrecoverable file break the page (show ? instead of a size) warner@allmydata.com**20081107045117] [checker: add is_recoverable() to checker results, make our stub immutable-verifier not throw an exception on unrecoverable files, add tests warner@allmydata.com**20081107043547] [monitor: update interface definition: get_status() can return a Failure warner@allmydata.com**20081107035452] [web/operations.py: if the operation failed, render the Failure warner@allmydata.com**20081107035309] [undoing test change for native_client.php secorp@allmydata.com**20081106220310] [NEWS: more minor edits warner@allmydata.com**20081106223517] [NEWS: minor edits warner@allmydata.com**20081106223356] [NEWS: mention SFTP server warner@allmydata.com**20081106014153] [client.py: oops, update FTP/SFTP config names to match current docs warner@allmydata.com**20081106013442] [remove duplicate+old docs/NEWS. The top-level NEWS file is the canonical one. warner@allmydata.com**20081106013224] [SFTP/FTP: merge user/account code, merge docs warner@allmydata.com**20081106012558] [docs: move webapi/ftp/sftp into a new frontends/ directory warner@allmydata.com**20081105233050] [ftp/sftp: move to a new frontends/ directory in preparation for factoring out password-auth component warner@allmydata.com**20081105200733] [sftpd: minor debug-logging tweak warner@allmydata.com**20081105194511] [confwiz.py - trying out a new configuration site secorp@allmydata.com**20081105011830] [ftpd: include an (unused) avatar logout callback warner@allmydata.com**20081105000104] [#531: implement an SFTP frontend. Mostly works, still lots of debug messages. Still needs tests and auth-by-pubkey in accounts.file warner@allmydata.com**20081105000022] [docs/ftp.txt: correct Twisted dependency: we don't need VFS, we can use a release, as long as you apply the patch warner@allmydata.com**20081104235840] [shebang: replace "/usr/bin/python" with "/usr/bin/env python" zooko@zooko.com**20081105000306 Ignore-this: 8ae33a8a7828fa7423422e252f2cfd74 ] [misc/fixshebangs.py zooko@zooko.com**20081105000130 Ignore-this: 13b03ea2d2ed8982f8346a827b46bd2e ] [util: copy in pyutil.fileutil.ReopenableNamedTemporaryFile zooko@zooko.com**20081104234715 Ignore-this: f1131e9b8f249b5f10be4cba2aeb6118 ] [immutable: tolerate filenode.read() with a size= that's too big, rather than hanging warner@allmydata.com**20081104212919] [util: copy in nummedobj from pyutil zooko@zooko.com**20081104195550] [util: copy in dictutil from pyutil zooko@zooko.com**20081104195327] [rollback change... move allmydatacontextmenu registration to installer.tmpl in tahoe-w32-client\installer booker@allmydata.com**20081103213647] [register the AllmydataContextMenu.dll for the context menu handler file sharing shell extension booker@allmydata.com**20081103200027] [debug catalog-shares: tolerate even more errors on bad files/directories warner@allmydata.com**20081030215447] [NEWS: update with all user-visible changes since the last update warner@allmydata.com**20081030213604] [#527: expire the cached files that are used to support Range: headers, every hour, when the file is unused and older than an hour warner@allmydata.com**20081030203909] [util/cachedir.py: add a cache-directory manager class, which expires+deletes unused files after a while warner@allmydata.com**20081030200120] [test_cli: try to fix windows again warner@allmydata.com**20081030193204] [debug/test_cli: fix error handling for catalog-shares, to make the test stop failing on windows warner@allmydata.com**20081030190651] [web: add 'Repair' button to checker results when they indicate unhealthyness. Also add the object's uri to the CheckerResults instance. warner@allmydata.com**20081030010917] [create_node.py: add 'web.static = public_html' to the initial tahoe.cfg warner@allmydata.com**20081030001336] [webapi: serve the /static URL tree from /public_html (configurable) warner@allmydata.com**20081029223431] [catalog-shares command: tolerate errors, log them to stderr, handle v2-immutable shares warner@allmydata.com**20081029221010] [test_web.py: one more line of test coverage warner@allmydata.com**20081029050015] [test_web: improve test coverage of PUT DIRURL t=uri replace=false warner@allmydata.com**20081029045744] [web: test (and fix) PUT DIRURL t=uri, which replaces a directory in-place with some other cap warner@allmydata.com**20081029045446] [web/directory.py: slight shuffle to improve test coverage warner@allmydata.com**20081029045406] [test_client.py: improve test coverage a bit warner@allmydata.com**20081029044335] [node.py: remove unused old_log() function warner@allmydata.com**20081029043558] [node.py: remove support for the old BASEDIR/authorized_keys.PORT file warner@allmydata.com**20081029043420] [move testutil into test/common_util.py, since it doesn't count as 'code under test' for our pyflakes numbers warner@allmydata.com**20081029042831] [util: move PollMixin to a separate file (pollmixin.py), so testutil can be moved into test/ warner@allmydata.com**20081029041548] [control.py: removed unused testutil.PollMixin warner@allmydata.com**20081029040359] [web/filenode: oops, fix test failures, not everything has a storage index warner@allmydata.com**20081029011720] [web/filenode: add Accept-Ranges and ETag (for immutable files) headers to GET responses warner@allmydata.com**20081029010103] [#527: respond to GETs with early ranges quickly, without waiting for the whole file to download. Fixes the alacrity problems with the earlier code. Still needs cache expiration. warner@allmydata.com**20081029005618] [#527: support HTTP 'Range:' requests, using a cachefile. Adds filenode.read(consumer, offset, size) method. Still needs: cache expiration, reduced alacrity. warner@lothar.com**20081028204104] [iputil.py: avoid a DNS lookup at startup (which may timeout tests when run on a partially-offline host) by using 198.41.0.4 instead of A.ROOT-SERVERS.NET warner@lothar.com**20081028203646] [interfaces.py: promote immutable.encode.NotEnoughSharesError.. it isn't just for immutable files any more warner@lothar.com**20081027203449] [interfaces.IMutableFileNode.download_best_version(): fix return value warner@lothar.com**20081027202046] [dirnode lookup: use distinct NoSuchChildError instead of the generic KeyError when a child can't be found warner@lothar.com**20081027201525] [storage: don't use colons in the corruption-advisory filename, since windows can't tolerate them warner@lothar.com**20081026024633] [mutable: call remove_advise_corrupt_share when we see share corruption in mapupdate/download/check, tolerate servers that do not implement it warner@lothar.com**20081024202128] [storage: add remote_advise_corrupt_share, for clients to tell storage servers about share corruption that they've discovered. The server logs the report. warner@lothar.com**20081024185248] [mutable/servermap.py: fix needs_merge(), it was incorrectly claiming that mixed shares with distinct seqnums needed a merge, causing repair(force=False) to fail warner@lothar.com**20081024040024] [test_web.test_POST_DIRURL_deepcheck: confirm that /operations/HANDLE/ works with or without the slash warner@lothar.com**20081024021759] [web/checker_results.py: remove dead code warner@lothar.com**20081024001717] [test_web: more test coverage warner@lothar.com**20081024001118] [webapi: fix t=rename from==to, it used to delete the file warner@lothar.com**20081023233236] [test_system: update test to match web checker results warner@lothar.com**20081023233202] [webapi deep-check: show the root as , rather than an empty path string warner@lothar.com**20081023230359] [mutable/checker: announce the mapupdate op on the 'recent uploads+downloads' page warner@lothar.com**20081023230319] [scripts/create_node.py: remove empty-string defaults for --introducer= and --nickname= warner@lothar.com**20081023230235] [deep-check: add webapi links to detailed per-file/dir results warner@lothar.com**20081023230031] [interface.py: fix typo warner@lothar.com**20081023225936] [webapi: make the /operations/ 't=status' qualifier optional, remove it from examples warner@lothar.com**20081023225658] [setup: require the latest version of the setuptools bootstrap egg zooko@zooko.com**20081025152858 Ignore-this: c0c9923ba3008f410d5cc56f2236edb9 ] [setup: include _pkgutil.py in setuptools bootstrap egg so that it will work on Python 2.4 zooko@zooko.com**20081025152839 Ignore-this: 38d81a037c1a3413d69d580ccb13fd67 ] [setup: pretend the tahoe requires twisted to set up, so that twisted will be there for nevow zooko@zooko.com**20081025135042 Ignore-this: 4e6c7e580f7e30df571e2e63be663734 ] [setup: require the SVN snapshot of setuptools to build zooko@zooko.com**20081025134959 Ignore-this: f68077dd10d85a71a1e06678365e6753 ] [setup: remove old bundled setuptools-0.6c9 zooko@zooko.com**20081025134947 Ignore-this: 3a95dd72346a60b39ffd6ddfadd1b3a8 ] [setup: bundle an SVN snapshot of setuptools instead of the most recent stable release of setuptools zooko@zooko.com**20081025134837 Ignore-this: 9a0c9a34b186b972650cf9455edb0d28 This SVN snapshot fixes a problem that prevents the setting up of nevow: http://bugs.python.org/setuptools/issue20 ] [setup: reorder dependencies to be sort of increasing order of how much they depend on other stuff zooko@zooko.com**20081025134739 Ignore-this: 6d636aaf5deb37cbf18172824b0bbf87 Not that the order makes any different to how it gets installed, as far as I can tell. ] [docs: add a note that when you make a new tahoe release, you should send the announcement to fuse-devel@lists.sourceforge.net zooko@zooko.com**20081023213658] [web/info.py: fix 'Check This Object' link, for files it was checking the parent directory by mistake warner@lothar.com**20081022171056] [#514: add meta-refresh=60 tag to t=status page for incomplete operations warner@lothar.com**20081022164842] [test_dirnode.py: oops, missed a Monitor(), unbreak tests warner@lothar.com**20081022085054] [immutable/filenode.py: add TODO note about the #514 monitor to check(), rather than going through the checker/verifier code and adding it, since Zooko is currently working on that code warner@lothar.com**20081022084237] [more #514: pass a Monitor to all checker operations, make mutable-checker honor the cancel flag warner@lothar.com**20081022083818] [dirnode.py: check for cancel during deep-traverse operations, and don't initiate any new ones if we've been cancelled. Gets us closer to #514. warner@lothar.com**20081022075552] [more #514 log-webop status/cancel: add handle-expiration, test coverage warner@lothar.com**20081022051354] [webapi.txt: improve t=deep-size output docs warner@lothar.com**20081022005331] [#514: improve test coverage warner@lothar.com**20081022005256] [Change deep-size/stats/check/manifest to a start+poll model instead of a single long-running synchronous operation. No cancel or handle-expiration yet. #514. warner@lothar.com**20081022000307] [setup: change ez_setup.py to install setuptools-0.6c9 zooko@zooko.com**20080930200502] [setup: bundle setuptools-0.6c9 zooko@zooko.com**20080930200448] [setup: remove bundled setuptools-0.6c8 zooko@zooko.com**20080930200336] [setup: remove the developer note about doing without GNU make (the GNU make requirement is about to hurt Peter if he tries to follow this doc, by the way) zooko@zooko.com**20081021163200 add classifiers showing with which versions of Python it is known to work. ] [* fuse/runtests: added --catch-up-pause option robk-tahoe@allmydata.com**20081021002902 On linux, write tests are failing because data written to fuse isn't showing up in tahoe by the time it's checked. it's not clear where this is originating, since the fuse implementation [should be] waiting for completion of tahoe operations before returning from its calls. This adds an option to control the duration of a pause between the fuse write and the check of tahoe, which is by default set to 2s on linux, which - somewhat inexplicably - seems to 'fix' the problem, in as far as it allows tests to complete. ] [fuse/runtests: include length in drepr() output robk-tahoe@allmydata.com**20081021000159] [fuse/runtests: make exceptions in 'read_in_random_order' into TestFailures robk-tahoe@allmydata.com**20081020235235] [fuse/blackmatch: added asynchronous (background) file download robk-tahoe@allmydata.com**20081020233333 previously, upon opening a file for reading, the open() call would block while the entire file was retrieved from tahoe into the cache directory. This change adds a DownloaderWithReadQueue class, and associated plumbing, such that an open() will return promptly with the download initiated 'in the background'. Subsequent read() operations will block until enough data has been downloaded to satisfy that request. This provides a behaviour similar to streaming, i.e. the client application will be able to read data from the fuse interface while the remainder of the file is still being downloaded. ] [fuse/runtests: added 'read_in_random_order' test robk-tahoe@allmydata.com**20081020232427 this test uploads a test file to tahoe, and then reads the file from fuse, but reads the blocks of the file in a random order; this is designed to exercise the asynchronous download feature of blackmatch - where the file is downloaded from tahoe asynchronously, and rather than blocking open() for the entirety of the download, instead individual read() calls are blocked until enough of the file has been downloaded to satisfy them ] [fuse/runtests: added a --no-cleanup option robk-tahoe@allmydata.com**20081020155120 the code had a 'fullcleanup' flag internally which controlled whether working directories were cleaned up. this promotes that to a command line option (negated) '--no-cleanup' defaulting to False, i.e. do cleanup ] [fuse/runtests: truncate expected file contents in reported error message robk-tahoe@allmydata.com**20081020144523 this avoids dumping the repr of 1Mb of random data to stdout in the event of a test failure, but rather just dumps the start/end of the errant strings if the amount of data is > 200 chars repr'd ] [fuse/blackmatch: fix platform specific problems in repr_flags robk-tahoe@allmydata.com**20081020143052 the repr_flags debug/logging function had a list of fields from the os module that might be passed into an open() call, but it included at least one which was available on the mac but not on linux. symmetrically linux has numerous flags which are not present on the mac. the repr_flags function is now tolerant of flags not being present, and has an expanded list of flags ] [makefile: added 'fuse-test' target to makefile, to run 'runtests' robk-tahoe@allmydata.com**20081019132518] [fuse/runtests: added a 'todo' flag, surpressing failure for implementations not expected to pass robk-tahoe@allmydata.com**20081019131600 since the current tests assume that the implementation responds to changes made to tahoe after mount, and impl_b prefetches and cached directory data, impl_b fails the current 'read' test suite. rather than reflect that problem in the overall failure of the runtests exit code, this adds a 'todo' flag to the implementations table, and sets the todo flag for impl_b. Thus errors will therein be reported in output, but not cause a failing exit code. ] [fuse/runtests: made runtests exit code depend on success robk-tahoe@allmydata.com**20081017180058 return an exit code of 0 only if no tests failed, and 1 in the case of linkage error, test setup failure, or individual test case failure ] [storage.py: assert that immutable share size will fit in the 4-byte v1 container (see #346). The struct module in py2.4 raises an error on overflow, but py2.5 merely emits a warning warner@lothar.com**20081020172208] [NEWS: update to summarize all changes since the last update warner@lothar.com**20081020164047] [fuse/runtest: make removal of webport file soft robk-tahoe@allmydata.com**20081017030154 previously the runtests suite removed the webport file created by tahoe create-client in all but the first node. now that the node config is in tahoe.cfg by default this file might not exist. ] [fuse/blackmatch: update json handling to support simplejson v2 robk-tahoe@allmydata.com**20081017025931 simplejson v2 returns strings as either unicode or str, depending upon its mood. thus the interpretation of the node's json repr of a directory, and the serialisation of strings in the json based rpc both exploded when built against simplejson v2. this makes both of these places liberal in their acceptance of either str or unicode. ] [fuse/blackmatch: log exception in server startup robk-tahoe@allmydata.com**20081017014650 humphf. my build runs the fuse stuff fine, but the build from the buildslave doesn't seem to start up properly. hopefully this will elicit some useful info ] [fuse/blackmatch: add readability to some logging, fix a permissions problem robk-tahoe@allmydata.com**20081017004421 adds a couple of functions to unpack 'mode' and 'flags' for open() calls, to facilitate debugging. adds a fix to ensure that all tmp files created for writing are opened with permissions 0600 - one problem I had with testing with the Finder was that files were being opened write only (0200) and were then failing to upload to tahoe due to internal permission denied errors. there remain a variety of problems with finder access which I'm unable to comprehend at this time. sometimes copies to tahoe will work fine, sometimes they yield "the finder cannot complete the operation because some data ... could not be read or written. (Error code -36)" sometimes "You may need to enter the name and password for an administrator on this computer to change the item" sometimes "The operation cannot be completed because an item with the name ... already exists." and sometimes "The operation cannot be completed because the item ... is locked." What seems to be absent is rhyme or reason. unix operations (cp, mv) work fine, rsync works fine. ] [fuse/blackmatch: fix linkage problems with daemonize robk-tahoe@allmydata.com**20081016163637 the daemonize() function imported from twisted was causing problems when run from a frozen (py2app) build. I simply copied the daemonize function into this file, and that fixes the problem. also removed a couple of lines of debugging spam that slipped through. ] [gui/macapp: minor bugfixes robk-tahoe@allmydata.com**20081016163052 though it seemed to work before the 'fstype' passed to fuse of 'allmydata' was today throwing errors that len(fstype) must be at most 7. fixed a typo in changes to 'mount_filesystem()' args bumped the delay between mounting a filesystem and 'open'ing it in Finder to 4s, as it seems to take a little longer to mount now the client and server fuse processes need to coordinate. ] [fuse/blackmatch: split into client/server (twisted server) robk-tahoe@allmydata.com**20081016150846 This implements a client/server split for blackmatch, where the client implements the fuse_main bindings and a simple blocking rpc client mechanism. The server implements the other half of that rpc mechanism, and contains all the actual logic for interpreting fuse requests in the context of the on disk cache and requests to the tahoe node. The server is based on a twisted reactor. The rpc mechanism implements a simple method dispatch including marshalling, using json, of basic inert data types, in a flat namespace (no objects). The client side is written in a blocking idiom, to interface with the threading model used by the fuse_main bindings, whereas the server side is written for a twisted reactor-based environment, intended to facilitate implementing more sophisticated logic in that paradigm. The two communicate over a unix domain socket, allocated within the nodedir. Command line usage is unchanged; the server is launched automatically by the client. The server daemonizes itself, to avoid preventing the original parent process (e.g. 'runtests') from waiting upon the server exiting. The client keeps open a 'keepalive' connection to the server; upon loss thereof the server will exit. This addresses the fact that the python-fuse bindings provide no notification of exit of the client process upon unmount. The client thus provides a relatively thin 'shim' proxying requests from the fuse_main bindings across the rpc to the server process, which handles the logic behind each request. For the time being, a '--no-split' option is provided to surpress the splitting into client/server, yielding the prior behaviour. Once the server logic gets more complex and more entrenched in a twisted idiom, this might be removed. The 'runtests' test harness currently tests both modes, as 'impl_c' and 'impl_c_no_split' ] [fuse/blackmatch: 'flatten' the fuse api implementation robk-tahoe@allmydata.com**20081016143547 the previous revision of blackmatch used a file_class to delegate all fuse api operations on files to a specific per-file class, which is an option given by the python-fuse bindings. this is a pre-cursor to the 'split' client/server version, which uses a simple, moreover flat, rpc mechanism to broker access to methods. ] [fuse/runtests: disable impl_a/impl_b on mac, as they don't actually work. robk-tahoe@allmydata.com**20081016143232] [fuse/runtests: added write_partial_overwrite test robk-tahoe@allmydata.com**20081016142926 this tests opening a file for update, overwriting a small part of it, and ensuring that the end result constitutes an overwrite of the original file. This tests, e.g. the implementation doesn' open a 'fresh' file but does in fact initialise the file to be uploaded with the contents of any extant file before applying updates ] [fuse/runtests: added --tests, renamed --suites robk-tahoe@allmydata.com**20081016142836 changed the --tests option to be --suites, as it takes a prefix, e.g. 'read' 'write' (or 'all', the default) and runs those suites which are applicable to each implementation being tested. added a --tests option, which takes a list of tests, e.g. 'read_file_contents' 'write_overlapping_large_writes' and runs all tests specified without regard to whether the implementation(s) under test are declared to support them. this is basically to allow a specific test or two to be run, saving time during development and debugging by not running the entire suite ] [fuse/runtests: added 'random scatter' write test robk-tahoe@allmydata.com**20081003233436 this writes the test file in a randomised order, with randomly sized writes. also for each 'slice' of the file written, a randomly chosen overlapping write is also made to the file. this ensures that the file will be written in its entirety in a thoroughly random order, with many overlapping writes. ] [fuse/runtests: add overlapping write tests robk-tahoe@allmydata.com**20081003224833 using both small and large blocksizes for writes, write a 1Mb file to fuse where every write overlaps another. This serves a useful purpose - in manual testing of blackmatch some time ago most operations e.g. bulk copies, worked fine, but using rsync caused data corruption on most files. it turned out to be that rsync writes in 64K blocks, but rather than making the last block short, the last block instead overlaps the preceding (already written) block. This revealed a problem where cache files were being opened 'append' rather than 'write' and hence the overlapping write to the fuse layer caused the overlapping portion of the file to be duplicated in cache, leading to oversized and corrupt files being uploaded. ] [fuse/runtests: remove write small file test, as it's subsumed by the tiny_file test robk-tahoe@allmydata.com**20081003223944] [fuse/runtests: added linear write tests for various block sizes robk-tahoe@allmydata.com**20081003223550 unit tests to test writing contiguous blocks linearly through the file, for a variety of block sizes; 'tiny_file' is an entire file fitting within a single io block / write operation. 'linear_{small,large}_writes' test a 1Mb file written with each write operation containing significantly less or more, respecitvely, data than fuse will pass into the implementation as a single operation (which on the mac at least is 64Kib) ] [fuse/runtests: add a very simple 'write' test robk-tahoe@allmydata.com**20081003172044 this performs a very simple write through the fuse layer and confirms that the file is stored correctly into the tahoe mesh. ('simple' in the sense that the entire file body fits trivially in a single write() operation, disk block etc) ] [fuse/runtests: added a --web-open option robk-tahoe@allmydata.com**20081003172026 similar to the --debug-wait option which causes the test harness to pause at various stages of the process to facilitate debugging, this option simplifies that debugging by automatically opening a web browser to the root dir of that implementation's tests when tests are commenced. in addition, if --web-open is specfied but --debug-wait is not, the harness will still pause after running tests but before tearing down the tahoe grid - this allows all tests to run to completion, but provide a debugging hook to investigate the end state of the grid's contents thereafter. ] [fuse/impl_a: fix a suspected bug in caching robk-tahoe@allmydata.com**20081003171309 from my examination of the tahoe_fuse ('impl_a') code, it looks like the intention is to cache the file contents in memory while it's open, since it does in fact do that. however it looks like it also ignored that cache entirely, and made an individual tahoe webapi GET request for each and every read() operation regardless of the relative size of the read block and the file in question. this changes that to make read() use the data in memory rather than fetch the data over again. if there's something more subtle going on, please let me know. ] [gui/macapp: slew of code cleanup; unmount filesystems on quit robk-tahoe@allmydata.com**20080925233235 a handful of code cleanup, renaming and refactoring. basically consolidating 'application logic' (mount/unmount fs) into the 'MacGuiApp' class (the wx.App) and cleaning up various scoping things around that. renamed all references to 'app' to refer more clearly to the 'AppContainer' or to the guiapp. globally renamed basedir -> nodedir also made the guiapp keep a note of each filesystem it mounts, and unmount them upon 'quit' so as to cleanup the user's environment before the tahoe node vanishes from out underneath the orphaned tahoe fuse processes ] [gui/macapp: make submenu of aliases for 'webopen' robk-tahoe@allmydata.com**20080925163919 this changes the 'open webroot' menu item to be a submenu listing all aliases defined in ~/.tahoe. Note that the dock menu does not support submenus, so it only offers a single 'open webroot' option for the default tahoe: alias. I had trouble with this at first and concluded that the submenus didn't work, and made it a distinct 'WebUI' menu in it's own right. on further inspection, there are still problems but they seem to be something like once the dock menu has been used, sometimes the app's main menubar menus will cease to function, and this happens regardless of whether submenus or plain simple menus are used. I have no idea what the peoblem is, but it's not submenu specific. ] [repairer: fix flaw in testutil.flip_one_bit() that Brian pointed out zooko@zooko.com**20081016194848] [misc/incident-gatherer: add classify_tahoe.py: a foolscap incident-gatherer classification plugin warner@allmydata.com**20081015220940] [repairer: test all different kinds of corruption that can happen to share files on disk zooko@zooko.com**20081014230920] [util/time_format.py: accept space separator, add unit tests warner@allmydata.com**20081013225258] [test_storage: use different filenames, poor stupid windows warner@allmydata.com**20081010021139] [scripts/debug.py: emit the immutable-share version number, tolerate v2 warner@allmydata.com**20081010013422] [storage.py: improve some precondition() error messages warner@allmydata.com**20081010011425] [storage: introduce v2 immutable shares, with 8-byte offsets fields, to remove two of the three size limitations in #346. This code handles v2 shares but does not generate them. We'll make a release with this v2-tolerance, wait a while, then make a second release that actually generates v2 shares, to avoid compatibility problems. warner@allmydata.com**20081010011327] [debug.py: oops, add missing import for ReadBucketProxy warner@allmydata.com**20081010002922] [storage: split WriteBucketProxy and ReadBucketProxy out into immutable/layout.py . No behavioral changes. warner@allmydata.com**20081010000800] [interfaces: loosen a few max-size constraints which would limit us to a mere 1.09 TB maximum file size zooko@zooko.com**20081009191357 These constraints were originally intended to protect against attacks on the storage server protocol layer which exhaust memory in the peer. However, defending against that sort of DoS is hard -- probably it isn't completely achieved -- and it costs development time to think about it, and it sometimes imposes limits on legitimate users which we don't necessarily want to impose. So, for now we forget about limiting the amount of RAM that a foolscap peer can cause you to start using. ] [util/limiter: add a repr warner@allmydata.com**20081007201945] [dirnode.build_manifest: include node.list in the limiter, that's the most important thing to slow down warner@allmydata.com**20081007201929] [web/directory: t=manifest output=html: make the caps into clickable hrefs warner@allmydata.com**20081007201845] [web/directory: factor out the get_root function warner@allmydata.com**20081007201742] [web/directory.py: remove unused imports warner@allmydata.com**20081007194820] [test_web: deep-size is more variable than I thought, so assert less warner@allmydata.com**20081007051147] [web: change t=manifest to return a list of (path,read/writecap) tuples, instead of a list of verifycaps. Add output=html,text,json. warner@allmydata.com**20081007043618] [web: rewrite t=deep-size in terms of deep-stats, update test to match inclusion of directory sizes warner@allmydata.com**20081007043539] [ftpd: hush pyflakes warner@allmydata.com**20081007014513] [ftpd: make sure we're using a patched/fixed Twisted, to avoid confusion later warner@allmydata.com**20081007011411] [ftp: change the twisted hack necessary for async-write-close, to one more agreeable to the twisted-dev folks, add a copy of the necessary patch to docs/ftp.txt warner@allmydata.com**20081007010605] [ftpd: remove debug messages warner@allmydata.com**20081006231620] [ftpd: add native_client.php -based HTTP authentication scheme warner@allmydata.com**20081006231511] [ftpd: add ftp.accounts checker, remove InMemoryPasswordChecker warner@allmydata.com**20081006225124] [test_system: add test coverage for immutable download.ConsumerAdapter, remove debug messages warner@allmydata.com**20081006225037] [ftp server: initial implementation. Still needs unit tests, custom Twisted patches. For #512 warner@allmydata.com**20081006195236] [test_cli.py: remove unused imports warner@allmydata.com**20081007004204] [CLI: remove 'tahoe admin generate-keypair', since the pycryptopp ecdsa API is about to change incompatibly. We'll undo this once pycryptopp is updated warner@allmydata.com**20081007002320] [docs: update architecture.txt 's section on the vdrive a.k.a. filesystem layer zooko@zooko.com**20081006210500 Remove some obsolete parts (correct at the time, now incorrect), change terminology to reflect my preference: s/vdrive/filesystem/ and s/dirnode/directory/, and make a few other small changes. ] [dirnode: fix my remarkably-consistent 'metdadata' typo warner@allmydata.com**20081003010845] [interfaces: fix minor typo warner@allmydata.com**20081003005249] [dirnode: add get_child_and_metadata_at_path warner@allmydata.com**20081003005203] [stop using 'as' as an identifier: as with 'with', 'as' has become a reserved word in python 2.6 warner@allmydata.com**20081003002749] [scripts/admin: split up generate_keypair code so that unit tests can use it more easily warner@allmydata.com**20081001235238] [docs: add some notes about things to do for a Tahoe release on pypi, freshmeat, and launchpad zooko@zooko.com**20081001210703] [misc/cpu-watcher.tac: use writeaside-and-rename for the history.pickle file warner@allmydata.com**20081001003053] [misc/spacetime: use async polling so we can add a 60-second timeout, add an index to the 'url' Axiom column for 2x speedup warner@allmydata.com**20080930233448] [#518: replace various BASEDIR/* config files with a single BASEDIR/tahoe.cfg, with backwards-compatibility of course warner@allmydata.com**20080930232149] [tolerate simplejson-2.0.0 and newer, which frequently return bytestrings instead of unicode objects. Closes #523 warner@allmydata.com**20080930222106] [munin/tahoe_doomsday: oops, tolerate 'null' in the timeleft results, to unbreak the 2wk/4wk graphs warner@allmydata.com**20080930202051] [test_node: improve coverage of advertised_ip_addresses a bit warner@allmydata.com**20080930060816] [testutil.PollMixin: set default timeout (to 100s), emit a more helpful error when the timeout is hit warner@allmydata.com**20080930052309] [repair: fix test to map from storage index to directory structure properly (thanks, cygwin buildbot, for being so kloodgey that you won't accept random binary filenames and thus making me notice this bug) zooko@zooko.com**20080926224913] [repairer: assert that the test code isn't accidentally allowing the repairer code which is being tested to do impossible things zooko@zooko.com**20080926222353] [repairer: enhance the repairer tests zooko@zooko.com**20080926174719 Make sure the file can actually be downloaded afterward, that it used one of the deleted and then repaired shares to do so, and that it repairs from multiple deletions at once (without using more than a reasonable amount of calls to storage server allocate). ] [netstring: add required_trailer= argument warner@allmydata.com**20080926165754] [test_netstring.py: move netstring tests to a separate file warner@allmydata.com**20080926165526] [move netstring() and split_netstring() into a separate util.netstring module warner@allmydata.com**20080926043824] [repairer: remove a test that doesn't apply to the repair-from-corruption case zooko@zooko.com**20080925220954] [repairer: add a test that repairer fixes corrupted shares (in addition to the test that it fixes deleted shares) zooko@zooko.com**20080925220712] [docs: proposed mutable file crypto design with ECDSA, 96-bit private keys, and semi-private keys (from http://allmydata.org/~zooko/lafs.pdf ) zooko@zooko.com**20080925213457] [docs: mutable file crypto design (from http://allmydata.org/~zooko/lafs.pdf ) zooko@zooko.com**20080925213433] [repairer: fix swapped docstrings; thanks Brian zooko@zooko.com**20080925182436] [trivial: remove unused imports; thanks, pyflakes zooko@zooko.com**20080925180422] [trivial: remove unused imports -- thanks, pyflakes zooko@zooko.com**20080925173453] [repairer: add basic test of repairer, move tests of immutable checker/repairer from test_system to test_immutable_checker, remove obsolete test helper code from test_filenode zooko@zooko.com**20080925171653 Hm... "Checker" ought to be renamed to "CheckerRepairer" or "Repairer" at some point... ] [setup: remove a few minimal unit tests from test_filenode which have been obviated by much better tests in test_mutable and test_system zooko@zooko.com**20080925161544] [gui/macapp: rough cut of ui tweaks; configurability, auto-mount robk-tahoe@allmydata.com**20080925141224 chatting with peter, two things the mac gui needed were the ability to mount the 'allmydata drive' automatically upon launching the app, and open the Finder to reveal it. (also a request to hide the debug 'open webroot' stuff) this (somewhat rough) patch implements all the above as default behaviour it also contains a quick configuration mechanism for the gui - rather than a preferences gui, running with a more 'tahoe' styled mechanism, the contents of a few optional files can modify the default behaviour, specifically file in ~/.tahoe/gui.conf control behaviour as follows: auto-mount (bool): if set (the default) then the mac app will, upon launch automatically mount the 'tahoe:' alias with the display name 'Allmydata' using a mountpoint of ~/.tahoe/mnt/__auto__ auto-open (bool): if set (the default) then upon mounting a file system (including the auto-mount if set) finder will be opened to the mountpoint of the filesystem, which essentially reveals the newly mounted drive in a Finder window show-webopen (bool): if set (false by default) then the 'open webroot' action will be made available in both the dock and file menus of the app daemon-timout (int): sets the daemon-timeout option passed into tahoe fuse when a filesystem is mounted. this defaults to 5 min files of type (int) much, naturally contain a parsable int representation. files of type (bool) are considered true if their (case-insensitive) contents are any of ['y', 'yes', 'true', 'on', '1'] and considered false otherwise. ] [gui/macapp: improve 'about' box robk-tahoe@allmydata.com**20080925135415 adds exactly 1 metric dollop of professionalism to the previously rather amateurish looking about box. ] [fuse/impl_c: UNDO --auto-fsid option robk-tahoe@allmydata.com**20080925134730 rolling back: Thu Sep 25 14:42:23 BST 2008 robk-tahoe@allmydata.com * fuse/impl_c: add --auto-fsid option this was inspired by reading the fuse docs and discovering the 'fsid' option to fuse_main, and was _intended_ to support a sort of 'stability' to the filesystem (specifically derived from the root-uri mounted, whether directly or via an alias) to support mac aliases across unmount/remount etc. some experimentation shows that that doesn't actually work, and that, at least for mac aliases in my testing, they're tied to path-to-mountpoint and not to the fsid - which seems to have no bearing. perhaps the 'local' flag is causing weirdness therein. at any rate, I'm recording it simply for posterity, in case it turns out to be useful after all somewhere down the road. M ./contrib/fuse/impl_c/blackmatch.py +13 ] [fuse/impl_c: add --auto-fsid option robk-tahoe@allmydata.com**20080925134223 this was inspired by reading the fuse docs and discovering the 'fsid' option to fuse_main, and was _intended_ to support a sort of 'stability' to the filesystem (specifically derived from the root-uri mounted, whether directly or via an alias) to support mac aliases across unmount/remount etc. some experimentation shows that that doesn't actually work, and that, at least for mac aliases in my testing, they're tied to path-to-mountpoint and not to the fsid - which seems to have no bearing. perhaps the 'local' flag is causing weirdness therein. at any rate, I'm recording it simply for posterity, in case it turns out to be useful after all somewhere down the road. ] [manhole: be more tolerant of authorized_keys. files in .tahoe robk-tahoe@allmydata.com**20080925031149 both peter and I independently tried to do the same thing to eliminate the authorized_keys file which was causing problems with the broken mac build (c.f. #522) namely mv authorized_keys.8223{,.bak} but the node is, ahem, let's say 'intolerant' of the trailing .bak - rather than disable the manhole as one might expect, it instead causes the node to explode on startup. this patch makes it skip over anything that doesn't pass the 'parse this trailing stuff as an int' test. ] [fuse/impl_c: move mac tahoefuse impl out into contrib/fuse robk-tahoe@allmydata.com**20080925014214 For a variety of reasons, high amongst them the fact that many people interested in fuse support for tahoe seem to have missed its existence, the existing fuse implementation for tahoe, previously 'mac/tahoefuse.py' has been renamed and moved. It was suggested that, even though the mac build depends upon it, that the mac/tahoefuse implementation be moved into contrib/fuse along with the other fuse implementations. The fact that it's not as extensively covered by unit tests as mainline tahoe was given as corroboration. In a bid to try and stem the confusion inherent in having tahoe_fuse, tfuse and tahoefuse jumbled together (not necessarily helped by referring to them as impl_a, b and c respectively) I'm hereby renaming tahoefuse as 'blackmatch' (black match is, per wikipedia "a type of crude fuse" hey, I'm a punny guy) Maybe one day it'll be promoted to be 'quickmatch' instead... Anyway, this patch moves mac/tahoefuse.py out to contrib/fuse/impl_c/ as blackmatch.py, and makes appropriate changes to the mac build process to transclude blackmatch therein. this leaves the extant fuse.py and fuseparts business in mac/ as-is and doesn't attempt to address such issues in contrib/fuse/impl_c. it is left as an exercise to the reader (or the reader of a message to follow) as to how to deal with the 'fuse' python module on the mac. as of this time, blackmatch should work on both mac and linux, and passes the four extant tests in runtests. (fwiw neither impl_a nor impl_b have I managed to get working on the mac yet) since blackmatch supports a read-write and caching fuse interface to tahoe, some write tests obviously need to be added to runtests. ] [macapp: changes to support aliases, updated tahoefuse command line options robk-tahoe@allmydata.com**20080925010128 the tahoefuse command line options changed to support the runtests harness, and as part of that gained support for named aliases via --alias this changes the mac app's invocation of tahoefuse to match that, and also changes the gui to present the list of defined aliases as valid mounts this replaces the previous logic which examined the ~/.tahoe/private directory looking for files ending in '.cap' - an ad-hoc alias mechanism. if a file is found matching ~/.tahoe/private/ALIASNAME.icns then that will still be passed to tahoefuse as the icon to display for that filesystem. if no such file is found, the allmydata icon will be used by default. the '-olocal' option is passed to tahoefuse. this is potentially contentious. specifically this is telling the OS that this is a 'local' filesystem, which is intended to be used to locally attached devices. however leopard (OSX 10.5) will only display non-local filesystems in the Finder's side bar if they are of fs types specifically known by Finder to be network file systems (nfs, cifs, webdav, afp) hence the -olocal flag is the only way on leopard to cause finder to display the mounted filesystem in the sidebar, but it displays as a 'device'. there is a potential (i.e. the fuse docs carry warnings) that this may cause vague and unspecified undesirable behaviour. (c.f. http://code.google.com/p/macfuse/wiki/FAQ specifically Q4.3 and Q4.1) ] [fuse/impl_c: reworking of mac/tahoefuse, command line options, test integration robk-tahoe@allmydata.com**20080925001535 a handful of changes to the tahoefuse implementation used by the mac build, to make command line option parsing more flexible and robust, and moreover to facilitate integration of this implementation with the 'runtests' test harness used to test the other two implementations. this patch includes; - improvements to command line option parsing [ see below ] - support for 'aliases' akin to other tahoe tools - tweaks to support linux (ubuntu hardy) the linux support tweaks are, or at least seem to be, a result of the fact that hardy ships with fuse 0.2pre3, as opposed to the fuse0.2 that macfuse is based upon. at least the versions I was working with have discrepencies in their interfaces, but on reflection this is probably a 'python-fuse' version issue rather than fuse per se. At any rate, the fixes to handling the Stat objects should be safe against either version, it's just that the bindings on hardy lacked code that was in the 'fuse' python module on the mac... command line options: the need for more flexible invocation in support of the runtests harness led me to rework the argument parsing from some simple positional hacks with a pass-through of the remainder to the fuse binding's 'fuse_main' to a system using twisted.usage to parse arguments, and having just one option '-o' being explicitly a pass-through for -o options to fuse_main. the options are now: --node-directory NODEDIR : this is used to look up the node-url to connect to if that's not specified concretely on the command line, and also used to determine the location of the cache directory used by the implementation, specifically '_cache' within the nodedir. default value: ~/.tahoe --node-url NODEURL : specify a node-url taking precendence over that found in the node.url file within the nodedir --alias ALIAS : specifies the named alias should be mounted. a lookup is performed in the alias table within 'nodedir' to find the root dir cap the named alias must exist in the alias table of the specified nodedir --root-uri ROOTURI : specifies that the given directory uri should be mounted at least one of --alias and --root-uri must be given (which directory to mount must be specified somehow) if both are given --alias takes precedence. --cache-timeout TIMEOUTSECS : specifies the number of seconds that cached directory data should be considered valid for. this tahoefuse implementation implements directory caching for a limited time; largely because the mac (i.e. the Finder in particular) tends to make a large number of requests in quick successsion when browsing the filesystem. on the flip side, the 'runtests' unit tests fail in the face of such caching because the changes made to the underlying tahoe directories are not reflected in the fuse presentation. by specifying a cache-timeout of 0 seconds, runtests can force the fuse layer into refetching directory data upon each request. any number of -oname=value options may be specified on the command line, and they will all be passed into the underlying fuse_main call. a single non-optional argument, the mountpoint, must also be given. ] [fuse/tests: slew of changes to fuse 'runtests' robk-tahoe@allmydata.com**20080924183601 This patch makes a significant number of changes to the fuse 'runtests' script which stem from my efforts to integrate the third fuse implementation into this framework. Perhaps not all were necessary to that end, and I beg nejucomo's forebearance if I got too carried away. - cleaned up the blank lines; imho blank lines should be empty - made the unmount command switch based on platform, since macfuse just uses 'umount' not the 'fusermount' command (which doesn't exist) - made the expected working dir for runtests the contrib/fuse dir, not the top-level tahoe source tree - see also discussion of --path-to-tahoe below - significantly reworked the ImplProcManager class. rather than subclassing for each fuse implementation to be tested, the new version is based on instantiating objects and providing relevant config info to the constructor. this was motivated by a desire to eliminate the duplication of similar but subtly different code between instances, framed by consideration of increasing the number of platforms and implementations involved. each implementation to test is thus reduced to the pertinent import and an entry in the 'implementations' table defining how to handle that implementation. this also provides a way to specify which sets of tests to run for each implementation, more on that below. - significantly reworked the command line options parsing, using twisted.usage; what used to be a single optional argument is now represented by the --test-type option which allows one to choose between running unittests, the system tests, or both. the --implementations option allows for a specific (comma-separated) list of implemenations to be tested, or the default 'all' the --tests option allows for a specific (comma-separated) list of tests sets to be run, or the default 'all'. note that only the intersection of tests requested on the command line and tests relevant to each implementation will be run. see below for more on tests sets. the --path-to-tahoe open allows for the path to the 'tahoe' executable to be specified. it defaults to '../../bin/tahoe' which is the location of the tahoe script in the source tree relative to the contrib/fuse dir by default. the --tmp-dir option controls where temporary directories (and hence mountpoints) are created during the test. this defaults to /tmp - a change from the previous behaviour of using the system default dir for calls to tempfile.mkdtemp(), a behaviour which can be obtained by providing an empty value, e.g. "--tmp-dir=" the --debug-wait flag causes the test runner to pause waiting upon user input at various stages through the testing, which facilitates debugging e.g. by allowing the user to open a browser and explore or modify the contents of the ephemeral grid after it has been instantiated but before tests are run, or make environmental adjustments before actually triggering fuse mounts etc. note that the webapi url for the first client node is printed out upon its startup to facilitate this sort of debugging also. - the default tmp dir was changed, and made configurable. previously the default behaviour of tempfile.mkdtemp() was used. it turns out that, at least on the mac, that led to temporary directories to be created in a location which ultimately led to mountpoint paths longer than could be handled by macfuse - specifically mounted filesystems could not be unmounted and would 'leak'. by changing the default location to be rooted at /tmp this leads to mountpoint paths short enough to be supported without problems. - tests are now grouped into 'sets' by method name prefix. all the existing tests have been moved into the 'read' set, i.e. with method names starting 'test_read_'. this is intended to facilitate the fact that some implementations are read-only, and some support write, so the applicability of tests will vary by implementation. the 'implementations' table, which governs the configuration of the ImplProcManager responsible for a given implementation, provides a list of 'test' (i.e test set names) which are applicable to that implementation. note no 'write' tests yet exist, this is merely laying the groundwork. - the 'expected output' of the tahoe command, which is checked for 'surprising' output by regex match, can be confused by spurious output from libraries. specfically, testing on the mac produced a warning message about zope interface resolution various multiple eggs. the 'check_tahoe_output()' function now has a list of 'ignorable_lines' (each a regex) which will be discarded before the remainder of the output of the tahoe script is matched against expectation. - cleaned up a typo, and a few spurious imports caught by pyflakes ] [fuse/impl_{a,b}: improve node-url handling robk-tahoe@allmydata.com**20080924182854 specifically change the expectation of the code to be such that the node-url (self.url) always includes the trailing slash to be a correctly formed url moreover read the node-url from the 'node.url' file found in the node 'basedir' and only if that doesn't exist, then fall back to reading the 'webport' file from therein and assuming localhost. This then supports the general tahoe pattern that tools needing only a webapi server can be pointed at a directory containing the node.url file, which can optionally point to another server, rather than requiring a complete node dir and locally running node instance. ] [fuse/impl_b: tweaks from testing on hardy robk-tahoe@allmydata.com**20080924180738 from testing on linux (specifically ubuntu hardy) the libfuse dll has a different name, specifically libfuse.so.2. this patch tries libfuse.so and then falls back to trying .2 if the former fails. it also changes the unmount behaviour, to simply return from the handler's loop_forever() loop upon being unmounted, rather than raising an EOFError, since none of the client code I looked at actually handled that exception, but did seem to expect to fall off of main() when loop_forever() returned. Additionally, from my testing unmount typically led to an OSError from the fuse fd read, rather than an empty read, as the code seemed to expect. also removed a spurious import pyflakes quibbled about. ] [setup: fix site-dirs to find system installed twisted on mac. robk-tahoe@allmydata.com**20080924174255 zooko helped me unravel a build weirdness today. somehow the system installed twisted (/System/Library) was pulling in parts of the other twisted (/Library) which had been installed by easy_install, and exploding. getting rid of the latter helped, but it took this change to get the tahoe build to stop trying to rebuild twisted and instead use the one that was already installed. c.f. tkt #229 ] [CLI: rework webopen, and moreover its tests w.r.t. path handling robk-tahoe@allmydata.com**20080924164523 in the recent reconciliation of webopen patches, I wound up adjusting webopen to 'pass through' the state of the trailing slash on the given argument to the resultant url passed to the browser. this change removes the requirement that arguments must be directories, and allows webopen to be used with files. it also broke the tests that assumed that webopen would always normalise the url to have a trailing slash. in fixing the tests, I realised that, IMHO, there's something deeply awry with the way tahoe handles paths; specifically in the combination of '/' being the name of the root path within an alias, but a leading slash on paths, e.g. 'alias:/path', is catagorically incorrect. i.e. 'tahoe:' == 'tahoe:/' == '/' but 'tahoe:/foo' is an invalid path, and must be 'tahoe:foo' I wound up making the internals of webopen simply spot a 'path' of '/' and smash it to '', which 'fixes' webopen to match the behaviour of tahoe's path handling elsewhere, but that special case sort of points to the weirdness. (fwiw, I personally found the fact that the leading / in a path was disallowed to be weird - I'm just used to seeing paths qualified by the leading / I guess - so in a debate about normalising path handling I'd vote to include the /) ] [CLI: reconcile webopen changes robk-tahoe@allmydata.com**20080924152002 I think this is largely attributable to a cleanup patch I'd made which never got committed upstream somehow, but at any rate various conflicting changes to webopen had been made. This cleans up the conflicts therein, and hopefully brings 'tahoe webopen' in line with other cli commands. ] [cli: cleanup webopen command robk-tahoe@allmydata.com**20080618201940 moved the body of webopen out of cli.py into tahoe_webopen.py made its invocation consistent with the other cli commands, most notably replacing its 'vdrive path' with the same alias parsing, allowing usage such as 'tahoe webopen private:Pictures/xti' ] [macapp: changed to remove 'Tahoe' from .app name robk-tahoe@allmydata.com**20080611003145 Change the build product from 'Allmydata Tahoe' to 'Allmydata' more inkeeping with the branding of the Allmydata product ] [add --syslog argument to 'tahoe start' and 'tahoe restart', used to pass --syslog to twistd for non-Tahoe nodes (like cpu-watcher) warner@allmydata.com**20080925010302] [misc/make-canary-files.py: tool to create 'canary files', explained in the docstring warner@allmydata.com**20080925004716] [webapi: survive slashes in filenames better: make t=info and t=delete to work, and let t=rename fix the problem warner@allmydata.com**20080924203505] [setup: when detecting platform, ask the Python Standard Library's platform.dist() before executing lsb_release, and cache the result in global (module) variables zooko@zooko.com**20080924180922 This should make it sufficiently fast, while still giving a better answer on Ubuntu than platform.dist() currently does, and also falling back to lsb_release if platform.dist() says that it doesn't know. ] [node.py: add BASEDIR/keepalive_timeout and BASEDIR/disconnect_timeout, to set/enable the foolscap timers, for #521 warner@allmydata.com**20080924175112] [setup: stop catching EnvironmentError when attempting to copy ./_auto_deps.py to ./src/allmydata/_auto_deps.py zooko@zooko.com**20080924000402 It is no longer the case that we can run okay without _auto_deps.py being in place in ./src/allmydata, so if that cp fails then the build should fail. ] [immutable: remove unused imports (thanks, pyflakes) zooko@zooko.com**20080923192610] [immutable: refactor immutable filenodes and comparison thereof zooko@zooko.com**20080923185249 * the two kinds of immutable filenode now have a common base class * they store only an instance of their URI, not both an instance and a string * they delegate comparison to that instance ] [setup: try parsing /etc/lsb-release first, then invoking lsb_release, because the latter takes half-a-second on my workstation, which is too long zooko@zooko.com**20080923171431 Also because in some cases the former will work and the latter won't. This patch also tightens the regexes so it won't match random junk. ] [setup: fix a cut-and-paste error in the fallback to parsing /etc/lsb-release zooko@zooko.com**20080923165551] [setup: if executing lsb_release doesn't work, fall back to parsing /etc/lsb-release before falling back to platform.dist() zooko@zooko.com**20080923162858 An explanatio of why we do it this way is in the docstring. ] [setup: if invoking lsb_release doesn't work (which it doesn't on our etch buildslave), then fall back to the Python Standard Library's platform.dist() function zooko@zooko.com**20080923154820] [setup: fix bug in recent patch to use allmydata.get_package_versions() to tell the foolscap app-version-tracking what's what zooko@zooko.com**20080923001347] [setup: when using the foolscap "what versions are here?" feature, use allmydata.get_package_versions() instead of specifically importing allmydata, pycryptopp, and zfec zooko@zooko.com**20080923000351] [setup: simplify the implementation of allmydata.get_package_versions() and add "platform" which is a human-oriented summary of the underlying operating system and machine zooko@zooko.com**20080922235354] [misc/make_umid: change docs, make elisp code easier to grab warner@lothar.com**20080920183933] [use foolscap's new app_versions API, require foolscap-0.3.1 warner@lothar.com**20080920183853] [BASEDIR/nickname is now UTF-8 encoded warner@lothar.com**20080920183713] [various: use util.log.err instead of twisted.log.err, so we get both Incidents and trial-test-flunking warner@lothar.com**20080920173545] [logging.txt: explain how to put log.err at the end of Deferred chains, explain FLOGTOTWISTED=1 warner@lothar.com**20080920173500] [util.log: send log.err to Twisted too, so that Trial tests are flunked warner@lothar.com**20080920173427] [setup.py trial: improve --verbose suggestion a bit warner@lothar.com**20080919193922] [test_cli: disable generate-keypair test on OS-X, pycryptopp still has a bug warner@lothar.com**20080919193855] [NEWS: finish editing for the upcoming 1.3.0 release warner@lothar.com**20080919193053] [NEWS: more edits, almost done warner@lothar.com**20080919010036] [NEWS: describe all changes since the last release. Still needs editing. warner@lothar.com**20080919002755] [CLI: add 'tahoe admin generate-keypair' command warner@lothar.com**20080919001133] [web: add 'more info' pages for files and directories, move URI/checker-buttons/deep-size/etc off to them warner@lothar.com**20080918050041] [setup.py: remove unused 'Extension' import warner@lothar.com**20080917230829] [setup.py,Makefile: move the 'chmod +x bin/tahoe' into setup.py warner@lothar.com**20080917230756] [docs/install.html: reference InstallDetails instead of debian-specific stuff warner@lothar.com**20080917225742] [Makefile,docs: tahoe-deps.tar.gz now lives in separate source/deps/ directory on http://allmydata.org warner@lothar.com**20080917204452] [docs: mention -SUMO tarballs, point users at release tarballs instead of development ones warner@lothar.com**20080917203631] [setup.py,Makefile: teat sdist --sumo about tahoe-deps/, use -SUMO suffix on tarballs, add sumo to 'make tarballs' target warner@lothar.com**20080917200119] [.darcs-boringfile ignore tahoe-deps and tahoe-deps.tar.gz warner@lothar.com**20080917195938] [docs: add a note about the process of making a new Tahoe release zooko@zooko.com**20080917170839] [Makefile: pyutil from a dependent lib causes a #455-ish problem, the workaround is to run build-once *three* times warner@lothar.com**20080917053643] [Makefile: desert-island: don't re-fetch tahoe-deps.tar.gz if it's already there, remove the tahoe-deps/ before untarring directory to avoid unpacking weirdness warner@lothar.com**20080917052204] [misc/check-build.py: ignore the 'Downloading file:..' line that occurs for the setup_requires= -triggered handling of the setuptools egg warner@lothar.com**20080917051725] [#249: add 'test-desert-island', to assert that a tahoe-deps.tar.gz -enabled build does not download anything warner@lothar.com**20080917013702] [#249: get dependent libs from tahoe-deps and ../tahoe-deps warner@lothar.com**20080917013627] [#249: move dependent libs out of misc/dependencies/, get them from tahoe-deps.tar.gz instead warner@allmydata.com**20080917012545] [conf_wiz.py - updating version numbers in file, should really get these from a TAG or conf file secorp@allmydata.com**20080917004547] [webish: add an extra newline to JSON output warner@lothar.com**20080915204314] [windows/Makefile: fix dependencies: windows-installer must cause windows-exe to run warner@allmydata.com**20080912052151] [Makefile: fix windows issues warner@allmydata.com**20080912050919] [Makefile: use run_with_pythonpath, move windows targets into a separate Makefile warner@allmydata.com**20080912044508] [setup.py: add 'setup.py run_with_pythonpath', to run other commands with PYTHONPATH set usefully warner@allmydata.com**20080912044418] [Makefile: convert check-auto-deps target into 'setup.py check_auto_deps' warner@allmydata.com**20080912035904] [startstop_node.py: find twistd in our supportlib if we had to build Twisted as a setuptools dependency. This is a form of cgalvan's #505 patch, simplified because now 'setup.py trial' takes care of sys.path and PYTHONPATH warner@allmydata.com**20080912025138] [rewrite parts of the Makefile in setup.py. Add 'build_tahoe' and 'trial' subcommands. warner@allmydata.com**20080912010321 The 'make build' target now runs 'setup.py build_tahoe', which figures out where the target 'supportlib' directory should go, and invokes 'setup.py develop' with the appropriate arguments. The 'make test' target now runs 'setup.py trial', which manages sys.path and runs trial as a subroutine instead of spawning an external process. This simplifies the case where Twisted was built as a dependent library (and thus the 'trial' executable is not on PATH). setup.py now manages sys.path and PYTHONPATH for its internal subcommands, so the $(PP) prefix was removed from all Makefile targets that invoke setup.py . For the remaining ones, the 'setup.py -q show_pythonpath' subcommand was added to compute this prefix with python rather than with fragile shell/Makefile syntax. ] [bin/tahoe: reflow error messages warner@allmydata.com**20080912010225] [mac/Makefile: remove the verbose hdiutil diagnostics now that we resolved the problem warner@allmydata.com**20080912004622] [Makefile: give setup.py develop a '--site-dirs' arg to work around the #249 setuptools bug which causes us to unnecessarily rebuild pyopenssl and other support libs installed via debian's python-support. Should be harmless on other platforms. warner@allmydata.com**20080910233432] [web: fix output=JSON, add buttons for repair/json to the 'run deep-check' form warner@allmydata.com**20080910211137] [disallow deep-check on non-directories, simplifies the code a bit warner@allmydata.com**20080910204458] [dirnode: refactor recursive-traversal methods, add stats to deep_check() method results and t=deep-check webapi warner@lothar.com**20080910084504] [dirnode: cleanup, make get_verifier() always return a URI instance, not a string warner@lothar.com**20080910083755] [test_system: check t=deep-stats too warner@lothar.com**20080910065457] [test_system: add deep-check-JSON tests, fix a bug warner@lothar.com**20080910061416] [test_system: oops, re-enable some tests that got bypassed warner@lothar.com**20080910060245] [test_system: add deep-stats test warner@lothar.com**20080910055634] [hush pyflakes warner@allmydata.com**20080910025017] [checker results: add output=JSON to webapi, add tests, clean up APIs warner@allmydata.com**20080910024517 to make the internal ones use binary strings (nodeid, storage index) and the web/JSON ones use base32-encoded strings. The immutable verifier is still incomplete (it returns imaginary healty results). ] [immutable verifier: provide some dummy results so deep-check works, make the tests ignore these results until we finish it off warner@allmydata.com**20080910010827] [mutable checker: even more tests. Everything in ICheckerResults should be covered now, except for immutable-verify which is incomplete warner@allmydata.com**20080910005706] [checker results: more tests, update interface docs warner@allmydata.com**20080910003010] [mutable checker: oops, fix redefinition of 'healthy' (numshares < N, not numshares < k, which is 'recoverable' not 'healthy') warner@allmydata.com**20080910002853] [checker results: more tests, more results. immutable verifier tests are disabled until they emit more complete results warner@allmydata.com**20080910001546] [checker: add tests, add stub for immutable check_and_repair warner@allmydata.com**20080909233449] [interfaces.py: minor improvement to IDirectoryNode.set_node warner@allmydata.com**20080909233416] [mac/Makefile: upload the .dmg file with foolscap xfer-client.py instead of scp warner@allmydata.com**20080908231943] [misc/xfer-client.py: small foolscap utility to transfer a file to a waiting server warner@allmydata.com**20080908231903] [setup: add excited DEVELOPER NOTE to install.html zooko@zooko.com**20080908215603 It should be removed before 1.3.0 release, of course... ] [setup: edit the text of install.html zooko@zooko.com**20080908215549] [setup: add link to the DownloadDebianPackages page zooko@zooko.com**20080908215451 Because I want that link off of the front page of the wiki... ] [setup: change URL from which to get source tarballs zooko@zooko.com**20080908215409 So that when you look at that directory you won't see distracting other things such as darcs repositories. ] [test_system: make log() tolerate the format= form warner@lothar.com**20080908030336] [immutable/checker: make log() tolerate the format= form warner@lothar.com**20080908030308] [checker: overhaul checker results, split check/check_and_repair into separate methods, improve web displays warner@allmydata.com**20080907194456] [webapi.txt: explain that t=manifest gives verifycaps warner@allmydata.com**20080907192950] [introducer: add get_nickname_for_peerid warner@allmydata.com**20080906050700] [docs/logging.txt: explain tahoe/foolscap logging. Addresses #239. warner@allmydata.com**20080904002531] [setup: don't assert that trial is present when the Makefile is evaluated zooko@zooko.com**20080903171837 This should fix #506, but it means that if (for some weird reason) Twisted can't be auto-installed and the find_trial.py script doesn't work, the user will get a weird failure message instead of a clean failure message explaining that trial couldn't be found. Oh well. Chris Galvan is working on a much nicer fix to all these issues -- see #505. ] [testutil.PollMixin: use a custom exception (and convert it) to avoid the ugly 'stash' cycle warner@allmydata.com**20080903033251] [mac/Makefile: more attempts to debug the buildslave failure warner@allmydata.com**20080829220614] [mac: add -verbose to the hdiutil call, to figure out why it's failing on the buildslave warner@allmydata.com**20080829205243] [setup: simplify parsing of python version number zooko@zooko.com**20080829000045] [setup: emit the version of python in the list of versions zooko@zooko.com**20080828220454] [munin: add tahoe_diskleft plugin, update spacetime/diskwatcher.tac to support it warner@allmydata.com**20080828203236] [docs: how_to_make_a_tahoe_release.txt zooko@zooko.com**20080828202109 Just some cryptic notes to self, but if I get hit by a truck then someone else might be able to decode them. ] [debian: include misc/cpu-watcher.tac in the debian package warner@allmydata.com**20080827223026] [munin/tahoe_doomsday: change the graph title, 'time predictor' is more accurate than 'space predictor' warner@allmydata.com**20080827213013] [munin/tahoe_diskusage: clip the graph at zero, to prevent transient negative excursions (such as when a lot of old logfiles are deleted from a storage server's disk) from scaling the graph into unusability warner@allmydata.com**20080827193543] [CREDITS: thanks to Chris Galvan zooko@zooko.com**20080827183950] [setup: patch from Chris Galvan to build sdists with no deps in them normally, but include deps if --sumo zooko@zooko.com**20080827182644] [servermap: don't log late arrivals, and don't log DeadReferenceError at log.WEIRD warner@allmydata.com**20080827003729] [mutable: make mutable-repair work for non-verifier runs, add tests warner@allmydata.com**20080826233454] [mutable: remove work-around for a flaw in an older version of foolscap zooko@zooko.com**20080826155055 We now require "foolscap[secure_connections] >= 0.3.0", per [source:_auto_deps.py]. ] [docs: edit install.html a tad zooko@zooko.com**20080826154929] [misc/make_umid: little script and elisp fragment to insert umid= arguments warner@allmydata.com**20080826015918] [logging: add 'unique-message-ids' (or 'umids') to each WEIRD-or-higher log.msg call, to make it easier to correlate log message with source code warner@allmydata.com**20080826015759] [logging cleanups: lower DeadReferenceError from WEIRD (which provokes Incidents) to merely UNUSUAL, don't pre-format Failures in others warner@allmydata.com**20080826005155] [checker: make the log() function of SimpleCHKFileVerifier compatible with the log() function of its superclasses and subclasses zooko@zooko.com**20080825214407] [docs: warn that the "garbage-collection and accounting" section of architecture.txt is out of date, and clarify that "deleted" therein means ciphertext getting garbage-collected zooko@zooko.com**20080822154605] [docs/filesystem-notes.txt: add notes about enabling the 'directory index' feature on ext3 filesystems for storage server lookup speed warner@allmydata.com**20080821205901] [setup: doc string describing what the require_auto_deps() function is for zooko@zooko.com**20080815172234] [mutable/checker: log a WEIRD-level event when we see a hash failure, to trigger an Incident warner@allmydata.com**20080813035020] [immutable checker: add a status_report field warner@allmydata.com**20080813033530] [mutable/servermap: lower the priority of many log messages warner@allmydata.com**20080813033506] [web/deep-check: show the webapi runtime at the bottom of the page warner@allmydata.com**20080813033426] [CLI: tolerate blank lines in the aliases file warner@allmydata.com**20080813025050] [test_web: workaround broken HEAD behavior in twisted-2.5.0 and earlier warner@allmydata.com**20080813024520] [test_web: oops, actually use HEAD (instead of GET) in the HEAD test warner@allmydata.com**20080813020451] [web: use get_size_of_best_version for HEAD requests, provide correct content-type warner@allmydata.com**20080813020410] [mutable: add get_size_of_best_version to the interface, to simplify the web HEAD code, and tests warner@allmydata.com**20080813020252] [CLI: add 'tahoe debug corrupt-share', and use it for deep-verify tests, and fix non-deep web checker API to pass verify=true into node warner@allmydata.com**20080813000501] [IFilesystemNode: add get_storage_index(), it makes tests easier warner@allmydata.com**20080812231407] [test_system: rename Checker to ImmutableChecker, to make room for a mutable one warner@allmydata.com**20080812225932] ['tahoe debug dump-share': add --offsets, to show section offsets warner@allmydata.com**20080812214656] [test_cli: oops, fix tests after recent stdout/stderr cleanup warner@allmydata.com**20080812214634] [scripts/debug: split out dump_immutable_share warner@allmydata.com**20080812205517] [scripts/debug: clean up use of stdout/stderr warner@allmydata.com**20080812205242] [CLI: move the 'repl' command to 'tahoe debug repl' warner@allmydata.com**20080812204017] [CLI: move all debug commands (dump-share, dump-cap, find-shares, catalog-shares) into a 'debug' subcommand, and improve --help output warner@allmydata.com**20080812203732] [hush a pyflakes warning warner@allmydata.com**20080812042423] [web/directory: enable verify=true in t=deep-check warner@allmydata.com**20080812042409] [dirnode: add some deep-check logging warner@allmydata.com**20080812042338] [checker_results.problems: don't str the whole Failure, just extract the reason string warner@allmydata.com**20080812042306] [checker: add information to results, add some deep-check tests, fix a bug in which unhealthy files were not counted warner@allmydata.com**20080812040326] [mutable/checker: rearrange a bit, change checker-results to have a status_report string warner@allmydata.com**20080812032033] [mutable/servermap: add summarize_version warner@allmydata.com**20080812031930] [CLI: make 'tahoe webopen' command accept aliases like 'tahoe ls' warner@allmydata.com**20080812012023] [munin diskusage/doomsday: oops, fix labels, everything was reported in the 1hr column warner@allmydata.com**20080811203431] [munin/tahoe_overhead: don't emit non-sensicial numbers warner@lothar.com**20080807214008] [munin: add tahoe_overhead plugin, to measure effectiveness of GC and deleting data from inactive accounts warner@lothar.com**20080807203925] [diskwatcher.tac: include total-bytes-used warner@lothar.com**20080807201214] [setup: remove accidentally duplicated lines from Makefile zooko@zooko.com**20080807193029] [misc/dependencies: remove the no-longer-useful foolscap-0.2.5 tarball warner@lothar.com**20080807184546] [Makefile: avoid bare quotes, since the emacs syntax-highlighter gets confused by them warner@lothar.com**20080807183001] [diskwatcher.tac: don't report negative timeleft warner@lothar.com**20080807173433] [diskwatcher.tac: reduce the polling rate to once per hour warner@lothar.com**20080807062021] [misc/spacetime: add munin plugins, add everything to .deb warner@lothar.com**20080807060003] [diskwatcher.tac: hush pyflakes warner@lothar.com**20080807050427] [diskwatcher.tac: add async-GET code, but leave it commented out: urlopen() seems to work better for now warner@lothar.com**20080807050327] [cpu-watcher.tac: improve error message warner@lothar.com**20080807043801] [disk-watcher: first draft of a daemon to use the HTTP stats interface and its new storage_server.disk_avail feature, to track changes in disk space over time warner@lothar.com**20080807042222] [misc/cpu-watcher.tac: tolerate missing pidfiles, just skip over that sample warner@lothar.com**20080807041705] [setup: don't attempt to escape quote marks, just delete them. Ugly, but it works okay. zooko@zooko.com**20080806232742] [setup: escape any double-quote chars in the PATH before using the PATH to find and invoke trial zooko@zooko.com**20080806231143] [storage: include disk-free information in the stats-gatherer output warner@lothar.com**20080806210602] [mutable: more repair tests, one with force=True to check out merging warner@lothar.com**20080806190607] [test/common: add ShouldFailMixin warner@lothar.com**20080806190552] [test_mutable: add comment about minimal-bandwidth repairer, comma lack of warner@lothar.com**20080806173850] [test_mutable: factor out common setup code warner@lothar.com**20080806173804] [mutable: start adding Repair tests, fix a simple bug warner@lothar.com**20080806061239] [mutable.txt: add warning about out-of-date section warner@lothar.com**20080806061219] [test_system: factor out find_shares/replace_shares to a common class, so they can be used by other tests warner@lothar.com**20080806014958] [debian/control: update dependencies to match _auto_deps: foolscap-0.3.0, pycryptopp-0.5 warner@lothar.com**20080806013222] [bump foolscap dependency to 0.3.0, for the new incident-gathering interfaces warner@lothar.com**20080805235828] [web: add 'report incident' button at the bottom of the welcome page warner@lothar.com**20080805190921] [test_cli: more coverage for 'tahoe put' modifying a mutable file in-place, by filename, closes #441 warner@lothar.com**20080804202643] [check_grid.py: update to match new CLI: 'put - TARGET' instead of 'put TARGET' warner@lothar.com**20080802024856] [test_cli: remove windows-worrying newlines from test data warner@lothar.com**20080802024734] [test_cli.py: factor out CLITestMixin warner@lothar.com**20080802022938] [CLI: change one-arg forms of 'tahoe put' to make an unlinked file, fix replace-mutable #441 warner@lothar.com**20080802022729] [CLI: add create-alias command, to merge mkdir and add-alias into a single (secure-from-argv-snooping) step warner@lothar.com**20080802021041] [test_cli: add system-based tests for PUT, including a mutable put that fails/todo (#441) warner@lothar.com**20080801221009] [tests: simplify CLI tests that use stdin, now that runner supports it warner@lothar.com**20080801220514] [CLI: simplify argument-passing, use options= for everthing, including stdout warner@lothar.com**20080801184624] [tests: add test that verifier notices any (randomly chosen) bit flipped in the verifiable part of any (randomly chosen) share zooko@zooko.com**20080731002015 The currently verifier doesn't (usually) pass this randomized test, hence the TODO. ] [tests: test that checker doesn't cause reads on the storage servers zooko@zooko.com**20080730235420 It would still pass the test if it noticed a corrupted share. (It won't notice, of course.) But it is required to do its work without causing storage servers to read blocks from the filesystem. ] [storage: make storage servers declare oldest supported version == 1.0, and storage clients declare oldest supported version == 1.0 zooko@zooko.com**20080730225107 See comments in patch for intended semantics. ] [tests: use the handy dandy TestCase.mktemp() function from trial to give unique and nicely named directories for each testcase zooko@zooko.com**20080730224920] [tests: don't use SignalMixin zooko@zooko.com**20080730223536 It seems like we no longer need it, and it screws up something internal in trial which causes trial's TestCase.mktemp() method to exhibit wrong behavior (always using a certain test method name instead of using the current test method name), and I wish to use TestCase.mktemp(). Of course, it is possible that the buildbot is about to tell me that we do still require SignalMixin on some of our platforms... ] [setup: if the user passes a TRIALOPT env var then pass that on to trial zooko@zooko.com**20080730205806 This is useful for --reporter=bwverbose, for example. ] [setup: turn back on reactor=poll for cygwin trial (else it runs out of fds) zooko@zooko.com**20080730181217] [setup: fix bug in Makefile -- ifeq, not ifneq -- so that now it sets poll reactor only if the user hasn't specified a REACTOR variable, instead of setting poll reactor only if the user has specified a REACTOR variable zooko@zooko.com**20080730160429] [setup: whoops, really remove the default reactor=poll this time zooko@zooko.com**20080730032358] [setup: instead of setting --reactor=poll for trial in all cases (which fails on platforms that don't have poll reactor, such as Windows and some Mac OS X), just set --reactor=poll for linux2. zooko@zooko.com**20080730031656 ] [setup: pass --reactor=poll to trial unless REACTOR variable is set, in which case pass --reactor=$(REACTOR) zooko@zooko.com**20080730023906 This hopefully works around the problem that Twisted v8.1.0 has a bug when used with pyOpenSSL v0.7 which bug causes some unit tests to spuriously fail -- see known_issues.txt r2788: http://allmydata.org/trac/tahoe/browser/docs/known_issues.txt?rev=2788#L122 Also it matches with the fact that --reactor=poll is required on cygwin. ] [setup: require secure_connections from foolscap zooko@zooko.com**20080730021041 This causes a problem on debian sid, since the pyOpenSSL v0.6 .deb doesn't come with .egg-info, so setuptools will not know that it is already installed and will try to install pyOpenSSL, and if it installs pyOpenSSL v0.7, then this will trigger the bug in Twisted v8.1.0 when used with pyOpenSSL v0.7. http://twistedmatrix.com/trac/ticket/3218 Now the comments in twisted #3218 suggest that it happens only with the select reactor, so maybe using --reactor=poll will avoid it. ] [tests: add test_system.Checker which tests basic checking (without verification) functionality zooko@zooko.com**20080728234317] [test: add testutil.flip_one_bit which flips a randomly chosen bit of the input string zooko@zooko.com**20080728234217] [tests: make it so that you can use common.py's SystemTestMixin.set_up_nodes() more than once with the same introducer zooko@zooko.com**20080728234029] [download.py: set up self._paused before registering the producer, since they might call pauseProducing right away warner@lothar.com**20080728215731] [test/common.py: use pre-computed Tub certificates for the system-test mixin, to speed such tests up by maybe 15%. The goal is to encourage more full-grid tests. warner@allmydata.com**20080728194421] [munin/tahoe_spacetime: show 2wk data even if 4wk data is unavailable warner@allmydata.com**20080728194233] [web: add /status/?t=json, with active upload/download ops. Addresses #493. warner@allmydata.com**20080726004110] [web: make t=json stats pages use text/plain, instead of leaving it at text/html warner@allmydata.com**20080726002427] [test_system.py: factor SystemTestMixin out of SystemTest warner@allmydata.com**20080725223349] [test_system.py: modify system-test setup code in preparation for merge with common.SystemTestMixin warner@allmydata.com**20080725222931] [test_system.py: move SystemTestMixin out into common.py, where further improvements will occur warner@allmydata.com**20080725221758] [test_system.py: create SystemTestMixin, with less cruft, for faster system-like tests warner@allmydata.com**20080725221300] [TAG allmydata-tahoe-1.2.0 zooko@zooko.com**20080722014608] Patch bundle hash: 9ffee3526159795f1692c2d98097fcd60ac9b2c6