id	summary	reporter	owner	description	type	status	priority	milestone	component	version	resolution	keywords	cc	launchpad_bug
1834	stop using share crawler for anything except constructing a leasedb	zooko		"I think we should stop using a ""share crawler"" — a long-running, persistent, duty-cycle-limited process that visits every share held by a storage server — for everything that we can.

And, I think that the only thing that we can't do in a different way is: construct a leasedb when we are first upgrading the server to a leasedb-capable version, or the leasedb has been lost or corrupted.

Here are the other things that are currently done by crawlers and how I think they should be done differently:

* Updating and/or checking the leases on shares to see if they have expired;

On David-Sarah's 666-accounting branch, this is now done for all shares by a single, synchronous command/query to leasedb. (#666)

* Delete shares that have lost all their leases (by cancellation or expiry);

I propose that this be done instead by the storage server maintaining a persistent set of shares to be deleted. When lease-updating step (which, in #666, is synchronous and fast) has identified a share that has no more leases, the share's id gets added to the persistent set of shares to delete. A long-running, persistent, duty-cycle-limited processes deletes those shares from the backend and removes their ids from the set of shares-to-delete. This is cleaner and more efficient than using a crawler, which has to visit ''all'' shares and which never stops twitching, since this has to visit only shares that have been marked as to-delete, and it quiesces when there is nothing to delete. (#1833)

* Discover newly added shares that the operator copied into the backend without notifying the storage server;

I propose that we stop supporting this method of moving shares around. If we stop supporting this, that would leave two options for if you want to add a share to a server:

1. Send it through the front door — 

I'm going to create this ticket in order to get a ticket number (probably #1834) that other tickets can reference, then come back and write more of this Description..."	defect	new	normal	undecided	code-storage	1.9.2		leases garbage-collection accounting