id	summary	reporter	owner	description	type	status	priority	milestone	component	version	resolution	keywords	cc	launchpad_bug
823	WUI server should have a disallow-all robots.txt	davidsarah		"Currently, if a web crawler gets access to a Tahoe WUI gateway server then it will crawl all reachable links. This is probably undesirable, or at least not a sensible default (even though it is understood that {{{robots.txt}}} is not meant as a security mechanism).

WUI servers should have a disallow-all {{{robots.txt}}}:

{{{
User-agent: *
Disallow: /
}}}

The robots.txt specification is at http://www.robotstxt.org/orig.html"	defect	new	major	undecided	code-frontend-web	1.5.0		privacy