# http://www.robotstxt.org/wc/norobots.html # only disallow wall, allow access to the rest of the site # (the swift archives) User-agent: * Disallow: /wall/ Disallow: /cgi-bin/ User-agent: Webcheck Disallow: User-agent: W3C-checklink Disallow: User-agent: htdig Disallow: