Jeremie Bouillon wrote:
About the robot.txt, can we find the one used by large
wiki (like
Wikipedia, or others) to help Wikimedia newbie exclube spiderbot from
where there aren't supposed to be ?
It's in the customary location, of course:
http://en.wikipedia.org/robots.txt
robots.txt files are fun... If you look at
http://www.whitehouse.gov/robots.txt you'll find the White House is
hiding WMDs!!!! ("Disallow: /wmd/text")
Other people just don't like having their documents archived for later
reference:
http://www.sco.com/robots.txt
Not that some people should complain... ;)
http://www.groklaw.net/robots.txt
-- brion vibber (brion @
pobox.com)