[Mediawiki-l] Re: rel=nofollow for edit, history and diff links

Fri Apr 29 22:25:49 UTC 2005

Jeremie Bouillon wrote:
> About the robot.txt, can we find the one used by large wiki (like
> Wikipedia, or others) to help Wikimedia newbie exclube spiderbot from
> where there aren't supposed to be ?

It's in the customary location, of course:
http://en.wikipedia.org/robots.txt

robots.txt files are fun... If you look at
http://www.whitehouse.gov/robots.txt you'll find the White House is
hiding WMDs!!!! ("Disallow:	/wmd/text")

Other people just don't like having their documents archived for later
reference: http://www.sco.com/robots.txt

Not that some people should complain... ;)
http://www.groklaw.net/robots.txt

-- brion vibber (brion @ pobox.com)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 253 bytes
Desc: OpenPGP digital signature
Url : http://lists.wikimedia.org/pipermail/mediawiki-l/attachments/20050429/b90acf39/attachment.pgp 