[Mediawiki-l] Re: rel=nofollow for edit, history and diff links
Brion Vibber
brion at pobox.com
Fri Apr 29 22:25:49 UTC 2005
Jeremie Bouillon wrote:
> About the robot.txt, can we find the one used by large wiki (like
> Wikipedia, or others) to help Wikimedia newbie exclube spiderbot from
> where there aren't supposed to be ?
It's in the customary location, of course:
http://en.wikipedia.org/robots.txt
robots.txt files are fun... If you look at
http://www.whitehouse.gov/robots.txt you'll find the White House is
hiding WMDs!!!! ("Disallow: /wmd/text")
Other people just don't like having their documents archived for later
reference: http://www.sco.com/robots.txt
Not that some people should complain... ;)
http://www.groklaw.net/robots.txt
-- brion vibber (brion @ pobox.com)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 253 bytes
Desc: OpenPGP digital signature
Url : http://lists.wikimedia.org/pipermail/mediawiki-l/attachments/20050429/b90acf39/attachment.pgp
More information about the MediaWiki-l
mailing list