Very nice of them to let us know!
----- Forwarded message from Warren Brown <wbrown(a)inktomi.com> -----
From: "Warren Brown" <wbrown(a)inktomi.com>
Date: Mon, 18 Aug 2003 17:25:08 -0700
To: root(a)wikipedia.org
Subject: Inktomi web crawler
The
wikipedia.org server is blocking Inktomi's "Slurp" web crawler by
returning 403 errors for all access attempts. Presumably, this block
was set up because we were crawling the site too aggressively at some
time in the past. We would like to include
wikipedia.org content in our
search database, and would be happy to work with you to match whatever
crawling limits you need to set.
Slurp observes /robots.txt rules for user-agent "Slurp". The crawler
access rate is normally limited to 4 pages per minute from a web server;
we can set that rate lower if you require. The Slurp access rate can
also be controlled by a "crawldelay" instruction in /robots.txt.
Inktomi search service is used MSN Search and a number of other web
portal and business sites worldwide. We are now a subsidiary company of
Yahoo!
Regards,
Warren Brown
Partner Service and Support
Inktomi, a Yahoo! Company
----- End forwarded message -----