Do we forbid certain spiders access to the site based on User-Agent? A
user in a German forum reported recently that he couldn't access
Wikipedia at all, always receiving a "Forbidden" message. It turned out
that his webwasher proxy was to blame (an ad banner block). The proxy
sends the User-Agent
"Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt) WebWasher 3.0"
Webwasher cannot be used to spider and download sites.
Axel