[Mediawiki-l] googlebot

Tue Aug 16 15:40:13 UTC 2005

On Tue, 16 Aug 2005, andres wrote:
> Accidentally i watched my logfiles (tail -f) and noticed a weird
> behaviour of the google spider.  It spends literally days following
> all links in Spezial:Recentchanges.

there is one problem with Special:Recentchanges where you can click
further and further on "next pages", even if there are no more.

> How can i disallow search engines from indexing all the recent
> changes?  They are worthless to index anyway.

check out: http://en.wikipedia.org/robots.txt for an example. You
could use:

User-agent: *
Disallow: /wiki/Special:Randompage
Disallow: /wiki/Special%3ARandompage
Disallow: /wiki/Special:Recentchanges
Disallow: /wiki/Special%3ARecentchanges

christof

-- 
Christof Damian         
christof at damian.net