[WikiEN-l] AFD courtesy problem

Wed Jan 18 13:54:11 UTC 2006

On Tue, 17 Jan 2006 22:53:13 +0100, Andrew Gray <shimgray at gmail.com> wrote:

> On 17/01/06, slimvirgin at gmail.com <slimvirgin at gmail.com> wrote:
>
>> The broader issue is why we allow Google to pick up any of our talk or
>> project pages, where NPOV, NOR, and V don't apply. Can anyone explain
>> why we allow that?
>
> It's logistically quite tricky to arrange matters so the spiders
> understand the difference between a talk page and a "real" page;
> "allowing" isn't the key, it's "why don't we prevent it", and the
> answer is "if we tried it probably wouldn't work very well".
>
> Not to stop anyone attempting something, but...

It's not nessesary for the spiders to know the differense between the  
namespaces. As long as MediaWiki know the difference it should be trivial  
to make it include:

<meta name="robots" content="noindex" />

in the header when rendering a talkpage (and any other namespace we don't  
want indexed).

All the spider need to know is not to index pages that contain that  
directive, and most major web indexers (including Google) do honor that  
tag.

It would require a one time purge of the cache of all the affected  
namespaces after it was implemented to take effect though, and stuff that  
have already been indexed might hang around in search results for months  
before the effects fully propegate.

-- 
[[User:Sherool]]