[Labs-l] Google bot

Mon Oct 20 14:41:38 UTC 2014

Magnus Manske (2014-10-20 13:33):
> [...]
> AFAIK, all tools use a default .lighttp configuration by default. Is 
> that replaced or extended by a local config file?
> If it's replaced, the default config could exclude Googlebot, and even 
> a blank local config file would re-enable Googlebot again, for those 
> who want it.

Options are merged not replaced (as docs say [docs]), but +1 for default 
option to block crawlers would be fine. Though there should be 
information on how to unblock them (probably in those docs).

[docs] = 
https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Configuring_the_web_server