[Foundation-l] Hiding namespaces from search engines

Wed Mar 19 21:51:51 UTC 2008

I think it's a great idea, personally. Robots.txt has (hopefully)
been a way for us to keep things like this out of search engines.
Keeping it out of our internal searches (ie: for our anon users)
would be beneficial as well, methinks.

-Chad

On Wed, Mar 19, 2008 at 5:21 PM, Brian McNeil
<brian.mcneil at wikinewsie.org> wrote:
> If the search back-end knows what matches with results from robots.txt then
>  it can hide these results, offer an option to search including hidden
>  results, and for the paranoid explain these are pages that search engines
>  are requested not to index. By this I mean even when results are returned,
>  not just the set of checkboxes when you get no results.
>
>  Where I see this applying on English Wikinews as a simple example is:
>
>  /wiki/Story_preparation/
>  /wiki/Portal:Prepared_stories/
>
>  One is main namespace, the other Portal. Both are searched by default.
>
>  Try going to Wikinews and searching for "Carter", 6th result is the prepared
>  obituary. With Shimon Peres it was until I made some changes recently 3rd or
>  4th and will likely pop up again in a day or so.
>
>  Question is, on the above basis do people think this is a worthwhile entry
>  to add to bugzilla? Will it in any way benefit other projects?
>
>
>
>  Brian McNeil
>  -----Original Message-----
>  From: foundation-l-bounces at lists.wikimedia.org
>
> [mailto:foundation-l-bounces at lists.wikimedia.org] On Behalf Of Chad
>  Sent: 19 March 2008 21:42
>  To: Wikimedia Foundation Mailing List
>  Subject: Re: [Foundation-l] Hiding namespaces from search engines
>
>
>
> I guess the question is:
>
>  Would we hide it from indexing or only from returning the results? The
>  latter seems easier than the former (and was where I was going with it).
>
>  -Chad
>
>  On Wed, Mar 19, 2008 at 3:24 PM, Bryan Tong Minh
>  <bryan.tongminh at gmail.com> wrote:
>  >
>  > On Wed, Mar 19, 2008 at 8:12 PM, Chad <innocentkiller at gmail.com> wrote:
>  >  > Likewise. After I said that, I started looking
>  >  >  at the code in my local MW install, not entirely
>  >  >  sure where it would go. I'll keep looking around,
>  >  >  as this would be a great extension to have.
>  >  >
>  >  >  -Chad
>  >  >
>  >  >
>  >  >
>  >  >  On Wed, Mar 19, 2008 at 3:03 PM, Bryan Tong Minh
>  >  >  <bryan.tongminh at gmail.com> wrote:
>  >  >  > On Wed, Mar 19, 2008 at 4:10 PM, Chad <innocentkiller at gmail.com>
>  wrote:
>  >  >  >  > Not currently, no.
>  >  >  >  >
>  >  >  >  >  Although, an extension could easily be written I
>  >  >  >  >  would think.
>  >  >  >  >
>  >  >  >  >  -Chad
>  >  >  >  >
>  >  >  >  >  On Wed, Mar 19, 2008 at 10:40 AM, Brian McNeil
>  >  >  >  >
>  >  >  >  >
>  >  >  >  > <brian.mcneil at wikinewsie.org> wrote:
>  >  >  >  >  > Which leads to the question...
>  >  >  >  >  >
>  >  >  >  >  >  Is there any way to get the internal search to honour some
>  sort of ranking
>  >  >  >  >  >  to put stuff in robots.txt at the very bottom?
>  >  >  >  >  >
>  >  >  >  >  >
>  >  >  >  I doubt the easiness.
>  >  >  >
>  >  >  >
>  >  >  >
>  >  >
>  >  >
>  >  > >  _______________________________________________
>  >  >  >  foundation-l mailing list
>  >  >  >  foundation-l at lists.wikimedia.org
>  >  >  >  Unsubscribe:
>  https://lists.wikimedia.org/mailman/listinfo/foundation-l
>  >  >  >
>  >  >
>  >  >  _______________________________________________
>  >  >  foundation-l mailing list
>  >  >  foundation-l at lists.wikimedia.org
>  >  >  Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
>  >  >
>  >  You should look in the MWSearch extension. However this frontend
>  >  relies on the lucene backend. The current version is 2.0, but in a
>  >  separate branch the 2.1 version is on track. That's were you should
>  >  look (It's Java).
>  >
>  >  Bryan
>  >
>  >
>  >
>  >  _______________________________________________
>  >  foundation-l mailing list
>  >  foundation-l at lists.wikimedia.org
>  >  Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
>  >
>
>  _______________________________________________
>  foundation-l mailing list
>  foundation-l at lists.wikimedia.org
>  Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
>
>
>  _______________________________________________
>  foundation-l mailing list
>  foundation-l at lists.wikimedia.org
>  Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
>