[Wikipedia-l] Re: Robots and special pages

On Saturday 18 May 2002 12:01 pm, you wrote: My two cents (well maybe a bit more), On talk pages: OPEN to bots Its A OK for bots to index talk pages -- these pages often have interesting discussion that should be on search engines. Of course, if this becomes a performance issue then we could prevent bots from indexing these. On wikipedia pages: OPEN to bots I STRONGLY feel that wikipedia pages should be open to bots -- remember we are also trying to expand our community here and people do search for those things on the net. On user pages: OPEN to bots I also don't see anything wrong with letting bots crawl all over user pages -- I occasionally browse personal home pages of other people that have similar interests to myself. This project isn't just about the articles it is also about community building. On log, history, print and special pages: CLOSED to bots (closed at least for indexing -- not sure about allowing the 'follow links' function. Would closing this allow bots to do their thing faster or slower? Is this at all important for us to consider? If a bot can index our site fast, will it do it more often?) I think that the wikipedia pages are FAR better at naturally explaining what the project is about than are the log, history and special pages are -- these pages are far too technical and change too quickly to be useful for any search performed on a search engine. There is also limited utility of having direct links to the Printable version of articles -- these don't have any active wiki links in them which obscures the fact that the page is from a wiki. Having history pages in the search results of external search engines is potentially dangerous, since somebody could easily click into an older version and save it -- thus reverting the article and unwittingly "earning" the label of VANDAL (even if they did make a stab at improving the version they read). Another reason to disallow bots access to history is because there often is copyrighted material in the history of pages that has since been removed from the current article version (it would be nice for an admin to able to delete just an older version of an article BTW). On Edit links: CLOSED to bots (for index and probably follow links) The edit links REALLY should NOT be allowed to be indexed by any bot: When somebody searches for something on a search engine, gets a link to our site, and clicks on it; do we want them to be greeted with an edit window? They want information -- not an edit window. No wonder we have so many pages that only have "Describe the new page here" as their only content. I've been tracking this for awhile and almost every one of these pages that are created, are created by an IP that never returns to edit again. Many (if not most) of these "mysteriously" created pages are probably from someone clicking from a search engine, becoming puzzled by the edit window, and hitting the save button in frustration. Heck, I think I may have created a few of these in my pre-wiki days. This has become a bit of a maintenance issue for the admins -- we can't delete these pages fast enough, let alone create stubs for them. If left unchecked, this could reduce the average quality of wikipedia articles and give people doubt as to whether an "active" wiki link really has an article (or even a stub) behind it. There could, of course, be a purely technical fix for this by having the software not recognize newly created blank or "Describe the new page here" pages as being real pages (a Good Idea BTW). But then we still would have frustrated people who were looking for actual info that in the future may avoid clicking through to our site because of a previous "edit window experience". Conclusion: We should try to put our best foot forward when allowing bots to index the site and only allow indexing of pages that have information which is potentially useful for the person searching. Edit widows and outdated lists are NOT useful to somebody clicking through for the first time (Recent Changes might be the only exception: Even though any index of this will be outdated, it is centrally important to the project and fairly self-explanatory). Links to older versions of articles and to history pages also sets-up would-be contributors into becoming labeled as "vandals" when trying to edit an older version -- thus turning them away forever. Let visitors explore a real article first and discover the difference between an edit window an an actual article -- then they can decide about becoming a contributor, visitor or even a developer for that matter. maveric149