On 09/20/2012 07:40 PM, MZMcBride wrote:
Scanning dumps (or really dealing with them in any
form) is pretty awful.
There's been some brainstorming in the past for how to set up a system where
users (or operators) could run arbitrary regular expressions on all of the
current wikitext regularly, but such a setup requires _a lot_ of anything
involved (disk space, RAM, bandwidth, processing power, etc.). Maybe one day
Labs will have something like this.
We have a dump grepper tool in the Parsoid codebase (see
js/tests/dumpGrepper.js) that takes about 25 minutes to grep an XML dump
of the English Wikipedia. The memory involved is minimal and constant,
the thing is mostly CPU-bound.
It should not be hard to hook this up to a web service. Our parser web
service in js/api could serve as a template for that.
Gabriel