There's been some questions recently about public backup dumps (or the
lack thereof). I've been working for the last few days on getting the
dump generation infrastructure up and running in a more consistent,
effective fashion.
Here's what's currently on my plate and the status thereof:
* Title corrections: some of the databases contain invalid page titles
left over from old bugs. This can sometimes break the import or export
process, so I'm writing a fixup script to find and rename them.
STATUS: Finding done, fixing to come.
Should be done with this later today.
* Dump filtering/processing: currently the dump has to run twice to
produce the current-only and all-revisions dump files. I'm working on a
postprocessing tool which will be able to split these two from a single
runthrough, as well as produce a filtered dump with the talk and user
pages removed.
Producing the split versions from one run should also mean that the dump
can run without having replication stopped the whole time.
It can also produce SQL for importing a dump directly into a database in
either 1.4 or 1.5 schema, for those using software based on the old
database layout. (We probably won't be hosting such files on our server
but you can run the program locally to filter XML-to-MySQL.)
STATUS: Mostly done. Some more testing and actually hooking up multiple
simultaneous outputs remains.
Should be done tonight or tomorrow.
* Progress and error reporting: The old backup script was a hacky shell
script with no error detection or recovery, requireing manually stopping
replication on a database server and reconfiguring the wiki cluster for
the duration. If something went awry, maybe nobody noticed... the
hackiness of this is a large part of why we've never just let it run
automatically on a cronjob.
I want to rework this for better automation and to provide useful
indications of what it's doing, where it's up to, and if something went
wrong.
STATUS: Not yet started. Hope to have done tomorrow or Friday.
* Clean up
download.wikimedia.org further, make use of status files left
by the updated backup runner script.
STATUS: Not yet started. (Doesn't have to be up before the backup starts.)
-- brion vibber (brion @
pobox.com)