[Wikitech-l] Re: Data dumps

I'm doing a test run of the new data dump script on our Korean cluster; currently jawiki (ja.wikipedia.org) is in progress: http://amaryllis.yaseo.wikimedia.org/backup/jawiki/20060118/ Any comments on the page layout and information included in the progress page? A couple notes: * The file naming has been changed so they include the database name and date. This should make it easier to figure out what the hell you just downloaded. * The directory structure is different; the database names are used instead of the weird mix of sites, languages, and database names which was hard to reliably get the scripts to run. Each database has subdirectories for each day it was dumped, plus a 'latest' subdirectory with symbolic links to the files from the last completed dump. * I renamed 'pages_current' and 'pages_full' to 'pages-meta-current' and 'pages-meta-history'. In addition to the big explanatory labels, this should emphasize that these dumps contain metapages such as discussion and user pages, distancing it from the pages-articles dump. * I've discontinued 7-Zip compression for the current-versions dumps, since it doesn't do better than bzip2 for those. They are still generated for the history dump, where it compresses significantly better (about 3 vs 11 GB for enwiki) * Upload tarballs are still not included at the moment. The backup runner script is written in Python, and is in our CVS in the 'backup' module should anyone feel like laughing at my code. A few more things need to be fixed up before I start running it on the main cluster, but it's pretty close! (A list of databases in progress, some locking, emailing me on error, and finding the prior XML dump to speed dump generation.) -- brion vibber (brion @ pobox.com) ------------------------------------------------------------------------ _______________________________________________ Wikitech-l mailing list Wikitech-l(a)wikimedia.org http://mail.wikipedia.org/mailman/listinfo/wikitech-l