On Wed, Dec 15, 2010 at 12:01 PM, Andrew Dunbar &lt;hippytrail(a)gmail.com&gt; wrote: I've written something similar for .xz, which uses LZMA2 same as .7z. It creates a virtual read-only filesystem using FUSE (the FUSE part is in perl, which uses pipes to dd and xzcat). Only real problem is that it doesn't use a stock .xz file, it uses a specially created one which concatenates lots of smaller .xz files (currently I concatenate between 5 and 20 or so 900K bz2 blocks into one .xz stream - between 5 and 20 because there's a preference to split on </page><page> boundaries). Apparently the folks at openzim have done something similar, using LZMA2. If anyone is interesting in working with me to make a package capable of being released to the public, I'd be willing to share my code. But it sounds like I'm just reinventing a wheel already invented by opensim. With concatenated .xz files, creating the index is *much* faster, because the .xz format puts the stream size at the end of each stream. Plus with .xz all streams are broken on 4-byte boundaries, whereas with .bz2 blocks can end at any *bit* (which means you have to do painful bit shifting to create the index). The file is also *much* smaller, on the order of 5-10% of bzip2 for a full history dump.

On Thu, Dec 16, 2010 at 12:47 AM, Andrew Dunbar &lt;hippytrail(a)gmail.com&gt; wrote: I'm fairly certain the specific 7z format which Wikimedia uses doesn't allow for random access, because the dictionary is never reset. No, there's no off-the-shelf tool to create these files - the standard .xz file created by xz utils puts everything in one stream, which is basically equivalent to the .7z files already being made. I'm sure "patches are welcome", but I don't have the time to create the patch. At the default settings, it's quite slow. I believe it's pretty much the same as 7zip with its default settings. The main reason I was using xz instead of 7zip is that xz handles pipes better - specifically, 7zip doesn't allow you to pipe from stdin to stdout. (See https://bugs.launchpad.net/ubuntu/+source/p7zip/+bug/383667 and the response - "You should use lzma." - well, lzma utils has been replaced by xz utils.) For decompression, .xz is generally faster than .bz2, slower than .gz No, it uses LZMA2.

On 15 December 2010 20:24, Manuel Schneider &lt;manuel.schneider(a)wikimedia.ch&gt; wrote: This is very interesting and I'll be watching it. Where do the HTML dumps come from? I'm pretty sure I've only seen "static" for Wikipedia and not for Wiktionary for example. I am also looking at adapting the parser for offline use to generate HTML from the dump file wikitext. Andrew Dunbar (hippietrail)

Hi, Am 16.12.2010 06:50, schrieb Andrew Dunbar: there are several ways to get the HTML data. Emmanuel (Kelson) who is doing Kiwix and WP1.0 uses SQL dumps to set up a seperate instance of MediaWiki to dump the data on the command line. Ralf from Pediapress has written a wrapper for zimlib to use it with Python, in order to integrate a ZIM export to the Collection Extension. * http://github.com/schmir/pyzim A few weeks ago Tommi from openZIM has committed "wikizim" to the zimwriter codebase, a command line tool that dumps a whole wiki into a ZIM file using the MediaWiki API. * http://svn.openzim.org/viewvc.cgi/trunk/zimwriter/ There have been discussions with Roan Kattouw about how to use wikizim on the Wikimedia wikis in the best way. His approach is to integrate the relevant code from wikizim into the MediaWiki code to avoid abstraction layers as much as possible in favour of performance and system load. /Manuel -- Regards Manuel Schneider Wikimedia CH - Verein zur Förderung Freien Wissens Wikimedia CH - Association for the advancement of free knowledge www.wikimedia.ch