Hi all
When trying to process the current enwiki dump (specifically,
enwiki-20080312-pages-articles.xml.bz2) using mwdumper, it crashed on me with an
UTF-8-related I/O Error in Xerces. The problem occurrs (with slightly different
symptoms) with Xerces 2.7.1 and 2.9.1, it is described in detail at [1].
Basically, Xerces' UTF-8 decoder is broken for the case that a surrogate pair is
split across buffer reads. The problem was reported by Robert Stojnic aka
rainman last year, but apparently the attempt to fix it only changed the way it
is broken.
Anyway, the current dump isn't usable with mwdumper. This is not good. And it is
likely to happen again.
I see two ways to solve it:
1) Ship a patched version of Xerces with mwdumper, with Robert's patch applied
(see bug report at [1]). But there seems to be some problem with that patch
(dicussed in the bug report), and relying on a patched version of a (supposedly)
standard lib feels a but dirty.
2) Don't use Xerces' UTF-8 decoder, use the JRE's built in one. I have
prepared
a patch for this (see [2]), but I have not tested it excessively (only so far as
that it doesn't blow up in my face). I havn't even verified that it actually
fixes the problem (it takes half a day to get that far in processing the dump -
that thing is huge, a small test case would be great for this). It would also be
good to know how using the JRE's decoder impacts performance. Because of these
questions, I havn't committed the patch yet. Please play with it if you have a
couple of minutes for this kind of thing.
Regards,
Daniel
[1] Xerces bug report <https://issues.apache.org/jira/browse/XERCESJ-1257>
[2] mwdumper patch: <http://rafb.net/p/7c5bkg52.html>