Btw: has adding thumbnail image support to the ZIM file format been ZIM is able to store anything, also pictures. Most of the ZIM files are with thumbnails.

Hi Douglas On 01/01/2013 02:22 AM, Douglas Crosher wrote: No, this is complicated to do... although this could be practical. I'm also not sure if we could achieve to get acceptable performances.

Great. Yes, xz decompression is the most complicated part.

Douglas, There are multiple JS LZMA libraries. I haven't looked at any of them but have you ? It might be enough for you to get a sens of performances. renaud On Tue, Jan 1, 2013 at 1:18 PM, Douglas Crosher &lt;dtc(a)scieneer.com <mailto:dtc@scieneer.com>> wrote: On 01/01/2013 08:09 PM, Emmanuel Engelhart wrote: I'll hack something together to explore the performance question, and follow up. file and Would it be very limiting on ZIM files if the XZ decoder were restricted to the 'XZ embedded' format, supporting only the 'LZMA2' filter? See: http://tukaani.org/xz/embedded.html Do ZIM files really need the XZ/LZMA2 containers, or could they just use raw LZMA1 compression? This could be added as a new cluster compression type for compatibility. Two possible uses for XZ/LZMA2 may be for large entries and/or entries with distinct regions that are compressible and not compressible. However perhaps a significant amount of content does not need this. I expect that typical HTML entries would be relatively small. It would seem pointless for a cluster to use multiple XZ blocks and/or streams when these could be avoided by placing entries in separate clusters. So perhaps there is a case for clusters with just one LZMA1 block. Further entries are likely to either be compressible or not, and could be placed in separate clusters rather than exploiting the LZMA2 support for such content. It might even save space not having the XZ container overhead. Regards Douglas Crosher _______________________________________________ Offline-l mailing list Offline-l(a)lists.wikimedia.org <mailto:Offline-l@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/offline-l _______________________________________________ Offline-l mailing list Offline-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/offline-l

Hi Renaud, Here are some I have seen: * LZMA-JS https://github.com/nmrugg/LZMA-JS/ Online demo: http://nmrugg.github.com/LZMA-JS/demos/advanced_demo.html * js-lzma http://code.google.com/p/js-lzma/ I have not seen any LZMA2 or XZ Javascript libraries. I've written much of the XZ container support, and are looking at the LZMA2 support, but there seems to be a lot of unnecessary baggage in the XZ/LZMA2 containers. First tests suggest LZMA decompression in JS is very slow, but I would not want to draw any conclusions until analysing it further. If raw LZMA1 were an option then it could also be useful to include the CRC32 for each cluster blob so that the decompresser could exit as soon as the blob was decoded rather than having to decompress the entire cluster to verify the XZ checks. The CRC32 could be included after each blob offset. If the JS decompresser is slow then such optimizations might be essential, and might help battery life on mobile devices. Regards Douglas Crosher On 01/02/2013 12:23 AM, renaud gaudin wrote: > Douglas, > > There are multiple JS LZMA libraries. I haven't looked at any of them > but have you ? It might be enough for you to get a sens of performances. > > renaud > > > On Tue, Jan 1, 2013 at 1:18 PM, Douglas Crosher &lt;dtc(a)scieneer.com > <mailto:dtc@scieneer.com>> wrote: > > On 01/01/2013 08:09 PM, Emmanuel Engelhart wrote: > > Hi Douglas > > > > On 01/01/2013 02:22 AM, Douglas Crosher wrote: > >> Has anyone considered a pure Javascript ZIM file reader and Wikipedia > >> reader? > > > > No, this is complicated to do... although this could be practical. I'm > > also not sure if we could achieve to get acceptable performances. > > I'll hack something together to explore the performance question, and > follow up. > > >> I have made a small start, writing some hack code to open a ZIM > file and > >> it gets to the point of needing to uncompress a cluster. A start has > >> been made on the needed XZ decompress code but it's not done yet. > > > > Great. Yes, xz decompression is the most complicated part. > > Would it be very limiting on ZIM files if the XZ decoder were restricted > to the 'XZ embedded' format, supporting only the 'LZMA2' filter? See: > http://tukaani.org/xz/embedded.html > > Do ZIM files really need the XZ/LZMA2 containers, or could they just use > raw LZMA1 compression? This could be added as a new cluster compression > type for compatibility. > > Two possible uses for XZ/LZMA2 may be for large entries and/or entries > with distinct regions that are compressible and not compressible. > However perhaps a significant amount of content does not need this. > > I expect that typical HTML entries would be relatively small. It would > seem pointless for a cluster to use multiple XZ blocks and/or streams > when these could be avoided by placing entries in separate clusters. So > perhaps there is a case for clusters with just one LZMA1 block. Further > entries are likely to either be compressible or not, and could be placed > in separate clusters rather than exploiting the LZMA2 support for such > content. > > It might even save space not having the XZ container overhead. > > Regards > Douglas Crosher

Would it be very limiting on ZIM files if the XZ decoder were restricted to the 'XZ embedded' format, supporting only the 'LZMA2' filter? See: http://tukaani.org/xz/embedded.html

Do ZIM files really need the XZ/LZMA2 containers, or could they just use raw LZMA1 compression? This could be added as a new cluster compression type for compatibility.

Two possible uses for XZ/LZMA2 may be for large entries and/or entries with distinct regions that are compressible and not compressible. However perhaps a significant amount of content does not need this.

I expect that typical HTML entries would be relatively small. It would seem pointless for a cluster to use multiple XZ blocks and/or streams when these could be avoided by placing entries in separate clusters. So perhaps there is a case for clusters with just one LZMA1 block.

Further entries are likely to either be compressible or not, and could be placed in separate clusters rather than exploiting the LZMA2 support for such content.

It might even save space not having the XZ container overhead.

Hi, there should be only one compression algorithm. Otherwise a reader must be able to handle every supported algorithm. What is the point of having a standard format where some readers could read only part of them?

The zimwriter makes clusters of 1MB of html files and compresses them with lzma2. Actually no xz overhead is used here. The 1MB cluster size is choosen because lzma2 uses it. Larger clusters do not increase the compression ratioa at all.

The writer has a fixed list of mime types, which are not compressed. The mime types are "image/jpeg", "image/png", "image/tiff", "image/gif" and"application/zip". The writer do not try to compress them further but they are stored as is in a separate cluster.

On 01/03/2013 10:29 PM, Douglas Crosher wrote: I am positive that LZMA2 is quite suited for JS in terms of performance (or at least not worse than other compressors). When we developed the ZIM file format we had small devices with only little ressources in mind, the first users of openZIM were small gadgets like the Ben NanoNote. LZMA has the disadvantage that it is quite expensive in compressing but gets better compression ration than others. On the other hand decompression is quite cheap compared with others, so it was the perfect choice for these small platforms. There was a discussion whether we could go a step further for devices with very limited memory and not decompress and cache a whole cluster when accessing it but just use the parts from the decompression stream we need and stop reading once we have all data needed, forgetting everything else. I think that has been implemented in zimlib.

Actually no xz overhead is used here.

Hi Douglas I will try to answer you, tell it if somehow I misunderstand you. Le 01/01/2013 14:18, Douglas Crosher a écrit : Why? ZIM only supports the LZMA2 compression. So this would be perfect. As far as I have understood xz-embedded is only a decompressor with limited features.

We can not change the chosen compression algorithm for the ZIM format.

We recommend to compress only text content and consequently pictures are usually not compressed in ZIM files. The amount of text compressed in one cluster is chosen by the ZIM creator, at Kiwix it's 1MB (a size we should maybe reconsider and increase).

Not sure to understand you right, but HTML entries are concatenated *before* being compressed.

As far as I know stream are LZMA2 encoded and do not use the XZ format.