Re: [Lzip-bug] On Windows unpacking does not use all cores

[Antonio Diaz Diaz]

Romano wrote:
> May I ask one last thing, is there a reason to use LC - literal
> context parameter lower than 8? Default is 3 and in lzma2 its not
> even allowed above 4. But lc8 give better ratio(~2.2% in my tests and
> many others on fileforums) for most data especially game repacks. I
> changed parameter in the header and it seems workign fine(I guess
> your lzlib is based on lzma1?), but I never understood why would 3 be
> the default promoted choice especially since lc does not affect speed
> and seems to be only benefit at 8.

The reason to not use lc = 8 is that it requires 768 KiB of fast RAM vs 
the 24 KiB required by lc = 3. This would make decompression slow on 
machines with, say, 64 KiB of cache. It also does not increase the 
compression ratio of tarballs by much. I just made a little test with a 
bunch of tarballs:

Total size of tar files = 2_649_872_384
Compressed sizes:
level         -0             -6             -9
lc = 3    576_393_065    391_449_171    367_964_614
lc = 8    569_160_271    388_354_250    365_461_202
             -1.2%          -0.8%          -0.7


> Lastly, pls consider to add more lzma options to the command line in
> next versions. Specifically mc and lc parameter would be very useful.

Adding backward-incompatible options is out of the scope of lzip. As you 
can read here[1], "The lzip format is designed for long-term archiving. 
Therefore it excludes any unneeded features that may interfere with the 
future extraction of the decompressed data."

[1] http://www.nongnu.org/lzip/manual/lzip_manual.html#Quality-assurance

There are lots of tweaks that can increase slightly the compression 
ratio of some kinds of data, generally at the cost of increasing 
resource consumption (memory/cpu), and worse, at the cost of producing 
incompatible files.


> There is also one tweak that 4x4 use that have significant effect on
> speed of compression: it test incoming data that are to be processed by
> lzma with some fast "order 0 model" first. If it cannot compress better
> than to 99% then data are copied instead. This give anywhere between
> 50%-300% speed increase.

These clever tweaks are fine for experimenting, but are also out of the 
scope of a stable standard general-purpose compressor like lzip. See for 
example how neither gzip nor bzip2 have made such changes to their formats.


Best regards,
Antonio.