Re: [Lzip-bug] bug report: 'plzip -vf9' fails to compress files over a certain size

[Antonio Diaz Diaz]

Hello Erik,

Erik Jan Tromp wrote:
> I believe I've found a bug in plzip. Apologies for rambling.

Thanks for reporting the issue. Better safe than sorry. :-)


> software:
> Slackware 14.2 (32 bit)
> gcc (GCC) 5.3.0
> plzip 1.6
>
> hardware:
> Intel(R) Core(TM) i7 CPU 930 @ 2.80GHz
> quad core with hyperthreading
> 6GB RAM (not a typo)

According to [1], compressing at level 9 requires about 32 * 11 + 64 * 3 
= 544 MiB. For 8 threads this takes 4352 MiB (4.25 GiB). I suspect the 
problem is that a process can only use less than 4 GiB on a 32 bit system.

[1] http://www.nongnu.org/lzip/manual/plzip_manual.html#Memory-requirements


> test:
> Compress each tarball with 'plzip -vf9 foo.tar'
>
> Tarballs 421969920 bytes&  larger: fail to compress
> Tarballs 305926656 bytes&  smaller: compress successfully

This is because compressing a file of 306 MB at level 9 can use only 5 
threads. See [2].

[2] http://www.nongnu.org/lzip/manual/plzip_manual.html#Minimum-file-sizes


> Tests were also performed on the following hardware using 'plzip -vf9
> foo.tar' with no errors.

All of them seem to either be 64 bit or have too few processors to reach 
the per process memory limit of 32 bit. Am I wrong?


> Please note that while specifying '-n'&  '-s' options are functional
> solutions, they are unacceptable for the intended use case. Regardless
> of a machine's hardware, the resulting compressed files must be
> identical/predictable.

If you want identical results on 32 and 64 bit machines, I guess you'll 
need to limit the number of threads. Limiting the dictionary size may 
not be enough on 32 bit systems with many cores.


Best regards,
Antonio.