Re: [Lzip-bug] bug report: 'plzip -vf9' fails to compress files over a certain size

[Erik Jan Tromp]

On Mon, 11 Dec 2017 13:39:13 +0100
Antonio Diaz Diaz <address@hidden> wrote:

> Erik Jan Tromp wrote:
> > I had assumed (&  hoped) plzip would attempt to use as much ram as
> > possible, scale back the number of threads if/when necessary, then
> > throw an error only if there was insufficient memory for a single
> > thread.
> 
> I have just implemented such scaling down for the 32 bit case. Please, 
> test the attached patch against plzip-1.6. I plan to release shortly a 
> new version of plzip with this feature.
> 
> Best regards,
> Antonio.

Success! Of sorts :)

The troublesome test files compressed without error, but at a cost of
time on some machines (lesser thread count).

Following is a sample from compressing the largest test file on various
machines. While the exact times below are essentially meaningless,
their relation between unpatched & patched (as well as thread count)
are informative.

"system default" as listed below was derived thusly:
  plzip --help | grep -- "--threads=" | tr -cd 0-9

Side note: the system default value is unchanged between unpatched &
patched.

   test file name: tribes2-25034-i686-1_ejt.tlz
uncompressed size: 661166080 bytes
  compressed size: 485020872 bytes

atom n270 w/ 1.5gb ram
system default: 2 threads
     unpatched: 20m  6s (threads: 4, 3, 2, 1)
       patched: 20m  5s (threads: 4, 4, 2, 1)

atom d525 w/ 2gb ram
system default: 4 threads
     unpatched: 11m 39s (threads: 6, 5, 4, 3, 2, 1)
       patched: 11m  3s (threads: 5, 4, 3, 2) [1]

celeron 3205u w/ 8gb ram
system default: 2 threads
     unpatched:  4m 45s (threads: 4, 3, 2)
       patched:  4m 44s (threads: 4, 3, 2)

core i3 w/ 8gb ram
system default: 4 threads
     unpatched:  2m 45s (threads: 6, 5, 4, 3, 2)
       patched:  3m 27s (threads: 5, 4, 3, 2)

core i7 w/ 6gb ram
system default: 8 threads
     unpatched:  1m 26s (threads: 9, 7, 6, 5, 4, 2) [2]
       patched:  2m 20s (threads: 5, 4, 3, 2)

[1] Both unpatched & patched pushed the system into using swap, but
patched less so. Slightly less thrashing == faster, or so I assume.

[2] My test script initially attempts compression at '-9' with the
system default thread count. If the initial attempt fails, it retries
in a loop specifying number of threads from "system default minus 1" to
1. If the last attempt fails (hasn't happened yet, but it's handled),
the script throws an error.

For completeness, I've attached the thread counting script. This is new
territory for me & I'm curious as to the disparity between system
default & actual threads. Dispatcher(s) & workers?

Thus endeth another rambling post. :)

Regards,
Erik

-- 
Talent hits a target no one else can hit; Genius hits a target no one
else can see.
 -- Arthur Schopenhauer

count_threads.sh
Description: Binary data