Re: zlib-ng as a compat replacement for zlib

[Kevin Wolf]

[ Cc: qemu-block ]

Am 30.08.2023 um 20:26 hat Richard W.M. Jones geschrieben:
> On Tue, Aug 29, 2023 at 05:49:24PM -0000, Daniel Alley wrote:
> > > The background to this is I've spent far too long trying to optimize
> > > the conversion of qcow2 files to raw files.  Most existing qcow2 files
> > > that you can find online are zlib compressed, including the qcow2
> > > images provided by Fedora.  Each cluster in the file is separately
> > > compressed as a zlib stream, and qemu uses zlib library functions to
> > > decompress them.  When downloading and decompressing these files, I
> > > measured 40%+ of the total CPU time is doing zlib decompression.
> > > 
> > > [You don't need to tell me how great Zstd is, qcow2 supports this for
> > > compression also, but it is not widely used by existing content.]

You make it sound like compressing each cluster individually has a big
impact. If so, does increasing the cluster size make a difference, too?
That could be an change with less compatibility concerns.

> > Independent from the decision to use zlib-ng as a distro-wide zlib
> > replacement, which is a good idea regardless.
> >
> > Are there reasons why Fedora's qcow2 images cannot switch to Zstd
> > compression?  Zstd support appears to have been added by QEMU 5.1 in
> > August 2020, and both EL8 and EL9 appear to have newer versions QEMU
> > available (therefore, they ought to be able to support those
> > images).
> 
> TBH I think the most probable reason is that people don't know about
> it and it is not obvious that you have to enable it.  To generate a
> zlib-compressed qcow2 file, you simply add the -c option, easy.  To
> use zstd compression you have to use this mouthful:
> 
>   qemu-img convert -f raw disk.img -O qcow2 disk.qcow2 -c -o 
> compression_type=zstd
> 
> The qemu-img man page doesn't even mention it.

Good point, that needs to be fixed.

(Though I don't think that '-o compression_type=zstd' in an unreasonable
mouthful for enabling a non-standard compression algorithm.)

> I think all recent qcow2-based tools should be fine with zstd, but I
> didn't check them all (RHEL 7 is still quite popular so that platform
> would no longer be compatible).

So my first thought was that maybe we can just change the default now
that the option has been there for quite a while. But then it occurred
to me that it's not a hard dependency. So at least, the default would
still have to be zlib if zstd isn't even compiled in, which makes it a
bit less nice in theory. In practice, it depends on what build options
distros actually use.

Unfortunately, we seem to build the RHEL packages with --disable-zstd
(I suppose just because we tend to disable everything nobody explicitly
asked for). Maybe we should check other distros. If zstd is commonly
enabled, we could still make it the default upstream (because honestly,
it probably does makes sense from an upstream perspective). Of course,
for RHEL this would mean that images out there are likely to use zstd
soon, so it might need to enable zstd in newer versions, too.

Kevin