Re: [PATCH] gitlab-ci: Replace Docker with Kaniko

[Camilla Conte]

On Fri, May 17, 2024 at 9:14 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Thu, May 16, 2024 at 07:24:04PM +0100, Daniel P. Berrangé wrote:
> > On Thu, May 16, 2024 at 05:52:43PM +0100, Camilla Conte wrote:
> > > Enables caching from the qemu-project repository.
> > >
> > > Uses a dedicated "$NAME-cache" tag for caching, to address limitations.
> > > See issue "when using --cache=true, kaniko fail to push cache layer 
> > > [...]":
> > > https://github.com/GoogleContainerTools/kaniko/issues/1459
> >
> > After investigating, this is a result of a different design approach
> > for caching in kaniko.
> >
> > In docker, it can leverage any existing image as a cache source,
> > reusing individual layers that were present. IOW, there's no
> > difference between a cache and a final image, they're one and the
> > same thing
> >
> > In kaniko, the cache is a distinct object type. IIUC, it is not
> > populated with the individual layers, instead it has a custom
> > format for storing the cached content. Therefore the concept of
> > storing the cache at the same location as the final image, is
> > completely inappropriate - you can't store two completely different
> > kinds of content at the same place.
> >
> > That is also why you can't just "git pull" the fetch the cache
> > image(s) beforehand, and also why it doesn't look like you can
> > use multiple cache sources with kaniko.
> >
> > None of this is inherantly a bad thing..... except when it comes
> > to data storage. By using Kaniko we would, at minimum, doubling
> > the amount of data storage we consume in the gitlab registry.
>
> Double is actually just the initial case. The cache is storing layers
> using docker tags, whose name appears based on a hash of the "RUN"
> command.
>
> IOW, the first time we build a container we have double the usage.
> When a dockerfile is updated changing a 'RUN' command, we now have
> triple the storage usage for cache. Update the RUN command again,
> and we now have quadruple the storage. etc.
>
> Kaniko does not appear to purge cache entries itself, and will rely
> on something else to do the cache purging.
>
> GitLab has support for purging old docker tags, but I'm not an
> admin on the QEMU project namespace, so can't tell if it can be
> enabled or not ? Many older projects have this permanently disabled
> due to historical compat issues in gitlab after they introduced the
> feature.

I'm pretty sure purging can be enabled. Gitlab itself proposes this
with a "set up cleanup" link on the registry page (1).
Can you recall what issues they were experiencing?

If this is the only issue blocking Kaniko adoption, and we can't solve
it by enabling the cleanup, I can write an additional step at the end
of the container build to explicitly remove old cache tags.

(1) https://gitlab.com/qemu-project/qemu/container_registry

>
> With regards,
> Daniel
> --
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
>