bug#28659: v0.13: guix pull fails; libgit2-0.26.0 and 0.25.1 content hashes fail

[Ludovic Courtès]

Leo Famulari <address@hidden> skribis:

> On Mon, Oct 02, 2017 at 10:00:33PM +0200, Ludovic Courtès wrote:
>> Right.  Jan suggested checking the content-addressed mirrors *before*
>> the real upstream address.  That would address the problem of upstream
>> sources modified in-place, but at the cost of privacy/self-sufficiency
>> as you note.  (Though it’s not really making “privacy” any worse in this
>> case: it’s gnu.org vs. github.com.)
>
> Yeah, I don't personally think there is a privacy issue with fetching
> sources from our mirrors at gnu.org, or other domains we control.
>
>> Perhaps we should make content-addressed mirrors configurable in a way
>> that’s orthogonal to derivations, something similar in spirit to
>> --substitute-urls?  The difficulty is that content-addressed mirrors are
>> not just URLs; see (guix download).
>>
>> Thoughts?
>
> I do think we should make it so that users don't suffer from unreliable
> upstream sources when we know the sources are available on our servers
> (or the Nix mirror), even with --no-substitutes.

The more I think about it, the more I’m inclined to simply move
content-addressed mirrors to the front of the list.  This means that
users, in practice, would be fetching all the source from
mirror.hydra.gnu.org.

The main issue is making it configurable.  Currently the
content-addressed mirror configuration for regular files in (guix
download) looks like this:

--8<---------------cut here---------------start------------->8---
(define %content-addressed-mirrors
  ;; List of content-addressed mirrors.  Each mirror is represented as a
  ;; procedure that takes a file name, an algorithm (symbol) and a hash
  ;; (bytevector), and returns a URL or #f.
  ;; Note: Avoid 'https' to mitigate <http://bugs.gnu.org/22774>.
  ;; TODO: Add more.
  '(list (lambda (file algo hash)
           ;; Files served by 'guix publish' are accessible under a single
           ;; hash algorithm.
           (string-append "http://mirror.hydra.gnu.org/file/";
                          file "/" (symbol->string algo) "/"
                          (bytevector->nix-base32-string hash)))
         (lambda (file algo hash)
           ;; 'tarballs.nixos.org' supports several algorithms.
           (string-append "http://tarballs.nixos.org/";
                          (symbol->string algo) "/"
                          (bytevector->nix-base32-string hash)))))
--8<---------------cut here---------------end--------------->8---

That for VCS checkouts in (guix build download-nar) looks like this:

--8<---------------cut here---------------start------------->8---
(define (urls-for-item item)
  "Return the fallback nar URL for ITEM--e.g.,
\"/gnu/store/cabbag3…-foo-1.2-checkout\"."
  ;; Here we hard-code nar URLs without checking narinfos.  That's probably OK
  ;; though.
  ;; TODO: Use HTTPS?  The downside is the extra dependency.
  (let ((bases '("http://mirror.hydra.gnu.org/guix";
                 "http://berlin.guixsd.org";))
        (item  (basename item)))
    (append (map (cut string-append <> "/nar/gzip/" item) bases)
            (map (cut string-append <> "/nar/" item) bases))))
--8<---------------cut here---------------end--------------->8---

The latter could be expressed by a command-line flag.  In fact it’s the
same as --substitute-urls.

(Time passes…)

Thinking more about it, why not simply always enable substitutes for
fixed-output derivations, like this:

diff --git a/nix/libstore/build.cc b/nix/libstore/build.cc
index d68e8b2bc..03a8f5080 100644
--- a/nix/libstore/build.cc
+++ b/nix/libstore/build.cc
@@ -1034,8 +1034,10 @@ void DerivationGoal::haveDerivation()
 
     /* We are first going to try to create the invalid output paths
        through substitutes.  If that doesn't work, we'll build
-       them. */
-    if (settings.useSubstitutes && substitutesAllowed(drv))
+       them.  Always enable substitutes for fixed-output derivations to
+       protect against disappearing files and in-place modifications on
+       upstream sites.  */
+    if ((fixedOutput || settings.useSubstitutes) && substitutesAllowed(drv))
         foreach (PathSet::iterator, i, invalidOutputs)
             addWaitee(worker.makeSubstitutionGoal(*i, buildMode == bmRepair));
 

This solves all our problems and makes download-nar.scm useless.

As an added bonus, it provides a improves the UI since we now always
see:

--8<---------------cut here---------------start------------->8---
0.1 MB will be downloaded:
   /gnu/store/plx9848n6waj6zghn3d54ybx8ihcn23k-guile-git-0.0-4.951a32c-checkout
--8<---------------cut here---------------end--------------->8---

… instead of:

--8<---------------cut here---------------start------------->8---
The following derivation will be built:
   
/gnu/store/y86rlb6pdm35im7q02y6479ca84zwylz-guile-git-000.0-4.951a32c-checkout.drv
--8<---------------cut here---------------end--------------->8---

The downside is that it still requires one to authorize the server’s
key, although it’s in theory unnecessary since it’s content addressed.
I’m not sure how to solve that because ‘guix substitute’ doesn’t know
that it’s substituting a fixed-output derivation.  I suppose we’d need
to modify the “protocol” between guix-daemon and ‘guix substitute’.

Thoughts?

Ludo’.