bug#30290: guix-daemon slows to a crawl when a substitute server is offline

[Maxim Cournoyer]

When a substitute server used by guix-daemon is offline, the daemon will
keep attempting to connect to it, even when it shouldn't need any data
(ran 'sudo guix system reconfigure my-config.scm' multiple times in a
row.

With the disconnected server (bayfront in my case), that command would
take close to 8 minutes, with many system calls like:

--8<---------------cut here---------------start------------->8---
connect(14, {sa_family=AF_INET, sin_port=htons(443), 
sin_addr=inet_addr("141.255.128.56")}, 16) = -1 EINPROGRESS
--8<---------------cut here---------------end--------------->8---

which wasted 5 seconds each time.

After removing this server from my substitute servers list, the same
operation (system reconfigure) is 8 times faster (1 minute).

Suggestion: the daemon should stop trying to use the offline substitute
server after trying for X times, and print a warning about it.

Maxim