bug#29335: 'guix publish' workers occasionally crash

[Ludovic Courtès]

Hello,

On berlin I’ve noticed that the ‘guix publish’ workers would
occasionally stop working: the main thread would keep replying to HTTP
requests, but the worker threads would no longer do anything, and would
leave behind them a bunch of .tmp files in /var/cache/guix/publish.

I captured the output of ‘guix publish’ (guix-0.13.0-8.357ab93) and the
only clue I have is this:

--8<---------------cut here---------------start------------->8---
GET /6kl9ydqmgklcqhxswg6v5isq5n1ih5gp.narinfo
In guix/workers.scm:
     74:9  2 (_)
    78:32  1 (_ srfi-34 #<condition &nix-connection-error [file: "/v…>)
In unknown file:
           0 (make-stack #t)
ERROR: In procedure make-stack:
ERROR: Throw to key `srfi-34' with args `(#<condition &nix-connection-error 
[file: "/var/guix/daemon-socket/socket" errno: 9] 3ba2ea0>)'.
GET /fgiih42mg2sr82mbmzf56grvrf021im6.narinfo
--8<---------------cut here---------------end--------------->8---

… where 9 is EBADF.

The open files are:

--8<---------------cut here---------------start------------->8---
address@hidden ~$ sudo ls -l /proc/4105/fd
total 0
l-wx------ 1 root root 64 Nov 17 04:00 0 -> /dev/null
l-wx------ 1 root root 64 Nov 17 04:00 1 -> /home/ludo/nohup.out
lr-x------ 1 root root 64 Nov 17 04:00 10 -> pipe:[9911558]
l-wx------ 1 root root 64 Nov 17 04:00 11 -> pipe:[9911558]
lr-x------ 1 root root 64 Nov 17 04:00 12 -> pipe:[9900923]
l-wx------ 1 root root 64 Nov 17 04:00 13 -> pipe:[9900923]
lrwx------ 1 root root 64 Nov 17 04:00 14 -> socket:[9911560]
lrwx------ 1 root root 64 Nov 17 04:00 15 -> socket:[9911563]
lr-x------ 1 root root 64 Nov 17 04:00 16 -> pipe:[9912323]
l-wx------ 1 root root 64 Nov 17 04:00 17 -> pipe:[9912323]
lr-x------ 1 root root 64 Nov 17 04:00 18 -> pipe:[9909649]
l-wx------ 1 root root 64 Nov 17 04:00 19 -> pipe:[9909649]
l-wx------ 1 root root 64 Nov 17 04:00 2 -> /home/ludo/nohup.out
lr-x------ 1 root root 64 Nov 17 04:00 20 -> pipe:[9908165]
l-wx------ 1 root root 64 Nov 17 04:00 21 -> pipe:[9908165]
lr-x------ 1 root root 64 Nov 17 04:00 22 -> pipe:[9899432]
l-wx------ 1 root root 64 Nov 17 04:00 23 -> pipe:[9899432]
lr-x------ 1 root root 64 Nov 17 04:00 24 -> pipe:[9911564]
l-wx------ 1 root root 64 Nov 17 04:00 25 -> pipe:[9911564]
lr-x------ 1 root root 64 Nov 17 04:00 26 -> pipe:[9908166]
l-wx------ 1 root root 64 Nov 17 04:00 27 -> pipe:[9908166]
l-wx------ 1 root root 64 Nov 17 04:00 28 -> 
/var/cache/guix/publish/gzip/11vik22bwzzvwksd1l08frhcm3v3hhmm-dealii-8.5.1.nar.tmp
l-wx------ 1 root root 64 Nov 17 04:00 29 -> 
/var/cache/guix/publish/gzip/k90h1bkmsiq8qxrhk31l5c8maiqmy5k0-zathura-pdf-mupdf-0.3.1.nar.tmp
lr-x------ 1 root root 64 Nov 17 04:00 3 -> pipe:[9902378]
lr-x------ 1 root root 64 Nov 17 04:00 30 -> 
/gnu/store/2xp8n6k6mbfgi8972lj0vvk52gfx4576-linux-libre-4.9.61/lib/modules/4.9.61-gnu/kernel/drivers/net/ethernet/intel/i40evf
lr-x------ 1 root root 64 Nov 17 04:00 31 -> 
/gnu/store/k90h1bkmsiq8qxrhk31l5c8maiqmy5k0-zathura-pdf-mupdf-0.3.1/share
l-wx------ 1 root root 64 Nov 17 04:00 32 -> 
/var/cache/guix/publish/gzip/xa6bwlgm82xwfg9icl4d9bs2bhj8cb1a-perl-safe-isa-1.000008.nar.tmp
l-wx------ 1 root root 64 Nov 17 04:00 33 -> 
/var/cache/guix/publish/gzip/2xp8n6k6mbfgi8972lj0vvk52gfx4576-linux-libre-4.9.61.nar.tmp
lr-x------ 1 root root 64 Nov 17 04:00 34 -> 
/gnu/store/11vik22bwzzvwksd1l08frhcm3v3hhmm-dealii-8.5.1/share
l-wx------ 1 root root 64 Nov 17 04:00 4 -> pipe:[9902378]
lr-x------ 1 root root 64 Nov 17 04:00 5 -> pipe:[9902379]
l-wx------ 1 root root 64 Nov 17 04:00 6 -> pipe:[9902379]
lr-x------ 1 root root 64 Nov 17 04:00 7 -> 
/gnu/store/sk1w5whws8xhvb03sm2h44f9fm6c2gkc-guix-0.13.0-8.357ab93/bin/.guix-real
lr-x------ 1 root root 64 Nov 17 04:00 8 -> pipe:[9910685]
l-wx------ 1 root root 64 Nov 17 04:00 9 -> pipe:[9910685]
--8<---------------cut here---------------end--------------->8---

Ludo’.