[bug#74394] [PATCH 0/2] Skip slow tests by default and run 'check' in Git pre-push hook.

[Ludovic Courtès]

Hi Maxim,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:

[...]

>> I agree with the goal, of course, but not with the method: even without
>> expensive tests, “make check” is going to take maybe 5–10 minutes, and
>> having that happen when you run “git push” can be a terrible development
>> experience (especially since the developer most likely either already
>> ran the test suite or part of it right before, or pushes package changes
>> that have infinitely small probability of breaking “make check”).
>
> As I wrote, 'make check' with this change takes about 1 minute on my
> machine;

Right now, without your patch, we have:

--8<---------------cut here---------------start------------->8---
$ wget -qO- $(guix build --log-file guix --no-grafts)|gunzip |grep "\`check'"
starting phase `check'
phase `check' succeeded after 2049.2 seconds
--8<---------------cut here---------------end--------------->8---

More than 30mn on the fast machines of the build farm, and with some of
the expensive tests already skipped (those that require network access:
time-machine, pack -RR, etc.).

This patch is not dividing wall-clock time by 30, is it?

>> This variable itself may still be useful though (I’d call it
>> ‘RUN_EXPENSIVE_TESTS’ or something like that—that’s the name used in
>> Coreutils—, “expensive” being the key word).  I would also keep it on by
>> default.
>
> One of the tests that was unbearably long when I measured was the
> time-machine test.  It took about 20 minutes to fetch the git repository
> with guile-git and run the test (which does extra work compared to the
> CLI like validating each object).  I don't think we want this kind of
> experience by default (because that'd probably convince people that
> running the test suite often is not a reasonable thing to do).  The
> other tests were more reasonable, with the longer ones in the 2-3
> minutes range on my machine, IIRC.  Perhaps we could have this 20 minute
> outlier skipped by default, maybe with a RUN_PROHIBITIVE_TESTS flag that
> would default to 0 (false).

Yeah okay, maybe we should skip them by default, and maybe we can find a
way to ensure developers do run them periodically.

> A long time ago I had read a blog post that argued that unit tests
> should be small and fast [0],

I actually agree.  :-)

Thanks!

Ludo’.