Re: "make -jN" requires mechanical changes to a Makefile [SOLVED]

[Bruno Haible]

Continuing this thread from May 2019
<https://lists.gnu.org/archive/html/bug-make/2019-05/msg00022.html>:
The problem was:

  How can a rule that generates multiple files be formulated so
  that it works with parallel make?

For example, a rule that invokes bison, or a rule that invokes
a different Makefile. For simplicity, here, use a rule that
creates 4 files copy1, copy2, copy3, copy4.

===========================================
all : copy1 copy2 copy3 copy4

copy1 copy2 copy3 copy4: Makefile
        install -c -m 644 Makefile copy1
        install -c -m 644 Makefile copy2
        install -c -m 644 Makefile copy3
        install -c -m 644 Makefile copy4
===========================================

Unfortunately, with "make -j8", it invokes the rule multiple times.

It is possible to change this Makefile so that
  (A) "rm -f copy?; make" executes the rule once.
  (B) "rm -f copy?; make -j8" executes the rule once as well.
  (C) After "make", another "make" just prints "Nothing to be done for 'all'."
  (D) After removing one of the files copy?, "make" executes the rule once.
      (This covers also the case of pressing Ctrl-C during "make", then
      doing "make" again.)
  (E) After replacing one of the files copy? with a file that is older than
      Makefile, "make" executes the rule once.

There are three possibilities:


(I) Assuming GNU make >= 4.3, the "Grouped explicit target" syntax does it.
Thanks to Kaz Kylheku and Paul Smith for having added this.

===========================================
all : copy1 copy2 copy3 copy4

copy1 copy2 copy3 copy4 &: Makefile
        install -c -m 644 Makefile copy1
        install -c -m 644 Makefile copy2
        install -c -m 644 Makefile copy3
        install -c -m 644 Makefile copy4
===========================================

But it will take a number of years until we can assume that all 'make'
programs that we care about support this syntax.


(II) It is possible to turn off parallel make. This does it:

===========================================
all : copy1 copy2 copy3 copy4

copy1 copy2 copy3 copy4 : Makefile
        install -c -m 644 Makefile copy1
        install -c -m 644 Makefile copy2
        install -c -m 644 Makefile copy3
        install -c -m 644 Makefile copy4

# This Makefile contains rules which don't work with parallel make.
# So, turn off parallel execution in this Makefile.
.NOTPARALLEL:
===========================================

But some people who want to minimize wall-clock execution time of
their builds may not like it.


(III) Use portable make syntax and still allow parallel make.

This is a bit harder. My solution is to first analyze in which order
the rule will generate the various files and thus what the expected
timestamp order is. In this case, it is:
  Makefile <= copy1 <= copy2 <= copy3 <= copy4

First some attempts that don't work:

===========================================
all : copy1 copy2 copy3 copy4

copy1: Makefile
        install -c -m 644 Makefile copy1
        install -c -m 644 Makefile copy2
        install -c -m 644 Makefile copy3
        install -c -m 644 Makefile copy4

copy2 copy3 copy4: copy1
===========================================

Fails (E).

===========================================
all : copy1 copy2 copy3 copy4

copy4: Makefile
        install -c -m 644 Makefile copy1
        install -c -m 644 Makefile copy2
        install -c -m 644 Makefile copy3
        install -c -m 644 Makefile copy4

copy1 copy2 copy3: copy4
===========================================

Fails (E) as well.

===========================================
all : copy1 copy2 copy3 copy4

copy4: Makefile
        install -c -m 644 Makefile copy1
        install -c -m 644 Makefile copy2
        install -c -m 644 Makefile copy3
        install -c -m 644 Makefile copy4

copy1: copy4
        test -f copy1 || { rm -f copy4; $(MAKE) copy4; }
copy2: copy4
        test -f copy2 || { rm -f copy4; $(MAKE) copy4; }
copy3: copy4
        test -f copy3 || { rm -f copy4; $(MAKE) copy4; }
===========================================

Fails (E) for copy1, ..., copy3.
And fails (C), i.e. clutters the "make" output.

===========================================
all : copy1 copy2 copy3 copy4

copy1: Makefile
        { test -f copy1 && test ! copy1 -ot Makefile; } || { rm -f copy4; 
$(MAKE) copies; }
copy2: copy1
        { test -f copy2 && test ! copy2 -ot copy1; } || { rm -f copy4; $(MAKE) 
copies; }
copy3: copy2
        { test -f copy3 && test ! copy3 -ot copy2; } || { rm -f copy4; $(MAKE) 
copies; }
copy4: copy3
        { test -f copy4 && test ! copy4 -ot copy3; } || { rm -f copy4; $(MAKE) 
copies; }

copies:
        install -c -m 644 Makefile copy1
        install -c -m 644 Makefile copy2
        install -c -m 644 Makefile copy3
        install -c -m 644 Makefile copy4
.PHONY: copies
===========================================

This solution fulfils all the requirements.

Can we use 'test FILE1 -ot FILE2'?
- POSIX 'test' does not support this option.
  See <https://pubs.opengroup.org/onlinepubs/9699919799/utilities/test.html>
- But all test programs in all current operating systems (Linux, macOS,
  FreeBSD, NetBSD, OpenBSD, AIX, Solaris, Haiku, Minix, Cygwin, even Busybox)
  support it. Except for /bin/sh on Solaris 10 — but in packages that use the
  GNU Build System, Autoconf sets SHELL = @CONFIG_SHELL@ = /bin/bash on such
  systems.

Bruno