Re: [bug #58734] gmake does not check for the existence of a file before complaining it is missing

[Paul Smith]

On Thu, 2020-07-09 at 16:00 +0000, Edward Welbourne wrote:
> By the sounds of it, you have four files that are all created by a
> single rule, which isn't a scenario make is good at.  So work-arounds
> are tricky:

If you want to use parallel builds you must find a workaround for this
issue, or you'll run the same recipe multiple times.  That has nothing
to do with the directory cache: that's because in POSIX there is no way
to explain to make that a single invocation of a recipe will generate
multiple files as output.

That lack doesn't bother non-parallel make, and since POSIX doesn't
define parallelism if you're using pure POSIX (no parallelism) it works
fine to just say:

    targ1 targ2 targ3: prereq1 prereq2 prereq3
            recipe

But if you use -j then you can't use this simple idiom and you'll have
to do something more sophisticated.

If you want your makefile to be POSIX-compatible AND work with -j AND
you have recipes that generate multiple outputs, you have only one real
solution I'm aware of: use a sentinel file like this:

   targ1 targ2 targ3: .sentinel ;

   .sentinel: prereq1 prereq2 prereq3
           recipe
           @touch .sentinel

As a happy side-effect, if you implement this fix for multiple commands
being run then you will ALSO have solved the directory cache problem.

The directory cache problem happens when some recipe does some work
behind make's back, with results that make needs to know about.  If you
explicitly state what your recipes are _really_ doing in your makefile,
you won't have directory cache problems.

> persuade the maintainer of gmake to introduce a significant
> pessimisation.

Well, in fact the directory cache IS a problem, and I've long
recognized that.  The many emails to this list and elsewhere make it
clear.

However, it is ALSO a big performance gain, and furthermore it only
impacts makefiles which don't provide full dependency information and
so are, at least, sub-optimal (still POSIX compliant, of course).

So I would like to fix things so it still provides benefits to well-
written makefiles but also doesn't have these downsides for other
makefiles.  I have considered many options but so far none have gotten
past concept stage:

* Add an option to simply disable it (maybe adding .POSIX: does that).
  Obviously a very blunt hammer.

* Add a facility to dirty the cache; say, every time make runs a
  process (either a recipe or $(shell ...)) it removes cached info for
  all directories.  This would work but also would potentially lose a
  lot of performance, depending on the build (basically the more
  updating your build does the less the directory cache helps).

* Figure out a way to detect whether a directory has been updated, and
  dirty its cache if so.  This is probably the best option, and for
  POSIX systems this is simple: you just look at the modification time
  of the directory.  But for Windows systems (and VMS) it's not nearly 
  so simple.  So we'd need to figure out how to manage that.

For best effect we can combine #2 and #3, and only check for updated
directories if we've invoked some process since the last time.