Re: Should make handle non-blocking jobserver and use socket pair instead?

[Paul Smith]

On Thu, 2019-08-15 at 17:26 +0300, Juha-Matti Tilli wrote:
> GNU make expects the jobserver to be blocking, whereas a program
> based on an event-loop architecture should really be using non-
> blocking file descriptors.

Well, GNU make invented the jobserver concept, and GNU make is not an
event-loop architecture (at least not in the way you mean).

The jobserver pipe is specifically and intentionally designed to be
blocking, so that when a program needs to get a token it will block in
the OS until there is one available, rather than having to maintain its
own unnecessary sleep loop to wait.

> I see two solutions:
> 
> 1. Create a socket pair instead of a pipe so that I can use
> MSG_DONTWAIT on a blocking socket

I'm not interested in rewriting the jobserver to use a socket instead
of a pipe.

> 2. Use a select/poll + read pair, and immediately before the read,
> turn on a 10 millisecond interval timer just in case some sub-make
> won the race to acquire the token

This also seems like a big change that would greatly increase the
complexity of the jobserver.

Also, note that there may be other users of the jobserver.  I don't
know if this ever happened but I had conversations with the GCC and
binutils folks, discussing the possibility of multithreaded compilers
or linkers that would interoperate with GNU make's jobserver feature to
control how many threads they would use.

Might I suggest a third option?  In your program, create the jobserver
as blocking as it is defined to be, and implement another method of
integrating it with your event loop.  For example, you could create a
separate thread that would read the blocking jobserver descriptor and
notify your event loop when a token was available.