Should make handle non-blocking jobserver and use socket pair instead?

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Should make handle non-blocking jobserver and use socket pair instead?

Juha-Matti Tilli
Hello,

(Please CC me, I’m not on the list)

I’m creating a new build system that is supposed to solve many of the
issues of standard make. Most programmers who create alternative build
systems (SCons, Rake, Shake) do so by selecting an alternative
programming language (Python, Ruby, Haskell), whereas I’m doing it in
C. Most programmers also create the system by incompatibility and
making it wildly different from make, whereas I’m keeping most of the
features in make that are actually good. That includes the jobserver.

My build system can fork recursive sub-makes, and has support for the
jobserver. While creating the support for jobserver, I noticed a
problem in GNU make handling of jobserver.

GNU make expects the jobserver to be blocking, whereas a program based
on an event-loop architecture should really be using non-blocking file
descriptors.

If the jobserver is set to be non-blocking (I’m creating the jobserver
in my build system and passing it to GNU make), GNU make gives this
error:

make: *** read jobs pipe: Resource temporarily unavailable.  Stop.
make: *** Waiting for unfinished jobs....

I’m wondering whether this error exit() is the right thing to do. In
the programming world, there is a lot of variability and different
choices. Some prefer C, some C++, some Rust. Similarly, some may
prefer GNU make and some may prefer alternative build systems. In my
opinion, GNU make, including parallel make, should attempt to be as
compatible as possible with all other build systems, because not every
programmer wants to use blocking file descriptors. The jobserver is
created by the top-level build system always, and it may be something
else than GNU make. Thus, I'm proposing a "be liberal in what you
accept; be conservative in what you do" approach. Now GNU make doesn't
accept a non-blocking jobserver.

Due to the reasons given in https://cr.yp.to/unix/nonblock.html I
cannot turn on O_NONBLOCK even after dup() because the flag hits the
entire ofile, not just the fd, and because GNU make does not work with
a non-blocking jobserver. I see two solutions:

1. Create a socket pair instead of a pipe so that I can use
MSG_DONTWAIT on a blocking socket

2. Use a select/poll + read pair, and immediately before the read,
turn on a 10 millisecond interval timer just in case some sub-make won
the race to acquire the token

Both of these solutions seem to work, and for compatibility with old
GNU make variants, I really have to use either solution for the next
10 years or so. In case the jobserver was created by make and not my
tool, the MSG_DONTWAIT doesn’t work, so in that case I have to revert
to the second solution. My current approach is to try MSG_DONTWAIT
first and in case of ENOTSOCK use the interval timer with read().

However, I think the issue should be fixed in GNU make too.
Specifically, I think GNU make should read(), and if it gives EAGAIN,
do a select()/poll() and read() again, iterating as long as read()
returns EAGAIN. Of course, some systems may not have a functioning
select()/poll(), so the special EAGAIN handling would apply only to
those systems that fully support select()/poll().

In fact, I argue that GNU make should too create a SOCK_STREAM socket
pair instead of a pipe, if the socket pair facility is available.
Socket pairs are superior in many aspects, including the possibility
to use MSG_DONTWAIT. I have tested that GNU make works perfectly with
a jobserver that is actually a socket pair instead of a pipe, but then
that jobserver needs to be created externally, as the GNU make
creating the jobserver creates a pipe and not a socket pair. As a
hack, a wrapper program could do it and then call GNU make.

What are your opinions of this? I can prepare a patch / patches for
GNU make if it would be acceptable. Of course I won't be wasting any
time reading GNU make sources if the improvement proposal is rejected.

BR, Juha-Matti

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: Should make handle non-blocking jobserver and use socket pair instead?

Gnu - Make - Bugs mailing list
Juha-Matti Tilli wrote:
> Hello,
>
> (Please CC me, I’m not on the list)

The jobserver was designed using a pipe because that is the least common denominator
for IPC across POSIX systems - it worked across all flavors of Unix in existence in 1991,
and continues to work unchanged to this day with no special maintenance requirements.

One you start deviating from least common denominator, support and maintenance demands
increase.

> I’m creating a new build system that is supposed to solve many of the
> issues of standard make. Most programmers who create alternative build
> systems (SCons, Rake, Shake) do so by selecting an alternative
> programming language (Python, Ruby, Haskell), whereas I’m doing it in
> C. Most programmers also create the system by incompatibility and
> making it wildly different from make, whereas I’m keeping most of the
> features in make that are actually good. That includes the jobserver.
>
> My build system can fork recursive sub-makes, and has support for the
> jobserver. While creating the support for jobserver, I noticed a
> problem in GNU make handling of jobserver.
>
> GNU make expects the jobserver to be blocking, whereas a program based
> on an event-loop architecture should really be using non-blocking file
> descriptors.
>
> If the jobserver is set to be non-blocking (I’m creating the jobserver
> in my build system and passing it to GNU make), GNU make gives this
> error:
>
> make: *** read jobs pipe: Resource temporarily unavailable.  Stop.
> make: *** Waiting for unfinished jobs....
>
> I’m wondering whether this error exit() is the right thing to do. In
> the programming world, there is a lot of variability and different
> choices. Some prefer C, some C++, some Rust. Similarly, some may
> prefer GNU make and some may prefer alternative build systems. In my
> opinion, GNU make, including parallel make, should attempt to be as
> compatible as possible with all other build systems, because not every
> programmer wants to use blocking file descriptors. The jobserver is
> created by the top-level build system always, and it may be something
> else than GNU make. Thus, I'm proposing a "be liberal in what you
> accept; be conservative in what you do" approach. Now GNU make doesn't
> accept a non-blocking jobserver.
>
> Due to the reasons given in https://cr.yp.to/unix/nonblock.html I
> cannot turn on O_NONBLOCK even after dup() because the flag hits the
> entire ofile, not just the fd, and because GNU make does not work with
> a non-blocking jobserver. I see two solutions:
>
> 1. Create a socket pair instead of a pipe so that I can use
> MSG_DONTWAIT on a blocking socket
>
> 2. Use a select/poll + read pair, and immediately before the read,
> turn on a 10 millisecond interval timer just in case some sub-make won
> the race to acquire the token
>
> Both of these solutions seem to work, and for compatibility with old
> GNU make variants, I really have to use either solution for the next
> 10 years or so. In case the jobserver was created by make and not my
> tool, the MSG_DONTWAIT doesn’t work, so in that case I have to revert
> to the second solution. My current approach is to try MSG_DONTWAIT
> first and in case of ENOTSOCK use the interval timer with read().
>
> However, I think the issue should be fixed in GNU make too.
> Specifically, I think GNU make should read(), and if it gives EAGAIN,
> do a select()/poll() and read() again, iterating as long as read()
> returns EAGAIN. Of course, some systems may not have a functioning
> select()/poll(), so the special EAGAIN handling would apply only to
> those systems that fully support select()/poll().
>
> In fact, I argue that GNU make should too create a SOCK_STREAM socket
> pair instead of a pipe, if the socket pair facility is available.
> Socket pairs are superior in many aspects, including the possibility
> to use MSG_DONTWAIT. I have tested that GNU make works perfectly with
> a jobserver that is actually a socket pair instead of a pipe, but then
> that jobserver needs to be created externally, as the GNU make
> creating the jobserver creates a pipe and not a socket pair. As a
> hack, a wrapper program could do it and then call GNU make.
>
> What are your opinions of this? I can prepare a patch / patches for
> GNU make if it would be acceptable. Of course I won't be wasting any
> time reading GNU make sources if the improvement proposal is rejected.

--
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: Should make handle non-blocking jobserver and use socket pair instead?

Juha-Matti Tilli
On Thu, Aug 15, 2019 at 7:04 PM Howard Chu <[hidden email]> wrote:
> The jobserver was designed using a pipe because that is the least common denominator
> for IPC across POSIX systems - it worked across all flavors of Unix in existence in 1991,
> and continues to work unchanged to this day with no special maintenance requirements.
>
> One you start deviating from least common denominator, support and maintenance demands
> increase.

Hi,

I guess that means NACK for the proposal to use a socket pair if such
a facility is available?

But how about the proposal to support non-blocking jobserver file
descriptors, should the descriptors be in the non-blocking mode?

BR, Juha-Matti

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: Should make handle non-blocking jobserver and use socket pair instead?

Paul Smith-20
In reply to this post by Juha-Matti Tilli
On Thu, 2019-08-15 at 17:26 +0300, Juha-Matti Tilli wrote:
> GNU make expects the jobserver to be blocking, whereas a program
> based on an event-loop architecture should really be using non-
> blocking file descriptors.

Well, GNU make invented the jobserver concept, and GNU make is not an
event-loop architecture (at least not in the way you mean).

The jobserver pipe is specifically and intentionally designed to be
blocking, so that when a program needs to get a token it will block in
the OS until there is one available, rather than having to maintain its
own unnecessary sleep loop to wait.

> I see two solutions:
>
> 1. Create a socket pair instead of a pipe so that I can use
> MSG_DONTWAIT on a blocking socket

I'm not interested in rewriting the jobserver to use a socket instead
of a pipe.

> 2. Use a select/poll + read pair, and immediately before the read,
> turn on a 10 millisecond interval timer just in case some sub-make
> won the race to acquire the token

This also seems like a big change that would greatly increase the
complexity of the jobserver.

Also, note that there may be other users of the jobserver.  I don't
know if this ever happened but I had conversations with the GCC and
binutils folks, discussing the possibility of multithreaded compilers
or linkers that would interoperate with GNU make's jobserver feature to
control how many threads they would use.

Might I suggest a third option?  In your program, create the jobserver
as blocking as it is defined to be, and implement another method of
integrating it with your event loop.  For example, you could create a
separate thread that would read the blocking jobserver descriptor and
notify your event loop when a token was available.


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make