Passing jobserver file descriptors to all children

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Passing jobserver file descriptors to all children

R. Diez
Hi all:

I am trying to use GCC's -flto=jobserver flag, but I cannot easily add the '+' character in front of the makefile recipe with
Autoconf/Automake. See here for more details:

https://gcc.gnu.org/ml/gcc-help/2020-02/msg00069.html

The jobserver used to be a GNU Make-specific feature, but other things like GCC's LTO are starting to use it. GCC may use it during object
file compilation in the future too, see this project:

https://gcc.gnu.org/wiki/ParallelGcc

In fact, any stage during a build process may be parallelised and start using the jobserver. With the advent of multicore CPUs, this will
probably increase with time.

The requirement to prepend a '+' to the recipe or to use $(MAKE) then becomes a nuisance.

Is there a way to tell GNU Make to always pass those jobserver file descriptors to all children?

Thanks in advance,
   rdiez

Reply | Threaded
Open this post in threaded view
|

Re: Passing jobserver file descriptors to all children

Paul Smith-20
On Wed, 2020-02-19 at 09:14 +0100, R. Diez wrote:
> Is there a way to tell GNU Make to always pass those jobserver file
> descriptors to all children?

There is nothing like that at the moment.

Originally make did pass these file descriptors to all children, but there
were bugs filed because some tools invoked by make expected to have
specific file descriptors available; having make pass down open FDs caused
them to fail.

I wouldn't be adverse to adding a way to allow this.

There is also an open bug asking that we do not pass along the jobserver
options at all if it's disabled.

Another option is to switch to a named pipe instead of an old-school pipe.
In this case the thing we'd pass along to sub-makes would be the path of
the named pipe, so each tool that wanted to use the jobserver could open it
directly and we wouldn't need to pass along open file descriptors.  There
are some downsides to that, and we'd need to check the portability
situation with named pipes.

Are there any docs around the GCC usage of the jobserver?  I have a few
things I've been considering for jobserver enhancements; for example:

 * As above, switching to named pipes.  So, any code using the jobserver
   should be able to detect when the value of the jobserver argument is
   different than expected (not two integers), and handle it gracefully
   (ignore it, maybe with a warning or something).

 * Adding a new token type, maybe "-" or something, that was written¬†to the
   jobserver if a command failed.  This would let us "fail faster".  This
   makes it very important that any user of the jobserver preserve the
   exact token they got and put it back into the pipe when done, rather
   than assuming it's always "+".


Reply | Threaded
Open this post in threaded view
|

Re: Passing jobserver file descriptors to all children

R. Diez

 > [...]
> Originally make did pass these file descriptors to all children, but there
> were bugs filed because some tools invoked by make expected to have
> specific file descriptors available; having make pass down open FDs caused
> them to fail.

Relying on specific file descriptor numbers is asking for trouble. I do not think that allowing for such brittle scripts justifies having to add a '+'
to the make recipe. It is easy to forget. That is probably the reason why the heuristic $(MAKE) was added to assume you wanted '+' for that recipe.
Which means that, if you are using a single shell for all the recipe (automatically or manually joining all commands with &&), then all other child
processes will inherit the jobserver file descriptors too. Who is going to think about that side-effect with inherited jobserver file descriptors when
optimising recipes to fork less shells?

Furthermore, such '+' distinction force Makefile generators to become more complex. Or to just ignore it, like Autoconf/Automake seems to do. So no
jobserver descriptors when linking with gcc then.

I guess it is too late to reverse such decision. But that should not apply to a new "--always-pass-jobserver-descriptors" flag. If you have
compatibility issues with old scripts that expect particular file descriptors, then do not use that flag and lose some parallel performance.

 From the manual page, Bash seems to use descriptors above 10 internally, as only the low 10 seem guarantee stability for scripts:

"Redirections using file descriptors greater than 9 should be used with care, as they may conflict with file descriptors the shell uses internally."

Maybe if GNU Make moved those jobserver descriptors to higher numbers, like Bash does internally, there would be less problems in this area.


> Are there any docs around the GCC usage of the jobserver?

Unfortunately, there does not seem to be much documentation about how GCC uses the jobserver. I asked on the mailing list about tracing jobserver
usage, but I got no answer yet.

I am not sure that looking at GCC would help much. Like I said, parallelising build tools will probably increase in the future, and that could impact
any build step on any language. In fact, with the advent of multicore CPUs, I am surprised that this has not developed faster.

Without a jobserver to limit the parallel load, automatic builders (like continuous integration build servers) may easily overload. For example, GCC's
LTO uses loads of RAM. If you do not place some limit, you may blow all RAM. In fact, I already commented on this some years ago, and that was not
about LTO:

Subject: Thoughts on limiting the parallel build load, suggestion for a new "-j auto" option

https://lists.gnu.org/archive/html/bug-make/2012-02/msg00031.html

Using a jobserver helps directly or indirectly limit the machine load. That is why I believe that making the jobserver available all over the place is
the way to go. In fact, sometimes the jobserver may be the only reason GNU Make is running for (with no actual useful makefile).

I am not sure about using named pipes. Then you have to think about security, and possibly cleaning them up at the end. Inheriting file descriptors
actually seems like a simple, good concept.

Best regards,
   rdiez