filtering job options from MAKEFLAGS, manually

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

filtering job options from MAKEFLAGS, manually

Laszlo Ersek
Hi!

I've got a peculiar problem. There is an outer makefile and an inner
makefile. The inner makefile comes from a project (a git submodule) that
is maintained separately from the outer makefile (the git superproject).

The outer makefile is "make -j"-clean. The inner makefile isn't.

In the outer makefile, we can use .NOTPARALLEL to make sure the inner
make is launched only in isolation, to build the relevant artifacts one
by one. That is, there are no multiple inner "make" processes
"competing" on the subproject.

However, the inner make inherits "-j" and "--jobserver-fds=3,4" from the
outer make via MAKEFLAGS. And, if the job server in the outer make lets
it, the inner make will happily parallelize recipes from the inner
makefile, even just to build a single target (build artifact). The inner
makefile is not suitable for that however.

The GNU make documentation suggests that we should add .NOTPARALLEL to
the inner makefile explicitly.

https://www.gnu.org/software/make/manual/make.html#index-_002eNOTPARALLEL

"Any recursively invoked make command will still run recipes in parallel
(unless its makefile also contains this target)."

It also suggests that MAKEFLAGS can be filtered manually, for options
*other than* "-j" (and some other special options).

https://www.gnu.org/software/make/manual/make.html#Options_002fRecursion


Would it be safe / robust to filter out "-j" and "--jobserver-fds=3,4"
as well, manually? (E.g. by sticking a shell script between the outer
and inner make processes.) In particular, "--jobserver-fds" is not
documented in end-user documentation, apparently, so I get a feeling
this option could change at any time, as an implementation detail of
distributing jobs.

Thank you very much,
Laszlo

_______________________________________________
Help-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-make
Reply | Threaded
Open this post in threaded view
|

Re: filtering job options from MAKEFLAGS, manually

Paul Smith-20
On Wed, 2019-01-23 at 11:24 +0100, Laszlo Ersek wrote:
> Would it be safe / robust to filter out "-j" and "--jobserver-fds=3,4"
> as well, manually? (E.g. by sticking a shell script between the outer
> and inner make processes.)

Why not simply add "-j1" explicitly to the inner-make invocation?  That
will disable the job server.

> In particular, "--jobserver-fds" is not documented in end-user
> documentation, apparently, so I get a feeling this option could
> change at any time, as an implementation detail of distributing jobs.

In fact it DID change, in GNU make 4.2, to be --jobserver-auth.  At the
same time this new option was published in the documentation and made
an official part of the GNU make interface.


_______________________________________________
Help-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-make
Reply | Threaded
Open this post in threaded view
|

Re: filtering job options from MAKEFLAGS, manually

David Boyce-3
And don't forget that many clever things can be done with --exec, e.g.
"--exec .NOTPARALLEL:". But -j1 seems simpler in this case.

On Wed, Jan 23, 2019 at 8:51 AM Paul Smith <[hidden email]> wrote:

> On Wed, 2019-01-23 at 11:24 +0100, Laszlo Ersek wrote:
> > Would it be safe / robust to filter out "-j" and "--jobserver-fds=3,4"
> > as well, manually? (E.g. by sticking a shell script between the outer
> > and inner make processes.)
>
> Why not simply add "-j1" explicitly to the inner-make invocation?  That
> will disable the job server.
>
> > In particular, "--jobserver-fds" is not documented in end-user
> > documentation, apparently, so I get a feeling this option could
> > change at any time, as an implementation detail of distributing jobs.
>
> In fact it DID change, in GNU make 4.2, to be --jobserver-auth.  At the
> same time this new option was published in the documentation and made
> an official part of the GNU make interface.
>
>
> _______________________________________________
> Help-make mailing list
> [hidden email]
> https://lists.gnu.org/mailman/listinfo/help-make
>
_______________________________________________
Help-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-make
Reply | Threaded
Open this post in threaded view
|

Re: filtering job options from MAKEFLAGS, manually

Laszlo Ersek
On 01/23/19 15:45, David Boyce wrote:

> And don't forget that many clever things can be done with --exec, e.g.
> "--exec .NOTPARALLEL:". But -j1 seems simpler in this case.
>
> On Wed, Jan 23, 2019 at 8:51 AM Paul Smith <[hidden email]> wrote:
>
>> On Wed, 2019-01-23 at 11:24 +0100, Laszlo Ersek wrote:
>>> Would it be safe / robust to filter out "-j" and "--jobserver-fds=3,4"
>>> as well, manually? (E.g. by sticking a shell script between the outer
>>> and inner make processes.)
>>
>> Why not simply add "-j1" explicitly to the inner-make invocation?  That
>> will disable the job server.
>>
>>> In particular, "--jobserver-fds" is not documented in end-user
>>> documentation, apparently, so I get a feeling this option could
>>> change at any time, as an implementation detail of distributing jobs.
>>
>> In fact it DID change, in GNU make 4.2, to be --jobserver-auth.  At the
>> same time this new option was published in the documentation and made
>> an official part of the GNU make interface.

Awesome, thank you both. Appending "-j1" to the command line of the
inner make seems a lot simpler than spelling out the pre-requisites of
.NOTPARALLEL.

(In fact it crossed my mind that .NOTPARALLEL could be extended to a
prerequisite-less form, like .SECONDARY: it would then apply to all
targets. But, "-j1" should work fine.)

Thank you!
Laszlo

_______________________________________________
Help-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-make
Reply | Threaded
Open this post in threaded view
|

Re: filtering job options from MAKEFLAGS, manually

David Boyce-3
Yes, -j1 is the right and obvious solution but I think you have
.NOTPARALLEL completely backwards. Here's what the manual says:

If .NOTPARALLEL is mentioned as a target, then this invocation of make will
be run serially, even if the ā€˜-jā€™ option is given ... Any prerequisites on
this target are ignored.

On Wed, Jan 23, 2019 at 5:10 PM Laszlo Ersek <[hidden email]> wrote:

> On 01/23/19 15:45, David Boyce wrote:
> > And don't forget that many clever things can be done with --exec, e.g.
> > "--exec .NOTPARALLEL:". But -j1 seems simpler in this case.
> >
> > On Wed, Jan 23, 2019 at 8:51 AM Paul Smith <[hidden email]> wrote:
> >
> >> On Wed, 2019-01-23 at 11:24 +0100, Laszlo Ersek wrote:
> >>> Would it be safe / robust to filter out "-j" and "--jobserver-fds=3,4"
> >>> as well, manually? (E.g. by sticking a shell script between the outer
> >>> and inner make processes.)
> >>
> >> Why not simply add "-j1" explicitly to the inner-make invocation?  That
> >> will disable the job server.
> >>
> >>> In particular, "--jobserver-fds" is not documented in end-user
> >>> documentation, apparently, so I get a feeling this option could
> >>> change at any time, as an implementation detail of distributing jobs.
> >>
> >> In fact it DID change, in GNU make 4.2, to be --jobserver-auth.  At the
> >> same time this new option was published in the documentation and made
> >> an official part of the GNU make interface.
>
> Awesome, thank you both. Appending "-j1" to the command line of the
> inner make seems a lot simpler than spelling out the pre-requisites of
> .NOTPARALLEL.
>
> (In fact it crossed my mind that .NOTPARALLEL could be extended to a
> prerequisite-less form, like .SECONDARY: it would then apply to all
> targets. But, "-j1" should work fine.)
>
> Thank you!
> Laszlo
>
_______________________________________________
Help-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-make
Reply | Threaded
Open this post in threaded view
|

Re: filtering job options from MAKEFLAGS, manually

Laszlo Ersek
On 01/24/19 02:54, David Boyce wrote:
> Yes, -j1 is the right and obvious solution but I think you have
> .NOTPARALLEL completely backwards. Here's what the manual says:
>
> If .NOTPARALLEL is mentioned as a target, then this invocation of make will
> be run serially, even if the ā€˜-jā€™ option is given ... Any prerequisites on
> this target are ignored.

My bad, you are right!
Thanks!
Laszlo

> On Wed, Jan 23, 2019 at 5:10 PM Laszlo Ersek <[hidden email]> wrote:
>
>> On 01/23/19 15:45, David Boyce wrote:
>>> And don't forget that many clever things can be done with --exec, e.g.
>>> "--exec .NOTPARALLEL:". But -j1 seems simpler in this case.
>>>
>>> On Wed, Jan 23, 2019 at 8:51 AM Paul Smith <[hidden email]> wrote:
>>>
>>>> On Wed, 2019-01-23 at 11:24 +0100, Laszlo Ersek wrote:
>>>>> Would it be safe / robust to filter out "-j" and "--jobserver-fds=3,4"
>>>>> as well, manually? (E.g. by sticking a shell script between the outer
>>>>> and inner make processes.)
>>>>
>>>> Why not simply add "-j1" explicitly to the inner-make invocation?  That
>>>> will disable the job server.
>>>>
>>>>> In particular, "--jobserver-fds" is not documented in end-user
>>>>> documentation, apparently, so I get a feeling this option could
>>>>> change at any time, as an implementation detail of distributing jobs.
>>>>
>>>> In fact it DID change, in GNU make 4.2, to be --jobserver-auth.  At the
>>>> same time this new option was published in the documentation and made
>>>> an official part of the GNU make interface.
>>
>> Awesome, thank you both. Appending "-j1" to the command line of the
>> inner make seems a lot simpler than spelling out the pre-requisites of
>> .NOTPARALLEL.
>>
>> (In fact it crossed my mind that .NOTPARALLEL could be extended to a
>> prerequisite-less form, like .SECONDARY: it would then apply to all
>> targets. But, "-j1" should work fine.)
>>
>> Thank you!
>> Laszlo
>>
>


_______________________________________________
Help-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-make