"make -jN" requires mechanical changes to a Makefile

classic Classic list List threaded Threaded
23 messages Options
12
Reply | Threaded
Open this post in threaded view
|

"make -jN" requires mechanical changes to a Makefile

Bruno Haible
Hi,

The GNU standards [1] say:
  "Try to make the build and installation targets, at least (and all their
   subtargets) work correctly with a parallel make."

But supporting parallel requires, in some cases, mechanical changes to a
Makefile. How about if GNU make was improved to not require me to make these
changes?

Namely, consider this Makefile:
===================================================
all : copy1 copy2 copy3 copy4

copy1 copy2 copy3 copy4: Makefile
        install -c -m 644 Makefile copy1
        install -c -m 644 Makefile copy2
        install -c -m 644 Makefile copy3
        install -c -m 644 Makefile copy4
===================================================

Observe:
* "rm -f copy?; make" works fine.
* "rm -f copy?; make -j8" occasionally fails:
$ rm -f copy? ; make -j8
install -c -m 644 Makefile copy1
install -c -m 644 Makefile copy1
install -c -m 644 Makefile copy1
install -c -m 644 Makefile copy1
install -c -m 644 Makefile copy2
install -c -m 644 Makefile copy2
install -c -m 644 Makefile copy2
install -c -m 644 Makefile copy2
install: cannot change permissions of 'copy2': No such file or directory
install -c -m 644 Makefile copy3
Makefile:4: recipe for target 'copy2' failed
make: *** [copy2] Error 1
make: *** Waiting for unfinished jobs....
install -c -m 644 Makefile copy3
install -c -m 644 Makefile copy3
install -c -m 644 Makefile copy4
install -c -m 644 Makefile copy4
install -c -m 644 Makefile copy4

The workaround is to introduce an intermediate target:

===================================================
all : copy1 copy2 copy3 copy4

copy1 copy2 copy3 copy4: install-copies
.PHONY: install-copies
install-copies: Makefile
        install -c -m 644 Makefile copy1
        install -c -m 644 Makefile copy2
        install -c -m 644 Makefile copy3
        install -c -m 644 Makefile copy4
===================================================

This is tedious, mechanical work. Couldn't GNU make implement the workaround
by itself? Namely, when during a parallel make, it encounters a rule

target1 ... targetN : dependencies
        STATEMENTS

and none of the statements depends on the precise target ($@ or similar),
it should transform that to the form

target1 ... targetN : intermediate_target
.PHONY: intermediate_target
intermediate_target: dependencies
        STATEMENTS

internally.

$ make --version
GNU Make 4.1

Best regards,

              Bruno

[1] https://www.gnu.org/prep/standards/html_node/Makefile-Basics.html


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: "make -jN" requires mechanical changes to a Makefile

Paul Smith-20
On Fri, 2019-05-10 at 22:49 +0200, Bruno Haible wrote:

> But supporting parallel requires, in some cases, mechanical changes to a
> Makefile. How about if GNU make was improved to not require me to make these
> changes?
>
> Namely, consider this Makefile:
> ===================================================
> all : copy1 copy2 copy3 copy4
>
> copy1 copy2 copy3 copy4: Makefile
>         install -c -m 644 Makefile copy1
>         install -c -m 644 Makefile copy2
>         install -c -m 644 Makefile copy3
>         install -c -m 644 Makefile copy4
> ===================================================

Well, IMO this makefile is just wrong.  For example, in the above
makefile if you run "make copy3" it will install all the files, which
is incorrect.

This makefile should be written correctly, as:

  all : copy1 copy2 copy3 copy4

  copy1: Makefile
          install -c -m 644 Makefile copy1
  copy2: Makefile
          install -c -m 644 Makefile copy2
  copy3: Makefile
          install -c -m 644 Makefile copy3
  copy4: Makefile
          install -c -m 644 Makefile copy4

then it will work properly in both parallel and non-parallel modes, and
it will do the right thing for invocations like "make copy3".

I understand that this requires changes to the makefile... but
incorrect code needs to be changed to be correct, rather than having
the compiler try to fix it for you.  At least, that's my current
thinking about this.

Also in general I really don't like to try to parse recipes and modify
behavior depending on what is found.  It leads to a lot of unexpected
consequences and confusion.


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: "make -jN" requires mechanical changes to a Makefile

Bruno Haible
Hi Paul,

> This makefile should be written correctly, as:
>
>   all : copy1 copy2 copy3 copy4
>
>   copy1: Makefile
>           install -c -m 644 Makefile copy1
>   copy2: Makefile
>           install -c -m 644 Makefile copy2
>   copy3: Makefile
>           install -c -m 644 Makefile copy3
>   copy4: Makefile
>           install -c -m 644 Makefile copy4

The Makefile was simplified. In the real-world case [1], the STATEMENTS
is a recursive invocation of another Makefile, which is generated by
Automake, and for which I cannot create a separate targets
  install-textstyle.h
  install-textstyle_stdbool.h
  install-textstyle_version.h
  install-textstyle_woe32dll.h
(because Automake just does not offer this).

Another real-world example (where a parallel make problem surely
exists, but wasn't reported so far) is this rule from
gettext/gettext-tools/src/Makefile.am:

po-gram-gen.c po-gram-gen.h: po-gram-gen.y
        $(AM_V_GEN)$(SHELL) $(YLWRAP) $(srcdir)/po-gram-gen.y \
                                      y.tab.c po-gram-gen.c \
                                      y.tab.h po-gram-gen.h \
                                      y.output po-gram-gen.output \
                                      -- $(YACC) $(YFLAGS) $(AM_YFLAGS) \
        && sed -e 's|".*/po-gram-gen.y"|"po-gram-gen.y"|' < po-gram-gen.c > po-gram-gen.c-tmp \
        && rm -f po-gram-gen.c \
        && mv po-gram-gen.c-tmp $(srcdir)/po-gram-gen.c \
        && { test '$(srcdir)' = . || mv po-gram-gen.h $(srcdir)/po-gram-gen.h; }

It's the same type of problem, and the same mechanical change is needed
for proper support of parallel make.

> I really don't like to try to parse recipes and modify
> behavior depending on what is found.  It leads to a lot of unexpected
> consequences and confusion.

In the current state, supporting parallel make requires extra work
for the maintainer.

Or would you recommend that I add this snippet to the top-level
Makefile of all my projects?

# This package does not support parallel make.
# So, turn off parallel execution (at least in GNU make >= 4.0).
GNUMAKEFLAGS = -j1

Bruno

[1] https://lists.gnu.org/archive/html/bug-gettext/2019-05/msg00084.html


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: "make -jN" requires mechanical changes to a Makefile

Henrik Carlqvist-2
> In the current state, supporting parallel make requires extra work
> for the maintainer.
>
> Or would you recommend that I add this snippet to the top-level
> Makefile of all my projects?
>
> # This package does not support parallel make.
> # So, turn off parallel execution (at least in GNU make >= 4.0).
> GNUMAKEFLAGS = -j1

If you really prefer to write rules which generates more than one target
the "right" way to avoid parallel make would be to add the .NOTPARALLEL
target in the Makefile.

regards Henrik

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: "make -jN" requires mechanical changes to a Makefile

Bruno Haible
Henrik Carlqvist wrote:
> If you really prefer to write rules which generates more than one target
> the "right" way to avoid parallel make would be to add the .NOTPARALLEL
> target in the Makefile.

This way allows to turn off parallel make for a single Makefile.
Indeed, this might be a better compromise, when only few Makefiles
have problems with parallel make.

Whereas the snippet I showed turns it off for an entire package (when placed
in the top-level Makefile).

Thanks for the suggestion.

Bruno


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: "make -jN" requires mechanical changes to a Makefile

Bruno Haible
In reply to this post by Bruno Haible
Hi,

I wrote:

> The workaround is to introduce an intermediate target:
>
> ===================================================
> all : copy1 copy2 copy3 copy4
>
> copy1 copy2 copy3 copy4: install-copies
> .PHONY: install-copies
> install-copies: Makefile
> install -c -m 644 Makefile copy1
> install -c -m 644 Makefile copy2
> install -c -m 644 Makefile copy3
> install -c -m 644 Makefile copy4
> ===================================================

This workaround doesn't actually work (in the actual case of GNU gettext):
it fails the "make distcheck" verification.

The real workaround goes like this:

===================================================
all : copy1 copy2 copy3 copy4

copy1: Makefile
        install -c -m 644 Makefile copy1
        install -c -m 644 Makefile copy2
        install -c -m 644 Makefile copy3
        install -c -m 644 Makefile copy4
copy2 copy3 copy4: copy1
===================================================

Bruno


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: "make -jN" requires mechanical changes to a Makefile

Paul Smith-20
On Sun, 2019-05-12 at 18:07 +0200, Bruno Haible wrote:

> This workaround doesn't actually work (in the actual case of GNU gettext):
> it fails the "make distcheck" verification.
>
> The real workaround goes like this:
>
> ===================================================
> all : copy1 copy2 copy3 copy4
>
> copy1: Makefile
>         install -c -m 644 Makefile copy1
>         install -c -m 644 Makefile copy2
>         install -c -m 644 Makefile copy3
>         install -c -m 644 Makefile copy4
> copy2 copy3 copy4: copy1
> ===================================================

This is not fully-correct either:

  $ make
  install -c -m 644 Makefile copy1
  install -c -m 644 Makefile copy2
  install -c -m 644 Makefile copy3
  install -c -m 644 Makefile copy4

  $ rm copy3

  $ make
  make: Nothing to be done for 'all'.


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: "make -jN" requires mechanical changes to a Makefile

Bruno Haible
Hi Paul,

> > The real workaround goes like this:
> >
> > ===================================================
> > all : copy1 copy2 copy3 copy4
> >
> > copy1: Makefile
> >         install -c -m 644 Makefile copy1
> >         install -c -m 644 Makefile copy2
> >         install -c -m 644 Makefile copy3
> >         install -c -m 644 Makefile copy4
> > copy2 copy3 copy4: copy1
> > ===================================================
>
> This is not fully-correct either:
>
>   $ make
>   install -c -m 644 Makefile copy1
>   install -c -m 644 Makefile copy2
>   install -c -m 644 Makefile copy3
>   install -c -m 644 Makefile copy4
>
>   $ rm copy3
>
>   $ make
>   make: Nothing to be done for 'all'.

Indeed. Thank you for having spotted this; otherwise I would have put a
broken rule into GNU gettext.

Now, when my use-case is:
  - one rule that produces N files (N > 1),
  - I want "make" to execute the rule only once, not N times,
    even with parallel make.
What is the solution? The documentation page [1] explicitly does NOT mention
this use-case for multiple targets on the same rule.

You mentioned splitting the rule into one rule per file. But bison does not
work this way; it really produces two files at once. And for other rules
I mentioned Automake limitations.

Parallel make is of growing importance, because Debian now uses parallel builds
by default (quote: "debhelper >= 10 defaults to parallel build").

Bruno

[1] https://www.gnu.org/software/make/manual/html_node/Multiple-Targets.html


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: "make -jN" requires mechanical changes to a Makefile

Henrik Carlqvist-2
On Sun, 12 May 2019 22:23:12 +0200
Bruno Haible <[hidden email]> wrote:
> Now, when my use-case is:
>   - one rule that produces N files (N > 1),
>   - I want "make" to execute the rule only once, not N times,
>     even with parallel make.
> What is the solution?

I think that the only good solution is to make sure than only 1 of the N
created files is a known target for the Makefile. If you write single
rules that on one call creates multiple targets your Makefile will not be
compatible with parallel make.

Example with one rule creating 4 files:

all : copy1

copy1: Makefile
        install -c -m 644 Makefile copy1
        install -c -m 644 Makefile copy2
        install -c -m 644 Makefile copy3
        install -c -m 644 Makefile copy4

A better way would be to have one rule to create multiple targets, but
only on target for each call, example:

all: copy1 copy2 copy3 copy4

copy%: Makefile
        install -c -m 644 Makefile $@

The above simple Makefile would be fully compatible with parallel make.

regards Henrik

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: "make -jN" requires mechanical changes to a Makefile

Bruno Haible
Henrik Carlqvist wrote:
> Example with one rule creating 4 files:
>
> all : copy1
>
> copy1: Makefile
>         install -c -m 644 Makefile copy1
>         install -c -m 644 Makefile copy2
>         install -c -m 644 Makefile copy3
>         install -c -m 644 Makefile copy4

I think the "representative" file should be copy4 here, because it's the one
that gets created last.

Otherwise, when the user interrupts "make" after copy1 was created but before
copy2, copy3, copy4 are created, and then retries "make" again, the second
'make' invocation will fail somewhere when it references copy2...copy4.

Bruno


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: "make -jN" requires mechanical changes to a Makefile

Howard Chu
Bruno Haible wrote:

> Henrik Carlqvist wrote:
>> Example with one rule creating 4 files:
>>
>> all : copy1
>>
>> copy1: Makefile
>>         install -c -m 644 Makefile copy1
>>         install -c -m 644 Makefile copy2
>>         install -c -m 644 Makefile copy3
>>         install -c -m 644 Makefile copy4
>
> I think the "representative" file should be copy4 here, because it's the one
> that gets created last.

That sort of thing is only true in serial make, you can't rely on it in parallel make.

--
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: "make -jN" requires mechanical changes to a Makefile

Bruno Haible
Howard Chu wrote:

> >> Example with one rule creating 4 files:
> >>
> >> all : copy1
> >>
> >> copy1: Makefile
> >>         install -c -m 644 Makefile copy1
> >>         install -c -m 644 Makefile copy2
> >>         install -c -m 644 Makefile copy3
> >>         install -c -m 644 Makefile copy4
> >
> > I think the "representative" file should be copy4 here, because it's the one
> > that gets created last.
>
> That sort of thing is only true in serial make, you can't rely on it in parallel make.

The sequence of lines of the recipe of a rule gets executed in order,
even in parallel make, no?

Bruno


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: "make -jN" requires mechanical changes to a Makefile

Henrik Carlqvist-2
On Mon, 13 May 2019 00:05:59 +0200
Bruno Haible <[hidden email]> wrote:

> Howard Chu wrote:
> > >> Example with one rule creating 4 files:
> > >>
> > >> all : copy1
> > >>
> > >> copy1: Makefile
> > >>         install -c -m 644 Makefile copy1
> > >>         install -c -m 644 Makefile copy2
> > >>         install -c -m 644 Makefile copy3
> > >>         install -c -m 644 Makefile copy4
> > >
> > > I think the "representative" file should be copy4 here, because it's
> > > the one that gets created last.
> >
> > That sort of thing is only true in serial make, you can't rely on it
> > in parallel make.
>
> The sequence of lines of the recipe of a rule gets executed in order,
> even in parallel make, no?

Yes, they will be run in sequence even with parallel make and copy4 might
be a better known target than copy1.

regards Henrik

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: "make -jN" requires mechanical changes to a Makefile

Edward Welbourne-3
> Howard Chu wrote:
> > >> Example with one rule creating 4 files:
> > >>
> > >> all : copy1
> > >>
> > >> copy1: Makefile
> > >>         install -c -m 644 Makefile copy1
> > >>         install -c -m 644 Makefile copy2
> > >>         install -c -m 644 Makefile copy3
> > >>         install -c -m 644 Makefile copy4

For this (toy example) case,

# We want to create all 4 copies
all: copy1 copy2 copy3 copy4
# Each of the copies depends on the Makefile:
copy1 copy2 copy3 copy4: Makefile
        install -c -m 644 $< $@

should do the trick.  The rule means: "To update any one of the files
copy1...copy4 (it'll be called $@), you need the Makefile (it'll be
called $<); once $< is up to date, if it's newer than $@, copy it to the
target you're after."  Note that asking for any one of the files means
asking for the rule to be run, no matter how many of them it makes.

However, that's no use to you for the case of bison generating two
files.  Your actual command doesn't just make one file; it makes two
files.  So the target whose command it is needs to be a (single) "both
files" target, not the two files.  Writing

this that .PHONY: both
both: source
        COMMAND

says: if either this or that is older than source, run COMMAND (once).
Whereas

this that: source
        COMMAND

says: for each of this and that (call it $@), if $@ is older than
source, run COMMAND.  If both are older than source, this claims we need
to run COMMAND twice: indeed, serial make could check both this and
that, to see if either is older than source, before running COMMAND for
either of them.  For each that is older than source, make would then
perfectly reasonably queue a copy of COMMAND, with the stale file's name
substituted for $@ anywhere it appears in COMMAND, for subsequent
execution.  It would thus run COMMAND twice, once for each of the files;
and, indeed, this would be correct behaviour if COMMAND only updated $@
(which is what make expects from its rules).

As it happens, serial make leaves the staleness check until just before
it runs the command, so it manages to optimise away the second run, if
COMMAND actually does update both (hence also the one that wasn't $@ on
the first run).  That's more or less an accident, though; and it doesn't
happen for parallel make, which duly does run COMMAND twice, in
parallel; and the two instances try to update the same files, which can
cause problems.  The error isn't in make; it is in the make file, which
misdescribed the recipe as a way to make *each* file, when it's actually
a way to make *both*.

Which is why you need that .PHONY intermediate.
It's what says "this command updates *both* files".

As Paul said, writing code to guess "what the author meant" and
implement that is usually perilous.  There are way too many ways for the
guessing code to get it wrong, at least until your guessing is being
done by a true AI that knows to recognise when it doesn't know what the
author meant, so stops the build to ask what was meant.

So, rather than asking for make to "do what I mean, not what I said",
please take the time to tell make what you really meant.  Describe,
faithfully and fully, how the commands relate to the files, then make
can do its dumb but robust thing reliably.  Making things cleverer
usually leads to them breaking; making them tediously straightforward is
good for reliability,

        Eddy.
--
A shortcut is the longest distance between two points.

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: "make -jN" requires mechanical changes to a Makefile [SOLVED]

Bruno Haible
In reply to this post by Bruno Haible
Continuing this thread from May 2019
<https://lists.gnu.org/archive/html/bug-make/2019-05/msg00022.html>:
The problem was:

  How can a rule that generates multiple files be formulated so
  that it works with parallel make?

For example, a rule that invokes bison, or a rule that invokes
a different Makefile. For simplicity, here, use a rule that
creates 4 files copy1, copy2, copy3, copy4.

===========================================
all : copy1 copy2 copy3 copy4

copy1 copy2 copy3 copy4: Makefile
        install -c -m 644 Makefile copy1
        install -c -m 644 Makefile copy2
        install -c -m 644 Makefile copy3
        install -c -m 644 Makefile copy4
===========================================

Unfortunately, with "make -j8", it invokes the rule multiple times.

It is possible to change this Makefile so that
  (A) "rm -f copy?; make" executes the rule once.
  (B) "rm -f copy?; make -j8" executes the rule once as well.
  (C) After "make", another "make" just prints "Nothing to be done for 'all'."
  (D) After removing one of the files copy?, "make" executes the rule once.
      (This covers also the case of pressing Ctrl-C during "make", then
      doing "make" again.)
  (E) After replacing one of the files copy? with a file that is older than
      Makefile, "make" executes the rule once.

There are three possibilities:


(I) Assuming GNU make >= 4.3, the "Grouped explicit target" syntax does it.
Thanks to Kaz Kylheku and Paul Smith for having added this.

===========================================
all : copy1 copy2 copy3 copy4

copy1 copy2 copy3 copy4 &: Makefile
        install -c -m 644 Makefile copy1
        install -c -m 644 Makefile copy2
        install -c -m 644 Makefile copy3
        install -c -m 644 Makefile copy4
===========================================

But it will take a number of years until we can assume that all 'make'
programs that we care about support this syntax.


(II) It is possible to turn off parallel make. This does it:

===========================================
all : copy1 copy2 copy3 copy4

copy1 copy2 copy3 copy4 : Makefile
        install -c -m 644 Makefile copy1
        install -c -m 644 Makefile copy2
        install -c -m 644 Makefile copy3
        install -c -m 644 Makefile copy4

# This Makefile contains rules which don't work with parallel make.
# So, turn off parallel execution in this Makefile.
.NOTPARALLEL:
===========================================

But some people who want to minimize wall-clock execution time of
their builds may not like it.


(III) Use portable make syntax and still allow parallel make.

This is a bit harder. My solution is to first analyze in which order
the rule will generate the various files and thus what the expected
timestamp order is. In this case, it is:
  Makefile <= copy1 <= copy2 <= copy3 <= copy4

First some attempts that don't work:

===========================================
all : copy1 copy2 copy3 copy4

copy1: Makefile
        install -c -m 644 Makefile copy1
        install -c -m 644 Makefile copy2
        install -c -m 644 Makefile copy3
        install -c -m 644 Makefile copy4

copy2 copy3 copy4: copy1
===========================================

Fails (E).

===========================================
all : copy1 copy2 copy3 copy4

copy4: Makefile
        install -c -m 644 Makefile copy1
        install -c -m 644 Makefile copy2
        install -c -m 644 Makefile copy3
        install -c -m 644 Makefile copy4

copy1 copy2 copy3: copy4
===========================================

Fails (E) as well.

===========================================
all : copy1 copy2 copy3 copy4

copy4: Makefile
        install -c -m 644 Makefile copy1
        install -c -m 644 Makefile copy2
        install -c -m 644 Makefile copy3
        install -c -m 644 Makefile copy4

copy1: copy4
        test -f copy1 || { rm -f copy4; $(MAKE) copy4; }
copy2: copy4
        test -f copy2 || { rm -f copy4; $(MAKE) copy4; }
copy3: copy4
        test -f copy3 || { rm -f copy4; $(MAKE) copy4; }
===========================================

Fails (E) for copy1, ..., copy3.
And fails (C), i.e. clutters the "make" output.

===========================================
all : copy1 copy2 copy3 copy4

copy1: Makefile
        { test -f copy1 && test ! copy1 -ot Makefile; } || { rm -f copy4; $(MAKE) copies; }
copy2: copy1
        { test -f copy2 && test ! copy2 -ot copy1; } || { rm -f copy4; $(MAKE) copies; }
copy3: copy2
        { test -f copy3 && test ! copy3 -ot copy2; } || { rm -f copy4; $(MAKE) copies; }
copy4: copy3
        { test -f copy4 && test ! copy4 -ot copy3; } || { rm -f copy4; $(MAKE) copies; }

copies:
        install -c -m 644 Makefile copy1
        install -c -m 644 Makefile copy2
        install -c -m 644 Makefile copy3
        install -c -m 644 Makefile copy4
.PHONY: copies
===========================================

This solution fulfils all the requirements.

Can we use 'test FILE1 -ot FILE2'?
- POSIX 'test' does not support this option.
  See <https://pubs.opengroup.org/onlinepubs/9699919799/utilities/test.html>
- But all test programs in all current operating systems (Linux, macOS,
  FreeBSD, NetBSD, OpenBSD, AIX, Solaris, Haiku, Minix, Cygwin, even Busybox)
  support it. Except for /bin/sh on Solaris 10 — but in packages that use the
  GNU Build System, Autoconf sets SHELL = @CONFIG_SHELL@ = /bin/bash on such
  systems.

Bruno


Reply | Threaded
Open this post in threaded view
|

Re: "make -jN" requires mechanical changes to a Makefile [SOLVED]

Howard Chu
Bruno Haible wrote:

> Continuing this thread from May 2019
> <https://lists.gnu.org/archive/html/bug-make/2019-05/msg00022.html>:
> The problem was:
>
>   How can a rule that generates multiple files be formulated so
>   that it works with parallel make?
>
> For example, a rule that invokes bison, or a rule that invokes
> a different Makefile. For simplicity, here, use a rule that
> creates 4 files copy1, copy2, copy3, copy4.
>
> ===========================================
> all : copy1 copy2 copy3 copy4
>
> copy1 copy2 copy3 copy4: Makefile
> install -c -m 644 Makefile copy1
> install -c -m 644 Makefile copy2
> install -c -m 644 Makefile copy3
> install -c -m 644 Makefile copy4
> ===========================================
>
> Unfortunately, with "make -j8", it invokes the rule multiple times.
>
> It is possible to change this Makefile so that
>   (A) "rm -f copy?; make" executes the rule once.
>   (B) "rm -f copy?; make -j8" executes the rule once as well.
>   (C) After "make", another "make" just prints "Nothing to be done for 'all'."
>   (D) After removing one of the files copy?, "make" executes the rule once.
>       (This covers also the case of pressing Ctrl-C during "make", then
>       doing "make" again.)
>   (E) After replacing one of the files copy? with a file that is older than
>       Makefile, "make" executes the rule once.
>
> There are three possibilities:

You're thinking about this the wrong way. Your set of commands is inherently
serial, therefore you need to write serial dependencies.

===========================================
all: copy1 copy2 copy3 copy4

copy1: Makefile
        install -c -m 644 Makefile $@
copy2: copy1
        install -c -m 644 Makefile $@
copy3: copy2
        install -c -m 644 Makefile $@
copy4: copy3
        install -c -m 644 Makefile $@
===========================================

This satisfies all your conditions, because it is inherently correct.

More on writing proper parallel Makefiles here http://highlandsun.com/hyc/#Make

--
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/

Reply | Threaded
Open this post in threaded view
|

Re: "make -jN" requires mechanical changes to a Makefile [SOLVED]

Paul Smith-20
In reply to this post by Bruno Haible
On Sun, 2020-09-13 at 20:55 +0200, Bruno Haible wrote:
>   How can a rule that generates multiple files be formulated so
>   that it works with parallel make?
>
> For example, a rule that invokes bison, or a rule that invokes
> a different Makefile. For simplicity, here, use a rule that
> creates 4 files copy1, copy2, copy3, copy4.

Sorry, I think the last time this came up I got side-tracked by the
fact that the example was using install.

There is a straightforward and portable way to do this even with
traditional make, it's just not as nice (but, nicer than changing all
the recipes to use test IMO! :)).

If you have a rule like this:

  <a1> <b2> ... : <prereqs>
          <command>

where <command> generates all targets with one invocation, then your
best bet is to modify this rule into two rules:

  <a1> <b2> ... : .sentinel ;

  .sentinel: <prereqs>
          <command>
          @touch $@

Note, it's critical to include the semicolon here.  Or if you prefer to
be more explicit you can write something like:

  <a1> <b2> ... : .sentinel
          : do nothing

This will work properly.


Reply | Threaded
Open this post in threaded view
|

Re: "make -jN" requires mechanical changes to a Makefile [SOLVED]

Paul Smith-20
On Sun, 2020-09-13 at 15:08 -0400, Paul Smith wrote:
>   <a1> <b2> ... : .sentinel ;
>
>   .sentinel: <prereqs>
>           <command>
>           @touch $@

Just to be clear, you don't have to use ".sentinel" you can use any
target name, and obviously you must use a different name for each
"grouping" of targets that needs to be built.


Reply | Threaded
Open this post in threaded view
|

Re: "make -jN" requires mechanical changes to a Makefile [SOLVED]

Bruno Haible
In reply to this post by Bruno Haible
> ===========================================
> all : copy1 copy2 copy3 copy4
>
> copy1: Makefile
> { test -f copy1 && test ! copy1 -ot Makefile; } || { rm -f copy4; $(MAKE) copies; }
> copy2: copy1
> { test -f copy2 && test ! copy2 -ot copy1; } || { rm -f copy4; $(MAKE) copies; }
> copy3: copy2
> { test -f copy3 && test ! copy3 -ot copy2; } || { rm -f copy4; $(MAKE) copies; }
> copy4: copy3
> { test -f copy4 && test ! copy4 -ot copy3; } || { rm -f copy4; $(MAKE) copies; }
>
> copies:
> install -c -m 644 Makefile copy1
> install -c -m 644 Makefile copy2
> install -c -m 644 Makefile copy3
> install -c -m 644 Makefile copy4
> .PHONY: copies
> ===========================================
>
> This solution fulfils all the requirements.

It can be simplified a bit. And remove verbosity:

===========================================
all : copy1 copy2 copy3 copy4

copy1: Makefile
        @{ test -f copy1 && test ! copy1 -ot Makefile; } || $(MAKE) copies
copy2: copy1
        @{ test -f copy2 && test ! copy2 -ot copy1; } || $(MAKE) copies
copy3: copy2
        @{ test -f copy3 && test ! copy3 -ot copy2; } || $(MAKE) copies
copy4: copy3
        @{ test -f copy4 && test ! copy4 -ot copy3; } || $(MAKE) copies

copies:
        install -c -m 644 Makefile copy1
        install -c -m 644 Makefile copy2
        install -c -m 644 Makefile copy3
        install -c -m 644 Makefile copy4
.PHONY: copies
===========================================


Reply | Threaded
Open this post in threaded view
|

Re: "make -jN" requires mechanical changes to a Makefile [SOLVED]

Bruno Haible
In reply to this post by Paul Smith-20
Hi Paul,

> There is a straightforward and portable way to do this even with
> traditional make, it's just not as nice (but, nicer than changing all
> the recipes to use test IMO! :)).
>
> If you have a rule like this:
>
>   <a1> <b2> ... : <prereqs>
>           <command>
>
> where <command> generates all targets with one invocation, then your
> best bet is to modify this rule into two rules:
>
>   <a1> <b2> ... : .sentinel ;
>
>   .sentinel: <prereqs>
>           <command>
>           @touch $@
>
> Note, it's critical to include the semicolon here.  Or if you prefer to
> be more explicit you can write something like:
>
>   <a1> <b2> ... : .sentinel
>           : do nothing
>
> This will work properly.

Thanks for showing this idiom.

I think this can be improved a bit: Instead of 'touch .sentinel' I would
use 'echo > .sentinel'. Because if you use 'touch .sentinel' and the build
directory is on an NFS mount and the clock of the NFS server is ahead the
clock of localhost (I have seen this quite frequently), the timestamp of
.sentinel will be behind the timestamp of the newest <prereqs>, and thus
another "make" run will execute the rule again.

Still, I don't like these '.sentinel' or stamp files, because they
add complexity:
  * The developer has to decide whether the stamp file should go into
    the build dir or $(srcdir). The general rule should be that if
    <command> uses only build tools that are guaranteed to be present,
    then <a1> <b2> ... and the stamp file should go into the build dir,
    whereas if they use less common tools (e.g. bison), they should
    all go into $(srcdir).
  * The automake dist target needs to be adjusted to include this stamp
    file.
  * Version control (.gitignore) needs to be adjusted to include this
    stamp file.

I'll use the rule with a phony target and 'test ... -ot ...', unless you
find a problem with it (other than the potential portability problem
redarding 'test').

Bruno


12