Parallel problem

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Parallel problem

jeremiah
Hello I have a problem with parallel tests being run with make -j 42
with the following makefile:

https://notabug.org/deesix/M2-Planet/src/RELEASE_CANDIDATE-parallel_testing/makefile

The problem is that although none of the tests actually write to the
same files, they all depend on the program to built first but make seems
to cause GCC to try to compile the program they all depend upon more
than once at the same time and breaks with the error message:

> In file included from <command-line>:
> /usr/include/stdc-predef.h:1: fatal error: can’t create precompiled
> header bin/M2-Planet: No such file or directory
>  /* Copyright (C) 1991-2018 Free Software Foundation, Inc.
>
> compilation terminated.
> In file included from <command-line>:
> /usr/include/stdc-predef.h:1: fatal error: can’t create precompiled
> header bin/M2-Planet: No such file or directory
>  /* Copyright (C) 1991-2018 Free Software Foundation, Inc.
>
> compilation terminated.

This occurs only when I do:
make clean test -j42

but never when I do:
make clean M2-Planet test -j42

What can I do to ensure that make only tries to build M2-Planet once
before running the tests?

-Jeremiah

Reply | Threaded
Open this post in threaded view
|

RE: Parallel problem

Cook, Malcolm-2
After a quick squiz at your makefile I’m guessing line 42 should not read
       -o bin/M2-Planet
But rather
       -o M2-Planet
Or even better:
       -o $@
No?
Similar changes for line 67 are probably in order




From: Help-make <help-make-bounces+mec=[hidden email]> On Behalf Of [hidden email]
Sent: Monday, January 11, 2021 17:36
To: [hidden email]
Subject: Parallel problem

ATTENTION: This email came from an external source. Do not open attachments or click on links from unknown senders or unexpected emails.


Hello I have a problem with parallel tests being run with make -j 42
with the following makefile:

https://notabug.org/deesix/M2-Planet/src/RELEASE_CANDIDATE-parallel_testing/makefile

The problem is that although none of the tests actually write to the
same files, they all depend on the program to built first but make seems
to cause GCC to try to compile the program they all depend upon more
than once at the same time and breaks with the error message:

> In file included from <command-line>:
> /usr/include/stdc-predef.h:1: fatal error: can’t create precompiled
> header bin/M2-Planet: No such file or directory
> /* Copyright (C) 1991-2018 Free Software Foundation, Inc.
>
> compilation terminated.
> In file included from <command-line>:
> /usr/include/stdc-predef.h:1: fatal error: can’t create precompiled
> header bin/M2-Planet: No such file or directory
> /* Copyright (C) 1991-2018 Free Software Foundation, Inc.
>
> compilation terminated.

This occurs only when I do:
make clean test -j42

but never when I do:
make clean M2-Planet test -j42

What can I do to ensure that make only tries to build M2-Planet once
before running the tests?

-Jeremiah
Reply | Threaded
Open this post in threaded view
|

Re: Parallel problem

jeremiah
> After a quick squiz at your makefile I’m guessing line 42 should not read
>
>        -o bin/M2-Planet
>
> But rather
>
>        -o M2-Planet
>
> Or even better:
>
>        -o $@
>
> No?
>
> Similar changes for line 67 are probably in order

No that does not correct the issue reported.
As bin is in the VPATH on line 19.

Further it would break the all of the tests.
As they expect bin/M2-Planet.

-Jeremiah

Reply | Threaded
Open this post in threaded view
|

Re: Parallel problem

Paul Smith-20
On Tue, 2021-01-12 at 12:35 +0000, [hidden email] wrote:

> > After a quick squiz at your makefile I’m guessing line 42 should
> > not read
> >         -o bin/M2-Planet
> > But rather
> >         -o M2-Planet
> > Or even better:
> >         -o $@
> > No?
> > Similar changes for line 67 are probably in order
>
> No that does not correct the issue reported.
> As bin is in the VPATH on line 19.

Whether bin is on VPATH is not relevant.  That's not how VPATH works:
VPATH is _only_ useful for locating _source_ files (files that always
exist).  It cannot be used for locating generated files.

For more details read this:

http://make.mad-scientist.net/papers/how-not-to-use-vpath/

> Further it would break the all of the tests.
> As they expect bin/M2-Planet.

Your rule MUST ALWAYS build the target file that you told make they
would build.  It's invalid, and will lead to breakage, if you tell make
that your recipe will create M2-Planet but it instead creates bin/M2-
Planet, which is a completely different file.

As a result, as Malcolm says, all recipes should build $@ and never
some alternate version of that (like bin/$@ or whatever).

If you really need the generated file to be bin/M2-Planet then your
target should build bin/M2-Planet:

  all: bin/M2-Planet

  bin/M2-Planet: ...
          $(CC) ... -o $@

If you want to be able to run "make M2-Planet" you can create a
shortcut like this:

  M2-Planet: bin/M2-Planet


Further, it's not going to work reliably to write a make command line
like this:

> make clean test -j42

That's because make will build the command line targets in parallel as
well, which means (because "clean" is not a prerequisite so make
doesn't know to make sure it's completed first) "test" and "clean" will
be run at the same time, which could mean that things that "test" is
creating (such as bin/M2-Planet) will be deleted by "clean".

The simplest solution is to run this instead:

  make clean && make test -j42

If you really want to be able to run "make clean test" then you'll have
to do something fancy: ensure clean is a prerequisite of all the
targets but ONLY when specified on the command line.  You can use this:

  CLEAN_TARGET := $(filter clean,$(MAKECMDGOALS))

which will be set to "clean" if "clean" was a target on the command
line, or the empty string if not.  Then you have to add $(CLEAN_TARGET)
to all the targets (it's not enough to make it a prerequisite of "test"
because that doesn't make it a prerequisite of the things "test"
depends on).

In your makefile it may be enough to list $(CLEAN_TARGET) as a
prerequisite of M2-Planet because it seems most everything depends on
that, since you compile from source rather than building object files.


Reply | Threaded
Open this post in threaded view
|

Re: Parallel problem

Tomas Nordin-2
Paul Smith <[hidden email]> writes:

> On Tue, 2021-01-12 at 12:35 +0000, [hidden email] wrote:
>> > After a quick squiz at your makefile I’m guessing line 42 should
>> > not read
>> >         -o bin/M2-Planet
>> > But rather
>> >         -o M2-Planet
>> > Or even better:
>> >         -o $@
>> > No?
>> > Similar changes for line 67 are probably in order
>>
>> No that does not correct the issue reported.
>> As bin is in the VPATH on line 19.
>
> Whether bin is on VPATH is not relevant.  That's not how VPATH works:
> VPATH is _only_ useful for locating _source_ files (files that always
> exist).  It cannot be used for locating generated files.
>
> For more details read this:
>
> http://make.mad-scientist.net/papers/how-not-to-use-vpath/
>
>> Further it would break the all of the tests.
>> As they expect bin/M2-Planet.
>
> Your rule MUST ALWAYS build the target file that you told make they
> would build.  It's invalid, and will lead to breakage, if you tell make
> that your recipe will create M2-Planet but it instead creates bin/M2-
> Planet, which is a completely different file.
>
> As a result, as Malcolm says, all recipes should build $@ and never
> some alternate version of that (like bin/$@ or whatever).
>
> If you really need the generated file to be bin/M2-Planet then your
> target should build bin/M2-Planet:
>
>   all: bin/M2-Planet
>
>   bin/M2-Planet: ...
>           $(CC) ... -o $@
>
> If you want to be able to run "make M2-Planet" you can create a
> shortcut like this:
>
>   M2-Planet: bin/M2-Planet

OK! Is that understood as a shortcut by make? So the "MUST ALWAYS build
the target" does not apply in this case?

Best regards
--
Tomas

Reply | Threaded
Open this post in threaded view
|

Re: Parallel problem

Paul Smith-20
On Tue, 2021-01-12 at 19:50 +0100, Tomas Nordin wrote:
> > If you want to be able to run "make M2-Planet" you can create a
> > shortcut like this:
> >    M2-Planet: bin/M2-Planet
>
> OK! Is that understood as a shortcut by make? So the "MUST ALWAYS
> build the target" does not apply in this case?

That target has no recipe, so there's nothing to build the target.  I
guess a more detailed way to say it would be that any rule that builds
an actual target file on the disk must always build the one that the
rule says it will build.

If you have a target that doesn't actually create a file (like "all"
for example, or this shortcut target) then the rule that the created
file must match the target doesn't apply.


Reply | Threaded
Open this post in threaded view
|

RE: Parallel problem

Cook, Malcolm-2
I think you should want to declare such a target as .PHONY, agreed Paul?

From: Help-make <help-make-bounces+mec=[hidden email]> On Behalf Of Paul Smith
Sent: Tuesday, January 12, 2021 13:29
To: Tomas Nordin <[hidden email]>; [hidden email]
Subject: Re: Parallel problem

ATTENTION: This email came from an external source. Do not open attachments or click on links from unknown senders or unexpected emails.


On Tue, 2021-01-12 at 19:50 +0100, Tomas Nordin wrote:
> > If you want to be able to run "make M2-Planet" you can create a
> > shortcut like this:
> > M2-Planet: bin/M2-Planet
>
> OK! Is that understood as a shortcut by make? So the "MUST ALWAYS
> build the target" does not apply in this case?

That target has no recipe, so there's nothing to build the target. I
guess a more detailed way to say it would be that any rule that builds
an actual target file on the disk must always build the one that the
rule says it will build.

If you have a target that doesn't actually create a file (like "all"
for example, or this shortcut target) then the rule that the created
file must match the target doesn't apply.
Reply | Threaded
Open this post in threaded view
|

Re: Parallel problem

Paul Smith-20
On Tue, 2021-01-12 at 19:36 +0000, Cook, Malcolm wrote:
> I think you should want to declare such a target as .PHONY, agreed
> Paul?

Sure, that's probably a good idea.