Error in processing of :: targets

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Error in processing of :: targets

reid.madsen
Gnumake maintainers,
 
I've run into a functional difference between the standard unix make and GNUmake (3.80) involving how the :: rules are processed.
 
In the standard unix make (Solaris), the :: targets ( i.e., all::) are process in the order they are found in the Makefile serially, regardless of whether parallelism is used (i.e., -jN, where N is > 1).
 
In GNUmake (3.80), the when -j1 is used, the "::" targets are executed serially.  When -jN is used the "::" targets are executed in parallel.
 
Consider the following -- from an IMAKE generated Makefile:
 
all:: all.prologue
all:: all.serial all.parallel
all:: ; @echo $@; sleep 1; echo [hidden email]
all:: all.epilogue
 
all.epilogue all.prologue:: ; @echo $@; sleep 2; echo [hidden email]
all.serial:: ; @echo [hidden email]; sleep 1; echo [hidden email]
all.serial:: ; @echo [hidden email]; sleep 1; echo [hidden email]
all.parallel: aaa bbb ccc
aaa bbb ccc: ; @echo $@; sleep 1; echo [hidden email]
 
In a non-parallel make (i.e. -j1) in both Unix and Gnu make, the following is the result:
 
all.prologue
all.prologue-done
all.serial-1
all.serial-1-done
all.serial-2
all.serial-2-done
aaa
aaa-done
bbb
bbb-done
ccc
ccc-done
all
all-done
all.epilogue
all.epilogue-done
 
When run in parallel using Unix make (i.e., -j12), the following occurs:
 
all.prologue
all.prologue-done
all.serial-1
aaa
bbb
ccc
all.serial-1-done
aaa-done
bbb-done
ccc-done
all.serial-2
all.serial-2-done
all
all-done
all-epilogue
all-epilogue-done
 
When run in parallel using GNUmake (i.e., -j12), the following occurs:
 
all.prologue
all.serial-1
aaa
bbb
ccc
all.epilogue
all.prologue-done
all.serial-1-done
aaa-done
all.serial-2
all.epilogue-done
bbb-done
ccc-done
all.serial-2-done
all
all-done
 
It appears that all the :: targets that exist in the file are started at the same time and not executed sequentially as is done for standard Unix make.  The only exception to this is the all.serial target which has multiple command, which are executed in order... As a result of this I cannot port our IMAKE based make utility to use GNUmake.
 
The intent of the above targets is as follows:
 
+ all.prologue -- things which should be performed before any other all* targets are made
                        For example, creating, or removing a directory.
+ all.epilogue -- things which should occur after all* targets have completed -- for example sending email.
+ a.serial -- rules that MUST execute serially
+ a.parallel -- rules that may execute in parallel
 
The following defines the order of execution:
 
all: all.prologue                   
all: all.serial all.parallel
[ multiple all, all-serial, and all.parallel targets]
all: all.epilogue
 
I'm interested in your views on this problem.  I think it's a bug because GNUmake is clearly out of sync with standard make on the order in which :: targets are evaluated in a parallel build environment.
 
Reid Madsen
Principal Engineer - Architecture Group
Tektronix, Inc.
214-912-5152
 

_______________________________________________
Bug-make mailing list
[hidden email]
http://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: Error in processing of :: targets

Paul Smith-20
%% [hidden email] writes:

  rm> I've run into a functional difference between the standard unix
  rm> make and GNUmake (3.80) involving how the :: rules are processed.

By "standard unix make" you seem to mean SystemV make.  Note that the
only official Standard-with-a-capital-S (recognized by the FSF and GNU
project) for make is the POSIX standard.  The POSIX standard doesn't
address this issue (it doesn't address parallelism at all).

SystemV make is, at best, a de facto standard, and there are plenty of
other (incompatible) versions of make out there.

In fact, GNU make is not intended to be a drop-in replacement for
SystemV make and there are _several_ points at which the behavior of the
two versions diverge.

  rm> In the standard unix make (Solaris), the :: targets ( i.e., all::)
  rm> are process in the order they are found in the Makefile serially,
  rm> regardless of whether parallelism is used (i.e., -jN, where N is >
  rm> 1).

  rm> In GNUmake (3.80), the when -j1 is used, the "::" targets are
  rm> executed serially.  When -jN is used the "::" targets are executed
  rm> in parallel.

GNU make's behavior here is exactly in keeping with the definition of
double-colon rules in the GNU make manual, which states:

>    When a target appears in multiple rules, all the rules must be the
> same type: all ordinary, or all double-colon.  If they are
> double-colon, each of them is independent of the others.  Each
> double-colon rule's commands are executed if the target is older than
> any prerequisites of that rule.  If there are no prerequisites for that
> rule, its commands are always executed (even if the target already
> exists).  This can result in executing none, any, or all of the
> double-colon rules.
>
>    Double-colon rules with the same target are in fact completely
> separate from one another.  Each double-colon rule is processed
> individually, just as rules with different targets are processed.

If you think a different set of behavior is desirable, the best thing is
probably to file an enhancement request on the Savannah
(https://savannah.gnu.org/) project page for GNU make and someone will
look at it.

Cheers!

--
-------------------------------------------------------------------------------
 Paul D. Smith <[hidden email]>          Find some GNU make tips at:
 http://www.gnu.org                      http://make.paulandlesley.org
 "Please remain calm...I may be mad, but I am a professional." --Mad Scientist


_______________________________________________
Bug-make mailing list
[hidden email]
http://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: Error in processing of :: targets

Howard Chu
In reply to this post by reid.madsen
[hidden email] wrote:
> Gnumake maintainers,
>  
> I've run into a functional difference between the standard unix make and
> GNUmake (3.80) involving how the :: rules are processed.
>  
> In the standard unix make (Solaris), the :: targets ( i.e., all::) are
> process in the order they are found in the Makefile serially, regardless
> of whether parallelism is used (i.e., -jN, where N is > 1).

That's funny, my "standard Unix" (Solaris) make doesn't have a "-j"
switch, and doesn't support parallelism. In fact, -j is a GNU-specific
extension, and it Works As Designed.
>  
> In GNUmake (3.80), the when -j1 is used, the "::" targets are executed
> serially.  When -jN is used the "::" targets are executed in parallel.

> Consider the following -- from an IMAKE generated Makefile:

This, like most of the dreck that Imake produces, is bogus.

http://lists.gnu.org/archive/html/bug-make/2001-09/msg00013.html
http://lists.gnu.org/archive/html/bug-make/2004-09/msg00001.html

> all:: all.prologue
> all:: all.serial all.parallel
> all:: ; @echo $@; sleep 1; echo $@-done <mailto:$@-done>
> all:: all.epilogue
>  
> all.epilogue all.prologue:: ; @echo $@; sleep 2; echo $@-done
> <mailto:$@-done>
> all.serial:: ; @echo $@-1 <mailto:$@-1>; sleep 1; echo $@-1-done
> <mailto:$@-1-done>
> all.serial:: ; @echo $@-2 <mailto:$@-2>; sleep 1; echo $@-2-done
> <mailto:$@-2-done>
> all.parallel: aaa bbb ccc
> aaa bbb ccc: ; @echo $@; sleep 1; echo $@-done <mailto:$@-done>

--
   -- Howard Chu
   Chief Architect, Symas Corp.       Director, Highland Sun
   http://www.symas.com               http://highlandsun.com/hyc
   Symas: Premier OpenSource Development and Support


_______________________________________________
Bug-make mailing list
[hidden email]
http://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

RE: Error in processing of :: targets

reid.madsen
In reply to this post by reid.madsen
Paul,

I guess I should provide an example for my previous statement:

all:: first
        @echo one
all:: second
        @echo two
first:
        @echo first
second:
        @echo second

In GNUmake 3.75, this produced:

        first
        one
        second
        two

In GNUmake 3.80, this produced:

        first
        second
        one
        two

On the surface, this might not appear to be a big deal -- unless the
order of execution between first & second is important.

In the larger picture, we like to intermix serialism and parallelism in
the same makefile.  In 3.80, if you choose parallelism, serialism is
impossible.  However, in 3.75, this worked just fine.

Why did the functionality change?

Reid

-----Original Message-----
From: Paul Smith [mailto:[hidden email]] On Behalf Of Paul D. Smith
Sent: Tuesday, June 14, 2005 7:48 AM
To: Madsen, Reid
Cc: [hidden email]
Subject: Re: Error in processing of :: targets

%% [hidden email] writes:

  rm> I've run into a functional difference between the standard unix
  rm> make and GNUmake (3.80) involving how the :: rules are processed.

By "standard unix make" you seem to mean SystemV make.  Note that the
only official Standard-with-a-capital-S (recognized by the FSF and GNU
project) for make is the POSIX standard.  The POSIX standard doesn't
address this issue (it doesn't address parallelism at all).

SystemV make is, at best, a de facto standard, and there are plenty of
other (incompatible) versions of make out there.

In fact, GNU make is not intended to be a drop-in replacement for
SystemV make and there are _several_ points at which the behavior of the
two versions diverge.

  rm> In the standard unix make (Solaris), the :: targets ( i.e., all::)
  rm> are process in the order they are found in the Makefile serially,
  rm> regardless of whether parallelism is used (i.e., -jN, where N is >
  rm> 1).

  rm> In GNUmake (3.80), the when -j1 is used, the "::" targets are
  rm> executed serially.  When -jN is used the "::" targets are executed
  rm> in parallel.

GNU make's behavior here is exactly in keeping with the definition of
double-colon rules in the GNU make manual, which states:

>    When a target appears in multiple rules, all the rules must be the
> same type: all ordinary, or all double-colon.  If they are
> double-colon, each of them is independent of the others.  Each
> double-colon rule's commands are executed if the target is older than
> any prerequisites of that rule.  If there are no prerequisites for
> that rule, its commands are always executed (even if the target
> already exists).  This can result in executing none, any, or all of
> the double-colon rules.
>
>    Double-colon rules with the same target are in fact completely
> separate from one another.  Each double-colon rule is processed
> individually, just as rules with different targets are processed.

If you think a different set of behavior is desirable, the best thing is
probably to file an enhancement request on the Savannah
(https://savannah.gnu.org/) project page for GNU make and someone will
look at it.

Cheers!

--
------------------------------------------------------------------------
-------
 Paul D. Smith <[hidden email]>          Find some GNU make tips at:
 http://www.gnu.org                      http://make.paulandlesley.org
 "Please remain calm...I may be mad, but I am a professional." --Mad
Scientist


_______________________________________________
Bug-make mailing list
[hidden email]
http://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

RE: Error in processing of :: targets

reid.madsen
In reply to this post by reid.madsen

Howard,

I thought I was using Sun's make, but in fact I was using GNUmake V3.75.
And in that version this works perfectly.  Meaning that the commands for
all.prologue are executed to completion before aaa, bbb, and ccc, are
build, and the commands for all.epilogue are executed AFTER aaa, bbb,
and ccc are built.

So you think it is bogus.  Then how would you implement this makefile
(which worked fine in 3.75 GNUmake) to achieve the described ordering?
Specifically the commands of all.prologue must execute BEFORE aaa, bbb,
or ccc.

        all:: all.prologue
        all:: aaa bbb ccc
        all:: all.epilogue

        all.prologue:: ; @echo prologue
        all.epilogue:: ; @echo epilogue
        aaa bbb ccc: ; @echo $@

Sure, you could add:

        aaa: all.prologue
        bbb: all.prologue
        ccc: all.prologue
and
        all.epilogue:: aaa bbb ccc

But assuming that every rule writer -- in a complex build system -- will
remember to do this is an unsafe assumption.  And it makes no difference
whether this Makefile was generated by IMAKE or by hand -- the problem
is the same.

In version 3.75, each :: rule and its dependencies were treated
separately.  Meaning that for each :: rule, the dependencies for that
rule were evaulated, and then the commands for that rule were run.
Dependencies for subequent instances of the same target were deferred
until that :: rule was evaulated.  If multiple dependencies were
specified, then they were built in parallel when the :: rule they were
mentioned in was evaluated.

In version 3.80, the dependencies for all :: rules are evaluated in
parallel before any commands for the :: target are executed.  This is a
significant change from V3.75.

Bottom line, GNUmake V3.75 provided a very useful mechanism to establish
ordering of :: rules in a parallel environment.  Thus functionality has
been lost in later versions.

Reid Madsen

-----Original Message-----
From: Howard Chu [mailto:[hidden email]]
Sent: Tuesday, June 14, 2005 8:21 AM
To: Madsen, Reid
Cc: [hidden email]
Subject: Re: Error in processing of :: targets

[hidden email] wrote:
> Gnumake maintainers,
>  
> I've run into a functional difference between the standard unix make
> and GNUmake (3.80) involving how the :: rules are processed.
>  
> In the standard unix make (Solaris), the :: targets ( i.e., all::) are

> process in the order they are found in the Makefile serially,
> regardless of whether parallelism is used (i.e., -jN, where N is > 1).

That's funny, my "standard Unix" (Solaris) make doesn't have a "-j"
switch, and doesn't support parallelism. In fact, -j is a GNU-specific
extension, and it Works As Designed.
>  
> In GNUmake (3.80), the when -j1 is used, the "::" targets are executed

> serially.  When -jN is used the "::" targets are executed in parallel.

> Consider the following -- from an IMAKE generated Makefile:

This, like most of the dreck that Imake produces, is bogus.

http://lists.gnu.org/archive/html/bug-make/2001-09/msg00013.html
http://lists.gnu.org/archive/html/bug-make/2004-09/msg00001.html

> all:: all.prologue
> all:: all.serial all.parallel
> all:: ; @echo $@; sleep 1; echo $@-done <mailto:$@-done>
> all:: all.epilogue
>  
> all.epilogue all.prologue:: ; @echo $@; sleep 2; echo $@-done
> <mailto:$@-done>
> all.serial:: ; @echo $@-1 <mailto:$@-1>; sleep 1; echo $@-1-done
> <mailto:$@-1-done>
> all.serial:: ; @echo $@-2 <mailto:$@-2>; sleep 1; echo $@-2-done
> <mailto:$@-2-done>
> all.parallel: aaa bbb ccc
> aaa bbb ccc: ; @echo $@; sleep 1; echo $@-done <mailto:$@-done>

--
   -- Howard Chu
   Chief Architect, Symas Corp.       Director, Highland Sun
   http://www.symas.com               http://highlandsun.com/hyc
   Symas: Premier OpenSource Development and Support


_______________________________________________
Bug-make mailing list
[hidden email]
http://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

RE: Error in processing of :: targets

Paul Smith-20
In reply to this post by reid.madsen
%% [hidden email] writes:

  rm> all:: first
  rm> @echo one
  rm> all:: second
  rm> @echo two
  rm> first:
  rm> @echo first
  rm> second:
  rm> @echo second

  rm> In GNUmake 3.75, this produced:

  rm> first
  rm> one
  rm> second
  rm> two

  rm> In GNUmake 3.80, this produced:

  rm> first
  rm> second
  rm> one
  rm> two

I was looking through the NEWS file for GNU make and found this:

> Version 3.63
>
    [...]
>
> * Multiple double-colon rules for the same target will no longer have
>   their commands run simultaneously under -j, as this could result in
>   the two commands trying to change the file at the same time and
>   interfering with one another.

So, it looks like you're right: double-colon rules should be invoked
serially.  I guess the docs weren't changed to make this clear when the
code was updated.  Obviously something changed between 3.75 and now to
break that: I definitely didn't change it on purpose.

Please file a bug at Savannah against GNU make and I'll look into it at
some point.

--
-------------------------------------------------------------------------------
 Paul D. Smith <[hidden email]>          Find some GNU make tips at:
 http://www.gnu.org                      http://make.paulandlesley.org
 "Please remain calm...I may be mad, but I am a professional." --Mad Scientist


_______________________________________________
Bug-make mailing list
[hidden email]
http://lists.gnu.org/mailman/listinfo/bug-make