[bug #58735] if depending on include, gmake does not execute commands for out of date targets in the right order

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

[bug #58735] if depending on include, gmake does not execute commands for out of date targets in the right order

David Boyce-5
URL:
  <https://savannah.gnu.org/bugs/?58735>

                 Summary: if depending on include, gmake does not execute
commands for out of date targets in the right order
                 Project: make
            Submitted by: schily
            Submitted on: Wed 08 Jul 2020 03:10:27 PM UTC
                Severity: 3 - Normal
              Item Group: None
                  Status: None
                 Privacy: Public
             Assigned to: None
             Open/Closed: Open
         Discussion Lock: Any
       Component Version: None
        Operating System: None
           Fixed Release: None
           Triage Status: None

    _______________________________________________________

Details:

gmake, when in parallel mode, executes commands for out of date files needed
by the include directive in a random order instead of executing them in the
order needed by the include directives.

It even tries to read the "include file" before "file" has been created by the
related rule command.

This causes files to be frequently created after the "include" directive that
likes to read these files is run.

This is related to a bug report about a wrong include concept in gmake that I
made in 1998 already.

BTW: rules for out of date makefiles that are on the command line are handled
correctly.




    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?58735>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/


Reply | Threaded
Open this post in threaded view
|

[bug #58735] if depending on include, gmake does not execute commands for out of date targets in the right order

David Boyce-5
Follow-up Comment #1, bug #58735 (project make):

I'm not sure I understand the issue being reported: an example would help.

Make works by first reading the initial makefile(s) and all (existing)
included files.  During this step it doesn't complain about missing makefiles.
 Then after that is all completed, make tries to rebuild all the makefiles
(existing or not).  If any fail to rebuild (and were not included with
"-include" / "sinclude"), then make fails.  If all were rebuilt, then make
starts over from scratch and re-reads everything.

It is true that currently (due to an implementation detail) they are built in
reverse order, as if you'd typed "make inc3.mk inc2.mk inc1.mk Makefile".  If
you have correct dependencies for your included makefiles this shouldn't
matter, but that can easily be changed so they are built in the same order
they were encountered.

Of course if you enable parallelism you really have to have correct
dependencies anyway or you can't be sure what order things will actually run
in.

Is the reverse order the issue you're concerned with?  I'm not sure what you
meant by "wrong include concept" reported in 1998.

If you mean, you think that if make encounters a missing included file, it
should stop and try to build that makefile immediately, then continue to read,
well, that's one way it could work but that's not how it works.  There are
some advantages to that, but there are also advantages to the way make works
today.  I haven't seen anything showing it would be worth the huge
backward-compatibility break to change the way make works in this respect.

    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?58735>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/


Reply | Threaded
Open this post in threaded view
|

[bug #58735] if depending on include, gmake does not execute commands for out of date targets in the right order

David Boyce-5
Follow-up Comment #2, bug #58735 (project make):

Thank you for confirming that gmake executes the commands for out of date
targets that result from the include directive in the inverse (which is the
wrong) order. This wrong order causes gmake or the compiler to try to access
files that have not yet been made - or causes gmake to complain they are
missing even though they already exist at the time, the error message is
displayed (see my other bug report).

***
Make always executes the commands for all Makefiles and other files it reads
via the include directive just before it actually tries to open the related
file for the first time. This happens, whenever a related rule exists at that
time that has already been read from a Makefile, from an included file or from
the so called built-in rules.
***

Unfortunately this is not how gmake works under all circumstances. For
Makefiles, this also applies to gmake, a Makefile that is not present, but may
e.g. be retrieved via SCCS is correctly retrieved by gmake before it opens
that Makefile for reading. But for files related to the include directive, you
have been describing the behavior currently implemented by gmake, which is
incorrect.

Just to remind you, this is a bug that I reported (in a partly different way)
in 1998 already and at that time, you admitted this is a bug that would take
some time for fixing. 22 years is "some time", so when will this bug be
fixed?

Given that the include directive is a make feature that exists longer than
gmake, I expect gmake to implement the orthognal behavior people know from
other, older implemenations.

My general problem here is that gmake has more than a single bug that cause
problems on platforms that come with gmake installed as as "make". The
problems I see, partially result from attempts to work around the gmake
problems. Some of the problems appear in serial mode, others in parallel mode.
I currently have a workaround concept for gmake that works in serial mode, but
not in parallel mode and people are asking for parallel compilations.

    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?58735>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/


Reply | Threaded
Open this post in threaded view
|

[bug #58735] When rebuilding makefiles, make tries them in reverse order

David Boyce-5
Update of bug #58735 (project make):

                 Summary: if depending on include, gmake does not execute
commands for out of date targets in the right order => When rebuilding
makefiles, make tries them in reverse order

    _______________________________________________________

Follow-up Comment #3:

> This wrong order causes gmake or the compiler to try to access files that
have not yet been made

Well, this will only happen if you do not declare the dependencies of your
included makefiles.

If you need to have inc2.mk built before inc1.mk can be built, then you should
specify that in the makefile:


include inc1.mk
inc1.mk : inc2.mk ; build inc1.mk using inc2.mk


If you provide dependency information in your makefile, then the order in
which makefiles are attempted to be rebuilt is irrelevant.

I have no problem inverting the order, but any makefile which relies on
ordering of goal targets, rather than explicit dependency relationships, is
brittle at best.

> Make always executes the commands for all Makefiles and other files it reads
via the include directive just before it actually tries to open the related
file for the first time. This happens, whenever a related rule exists at that
time that has already been read from a Makefile, from an included file or from
the so called built-in rules.

I'm not sure where this quoted text comes from.  It doesn't come from anything
I ever said, or anything in the GNU make manual.  GNU make has never worked
like that, and it will never work like that.

> Just to remind you, this is a bug that I reported (in a partly different
way) in 1998 already and at that time, you admitted this is a bug

Unfortunately, I do not have archives going back past 2000 available to me.
If you can point me to such archives that would be helpful.  However, there
must have been a miscommunication.  At no time have I ever considered make's
meta-behavior here (reading all makefiles first, then rebuilding the out of
date ones) to be a bug.

> Given that the include directive is a make feature that exists longer than
gmake, I expect gmake to implement the orthognal behavior people know from
other, older implemenations.

Well, I can't speak to what Roland or Richard knew or did not know, or should
have known, about "other implementations" at the time they implemented GNU
make's current behavior back in the 1980's.  However, GNU make's current
algorithm for rebuilding out of date makefiles is well-defined and has
important advantages over the model you quote above, and those advantages are
actively being used by (many) thousands of makefiles.  Changing GNU make's
"include" to work in the way described above is completely out of the
question.

I'm sorry if an email I sent caused you to think that there was a chance that
this would happen someday: I certainly didn't intend to imply that.

    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?58735>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/


Reply | Threaded
Open this post in threaded view
|

[bug #58735] When rebuilding makefiles, make tries them in reverse order

David Boyce-5
Follow-up Comment #4, bug #58735 (project make):

If I provide more dependency information, gmake runs more commands in parallel
when in parallel mode and this will cause more problems since only one of
these commands needs to be run.

So this is also a problem that is related to an unhappy combination of
contradictory behavior in gmake.

The general problem is that gmake implements the include feature in a way that
is in conflict with the behavior of other, older implementations.

I cannot speak for UNIXv8 and this is not really useful to look at, since
extremely few people had access to the make experiments from UNIXv8.

I however can confirm that SunOS did introduce the so called "SunPro Make" in
January 1986 to a large community.

Gmake first appeared in late 1988. If gmake did have "include" from it's
beginning, this would have been 3 years later than SunPro Make.

Given that gmake implements a lot of the features that have been made
available to a larger community by SunPro Make in January 1986 (e.g. pattern
macro expansions and pattern matching rules for implicit targets) and given
that gmake has been originally developed on SunOS, it is obvious that gmake in
1988 intended to copy the features of SunPro Make.

You did not verify that the specific behavior of gmake is needed by any
makefile in the public. Other make implementations use a different method and
the other make implementations do not have resulting problems.

The way gmake works slows down work, as it re-execs gmake. I am however not
asking you to remove this implementation detail....

I just ask you to introduce orthogonal behavior in gmake and to evaluate
existing rules for include files the same way as it is done for Makefiles:
just before the file is opened the first time.

BTW: SunPro Make is not something from the past that no longer exists and it
is (as a result of a request from me) OpenSource since December 2006. I made
it highly portable and maintain it since 9 years and it is part of the
"schilytools" bundle; together with smake, the make implementation I wrote in
1984 and maintain since then.

So everybody can check the behavior of SunPro Make.

    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?58735>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/


Reply | Threaded
Open this post in threaded view
|

[bug #58735] When rebuilding makefiles, make tries them in reverse order

David Boyce-5
Follow-up Comment #5, bug #58735 (project make):

> If I provide more dependency information, gmake runs more commands in
parallel when in parallel mode and this will cause more problems since only
one of these commands needs to be run.

As discussed in the other bug, the right answer to this problem is to explain
to make what your target's recipes really do; hiding _more_ details from make
only make it less likely to perform the build correctly.

> The general problem is that gmake implements the include feature in a way
that is in conflict with the behavior of other, older implementations.

As I mentioned before, I'm not prepared to discuss decisions made by other
people 32+ years ago.  If you want to debate the appropriateness of that
decision you can reach out to Roland and Richard and have that conversation.
My suspicion is that they had their own ideas of how "include" should work,
and didn't think that the fact some other existing (proprietary, at the time)
make's "include" worked differently should keep them from implementing a
better method.  But, you'll have to ask them.

At this point the disparity has existed for the aforementioned 32+ years and I
have no interest in breaking GNU make makefiles in an effort to resolve it.

> You did not verify that the specific behavior of gmake is needed by any
makefile in the public.

Unless I'm misunderstanding your description of the behavior, using it would
break every GNU make makefile where the rule to create the include file
appeared _after_ the include line itself.  So every makefile that currently
contains:


include inc1.mk
inc1.mk: ; echo FOO = bar > $@


or anything similar, will break.

> I just ask you to introduce orthogonal behavior in gmake and to evaluate
existing rules for include files the same way as it is done for Makefiles:
just before the file is opened the first time.

I think there's some confusion.  GNU make only has one algorithm to rebuild
makefiles.  There's no difference in behavior for the initial makefile,
makefiles specified on the command line, or included makefiles: they all are
handled identically.

If there is no initial makefile, then obviously there are no other include
files to build and make must rely completely on built-in rules.  However, it
still follows exactly the same steps as when an initial makefile already
exists:

1) Try to load the makefile, don't fail if it doesn't exist
2) Set the goal target(s) to all makefiles (just Makefile in this case)
3) Try to rebuild those files
4) If any file needs to be rebuilt but cannot be, then:
   * If it was not included with -include or sinclude, fail
5) If any makefile was updated, start over from scratch.
6) Else, run the build

    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?58735>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/


Reply | Threaded
Open this post in threaded view
|

[bug #58735] When rebuilding makefiles, make tries them in reverse order

David Boyce-5
Follow-up Comment #6, bug #58735 (project make):

I just want to be clear: changing make to allow it to be able to rebuild
makefiles incrementally is not just a breaking change, it is also a sizeable
development effort.

Today GNU make works in three discrete stages: first it reads all makefiles,
then it organizes and completes the DAG, then it builds all the goal targets.

GNU make proceeds through each stage in order, every time.  The current
codebase does not allow going back and forth between these stages.  So it is
not possible to see an include operation, complete the DAG, rebuild the
included file, then go back and continue reading more makefiles.


    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?58735>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/


Reply | Threaded
Open this post in threaded view
|

[bug #58735] When rebuilding makefiles, make tries them in reverse order

David Boyce-5
Update of bug #58735 (project make):

                  Status:                    None => Fixed                  
             Assigned to:                    None => psmith                
             Open/Closed:                    Open => Closed                
        Operating System:                    None => Any                    
           Fixed Release:                    None => SCM                    
           Triage Status:                    None => Small Effort          


    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?58735>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/


Reply | Threaded
Open this post in threaded view
|

[bug #58735] When rebuilding makefiles, make tries them in reverse order

David Boyce-5
Follow-up Comment #7, bug #58735 (project make):

A fix for the "try to rebuild included makefiles in reverse order" issue has
been pushed to Git and will be in the next release.

Thanks for reporting!

    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?58735>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/