[bug #56892] toxic combination of -include and match-anything rules

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[bug #56892] toxic combination of -include and match-anything rules

Makoto Kato
URL:
  <https://savannah.gnu.org/bugs/?56892>

                 Summary: toxic combination of -include and match-anything
rules
                 Project: make
            Submitted by: boyski
            Submitted on: Sat 14 Sep 2019 08:28:32 PM UTC
                Severity: 3 - Normal
              Item Group: Bug
                  Status: None
                 Privacy: Public
             Assigned to: None
             Open/Closed: Open
         Discussion Lock: Any
       Component Version: 4.2.1
        Operating System: POSIX-Based
           Fixed Release: None
           Triage Status: None

    _______________________________________________________

Details:

I'm not saying this is a bug necessarily but want to submit it for some sort
of resolution. I an into an infinite loop situation recently. On my advice, in
a very old and complex recursive makefile suite (not of my design or under my
ownership) a co-worker added a construct like the following:

-include foobar.mk
foobar ?= XYZ

The idea is that foobar.mk would be a one-line include file which assigns the
"foobar" variable if it exists and if not we fall back to the default value
"XYZ". This is simple and solved his problem in unit testing but when plugged
into the large old suite it resulted in an infinite loop.

The reason is twofold: (a) if an included makefile doesn't exist make will try
to run a recipe to create it and (b) somewhere deep in this makefile suite was
a match-anything rule. From here it's obvious: make went looking for a recipe
that claimed to be able to make foobar.mk, landed on the match-anything rule
which does a recursive make invocation (and does not create foobar.mk), and
we're off to the races with an infinite make loop.

The situation is illustrated by the following test file:

$ cat Makefile
.PHONY: all
all:
        @:$(info making $@)

%:
# @test -f nosuchfile.mk
        $(MAKE) $@

#-include nosuchfile.mk

As it stands this works fine:

$ make
making all

If the -include line is enabled the infinite loop happens, but if the test -f
line is also uncommented things work correctly again.

My first thought is that it makes no sense to invoke a recipe to create a
nonexistent file included via -include because the whole point of -include is
to say "I know this file may not exist and that's ok". However, the
documentation says of -include that it:

"... acts like include in every way except that there is no error [...] if any
of the filenames [...] do not exist or cannot be remade."

I guess the "or cannot be remade" is a strong implication that -include files
are subject to remaking so maybe it would hard to "fix" this for compatibility
reasons? And of course there's a difference between a file being out of date
and not existing and it makes perfect sense to want to update a -included file
if it exists.

However, I still think that in the special case where foobar.mk does not exist
and is included with -include the remake attempt should be elided and the
inclusion silently skipped. Whether that's too much of a compatibility break
is a valid question though.

But even stepping back from those details, with either "include" or
"-include", why should make continue if it ran the recipe and the recipe did
not in fact create the file as it claimed it would? In other words, why not do
a check for existence after running the remake recipe (as illustrated in the
test case)? It seems to me that dying with "no such file or directory" would
be preferable to allowing an infinite loop, and furthermore I think this could
be added without breaking (documented) compatibility.  AFAICT the manual
discusses only situations where the included file "cannot be remade".
Currently "remaking" seems to be a synonym for "running the recipe which
claims to make it" but tightening that definition to to require that the
recipe actually creates the file seems both more robust and entirely
compatible, not to mention intuitive.

Thus I suggest one or both of these minor tweaks:

1. Fail (with include) or continue (with -include) when, after running the
remake recipe, the file does not exist.

2. Ignore (do not attempt to make) a file included by -include if it does not
exist at all. But if if does exist, go ahead and try to update it.




    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?56892>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

[bug #56892] toxic combination of -include and match-anything rules

Makoto Kato
Follow-up Comment #1, bug #56892 (project make):

It is definitely not the case that we can ignore -include if the makefile
doesn't exist.  In fact until recently, virtually every makefile that needed
to be rebuilt would be included with "-include", because otherwise you'd get a
warning that the file didn't exist before make would try to build it.

Only as of GNU make 4.2 will the warning be delayed until after make decides
whether it can be made so you don't need to use -include but lots and lots of
makefiles still do it.  See bug #102.

I'm not sure I see how your option #2 helps.  If I understand your setup
correctly it's not that make is ever re-execing itself even once.  What
happens is that make tries to rebuild nosuchfile.mk and finds a match-anything
rule that runs a recursive make, the recursive make tries to rebuild
nosuchfile.mk and finds a match-anything rule that runs a recursive make, etc.
 None of these sub-makes ever exit so we never re-exec ourselves.

In other words, it's not that the same make is re-execing itself forever but
that each instance of make is forking a new child make that forks another new
child make, etc. until presumably the system freaks out.  In other words this
seems like a fork bomb.


    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?56892>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

[bug #56892] toxic combination of -include and match-anything rules

Makoto Kato
Follow-up Comment #2, bug #56892 (project make):

Ok. WRT your second point, you are of course right and I didn't think it out
properly. WRT the delayed warning being new as of 4.2.1, I didn't know that
and agree that it's a serious compatibility issue, so I'm happy to drop this.

    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?56892>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

[bug #56892] toxic combination of -include and match-anything rules

Makoto Kato
Follow-up Comment #3, bug #56892 (project make):

I can think of one way to address these types of issues that might not be too
impactful:

Make knows both how deep inside a recursion it is (via MAKELEVEL) and also how
many times its re-exec'd itself (via MAKE_RESTARTS).  It could check to see if
either of those values seems outrageous and stop.  For example, if our
MAKELEVEL is > 100 or our MAKE_RESTARTS > 1000, or something, we'd fail with
an error.

I don't know if we'd need to provide a way for the user to reset these limits
in case they had a legitimate need for "outrageous" values.

    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?56892>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

[bug #56892] toxic combination of -include and match-anything rules

Makoto Kato
Update of bug #56892 (project make):

              Item Group:                     Bug => Enhancement            


    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?56892>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: [bug #56892] toxic combination of -include and match-anything rules

Edward Welbourne-3
In reply to this post by Makoto Kato
David Boyce (14 September 2019 22:28) wrote, on <https://savannah.gnu.org/bugs/?56892>:

> I'm not saying this is a bug necessarily but want to submit it for
> some sort of resolution. I an into an infinite loop situation
> recently. On my advice, in a very old and complex recursive makefile
> suite (not of my design or under my ownership) a co-worker added a
> construct like the following:
>
> -include foobar.mk
> foobar ?= XYZ
>
> The idea is that foobar.mk would be a one-line include file which
> assigns the "foobar" variable if it exists and if not we fall back to
> the default value "XYZ". This is simple and solved his problem in unit
> testing but when plugged into the large old suite it resulted in an
> infinite loop.
>
> The reason is twofold: (a) if an included makefile doesn't exist make
> will try to run a recipe to create it and (b) somewhere deep in this
> makefile suite was a match-anything rule. From here it's obvious: make
> went looking for a recipe that claimed to be able to make foobar.mk,
> landed on the match-anything rule which does a recursive make
> invocation (and does not create foobar.mk), and we're off to the races
> with an infinite make loop.

You can probably work round this by adding a wilfully failing rule,

foobar.mk:
        @echo "$@ can be hand-created to set foobar" >&2
        false

IIUC, since this rule is more specific than the match-anything, make
shall use it instead, fail, and happily get on with not caring that the
-include didn't happen.  The echo is, of course, optional.

        Eddy.

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make