Erroneously not updating intermediate/secondary dependency

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Erroneously not updating intermediate/secondary dependency

Luke Shumaker
I believe that I have found a bug present in both Make 4.2.1 (as shipped by Arch
Linux) and in the latest git commit (214865ed5c66d8e363b16ea74509f23d93456707).

Here is a simple Makefile demonstrating the bug:

        all: a-derived.out
        .PHONY: all
       
        kindaclean:
                rm -f -- *.out
        clean: kindaclean
                rm -f -- a-derived.src
        .PHONY: kindaclean clean
       
        %.out: %.src
                { echo 'build'; date; ls -l $^; } > $@
       
        a-derived.src: a.out
                { echo 'generate'; date; ls -l $^; } > $@
        a-derived.out: a.out # Make sure that "a.out" isn't skipped as intermediate
       
        .SECONDARY:

For clarity, here is the dependency graph; "==" indicates an explicit
rule, "--" indicates an implicit rule:

        all >===> a-derived.out >---> a-derived.src >===> a.out >---> a.src
                               `>=======================>'

The bug is that when running

        $ echo foo > a.src

        $ make
        { echo 'build'; date; ls -l a.src; } > a.out
        { echo 'generate'; date; ls -l a.out; } > a-derived.src
        { echo 'build'; date; ls -l a-derived.src a.out; } > a-derived.out

        $ make kindaclean
        rm -f -- *.out

        $ make
        { echo 'build'; date; ls -l a.src; } > a.out
        { echo 'build'; date; ls -l a-derived.src a.out; } > a-derived.out

The "a-derived.src" does not get updated, despite both of thes
conditions being met: (1) its dependency "a.out" getting updated, and
(2) it being used in a later recipe.

We can instrument the Makefile to more explicitly demonstrate the
failure:

        define sanitycheck
          if ! test -e $1; then \
            echo '=> dependency "$(strip $1): $(strip $2)" skipped because dependent "$(strip $1)" does not exist'; \
          elif ! test -e $2; then \
            echo '=> dependency "$(strip $1): $(strip $2)" skipped because dependency "$(strip $2)" does not exist'; \
          elif test $2 -nt $1; then \
            echo '=> dependency "$(strip $1): $(strip $2)" failed because dependency "$(strip $2)" is newer than dependant "$(strip $1)"'; \
            ls -l $1 $2; \
            exit 1; \
          else \
            echo '=> dependency "$(strip $1): $(strip $2)" passed'; \
          fi
        endef
       
        all: a-derived.out
                @$(call sanitycheck, a.out         , a.src         )
                @$(call sanitycheck, a-derived.src , a.out         )
                @$(call sanitycheck, a-derived.out , a-derived.src )
        .PHONY: all
       
        kindaclean:
                rm -f -- *.out
        clean: kindaclean
                rm -f -- a-derived.src
        .PHONY: kindaclean clean
       
        %.out: %.src
                { echo 'build'; date; ls -l $^; } > $@
       
        a-derived.src: a.out
                { echo 'generate'; date; ls -l $^; } > $@
        a-derived.out: a.out # Make sure that "a.out" isn't skipped as intermediate
       
        .SECONDARY:

Which yields:

        $ echo foo > a.src

        $ make
        { echo 'build'; date; ls -l a.src; } > a.out
        { echo 'generate'; date; ls -l a.out; } > a-derived.src
        { echo 'build'; date; ls -l a-derived.src a.out; } > a-derived.out
        => dependency "a.out: a.src" passed
        => dependency "a-derived.src: a.out" passed
        => dependency "a-derived.out: a-derived.src" passed

        $ make kindaclean
        rm -f -- *.out

    $ # wait a momenent, so the timestamps are visibly different

        $ make
        { echo 'build'; date; ls -l a.src; } > a.out
        { echo 'build'; date; ls -l a-derived.src a.out; } > a-derived.out
        => dependency "a.out: a.src" passed
        => dependency "a-derived.src: a.out" failed because dependency "a.out" is newer than dependant "a-derived.src"
        -rw-r--r-- 1 lukeshu lukeshu 89 Feb 22 13:45 a-derived.src
        -rw-r--r-- 1 lukeshu lukeshu 85 Feb 22 13:47 a.out
        make: *** [Makefile:17: all] Error 1

--
Happy hacking,
~ Luke Shumaker

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: Erroneously not updating intermediate/secondary dependency

Martin Dorey-2
make kindaclean; make --debug=all says:

    Finished prerequisites of target file 'a-derived.src'.
    Prerequisite 'a.out' of target 'a-derived.src' does not exist.
   No need to remake target 'a-derived.src'.


The first difference is what happens if the intermediate file does not exist. If an ordinary file b does not exist, and make considers a target that depends on b, it invariably creates b and then updates the target from b. But if b is an intermediate file, then make can leave well enough alone. It won’t bother updating b, or the ultimate target, unless some prerequisite of b is newer than that target or there is some other reason to update that target.

By using .SECONDARY with no prerequisites, you've marked everything as intermediate.  Marking just a.out as .SECONDARY seems enough to cause the problem.


From: Bug-make <bug-make-bounces+martin.dorey=[hidden email]> on behalf of Luke Shumaker <[hidden email]>
Sent: Friday, February 22, 2019 11:45
To: [hidden email]
Subject: Erroneously not updating intermediate/secondary dependency
 
***** EXTERNAL EMAIL *****

I believe that I have found a bug present in both Make 4.2.1 (as shipped by Arch
Linux) and in the latest git commit (214865ed5c66d8e363b16ea74509f23d93456707).

Here is a simple Makefile demonstrating the bug:

        all: a-derived.out
        .PHONY: all

        kindaclean:
                rm -f -- *.out
        clean: kindaclean
                rm -f -- a-derived.src
        .PHONY: kindaclean clean

        %.out: %.src
                { echo 'build'; date; ls -l $^; } > $@

        a-derived.src: a.out
                { echo 'generate'; date; ls -l $^; } > $@
        a-derived.out: a.out # Make sure that "a.out" isn't skipped as intermediate

        .SECONDARY:

For clarity, here is the dependency graph; "==" indicates an explicit
rule, "--" indicates an implicit rule:

        all >===> a-derived.out >---> a-derived.src >===> a.out >---> a.src
                               `>=======================>'

The bug is that when running

        $ echo foo > a.src

        $ make
        { echo 'build'; date; ls -l a.src; } > a.out
        { echo 'generate'; date; ls -l a.out; } > a-derived.src
        { echo 'build'; date; ls -l a-derived.src a.out; } > a-derived.out

        $ make kindaclean
        rm -f -- *.out

        $ make
        { echo 'build'; date; ls -l a.src; } > a.out
        { echo 'build'; date; ls -l a-derived.src a.out; } > a-derived.out

The "a-derived.src" does not get updated, despite both of thes
conditions being met: (1) its dependency "a.out" getting updated, and
(2) it being used in a later recipe.

We can instrument the Makefile to more explicitly demonstrate the
failure:

        define sanitycheck
          if ! test -e $1; then \
            echo '=> dependency "$(strip $1): $(strip $2)" skipped because dependent "$(strip $1)" does not exist'; \
          elif ! test -e $2; then \
            echo '=> dependency "$(strip $1): $(strip $2)" skipped because dependency "$(strip $2)" does not exist'; \
          elif test $2 -nt $1; then \
            echo '=> dependency "$(strip $1): $(strip $2)" failed because dependency "$(strip $2)" is newer than dependant "$(strip $1)"'; \
            ls -l $1 $2; \
            exit 1; \
          else \
            echo '=> dependency "$(strip $1): $(strip $2)" passed'; \
          fi
        endef

        all: a-derived.out
                @$(call sanitycheck, a.out         , a.src         )
                @$(call sanitycheck, a-derived.src , a.out         )
                @$(call sanitycheck, a-derived.out , a-derived.src )
        .PHONY: all

        kindaclean:
                rm -f -- *.out
        clean: kindaclean
                rm -f -- a-derived.src
        .PHONY: kindaclean clean

        %.out: %.src
                { echo 'build'; date; ls -l $^; } > $@

        a-derived.src: a.out
                { echo 'generate'; date; ls -l $^; } > $@
        a-derived.out: a.out # Make sure that "a.out" isn't skipped as intermediate

        .SECONDARY:

Which yields:

        $ echo foo > a.src

        $ make
        { echo 'build'; date; ls -l a.src; } > a.out
        { echo 'generate'; date; ls -l a.out; } > a-derived.src
        { echo 'build'; date; ls -l a-derived.src a.out; } > a-derived.out
        => dependency "a.out: a.src" passed
        => dependency "a-derived.src: a.out" passed
        => dependency "a-derived.out: a-derived.src" passed

        $ make kindaclean
        rm -f -- *.out

    $ # wait a momenent, so the timestamps are visibly different

        $ make
        { echo 'build'; date; ls -l a.src; } > a.out
        { echo 'build'; date; ls -l a-derived.src a.out; } > a-derived.out
        => dependency "a.out: a.src" passed
        => dependency "a-derived.src: a.out" failed because dependency "a.out" is newer than dependant "a-derived.src"
        -rw-r--r-- 1 lukeshu lukeshu 89 Feb 22 13:45 a-derived.src
        -rw-r--r-- 1 lukeshu lukeshu 85 Feb 22 13:47 a.out
        make: *** [Makefile:17: all] Error 1

--
Happy hacking,
~ Luke Shumaker

_______________________________________________
Bug-make mailing list
[hidden email]
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.gnu.org%2Fmailman%2Flistinfo%2Fbug-make&amp;data=01%7C01%7CMartin.Dorey%40hitachivantara.com%7C39c24e17d2914b68964308d698fed009%7C18791e1761594f52a8d4de814ca8284a%7C0&amp;sdata=6uJzMYFWkMGPLpN7OCR32wm2d7lCsdQ0vX7KhL3YRbA%3D&amp;reserved=0

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: Erroneously not updating intermediate/secondary dependency

Luke Shumaker
On Fri, 22 Feb 2019 15:39:25 -0500,
Martin Dorey wrote:
>
> make kindaclean; make --debug=all says:
>
>     Finished prerequisites of target file 'a-derived.src'.
>     Prerequisite 'a.out' of target 'a-derived.src' does not exist.
>    No need to remake target 'a-derived.src'.

Let's look at the full output, with --debug lines numbered so that we
may refer to them (and interesting lines marked with "!"):

             $ make
             …
             $ make kindaclean
             …
             $ make --debug=basic,implicit,verbose --no-builtin-rules -j1
          1  GNU Make 4.2.1
          2  Built for x86_64-pc-linux-gnu
          3  Copyright (C) 1988-2016 Free Software Foundation, Inc.
          4  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
          5  This is free software: you are free to change and redistribute it.
          6  There is NO WARRANTY, to the extent permitted by law.
          7  Reading makefiles...
          8  Reading makefile 'Makefile'...
          9  Updating makefiles....
         10  Updating goal targets....
         11  Considering target file 'all'.
         12   File 'all' does not exist.
         13    Looking for an implicit rule for 'a-derived.out'.
         14    Trying pattern rule with stem 'a-derived'.
         15    Trying implicit prerequisite 'a-derived.src'.
         16    Found an implicit rule for 'a-derived.out'.
         17     Looking for an implicit rule for 'a.out'.
         18     Trying pattern rule with stem 'a'.
         19     Trying implicit prerequisite 'a.src'.
         20     Found an implicit rule for 'a.out'.
         21      Considering target file 'a.src'.
         22       Looking for an implicit rule for 'a.src'.
         23       No implicit rule found for 'a.src'.
         24       Finished prerequisites of target file 'a.src'.
         25      No need to remake target 'a.src'.
         26   Considering target file 'a-derived.out'.
         27    File 'a-derived.out' does not exist.
         28     Considering target file 'a-derived.src'.
         29        Pruning file 'a.src'.
         30      Finished prerequisites of target file 'a-derived.src'.
        !31      Prerequisite 'a.out' of target 'a-derived.src' does not exist.
        !32     No need to remake target 'a-derived.src'.
         33      Pruning file 'a.src'.
         34    Considering target file 'a.out'.
         35     File 'a.out' does not exist.
         36      Pruning file 'a.src'.
         37     Finished prerequisites of target file 'a.out'.
        !38    Must remake target 'a.out'.
             { echo 'build'; date; ls -l a.src; } > a.out
         39    Successfully remade target file 'a.out'.
         40    Finished prerequisites of target file 'a-derived.out'.
         41   Must remake target 'a-derived.out'.
             { echo 'build'; date; ls -l a-derived.src a.out; } > a-derived.out
         42   Successfully remade target file 'a-derived.out'.
         43   Finished prerequisites of target file 'all'.
         44  Must remake target 'all'.
             => dependency "a.out: a.src" passed
             => dependency "a-derived.src: a.out" failed because dependency "a.out" is newer than dependant "a-derived.src"
             -rw-r--r-- 1 luke users 84 Feb 22 18:38 a-derived.src
             -rw-r--r-- 1 luke users 80 Feb 22 18:40 a.out
             make: *** [Makefile:17: all] Error 1

On line 32 it makes the decision to skip updating a-derived.src
because of the premise on line 31.  However, on line 38 it invalidates
the premise, but doesn't reconsider decisions made based on that
premise.

Put another way, it makes the decision on line 32 too early, before it
has enough information.

>
> https://www.gnu.org/software/make/manual/html_node/Chained-Rules.html#Chained-Rules teaches:
>
>  The first difference is what happens if the intermediate file does
>  not exist. If an ordinary file b does not exist, and make considers
>  a target that depends on b, it invariably creates b and then
>  updates the target from b. But if b is an intermediate file, then
>  make can leave well enough alone. It won’t bother updating b, or
>  the ultimate target, unless some prerequisite of b is newer than
>  that target or there is some other reason to update that target.

In this example b='a.out'.  Thus with reading the quoted text, if Make
had decided to not "bother updating b [a.out]", then that would be one
thing.  The problem is that it *does* update b=a.out, but doesn't
trickle that through the DAG (failing to update 'a-derived.src').

> By using .SECONDARY with no prerequisites, you've marked everything
> as intermediate.

That is accurate.

>                   Marking just a.out as .SECONDARY seems enough to
> cause the problem.

That is also accurate.

It seems that this may be an appropriate time to lament that there is
no way to globally disable intermediate file removal without opting
in to the "nodes in the DAG don't have to exist" semantics.

--
Happy hacking,
~ Luke Shumaker

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make