[bug #55243] Request for a way to indicate that the same recipe execution produces several targets

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[bug #55243] Request for a way to indicate that the same recipe execution produces several targets

Paul D. Smith
URL:
  <https://savannah.gnu.org/bugs/?55243>

                 Summary: Request for a way to indicate that the same recipe
execution produces several targets
                 Project: make
            Submitted by: pacalet
            Submitted on: Wed 19 Dec 2018 12:53:09 PM UTC
                Severity: 3 - Normal
              Item Group: Enhancement
                  Status: None
                 Privacy: Public
             Assigned to: None
             Open/Closed: Open
         Discussion Lock: Any
       Component Version: 4.2.1
        Operating System: Any
           Fixed Release: None
           Triage Status: None

    _______________________________________________________

Details:

This request for enhancement concerns cases where multiple targets are
produced by one single execution of a recipe.

I frequently encounter situations where a single recipe execution produces
several targets and I'd like make to know about all these targets.

I know there is a side-effect of pattern rules with several targets that leads
make to consider that all targets matched by one activation of the rule are
produced by a single execution of the recipe.

But exploiting this sometimes leads to clumsy Makefiles where pattern rules
are used without any other reason than this side effect. Moreover, building
the list of pattern targets is sometimes difficult.

It would be much more convenient to have a new type of rule dedicated to this
situation. For instance, a triple-colon rule:


a b c::: foo
    touch a b c


Of course, there are several related questions, like, for instance:

0 the semantics of the $@ automatic variable (list of targets, first target in
the list, other?),
0 the existence or not of a corresponding (static) pattern rule,
0 ...




    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?55243>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: [bug #55243] Request for a way to indicate that the same recipe execution produces several targets

Brian Vandenberg-2
I usually instruct my co-workers to avoid making recipes along those lines as I consider it to be a form of painting yourself into a corner, though there are times where it's unavoidable.

In the build I maintain it used to have something like this:

blah.o : blah.xyz
> some command | sed 'some stuff' | some other command > blah.cc
> ${compiler and flags} blah.cc -o blah.o

There were a few problems with it:

* The 1st line of the recipe also produced a header as a side effect (hidden dependency)
* This won't fail the build: /bin/true | /bin/false | /bin/true.  That can be changed if bash is the shell your build uses (.SHELL_FLAGS := -euo pipefail -c)
* Even if one command in that pipeline fails the build it might not be obvious what the failure was.

I changed it to (extensions changed to protect the innocent):

blah.x1 : blah.xyz
> some command > ${@}

blah.x2 : blah.x1
> sed 'some stuff' > ${@}

blah.cc : blah.x2
> some other command > ${@}

blah.h : blah.cc
> no real recipe was provided, just adding this note for emphasis; this last part is how I suggest you solve it

-brian

On Wed, Dec 19, 2018 at 5:53 AM Renaud Pacalet <[hidden email]> wrote:
URL:
  <https://savannah.gnu.org/bugs/?55243>

                 Summary: Request for a way to indicate that the same recipe
execution produces several targets
                 Project: make
            Submitted by: pacalet
            Submitted on: Wed 19 Dec 2018 12:53:09 PM UTC
                Severity: 3 - Normal
              Item Group: Enhancement
                  Status: None
                 Privacy: Public
             Assigned to: None
             Open/Closed: Open
         Discussion Lock: Any
       Component Version: 4.2.1
        Operating System: Any
           Fixed Release: None
           Triage Status: None

    _______________________________________________________

Details:

This request for enhancement concerns cases where multiple targets are
produced by one single execution of a recipe.

I frequently encounter situations where a single recipe execution produces
several targets and I'd like make to know about all these targets.

I know there is a side-effect of pattern rules with several targets that leads
make to consider that all targets matched by one activation of the rule are
produced by a single execution of the recipe.

But exploiting this sometimes leads to clumsy Makefiles where pattern rules
are used without any other reason than this side effect. Moreover, building
the list of pattern targets is sometimes difficult.

It would be much more convenient to have a new type of rule dedicated to this
situation. For instance, a triple-colon rule:


a b c::: foo
    touch a b c


Of course, there are several related questions, like, for instance:

0 the semantics of the $@ automatic variable (list of targets, first target in
the list, other?),
0 the existence or not of a corresponding (static) pattern rule,
0 ...




    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?55243>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: [bug #55243] Request for a way to indicate that the same recipe execution produces several targets

Renaud Pacalet
On 22/12/2018 02:10, Brian Vandenberg wrote:

> I usually instruct my co-workers to avoid making recipes along those
> lines as I consider it to be a form of painting yourself into a corner,
> though there are times where it's unavoidable.
>
> In the build I maintain it used to have something like this:
>
> blah.o : blah.xyz <http://blah.xyz>
>> some command | sed 'some stuff' | some other command > blah.cc
>> ${compiler and flags} blah.cc -o blah.o
>
> There were a few problems with it:
>
> * The 1st line of the recipe also produced a header as a side effect
> (hidden dependency)
> * This won't fail the build: /bin/true | /bin/false | /bin/true.  That
> can be changed if bash is the shell your build uses (.SHELL_FLAGS :=
> -euo pipefail -c)
> * Even if one command in that pipeline fails the build it might not be
> obvious what the failure was.
>
> I changed it to (extensions changed to protect the innocent):
>
> blah.x1 : blah.xyz <http://blah.xyz>
>> some command > ${@}
>
> blah.x2 : blah.x1
>> sed 'some stuff' > ${@}
>
> blah.cc : blah.x2
>> some other command > ${@}
>
> blah.h : blah.cc
>> no real recipe was provided, just adding this note for emphasis; this
> last part is how I suggest you solve it
Perfect but this does not work in the situations I am interested in,
that is where one single atomic command produces several files. Example:
pdflatex produces several files and there are cases where you want to
track some of them, e.g. document.pdf and document.aux.
--
Renaud

>
> -brian
>
> On Wed, Dec 19, 2018 at 5:53 AM Renaud Pacalet <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     URL:
>       <https://savannah.gnu.org/bugs/?55243>
>
>                      Summary: Request for a way to indicate that the
>     same recipe
>     execution produces several targets
>                      Project: make
>                 Submitted by: pacalet
>                 Submitted on: Wed 19 Dec 2018 12:53:09 PM UTC
>                     Severity: 3 - Normal
>                   Item Group: Enhancement
>                       Status: None
>                      Privacy: Public
>                  Assigned to: None
>                  Open/Closed: Open
>              Discussion Lock: Any
>            Component Version: 4.2.1
>             Operating System: Any
>                Fixed Release: None
>                Triage Status: None
>
>         _______________________________________________________
>
>     Details:
>
>     This request for enhancement concerns cases where multiple targets are
>     produced by one single execution of a recipe.
>
>     I frequently encounter situations where a single recipe execution
>     produces
>     several targets and I'd like make to know about all these targets.
>
>     I know there is a side-effect of pattern rules with several targets
>     that leads
>     make to consider that all targets matched by one activation of the
>     rule are
>     produced by a single execution of the recipe.
>
>     But exploiting this sometimes leads to clumsy Makefiles where
>     pattern rules
>     are used without any other reason than this side effect. Moreover,
>     building
>     the list of pattern targets is sometimes difficult.
>
>     It would be much more convenient to have a new type of rule
>     dedicated to this
>     situation. For instance, a triple-colon rule:
>
>
>     a b c::: foo
>         touch a b c
>
>
>     Of course, there are several related questions, like, for instance:
>
>     0 the semantics of the $@ automatic variable (list of targets, first
>     target in
>     the list, other?),
>     0 the existence or not of a corresponding (static) pattern rule,
>     0 ...
>
>
>
>
>         _______________________________________________________
>
>     Reply to this item at:
>
>       <https://savannah.gnu.org/bugs/?55243>
>
>     _______________________________________________
>       Message sent via Savannah
>       https://savannah.gnu.org/
>
>
>     _______________________________________________
>     Bug-make mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://lists.gnu.org/mailman/listinfo/bug-make
>

--
Renaud Pacalet
Télécom ParisTech - LabSoC, c/o EURECOM
Campus SophiaTech,
450 Route des Chappes, CS 50193 - 06904 Biot Sophia Antipolis cedex, FRANCE
Tel : +33 (0) 4 9300 8402
Web : http://www.telecom-paristech.fr/


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [bug #55243] Request for a way to indicate that the same recipe execution produces several targets

Brian Vandenberg-2
> blah.h : blah.cc
>> no real recipe was provided, just adding this note for emphasis; this
> last part is how I suggest you solve it

Perfect but this does not work in the situations I am interested in,
that is where one single atomic command produces several files. Example:
pdflatex produces several files and there are cases where you want to
track some of them, e.g. document.pdf and document.aux.
--
Renaud

I may be mistaken, but this sounds like a problem I tried tackling: force PDF's to re-build if any of its dependencies changed.  I started with pdftex's "-recorder" flag, but it doesn't allow you to control output filename / directory and there were other problems that made it completely un-suited to generating a dependency list.  After looking at all the docs, what others on the internet tried, etc, I was ready to throw in the towel.

The following ugly hack came about as I was thinking about how nice it'd be if I had something "magic" (eg, tup's use of fuse) to 'discover' what files were accessed; this is roughly what I threw together:

$ cat blah.sh
#!/bin/bash
set -euo pipefail
strace -y -f -o blah.dep.tmp pdflatex -halt-on-error -interaction=batchmode "${@}"
perl -ni -e 'while( m{<([^>]+)>}g ) { my $tmp = $1; $tmp =~ m{'^$(pwd)'} && print "$tmp\n"}' blah.dep.tmp
echo "${@} : $(sort -u blah.dep.tmp)" > blah.dep

$ cat makefile
-include $(shell find docs -name '*.dep')
# note lack of dependencies listed; they come from the .dep files
%.pdf :
> blah.sh ${@}

Explanation:
* First time the build runs .dep files don't exist and it'll just assume everything is ok; subsequent builds will have all necessary dependencies
* The "-y" flag in strace causes it to emit paths or descriptive text (eg, socket:12344321) in <>s for file descriptor arguments to functions:
fstat(3</path/to/some/file>, {st_mode=S_IFREG|0644, st_size=170187, ...}) = 0
* The paths within <> are always fully qualified; the perl one-liner extracts each path and if it is a descendent of $(pwd) it gets printed; obviously the paths need to be filtered further to distinguish between output -vs- input, etc; the above is just a rough sketch

Problems:
* It's stupid; clever, but stupid
* Platforms without strace would require a different solution
* The "-y" flag is a relatively recent addition to strace; without it, extracting valid paths is more error prone.

dtrace (solaris/bsd) and system tap (linux) scripts are also an option; in fact if I weren't looking at moving to tup I'd probably go that route.  The only hurdle there is that execution of the script would either require the user to have special permissions or the script needs to be owned by a user with those special permissions & marked setuid so normal users can execute it.

-brian


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: [bug #55243] Request for a way to indicate that the same recipe execution produces several targets

David Boyce-3
I've only had time to skim this so apologies if my reply is (even more) off topic but the use of strace implies that you're doing what I call file auditing, tracking file usage via open system calls and the like. In this context I could mention a couple of my old OSS projects:

1. Poor Man's File Audit: https://github.com/boyski/pmaudit

This is a very lightweight way of autodiscovering file usage by monitoring metadata (atime, mtime). I won't say more here since it would just repeat the README.


This is a more involved C program for Linux use which works by intercepting system calls in user space which I had to abandon some years ago due to day-job pressures. Roughly analogous to strace which at this point may be a better choice unless you get into advanced usage. You definitely don't want to touch the server side of this, which was being rewritten when I had to abandon it, but the client worked pretty well when I dropped it. Close to dead but mentioned for completeness.

David Boyce

On Fri, Dec 28, 2018 at 5:36 PM Brian Vandenberg <[hidden email]> wrote:
> blah.h : blah.cc
>> no real recipe was provided, just adding this note for emphasis; this
> last part is how I suggest you solve it

Perfect but this does not work in the situations I am interested in,
that is where one single atomic command produces several files. Example:
pdflatex produces several files and there are cases where you want to
track some of them, e.g. document.pdf and document.aux.
--
Renaud

I may be mistaken, but this sounds like a problem I tried tackling: force PDF's to re-build if any of its dependencies changed.  I started with pdftex's "-recorder" flag, but it doesn't allow you to control output filename / directory and there were other problems that made it completely un-suited to generating a dependency list.  After looking at all the docs, what others on the internet tried, etc, I was ready to throw in the towel.

The following ugly hack came about as I was thinking about how nice it'd be if I had something "magic" (eg, tup's use of fuse) to 'discover' what files were accessed; this is roughly what I threw together:

$ cat blah.sh
#!/bin/bash
set -euo pipefail
strace -y -f -o blah.dep.tmp pdflatex -halt-on-error -interaction=batchmode "${@}"
perl -ni -e 'while( m{<([^>]+)>}g ) { my $tmp = $1; $tmp =~ m{'^$(pwd)'} && print "$tmp\n"}' blah.dep.tmp
echo "${@} : $(sort -u blah.dep.tmp)" > blah.dep

$ cat makefile
-include $(shell find docs -name '*.dep')
# note lack of dependencies listed; they come from the .dep files
%.pdf :
> blah.sh ${@}

Explanation:
* First time the build runs .dep files don't exist and it'll just assume everything is ok; subsequent builds will have all necessary dependencies
* The "-y" flag in strace causes it to emit paths or descriptive text (eg, socket:12344321) in <>s for file descriptor arguments to functions:
fstat(3</path/to/some/file>, {st_mode=S_IFREG|0644, st_size=170187, ...}) = 0
* The paths within <> are always fully qualified; the perl one-liner extracts each path and if it is a descendent of $(pwd) it gets printed; obviously the paths need to be filtered further to distinguish between output -vs- input, etc; the above is just a rough sketch

Problems:
* It's stupid; clever, but stupid
* Platforms without strace would require a different solution
* The "-y" flag is a relatively recent addition to strace; without it, extracting valid paths is more error prone.

dtrace (solaris/bsd) and system tap (linux) scripts are also an option; in fact if I weren't looking at moving to tup I'd probably go that route.  The only hurdle there is that execution of the script would either require the user to have special permissions or the script needs to be owned by a user with those special permissions & marked setuid so normal users can execute it.

-brian

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

[bug #55243] Request for a way to indicate that the same recipe execution produces several targets

Paul D. Smith
In reply to this post by Paul D. Smith
Update of bug #55243 (project make):

                  Status:                    None => Duplicate              
             Open/Closed:                    Open => Closed                

    _______________________________________________________

Follow-up Comment #1:

Duplicate of bug #8297

    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?55243>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make