Feature request / patch: dependency-only prerequisites

classic Classic list List threaded Threaded
25 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Feature request / patch: dependency-only prerequisites

Christof Warlich

Dear all,

please bear with me if I'm doing something wrong here, this is the first time that I'm trying to contribute to GNU Make.

The attached patch would add a minor (but imho useful) feature to GNU Make. Here is an extract of the (changed) documentation (changes are in red), giving a quite comprehensive idea as to why this feature would be useful:

4.3 Types of Prerequisites

There are actually three different types of prerequisites understood by GNU make: normal prerequisites such as described in the previous section, order-only prerequisites, and dependency-only prerequisites.

A normal prerequisite makes two statements: first, it imposes an order in which recipes will be invoked: the recipes for all prerequisites of a target will be completed before the recipe for the target is run. Second, it imposes a dependency relationship: if any prerequisite is newer than the target, then the target is considered out-of-date and must be rebuilt.

Normally, this is exactly what you want: if a target’s prerequisite is updated, then the target should also be updated.

Occasionally, however, you have a situation where you want to impose a specific ordering on the rules to be invoked without forcing the target to be updated if one of those rules is executed. In that case, you want to define order-only prerequisites. Order-only prerequisites can be specified by placing a pipe symbol (|) in the prerequisites list: any prerequisites to the left of the pipe symbol are normal; any prerequisites to the right are order-only:

targets : normal-prerequisites | order-only-prerequisites

The normal prerequisites section may of course be empty. Also, you may still declare multiple lines of prerequisites for the same target: they are appended appropriately (normal prerequisites are appended to the list of normal prerequisites; order-only prerequisites are appended to the list of order-only prerequisites). Note that if you declare the same file to be both a normal and an order-only prerequisite, the normal prerequisite takes precedence (since they have a strict superset of the behavior of an order-only prerequisite).

Consider an example where your targets are to be placed in a separate directory, and that directory might not exist before make is run. In this situation, you want the directory to be created before any targets are placed into it but, because the timestamps on directories change whenever a file is added, removed, or renamed, we certainly don’t want to rebuild all the targets whenever the directory’s timestamp changes. One way to manage this is with order-only prerequisites: make the directory an order-only prerequisite on all the targets:

OBJDIR := objdir
OBJS := $(addprefix $(OBJDIR)/,foo.o bar.o baz.o)

$(OBJDIR)/%.o : %.c
        $(COMPILE.c) $(OUTPUT_OPTION) $<

all: $(OBJS)

$(OBJS): | $(OBJDIR)

$(OBJDIR):
        mkdir $(OBJDIR)

Now the rule to create the objdir directory will be run, if needed, before any ‘.o’ is built, but no ‘.o’ will be built because the objdir directory timestamp changed.

Finally, the third type of prerequisites, i.e. depenency-only prerequisites, may be specified by placing a "smaller than" symbol (<) in the prerequisite list: any prerequisites to the left of the "smaller than" symbol are normal (or order-only); any prerequisites to the right are dependency-only (and possibly order-only as well).

Dependency-only prerequisites behave almost identical to the other two prerequisite types, with one important exception: They do not contribute to any of their list-type related automatic variables. Thus, dependency-only prerequisites are not added to neither of the automatic variable lists $^, $+, $?, $*, $(^F), $(+F), $(?F), $(*F), $(^D), $(+D), $(?D) and $(*D), and prerequisites that are both dependency-only and order-only are not added to neither of the automatic variable lists $|, $(|F), $(|D).

The rationale behind dependency-only dependencies is to make it more easy to extend dependency lists of existing Makefiles. An example may illustrate this:

The following code may be considered as a snippet of a large and maybe rather complex Makefile:

myappl: main.o file1.o file2.o
	gcc -o $ $^

At a first glance, it lists all the relevant prerequisites, but a second thought reveals that this is just not true: The target certainly also depends on the compiler frontend, the linker backend and the Makefile itself.

Thus, a more complete snippet should look more like this:

myappl: main.o file1.o file2.o /usr/bin/gcc /usr/bin/ld Makefile
	gcc -o $ $(filter %.o, $^)

Please note the need for the newly introduced GNU Make’s $(filter ) function besides the additional prerequisites.

But for big projects, say the Linux kernel or a toolchain build, it would be rather laborious to change and fix all the Makefiles accordingly, and it would be more than questionable if such patches would be welcomed by every project. Fortunately, with dependency-only prerequisites at hand, the upstream Makefiles do not need to be changed at all. Instead, it’s sufficient to list the additional dependencies as dependency-only prerequisites in another Makefile that just includes the upstream Makefile. To continue with our example (and assuming the related upstream Makefile was just called Makefile, we could most conviniently add a GNUmakefile with the following content:

include Makefile
myappl: < /usr/bin/gcc /usr/bin/ld Makefile

Calling make now would prefer GNUmakefile over Makefile, thus respecting the additional prerequisites without affecting the related reciepe

What do you think: Do I have any chance to have my patch included? I'm certainly more that willing to do any modifications desired.

Thanks in advance for any help,

Chris



_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make

dependency_only.patch (10K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Feature request / patch: dependency-only prerequisites

Martin Dorey-2
The target certainly also depends on the compiler frontend, the linker backend and the Makefile itself

It's unimaginable that it doesn't depend on compiler libraries and myriad compiler-provided header files too.  Using header files in a compilation example would make the $(filter ...) solution less tenable, strengthening your case.  Sadly, though, adding all these things to the dependency list won't really help me.  We installed this system just three days ago, yet the mtime on stdio.h is months ago: 

devadmin@ch-ep1-3:~$ ls -l /usr/include/stdio.h
-rw-r--r-- 1 root root 31494 Feb  6 21:17 /usr/include/stdio.h
devadmin@ch-ep1-3:~$

That dates from when upstream built the package that included it:


If you decide to press on, then I have three minor suggestions:

the newly introduced GNU Make’s $(filter ) function

The revision history for make.texi shows that $(filter) was added some three decades ago:

[hidden email]            9f8301ae1ac6d9076e38ec86f12d59ba40b851bd:Revision 1.2  1988/04/23 16:16:04  roland
...
[hidden email]            9f8301ae1ac6d9076e38ec86f12d59ba40b851bd:* Added the `filter', `filter-out', `strip' and `join' expansion functions.

https://www.gnu.org/software/make/manual/html_node/Features.html#Features suggests that this function was an innovation of GNU make, true, but I found "newly introduced" misleading.  I'm a native English speaker.

gcc -o $ $(filter %.o, $^)

You've put a space after the comma.  I wouldn't do that in a Makefile, though I would in every other programming language.  https://www.gnu.org/software/make/manual/html_node/Text-Functions.html#Text-Functions mostly wouldn't have that space either.  $(filter) is one of those word-oriented functions where I don't think it would matter, but it's easier to have simple rules of thumb that keep you safe.  One of the spaces after a comma here is important:

martind@swiftboat:~/tmp/warlich-2019-07-05$ cat Makefile
INPUT = a.o b.o
fn = $(1)
OUTPUT = $(call fn, $(filter-out %.o, $(INPUT)))
$(if $(OUTPUT),$(error :$(OUTPUT): is non-empty!))
martind@swiftboat:~/tmp/warlich-2019-07-05$ make
Makefile:4: *** : : is non-empty!.  Stop.
martind@swiftboat:~/tmp/warlich-2019-07-05$

depenency
conviniently
reciepe

Spelling.


From: Bug-make <bug-make-bounces+martin.dorey=[hidden email]> on behalf of Christof Warlich <[hidden email]>
Sent: Thursday, July 4, 2019 10:13
To: [hidden email]
Subject: Feature request / patch: dependency-only prerequisites
 
***** EXTERNAL EMAIL *****

Dear all,

please bear with me if I'm doing something wrong here, this is the first time that I'm trying to contribute to GNU Make.

The attached patch would add a minor (but imho useful) feature to GNU Make. Here is an extract of the (changed) documentation (changes are in red), giving a quite comprehensive idea as to why this feature would be useful:

4.3 Types of Prerequisites

There are actually three different types of prerequisites understood by GNU make: normal prerequisites such as described in the previous section, order-only prerequisites, and dependency-only prerequisites.

A normal prerequisite makes two statements: first, it imposes an order in which recipes will be invoked: the recipes for all prerequisites of a target will be completed before the recipe for the target is run. Second, it imposes a dependency relationship: if any prerequisite is newer than the target, then the target is considered out-of-date and must be rebuilt.

Normally, this is exactly what you want: if a target’s prerequisite is updated, then the target should also be updated.

Occasionally, however, you have a situation where you want to impose a specific ordering on the rules to be invoked without forcing the target to be updated if one of those rules is executed. In that case, you want to define order-only prerequisites. Order-only prerequisites can be specified by placing a pipe symbol (|) in the prerequisites list: any prerequisites to the left of the pipe symbol are normal; any prerequisites to the right are order-only:

targets : normal-prerequisites | order-only-prerequisites

The normal prerequisites section may of course be empty. Also, you may still declare multiple lines of prerequisites for the same target: they are appended appropriately (normal prerequisites are appended to the list of normal prerequisites; order-only prerequisites are appended to the list of order-only prerequisites). Note that if you declare the same file to be both a normal and an order-only prerequisite, the normal prerequisite takes precedence (since they have a strict superset of the behavior of an order-only prerequisite).

Consider an example where your targets are to be placed in a separate directory, and that directory might not exist before make is run. In this situation, you want the directory to be created before any targets are placed into it but, because the timestamps on directories change whenever a file is added, removed, or renamed, we certainly don’t want to rebuild all the targets whenever the directory’s timestamp changes. One way to manage this is with order-only prerequisites: make the directory an order-only prerequisite on all the targets:

OBJDIR := objdir
OBJS := $(addprefix $(OBJDIR)/,foo.o bar.o baz.o)

$(OBJDIR)/%.o : %.c
        $(COMPILE.c) $(OUTPUT_OPTION) $<

all: $(OBJS)

$(OBJS): | $(OBJDIR)

$(OBJDIR):
        mkdir $(OBJDIR)

Now the rule to create the objdir directory will be run, if needed, before any ‘.o’ is built, but no ‘.o’ will be built because the objdir directory timestamp changed.

Finally, the third type of prerequisites, i.e. depenency-only prerequisites, may be specified by placing a "smaller than" symbol (<) in the prerequisite list: any prerequisites to the left of the "smaller than" symbol are normal (or order-only); any prerequisites to the right are dependency-only (and possibly order-only as well).

Dependency-only prerequisites behave almost identical to the other two prerequisite types, with one important exception: They do not contribute to any of their list-type related automatic variables. Thus, dependency-only prerequisites are not added to neither of the automatic variable lists $^, $+, $?, $*, $(^F), $(+F), $(?F), $(*F), $(^D), $(+D), $(?D) and $(*D), and prerequisites that are both dependency-only and order-only are not added to neither of the automatic variable lists $|, $(|F), $(|D).

The rationale behind dependency-only dependencies is to make it more easy to extend dependency lists of existing Makefiles. An example may illustrate this:

The following code may be considered as a snippet of a large and maybe rather complex Makefile:

myappl: main.o file1.o file2.o
	gcc -o $ $^

At a first glance, it lists all the relevant prerequisites, but a second thought reveals that this is just not true: The target certainly also depends on the compiler frontend, the linker backend and the Makefile itself.

Thus, a more complete snippet should look more like this:

myappl: main.o file1.o file2.o /usr/bin/gcc /usr/bin/ld Makefile
	gcc -o $ $(filter %.o, $^)

Please note the need for the newly introduced GNU Make’s $(filter ) function besides the additional prerequisites.

But for big projects, say the Linux kernel or a toolchain build, it would be rather laborious to change and fix all the Makefiles accordingly, and it would be more than questionable if such patches would be welcomed by every project. Fortunately, with dependency-only prerequisites at hand, the upstream Makefiles do not need to be changed at all. Instead, it’s sufficient to list the additional dependencies as dependency-only prerequisites in another Makefile that just includes the upstream Makefile. To continue with our example (and assuming the related upstream Makefile was just called Makefile, we could most conviniently add a GNUmakefile with the following content:

include Makefile
myappl: < /usr/bin/gcc /usr/bin/ld Makefile

Calling make now would prefer GNUmakefile over Makefile, thus respecting the additional prerequisites without affecting the related reciepe

What do you think: Do I have any chance to have my patch included? I'm certainly more that willing to do any modifications desired.

Thanks in advance for any help,

Chris



_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: Feature request / patch: dependency-only prerequisites

Christof Warlich

on 05.07.19 at 23:44 Martin Dorey wrote:

... Sadly, though, adding all these things to the dependency list won't really help me.  We installed this system just three days ago, yet the mtime on stdio.h is months ago: 

devadmin@ch-ep1-3:~$ ls -l /usr/include/stdio.h
-rw-r--r-- 1 root root 31494 Feb  6 21:17 /usr/include/stdio.h
devadmin@ch-ep1-3:~$

That dates from when upstream built the package that included it:

Yes, that's unfortunate, and the only generic fix would be checksum-based dependencies. Decent SCMs like git are also quite helpful in tackling this issue, see https://git.wiki.kernel.org/index.php/GitFaq#Why_isn.27t_Git_preserving_modification_time_on_files.3F


If you decide to press on, then I have three minor suggestions:

the newly introduced GNU Make’s $(filter ) function

The revision history for make.texi shows that $(filter) was added some three decades ago:

[hidden email]            9f8301ae1ac6d9076e38ec86f12d59ba40b851bd:Revision 1.2  1988/04/23 16:16:04  roland
...
[hidden email]            9f8301ae1ac6d9076e38ec86f12d59ba40b851bd:* Added the `filter', `filter-out', `strip' and `join' expansion functions.

https://www.gnu.org/software/make/manual/html_node/Features.html#Features suggests that this function was an innovation of GNU make, true, but I found "newly introduced" misleading.  I'm a native English speaker.
Yes, that's misleading: The "newly introduced" was meant to emphasize that a $(filter) function was added compared to the previous example, but you are right, it does more bad than good.

gcc -o $ $(filter %.o, $^)

You've put a space after the comma.  I wouldn't do that in a Makefile, though I would in every other programming language.  https://www.gnu.org/software/make/manual/html_node/Text-Functions.html#Text-Functions mostly wouldn't have that space either.  $(filter) is one of those word-oriented functions where I don't think it would matter, but it's easier to have simple rules of thumb that keep you safe.  One of the spaces after a comma here is important:

martind@swiftboat:~/tmp/warlich-2019-07-05$ cat Makefile
INPUT = a.o b.o
fn = $(1)
OUTPUT = $(call fn, $(filter-out %.o, $(INPUT)))
$(if $(OUTPUT),$(error :$(OUTPUT): is non-empty!))
martind@swiftboat:~/tmp/warlich-2019-07-05$ make
Makefile:4: *** : : is non-empty!.  Stop.
martind@swiftboat:~/tmp/warlich-2019-07-05$

depenency
conviniently
reciepe

Spelling.

Thanks for your suggestions, I'm happy to include them all (patch attached), together with some other doc fixes (hopefully) improving the grammer.



_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make

0001-Add-support-for-dependency-only-prerequisites.patch (10K) Download Attachment
0002-Fix-documentation-for-dependency-only-prerequisites.patch (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Feature request / patch: dependency-only prerequisites

Tim Murphy-4
I quite like this idea because so many of us work on integrating things that we have no permission to modify and we need ways to make them work cleanly without messing them up.  They are sort of "fix-up" or "patch" dependencies. I'm not sure the name makes this clear though.

Regards,

Tim


On Sat, 6 Jul 2019 at 09:28, Christof Warlich <[hidden email]> wrote:

on 05.07.19 at 23:44 Martin Dorey wrote:

... Sadly, though, adding all these things to the dependency list won't really help me.  We installed this system just three days ago, yet the mtime on stdio.h is months ago: 

devadmin@ch-ep1-3:~$ ls -l /usr/include/stdio.h
-rw-r--r-- 1 root root 31494 Feb  6 21:17 /usr/include/stdio.h
devadmin@ch-ep1-3:~$

That dates from when upstream built the package that included it:

Yes, that's unfortunate, and the only generic fix would be checksum-based dependencies. Decent SCMs like git are also quite helpful in tackling this issue, see https://git.wiki.kernel.org/index.php/GitFaq#Why_isn.27t_Git_preserving_modification_time_on_files.3F


If you decide to press on, then I have three minor suggestions:

the newly introduced GNU Make’s $(filter ) function

The revision history for make.texi shows that $(filter) was added some three decades ago:

[hidden email]            9f8301ae1ac6d9076e38ec86f12d59ba40b851bd:Revision 1.2  1988/04/23 16:16:04  roland
...
[hidden email]            9f8301ae1ac6d9076e38ec86f12d59ba40b851bd:* Added the `filter', `filter-out', `strip' and `join' expansion functions.

https://www.gnu.org/software/make/manual/html_node/Features.html#Features suggests that this function was an innovation of GNU make, true, but I found "newly introduced" misleading.  I'm a native English speaker.
Yes, that's misleading: The "newly introduced" was meant to emphasize that a $(filter) function was added compared to the previous example, but you are right, it does more bad than good.

gcc -o $ $(filter %.o, $^)

You've put a space after the comma.  I wouldn't do that in a Makefile, though I would in every other programming language.  https://www.gnu.org/software/make/manual/html_node/Text-Functions.html#Text-Functions mostly wouldn't have that space either.  $(filter) is one of those word-oriented functions where I don't think it would matter, but it's easier to have simple rules of thumb that keep you safe.  One of the spaces after a comma here is important:

martind@swiftboat:~/tmp/warlich-2019-07-05$ cat Makefile
INPUT = a.o b.o
fn = $(1)
OUTPUT = $(call fn, $(filter-out %.o, $(INPUT)))
$(if $(OUTPUT),$(error :$(OUTPUT): is non-empty!))
martind@swiftboat:~/tmp/warlich-2019-07-05$ make
Makefile:4: *** : : is non-empty!.  Stop.
martind@swiftboat:~/tmp/warlich-2019-07-05$

depenency
conviniently
reciepe

Spelling.

Thanks for your suggestions, I'm happy to include them all (patch attached), together with some other doc fixes (hopefully) improving the grammer.


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: Feature request / patch: dependency-only prerequisites

David Boyce-3
I also think the proposal is reasonable but I think it would be "stickier", i.e. less likely to get lost, if you filed it as an enhancement request via the Savannah bug reporting system.

On Tue, Jul 9, 2019 at 5:27 AM Tim Murphy <[hidden email]> wrote:
I quite like this idea because so many of us work on integrating things that we have no permission to modify and we need ways to make them work cleanly without messing them up.  They are sort of "fix-up" or "patch" dependencies. I'm not sure the name makes this clear though.

Regards,

Tim


On Sat, 6 Jul 2019 at 09:28, Christof Warlich <[hidden email]> wrote:

on 05.07.19 at 23:44 Martin Dorey wrote:

... Sadly, though, adding all these things to the dependency list won't really help me.  We installed this system just three days ago, yet the mtime on stdio.h is months ago: 

devadmin@ch-ep1-3:~$ ls -l /usr/include/stdio.h
-rw-r--r-- 1 root root 31494 Feb  6 21:17 /usr/include/stdio.h
devadmin@ch-ep1-3:~$

That dates from when upstream built the package that included it:

Yes, that's unfortunate, and the only generic fix would be checksum-based dependencies. Decent SCMs like git are also quite helpful in tackling this issue, see https://git.wiki.kernel.org/index.php/GitFaq#Why_isn.27t_Git_preserving_modification_time_on_files.3F


If you decide to press on, then I have three minor suggestions:

the newly introduced GNU Make’s $(filter ) function

The revision history for make.texi shows that $(filter) was added some three decades ago:

[hidden email]            9f8301ae1ac6d9076e38ec86f12d59ba40b851bd:Revision 1.2  1988/04/23 16:16:04  roland
...
[hidden email]            9f8301ae1ac6d9076e38ec86f12d59ba40b851bd:* Added the `filter', `filter-out', `strip' and `join' expansion functions.

https://www.gnu.org/software/make/manual/html_node/Features.html#Features suggests that this function was an innovation of GNU make, true, but I found "newly introduced" misleading.  I'm a native English speaker.
Yes, that's misleading: The "newly introduced" was meant to emphasize that a $(filter) function was added compared to the previous example, but you are right, it does more bad than good.

gcc -o $ $(filter %.o, $^)

You've put a space after the comma.  I wouldn't do that in a Makefile, though I would in every other programming language.  https://www.gnu.org/software/make/manual/html_node/Text-Functions.html#Text-Functions mostly wouldn't have that space either.  $(filter) is one of those word-oriented functions where I don't think it would matter, but it's easier to have simple rules of thumb that keep you safe.  One of the spaces after a comma here is important:

martind@swiftboat:~/tmp/warlich-2019-07-05$ cat Makefile
INPUT = a.o b.o
fn = $(1)
OUTPUT = $(call fn, $(filter-out %.o, $(INPUT)))
$(if $(OUTPUT),$(error :$(OUTPUT): is non-empty!))
martind@swiftboat:~/tmp/warlich-2019-07-05$ make
Makefile:4: *** : : is non-empty!.  Stop.
martind@swiftboat:~/tmp/warlich-2019-07-05$

depenency
conviniently
reciepe

Spelling.

Thanks for your suggestions, I'm happy to include them all (patch attached), together with some other doc fixes (hopefully) improving the grammer.


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: Feature request / patch: dependency-only prerequisites

Christof Warlich

Am 09.07.19 um 15:42 schrieb David Boyce:

I also think the proposal is reasonable but I think it would be "stickier", i.e. less likely to get lost, if you filed it as an enhancement request via the Savannah bug reporting system.
I'd be glad to do that if it increases the chance to get it in. But I couldn't find the Savannah bug reporting system (on https://savannah.gnu.org/). Do I need to create a new user and log in to be able to file an enhancement request?

On Tue, Jul 9, 2019 at 5:27 AM Tim Murphy <[hidden email]> wrote:
I quite like this idea because so many of us work on integrating things that we have no permission to modify and we need ways to make them work cleanly without messing them up.  They are sort of "fix-up" or "patch" dependencies. I'm not sure the name makes this clear though
Agreed, the name "dependency-only" is far from being a perfect choice, and I'd be happy with any other suggestion.


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: Feature request / patch: dependency-only prerequisites

David Boyce-3
I don't claim to know much about how Savannah works but I see anonymous bug reports and RFEs come in frequently so it seems you don't need a login to make one.


On Tue, Jul 9, 2019 at 11:19 AM Christof Warlich <[hidden email]> wrote:

Am 09.07.19 um 15:42 schrieb David Boyce:

I also think the proposal is reasonable but I think it would be "stickier", i.e. less likely to get lost, if you filed it as an enhancement request via the Savannah bug reporting system.
I'd be glad to do that if it increases the chance to get it in. But I couldn't find the Savannah bug reporting system (on https://savannah.gnu.org/). Do I need to create a new user and log in to be able to file an enhancement request?

On Tue, Jul 9, 2019 at 5:27 AM Tim Murphy <[hidden email]> wrote:
I quite like this idea because so many of us work on integrating things that we have no permission to modify and we need ways to make them work cleanly without messing them up.  They are sort of "fix-up" or "patch" dependencies. I'm not sure the name makes this clear though
Agreed, the name "dependency-only" is far from being a perfect choice, and I'd be happy with any other suggestion.


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: Feature request / patch: dependency-only prerequisites

Paul Smith-20
In reply to this post by Christof Warlich
On Tue, 2019-07-09 at 20:19 +0200, Christof Warlich wrote:
> Am 09.07.19 um 15:42 schrieb David Boyce:
> > I also think the proposal is reasonable but I think it would be
> > "stickier", i.e. less likely to get lost, if you filed it as an
> > enhancement request via the Savannah bug reporting system.
>  I'd be glad to do that if it increases the chance to get it in. But
> I couldn't find the Savannah bug reporting system (on
> https://savannah.gnu.org/). Do I need to create a new user and log in
> to be able to file an enhancement request?

https://savannah.gnu.org/bugs/?func=additem&group=make

You don't have to make an account in order to file a bug, but be sure
to add yourself to mail notifications if you don't create an account,
or you won't get any updates.

My first reaction is that this solution seems like a very big hammer
(in terms of changes to makefile syntax) to handle a relatively rare
and specific situation.  I wonder if there's an alternative that gives
the same result but is a bit less intrusive.

What if we do this instead: create a variable which can contain a list
of "hidden prerequisites": they will always be built first but they
won't cause any changes in the automatic variables.  If set globally it
will apply to all targets.  If set as a target-specific variable then
it applies only to that target (and its prerequisites, which can be
remediated with the private keyword).

This seems like it could be something generally useful, to ensure some
targets are always built first, as well as useful in this context.


On another topic, note that these changes are of a size that they do
meet the requirement for copyright assignment before they can be
applied.  I can forward you the details if that's something you're
interested in.

Cheers!


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: Feature request / patch: dependency-only prerequisites

Christof Warlich
Am 09.07.19 um 20:52 schrieb Paul Smith:

> https://savannah.gnu.org/bugs/?func=additem&group=make
> You don't have to make an account in order to file a bug, but be sure
> to add yourself to mail notifications if you don't create an account,
> or you won't get any updates.
Ok, but considering your suggestion below, I doubt that it
makes much sense to still file my initial approach.

> My first reaction is that this solution seems like a very big hammer
> (in terms of changes to makefile syntax) to handle a relatively rare
> and specific situation.  I wonder if there's an alternative that gives
> the same result but is a bit less intrusive.
>
> What if we do this instead: create a variable which can contain a list
> of "hidden prerequisites": they will always be built first but they
> won't cause any changes in the automatic variables.  If set globally it
> will apply to all targets.  If set as a target-specific variable then
> it applies only to that target (and its prerequisites, which can be
> remediated with the private keyword).
>
> This seems like it could be something generally useful, to ensure some
> targets are always built first, as well as useful in this context.

AFAICS, this would serve my desires at least equally well. And the
more I think about it, the more I get convinced that it would be
the superior approach for a couple of reasons, like better readability,
almost no syntactical change, ...

Furthermore, this variable may quite naturally even honor order-only
prerequisites by including the pipe character at any point in its
prerequisite list.

> On another topic, note that these changes are of a size that they do
> meet the requirement for copyright assignment before they can be
> applied.  I can forward you the details if that's something you're
> interested in.
>
> Cheers!
I certainly would be happy to sign any copyright requirements needed.
But just let me know what would be most conveniet for _you_ to proceed:

Would it really help if I try to supply a patch that implements your idea?
Or is the current "state of affairs" sufficient that the general idea (i.e.
allowing to add target specific prerequisites without changing automatic
variables) will eventually go into some future version of GNU Make?

Cheers,

Chris


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: Feature request / patch: dependency-only prerequisites

Christof Warlich
In reply to this post by David Boyce-3
Am 09.07.19 um 20:51 schrieb David Boyce:

> I don't claim to know much about how Savannah works but I see
> anonymous bug reports and RFEs come in frequently so it seems you
> don't need a login to make one.
>
> https://www.gnu.org/software/make/manual/html_node/Bugs.html
Thanks, that's want I was looking for. :-)

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: Feature request / patch: dependency-only prerequisites

Paul Smith-20
In reply to this post by Christof Warlich
On Wed, 2019-07-10 at 10:24 +0200, Christof Warlich wrote:

> > What if we do this instead: create a variable which can contain a list
> > of "hidden prerequisites": they will always be built first but they
> > won't cause any changes in the automatic variables.  If set globally it
> > will apply to all targets.  If set as a target-specific variable then
> > it applies only to that target (and its prerequisites, which can be
> > remediated with the private keyword).
> >
> > This seems like it could be something generally useful, to ensure some
> > targets are always built first, as well as useful in this context.
>
> AFAICS, this would serve my desires at least equally well. And the
> more I think about it, the more I get convinced that it would be
> the superior approach for a couple of reasons, like better readability,
> almost no syntactical change, ...
>
> Furthermore, this variable may quite naturally even honor order-only
> prerequisites by including the pipe character at any point in its
> prerequisite list.

True.  I hadn't fully considered this but you're right.  Possibly it
would work to expand the variable value and use something like
split_prereqs() in file.c on the result to get the list.

> > On another topic, note that these changes are of a size that they do
> > meet the requirement for copyright assignment before they can be
> > applied.  I can forward you the details if that's something you're
> > interested in.
> >
> > Cheers!
> I certainly would be happy to sign any copyright requirements needed.
> But just let me know what would be most conveniet for _you_ to proceed:
>
> Would it really help if I try to supply a patch that implements your idea?
> Or is the current "state of affairs" sufficient that the general idea (i.e.
> allowing to add target specific prerequisites without changing automatic
> variables) will eventually go into some future version of GNU Make?

Well, being honest, there's not a great chance I'll find the time to
implement this on my own.  Not because I have a problem with idea, but
just because I don't have a lot of free time.  If it's something you
want your best bet is to implement it yourself.

The more comprehensive the implementation (that is, docs, regression
tests, etc.) the more likely it is to be applied quickly.

Let me know how you want to proceed.

Thanks for your interest in GNU make!


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: Feature request / patch: dependency-only prerequisites

Christof Warlich
Am 10.07.19 um 14:44 schrieb Paul Smith:

>> Would it really help if I try to supply a patch that implements your idea?
>> Or is the current "state of affairs" sufficient that the general idea (i.e.
>> allowing to add target specific prerequisites without changing automatic
>> variables) will eventually go into some future version of GNU Make?
> Well, being honest, there's not a great chance I'll find the time to
> implement this on my own.  Not because I have a problem with idea, but
> just because I don't have a lot of free time.  If it's something you
> want your best bet is to implement it yourself.
Fair enough :-).
> The more comprehensive the implementation (that is, docs, regression
> tests, etc.) the more likely it is to be applied quickly.

I'll do my very best, but it may take some time. It will also most likely
require a couple of iterations and I may need to ask for some help, but
let's see ...

> Let me know how you want to proceed.

As I want to tackle the most challenging part first, I'd like to suggest to
do the copyright assignment after the technical side is in a decent shape.
But as I said, no problem to do it fist if that's more appropriate.

> Thanks for your interest in GNU make!
Thanks for considering the feature :-) and for maintaining GNU Make!

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: Feature request / patch: dependency-only prerequisites

Paul Smith-20
On Wed, 2019-07-10 at 18:15 +0200, Christof Warlich wrote:
> Am 10.07.19 um 14:44 schrieb Paul Smith:
> > Let me know how you want to proceed.
>
> As I want to tackle the most challenging part first, I'd like to suggest to
> do the copyright assignment after the technical side is in a decent shape.
> But as I said, no problem to do it fist if that's more appropriate.

That's no problem.  However you prefer is fine with me.

Copyright assignment these days is pretty straightforward and can be
done via email instead of sending paper documents around like in the
old days, so it's not as long a process.

One item to note: if you work for a company which has rights to your
work, then they would need to sign off on this as well.

Cheers!


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: Feature request / patch: dependency-only prerequisites

Christof Warlich

Am 13.07.19 um 14:46 schrieb Paul Smith:

On Wed, 2019-07-10 at 18:15 +0200, Christof Warlich wrote:
Am 10.07.19 um 14:44 schrieb Paul Smith:
Let me know how you want to proceed.
As I want to tackle the most challenging part first, I'd like to suggest to
do the copyright assignment after the technical side is in a decent shape.
But as I said, no problem to do it fist if that's more appropriate.
That's no problem.  However you prefer is fine with me.

Copyright assignment these days is pretty straightforward and can be
done via email instead of sending paper documents around like in the
old days, so it's not as long a process.

One item to note: if you work for a company which has rights to your
work, then they would need to sign off on this as well.

Cheers!

Hi Paul, all,

attached is a first "proof of concept" patch that uses the specific variable
PREREQUISTES to add prerequisites that do not affect automatic
variables. It's not complete (see the points below), but should be good
enough to ask for initial feedback on whether the general approach makes
sense.

Here are the points that I'm aware of that are missing:

  1. Currently, PREREQUISITES is only considered when it is defined
    as a target specific variable in an ordinary rule. The code to handle
    the case when PREREQUISITES is a global variable (or when in is
    defined in a pattern rule?!) is still missing.
  2. Variables and functions are not expanded. Could you give me a
    quick hint on how this should be done? I've found lookup_variable()
    to expand global variables, but this would require to do the parsing
    manually and would not be of much use with functions.
  3. What about the "struct file" pointer and the "struct variable" pointer
    returned by lookup_variable() and lookup_variable_in_set() respectively:
    Do I need to free memory when done with them, and if so, may I just
    use free()?

Thanks for any feedback and your precious time,

Chris

P.S.: There is a small Makefile that could be used for a quick test of the
current (limited) functionality:

target: PREREQUISITES=hi ho
target: foo bar
    @echo "dependency-only prerequisites: ${PREREQUISITES}"
    @echo "ordinary prerequisites: $^"
    rm -rf hi ho foo bar
hi ho foo bar:
    touch $@

Running make with this Makefile yields:

$ make target
touch foo
touch bar
touch hi
touch ho
dependency-only prerequisites: hi ho
ordinary prerequisites: foo bar
rm -rf hi ho foo bar

as desired, i.e. both "hi" and "ho" are recognized as prerequisites but were
not added to $^.


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make

prereq.patch (7K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Feature request / patch: dependency-only prerequisites

Christof Warlich
Am 13.07.19 um 14:46 schrieb Paul Smith:

> On Wed, 2019-07-10 at 18:15 +0200, Christof Warlich wrote:
>> Am 10.07.19 um 14:44 schrieb Paul Smith:
>>> Let me know how you want to proceed.
>> As I want to tackle the most challenging part first, I'd like to suggest to
>> do the copyright assignment after the technical side is in a decent shape.
>> But as I said, no problem to do it fist if that's more appropriate.
> That's no problem.  However you prefer is fine with me.
>
> Copyright assignment these days is pretty straightforward and can be
> done via email instead of sending paper documents around like in the
> old days, so it's not as long a process.
>
> One item to note: if you work for a company which has rights to your
> work, then they would need to sign off on this as well.
>
> Cheers!
>
Hi,

attached is the comprehensive (and significantly reworked) patch that
implements the feature of a new internal variable ( I called it
.PREREQUISITES)
allowing to add prerequisites that do not contribute to the automatic list
variables, including the related docu and tests.

 From my perspective, everything is complete and works as it should, so I
would definitely need your feedback if there is anything missing or wrong.

Otherwise (or in parallel), could you guide me to get the copyright
assignment
done so that the patch could be included?

In the hope to get some response,

Chris




_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make

prerequisites.patch (12K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Feature request / patch: dependency-only prerequisites

Paul Smith-20
On Fri, 2019-08-09 at 17:52 +0200, Christof Warlich wrote:
>  From my perspective, everything is complete and works as it should,
> so I would definitely need your feedback if there is anything missing
> or wrong.
>
> Otherwise (or in parallel), could you guide me to get the copyright
> assignment done so that the patch could be included?

I am on vacation until next week and will have limited internet
connectivity.

I'll try to send along the instructions you need for copyright
assignment.

Cheers!


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: Feature request / patch: dependency-only prerequisites

Christof Warlich
Am 14.08.19 um 20:31 schrieb Paul Smith:

> On Fri, 2019-08-09 at 17:52 +0200, Christof Warlich wrote:
>>   From my perspective, everything is complete and works as it should,
>> so I would definitely need your feedback if there is anything missing
>> or wrong.
>>
>> Otherwise (or in parallel), could you guide me to get the copyright
>> assignment done so that the patch could be included?
> I am on vacation until next week and will have limited internet
> connectivity.
Same with me, hope you'll enjoy :-)!
> I'll try to send along the instructions you need for copyright
> assignment.

Thanks, I've applied for permanent assignment, awaiting response.

Anyway, I found a bug in my previous patch, so you'll find a fixed version
attached. But no need to hurry!

In case anyone already had a closer look into the previous patch and / or
likes detailed information w.r.t. what was wrong, please let me know: I'm
happy to elaborate, together with sending an incremental patch towards
the previous one.

Chris



_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make

prerequisites.patch (12K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Feature request / patch: dependency-only prerequisites

Paul Smith-20
In reply to this post by Christof Warlich
On Fri, 2019-08-09 at 17:52 +0200, Christof Warlich wrote:
> attached is the comprehensive (and significantly reworked) patch that
> implements the feature of a new internal variable

Hi Christof; .EXTRA_PREREQS is provided in 4.2.93 please test in your
environment.

Note that I did rework some things to allow it to work with implicit rules
(previously it only worked for explicit rules) although it still can't be
used as a pattern-specific variable: that will require more effort.  Also
some changes were made to make things a bit more efficient.


Reply | Threaded
Open this post in threaded view
|

Re: Feature request / patch: dependency-only prerequisites

Christof Warlich

Hi Paul,

Am 03.01.20 um 18:24 schrieb Paul Smith:
Hi Christof; .EXTRA_PREREQS is provided in 4.2.93 please test in your
environment.

Note that I did rework some things to allow it to work with implicit rules
(previously it only worked for explicit rules) although it still can't be
used as a pattern-specific variable: that will require more effort.  Also
some changes were made to make things a bit more efficient.

many thanks for adding this into the release candidate . Again, I very much appreciate your dedicated work, and even more so after looking at the extra work that was still needed on your side to make the patch acceptable for inclusion into GNU make.

By the way, I stumbled into a minor issue entirely unrelated to the feature above that you may want to fix: The gnulib git repository referenced in bootstrap seems to have changed its location, so you may want it to change it accordingly:

$ git diff bootstrap
diff --git a/bootstrap b/bootstrap
index 05808ee..8f52cc1 100755
--- a/bootstrap
+++ b/bootstrap
@@ -47,7 +47,7 @@ PERL="${PERL-perl}"
 
 me=$0
 
-default_gnulib_url=git://git.sv.gnu.org/gnulib
+default_gnulib_url=https://git.savannah.gnu.org/git/gnulib.git
 
 usage() {
   cat <<EOF

Furthermore, as running ./bootstrap adds the gnulib directory, it may be appropriate to add it to .gitignore:

$ git diff .gitignore
diff --git a/.gitignore b/.gitignore
index df60c2c..f03287e 100644
--- a/.gitignore
+++ b/.gitignore
@@ -68,3 +68,4 @@ configh.dos
 make-[0-9]*/
 make-[0-9]*.tar.*
 checkcfg.*.log
+gnulib

And finally, running ./bootstrap also changes lib/.gitignore and m4/.gitignore, so you may want to consider to commit these changed versions:

$ git diff lib/.gitignore m4/.gitignore
diff --git a/lib/.gitignore b/lib/.gitignore
index cb391c4..a94190f 100644
--- a/lib/.gitignore
+++ b/lib/.gitignore
@@ -9,3 +9,63 @@
 !/glob.in.h
 /glob.h
 
+/Makefile.am
+/_Noreturn.h
+/access.c
+/alloca.c
+/alloca.in.h
+/arg-nonnull.h
+/basename-lgpl.c
+/c++defs.h
+/close.c
+/concat-filename.c
+/concat-filename.h
+/dirname-lgpl.c
+/dirname.h
+/dosname.h
+/dup2.c
+/errno.in.h
+/error.c
+/error.h
+/exitfail.c
+/exitfail.h
+/fcntl.c
+/fcntl.in.h
+/fd-hook.c
+/fd-hook.h
+/filename.h
+/findprog-in.c
+/findprog.h
+/getdtablesize.c
+/getloadavg.c
+/getprogname.c
+/getprogname.h
+/gettext.h
+/intprops.h
+/limits.in.h
+/malloc.c
+/msvc-inval.c
+/msvc-inval.h
+/msvc-nothrow.c
+/msvc-nothrow.h
+/stdbool.in.h
+/stddef.in.h
+/stdint.in.h
+/stdio.in.h
+/stdlib.in.h
+/stpcpy.c
+/strerror-override.c
+/strerror-override.h
+/strerror.c
+/string.in.h
+/stripslash.c
+/sys_types.in.h
+/unistd.c
+/unistd.in.h
+/verify.h
+/warn-on-use.h
+/xalloc-die.c
+/xalloc-oversized.h
+/xalloc.h
+/xconcat-filename.c
+/xmalloc.c
diff --git a/m4/.gitignore b/m4/.gitignore
index 244385a..be62484 100644
--- a/m4/.gitignore
+++ b/m4/.gitignore
@@ -3,3 +3,47 @@
 !/aclocal.m4
 !/dospaths.m4
 
+/00gnulib.m4
+/absolute-header.m4
+/access.m4
+/alloca.m4
+/asm-underscore.m4
+/close.m4
+/dirname.m4
+/double-slash-root.m4
+/dup2.m4
+/eaccess.m4
+/errno_h.m4
+/error.m4
+/extensions.m4
+/fcntl.m4
+/fcntl_h.m4
+/findprog-in.m4
+/getdtablesize.m4
+/getloadavg.m4
+/getprogname.m4
+/gnulib-common.m4
+/gnulib-comp.m4
+/gnulib-tool.m4
+/host-cpu-c-abi.m4
+/include_next.m4
+/limits-h.m4
+/malloc.m4
+/msvc-inval.m4
+/msvc-nothrow.m4
+/multiarch.m4
+/off_t.m4
+/ssize_t.m4
+/stdbool.m4
+/stddef_h.m4
+/stdint.m4
+/stdio_h.m4
+/stdlib_h.m4
+/stpcpy.m4
+/strerror.m4
+/string_h.m4
+/sys_socket_h.m4
+/sys_types_h.m4
+/unistd_h.m4
+/warn-on-use.m4
+/xalloc.m4

Cheers,

Christof


Reply | Threaded
Open this post in threaded view
|

Re: Feature request / patch: dependency-only prerequisites

Paul Smith-20
On Fri, 2020-01-03 at 20:42 +0100, Christof Warlich wrote:
> By the way, I stumbled into a minor issue entirely unrelated to the
> feature above that you may want to fix: The gnulib git repository
> referenced in bootstrap seems to have changed its location, so you may
> want it to change it accordingly:

Hm.  Does the original not work?  The version I have is the same one that
is currently being distributed with the gnulib library; see:

http://git.savannah.gnu.org/cgit/gnulib.git/tree/build-aux/bootstrap

sv.gnu.org is a valid alias for savannah.gnu.org so unless the problem is
with the git protocol vs. https, it should work fine.

I would prefer to see this change made in gnulib's upstream copy if it is a
correct change, rather than in my copy.

> Furthermore, as running ./bootstrap adds the gnulib directory, it may be
> appropriate to add it to .gitignore:

OK.  I always use a separate gnulib repo and set $GNULIB_SRCDIR, because
checking out gnulib takes a long time.  So I didn't notice this issue.

> And finally, running ./bootstrap also changes lib/.gitignore and
> m4/.gitignore, so you may want to consider to commit these changed
> versions:

I know about this but I don't want to do it.  My setting for .gitignore is
correct already.  I really wish I could find a way to tell gnulib modules
to stop messing around with my .gitignore files but until then I just don't
pay any attention to those changes.


12