Brittle -includes

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Brittle -includes

David Deutsch
Hi everybody,

I've put together a somewhat extensive build system on top of make which
relies heavily on generating and including Makefiles (partly to avoid
The Dreaded Recursive Make). The centerpiece is a complement to %-rules
where I have blueprints for certain filetypes that I match against a
path via regex in a custom script that writes an actual makefile. I
guess it's my way of having %-rules with multiple, named % variables
across a path, allowing you do do things like:

---- ---- ---- ----
target/{{ path }}-w{{ width }}-h{{ height }}.jpg: source/{{ path }}.jpg
    cat $< | convert -resize {{ width }}x{{ height }} - jpg:- > $@
---- ---- ---- ----

I use it to build trees of linked documents and assets, so there is a
lot of overlap in the assets across documents. My approach is loosely
based on the patterns laid out in "Auto-Dependency Generation" [0].

A simplified version of the .mk file printer recipe is:

---- ---- ---- ----
$(makedir)/%.mk:
    $(mybin)/mkautoprint $* > $@
---- ---- ---- ----

When such an .mk file is created, it also includes dependencies of
further to-be-created .mk files like so:

---- ---- ---- ----
THEFILE: PATHOFMYDEPENDENCY
    $(mybin)/buildfile $*

# This .mk includes no recipe for PATHOFMYDEPENDENCY, so the script
attaches:

ifeq (,$(findstring $(makedir)/PATHOFMYDEPENDENCY,$(MAKEFILE_LIST)))
-include $(makedir)/PATHOFMYDEPENDENCY.mk
endif
---- ---- ---- ----

(The MAKEFILE_LIST ifeq is used to suppress warnings because multiple
documents could be trying to build the same asset)


My problem is that I keep running into situations where an .mk is
-include'd as seen above, but make never attempts to... make it,
resulting in a 'No rule to make target' error. I suspect there is some
kind of race condition going on where it does "see" the include, but
missing dependency error is triggered before that can occur. This
happens consistently, no matter whether I use -j or not.

The odd thing is that it will happily build hundreds of those .mk files
and then fail at one of them. You'd think that it's due to dependencies,
but generating the .mk itself has none. Similarly, when I remove one of
those .mk files, make often does not remake them on the next turn.

Does anybody have a tip on how I could debug this? Is there a better
approach for this strategy that I could look into? It's quite likely
that I missed something when reading "Auto-Dependency Generation". Even
after years of using make heavily, now, I still can't answer questions
like "do i need to use include or -include?" with absolute certainty. (I
think I got that bit from "The GNU Make Book" by John Graham-Cumming and
just cargo-cult used it for months now.)

I hope I explained the problem well enough - do let me know if there is
anything I need to further clarify!

regards,
David


[0]
http://make.mad-scientist.net/papers/advanced-auto-dependency-generation/


_______________________________________________
Help-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-make
Reply | Threaded
Open this post in threaded view
|

Re: Brittle -includes

Paul Smith-20
On Wed, 2019-10-16 at 08:10 +0200, David Deutsch wrote:
> My problem is that I keep running into situations where an .mk is
> -include'd as seen above, but make never attempts to... make it,
> resulting in a 'No rule to make target' error. I suspect there is some
> kind of race condition going on where it does "see" the include, but
> missing dependency error is triggered before that can occur. This
> happens consistently, no matter whether I use -j or not.

If you don't use -j, then it can't be a race condition.  Make is not
multithreaded so without -j it should be completely deterministic.  Are
you saying that _without_ -j you get nondeterministic failures?

You didn't mention what version of GNU make you're using.  Perhaps
there's a bug in that version?

Also, note that with newer versions of GNU make you don't need to use
"-include" to avoid warnings on non-existent makefiles.

I recommend you get to a sufficiently-new version of GNU make and
change all your "-include" to just "include" and see if that helps.


_______________________________________________
Help-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-make
Reply | Threaded
Open this post in threaded view
|

Re: Brittle -includes

David Deutsch
Hi Paul,

Thank you for getting back to me (and thank you so much for all your work!).

Race Condition - That was poorly worded on my part. I was thinking more
in terms of how make processes each makefile. I currently put the
includes at the end of each (generated) makefile and was thinking that
maybe if it reads (and tries to execute) the recipe before reading (and
making) the "include" makefile, that could produce issues. The
nondeterministic behavior probably isn't one, but it seems weird to me
that the same generated makefiles work in one case and fail in another.

I've replaced all instances of "-include" with "include" and I'm now
running into problems creating .d files in another part of the setup. I
guess because while it's not throwing a warning, it doesn't write or
include that makefile, now? I've read the manual on "How Makefiles Are
Remade" a few dozen times now, but I'm still not too sure about the
initial case - at what point Makefiles that don't yet exist are written.

I'm using Version 4.2.1. Do you think updating to 4.2.92 would make a
difference?


Another question, if I may: As I wrote earlier, I'm mostly doing this to
extend the pattern rule behavior of make by creating makefiles (and thus
targets with their recipe) for paths that match a certain regular
expression. It seems a bit… excessive to write hundreds, if not
thousands of makefiles and I worry that I'm going to run into
scalability issues at some point (luckily, we live in an era of SSDs, so
it hasn't been too much of a pain so far).

Is there a different direction that you could point me to, here, like
using Guile or some other means to solve this problem? Or would I have
to change core make behavior to make something like this happen? Note
that a complicating factor is that I sometimes have make blueprints that
span several targets (because I sometimes have to not just find a
recipe, but that could itself require additional prerequisites).

Best regards,
David


On 10/16/19 2:18 PM, Paul Smith wrote:

> On Wed, 2019-10-16 at 08:10 +0200, David Deutsch wrote:
>> My problem is that I keep running into situations where an .mk is
>> -include'd as seen above, but make never attempts to... make it,
>> resulting in a 'No rule to make target' error. I suspect there is some
>> kind of race condition going on where it does "see" the include, but
>> missing dependency error is triggered before that can occur. This
>> happens consistently, no matter whether I use -j or not.
> If you don't use -j, then it can't be a race condition.  Make is not
> multithreaded so without -j it should be completely deterministic.  Are
> you saying that _without_ -j you get nondeterministic failures?
>
> You didn't mention what version of GNU make you're using.  Perhaps
> there's a bug in that version?
>
> Also, note that with newer versions of GNU make you don't need to use
> "-include" to avoid warnings on non-existent makefiles.
>
> I recommend you get to a sufficiently-new version of GNU make and
> change all your "-include" to just "include" and see if that helps.
>

_______________________________________________
Help-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-make
Reply | Threaded
Open this post in threaded view
|

Re: Brittle -includes

David Deutsch
To update on this: I managed to figure out the first issue. It was
simply two paths that looked alike, one working, the other not working.

On the top of my make wishlist would be to have it give me a little more
detail than "No rule to make X". In many cases, what it really means is
"No rule to make Y which I'd need to make X which is what I'm currently
trying to make". I might know what the issue with trying to make Y is,
but baffled over how it is stuck on making X. So I guess something like
a short trace on the current set of dependencies make is trying to resolve.

Any pointers on the add-on question would be very much appreciated.

best regards,
David


On 10/16/19 3:27 PM, David Deutsch wrote:

> Hi Paul,
>
> Thank you for getting back to me (and thank you so much for all your work!).
>
> Race Condition - That was poorly worded on my part. I was thinking more
> in terms of how make processes each makefile. I currently put the
> includes at the end of each (generated) makefile and was thinking that
> maybe if it reads (and tries to execute) the recipe before reading (and
> making) the "include" makefile, that could produce issues. The
> nondeterministic behavior probably isn't one, but it seems weird to me
> that the same generated makefiles work in one case and fail in another.
>
> I've replaced all instances of "-include" with "include" and I'm now
> running into problems creating .d files in another part of the setup. I
> guess because while it's not throwing a warning, it doesn't write or
> include that makefile, now? I've read the manual on "How Makefiles Are
> Remade" a few dozen times now, but I'm still not too sure about the
> initial case - at what point Makefiles that don't yet exist are written.
>
> I'm using Version 4.2.1. Do you think updating to 4.2.92 would make a
> difference?
>
>
> Another question, if I may: As I wrote earlier, I'm mostly doing this to
> extend the pattern rule behavior of make by creating makefiles (and thus
> targets with their recipe) for paths that match a certain regular
> expression. It seems a bit… excessive to write hundreds, if not
> thousands of makefiles and I worry that I'm going to run into
> scalability issues at some point (luckily, we live in an era of SSDs, so
> it hasn't been too much of a pain so far).
>
> Is there a different direction that you could point me to, here, like
> using Guile or some other means to solve this problem? Or would I have
> to change core make behavior to make something like this happen? Note
> that a complicating factor is that I sometimes have make blueprints that
> span several targets (because I sometimes have to not just find a
> recipe, but that could itself require additional prerequisites).
>
> Best regards,
> David
>
>
> On 10/16/19 2:18 PM, Paul Smith wrote:
>> On Wed, 2019-10-16 at 08:10 +0200, David Deutsch wrote:
>>> My problem is that I keep running into situations where an .mk is
>>> -include'd as seen above, but make never attempts to... make it,
>>> resulting in a 'No rule to make target' error. I suspect there is some
>>> kind of race condition going on where it does "see" the include, but
>>> missing dependency error is triggered before that can occur. This
>>> happens consistently, no matter whether I use -j or not.
>> If you don't use -j, then it can't be a race condition.  Make is not
>> multithreaded so without -j it should be completely deterministic.  Are
>> you saying that _without_ -j you get nondeterministic failures?
>>
>> You didn't mention what version of GNU make you're using.  Perhaps
>> there's a bug in that version?
>>
>> Also, note that with newer versions of GNU make you don't need to use
>> "-include" to avoid warnings on non-existent makefiles.
>>
>> I recommend you get to a sufficiently-new version of GNU make and
>> change all your "-include" to just "include" and see if that helps.
>>
> _______________________________________________
> Help-make mailing list
> [hidden email]
> https://lists.gnu.org/mailman/listinfo/help-make

_______________________________________________
Help-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-make
Reply | Threaded
Open this post in threaded view
|

Re: Brittle -includes

Paul Smith-20
On Mon, 2019-10-21 at 05:55 +0200, David Deutsch wrote:
> On the top of my make wishlist would be to have it give me a little more
> detail than "No rule to make X". In many cases, what it really means is
> "No rule to make Y which I'd need to make X which is what I'm currently
> trying to make". I might know what the issue with trying to make Y is,
> but baffled over how it is stuck on making X. So I guess something like
> a short trace on the current set of dependencies make is trying to resolve.

I can only assume that the rules in question are pattern rules.  If
they were explicit rules then make would indeed give you a more clear
declaration of which file is missing.

However, with pattern rules this is not possible, because there can be
many pattern rules that might match the same target.

Consider the pattern target "%.o".  A .o file can be built from many
different types of sources: assembly files, C source files, C++ source
files, FORTRAN files, etc. etc.  So, if a prerequisite for "foo.o"
exists in the makefile but make cannot detect any pattern rule that
would successfully build it, make doesn't know which of those pattern
rules you expected it to match.  All it knows is that it didn't match
any of them.

> Any pointers on the add-on question would be very much appreciated.

Unfortunately I didn't really understand the process you're using.  Why
do you need to write hundreds or thousands of makefiles?

But in any event, this is not an unusual occurrence.  On a typical make
build system with auto-generated prerequisites it's not at all unusual
to have one makefile per object file defining that object file'd
prerequisites, and there can easily be hundreds or even thousands of
these in a larger project.


_______________________________________________
Help-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-make
Reply | Threaded
Open this post in threaded view
|

Re: Brittle -includes

David Deutsch
Hi Paul,

On 10/21/19 7:29 AM, Paul Smith wrote:
> I can only assume that the rules in question are pattern rules.  If
> they were explicit rules then make would indeed give you a more clear
> declaration of which file is missing.

Actually, I'm quite sure that they are explicit rules. I will try to put
together a simple example to recreate the issue.

> Unfortunately I didn't really understand the process you're using.  Why
> do you need to write hundreds or thousands of makefiles?

I think the process I'm using is best understood by its requirement:
Pattern rules with multiple, named %'s. I call them blueprints and they
produce makefiles that give you the set of targets and recipes to create
complex prerequisites.

Think: A document will incorporate and thus depends on multiple versions
of the same source image with different aspect ratios. Those are
referenced as public/image/filename-[width]-[height].jpg (matching
src/filename.jpg and using [width] and [height] in their recipe) and
that path in turn matches against a blueprint to produce such a file.

Those rules, however, are very regular and it feels not that far removed
from pattern rules.

A fake syntax that is more make-like might look like this:

public/image/%filename%-%width%-%height%.jpg: src/%filename%.jpg
    mybin/magicaspectratio $< $(width) $(height) > $@

There is of course some overlap with regular variables - those "recipe
variables" are scoped only to the recipe. Maybe it'd make more sense to
have a syntax like $*(name) to match with the automatic $* stem variable.

Did that clarify my process?

best regards,
David


_______________________________________________
Help-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-make
Reply | Threaded
Open this post in threaded view
|

Re: Brittle -includes

Paul Smith-20
On Mon, 2019-10-21 at 07:51 +0200, David Deutsch wrote:
> On 10/21/19 7:29 AM, Paul Smith wrote:
> > I can only assume that the rules in question are pattern rules.  If
> > they were explicit rules then make would indeed give you a more clear
> > declaration of which file is missing.
>
> Actually, I'm quite sure that they are explicit rules. I will try to put
> together a simple example to recreate the issue.

OK.

> > Unfortunately I didn't really understand the process you're using.  Why
> > do you need to write hundreds or thousands of makefiles?
>
> I think the process I'm using is best understood by its requirement:
> Pattern rules with multiple, named %'s. I call them blueprints and they
> produce makefiles that give you the set of targets and recipes to create
> complex prerequisites.

OK, but, make doesn't support multiple pattern characters in a single
target, as you're aware.  So I'm not sure what exactly you're asking us
to comment on.


_______________________________________________
Help-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-make
Reply | Threaded
Open this post in threaded view
|

Re: Brittle -includes

David Deutsch
Hi Paul,

Sorry, that was part of my earlier email - I understand that this is not possible currently. The question was - if I wanted to implement the behavior in make itself, would I have to basically write my own make? I know that Guile is a choice for extending make with custom code, but it seems to me like that is only for adding functions, not adding to core behavior like pattern rules, or am I missing something there?

best regards,
David

Am 21. Oktober 2019 14:36:04 MESZ schrieb Paul Smith <[hidden email]>:

>On Mon, 2019-10-21 at 07:51 +0200, David Deutsch wrote:
>> On 10/21/19 7:29 AM, Paul Smith wrote:
>> > I can only assume that the rules in question are pattern rules.  If
>> > they were explicit rules then make would indeed give you a more
>clear
>> > declaration of which file is missing.
>>
>> Actually, I'm quite sure that they are explicit rules. I will try to
>put
>> together a simple example to recreate the issue.
>
>OK.
>
>> > Unfortunately I didn't really understand the process you're using.
>Why
>> > do you need to write hundreds or thousands of makefiles?
>>
>> I think the process I'm using is best understood by its requirement:
>> Pattern rules with multiple, named %'s. I call them blueprints and
>they
>> produce makefiles that give you the set of targets and recipes to
>create
>> complex prerequisites.
>
>OK, but, make doesn't support multiple pattern characters in a single
>target, as you're aware.  So I'm not sure what exactly you're asking us
>to comment on.

--
Diese Nachricht wurde von meinem Android-Gerät mit K-9 Mail gesendet.
_______________________________________________
Help-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-make