Evaluating a variable only once after a recipe has run

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Evaluating a variable only once after a recipe has run

R. Diez
Hi all:

I am investigating the OpenWrt build system, which is a big, complex makefile.

I recently came across the following definition in this file:

https://git.openwrt.org/?p=openwrt/openwrt.git;a=blob;f=toolchain/musl/Makefile;h=2b9312bcbf123c03cf8947c52044557e27377e79;hb=HEAD

MUSL_MAKEOPTS = -C $(HOST_BUILD_DIR) \
          DESTDIR="$(TOOLCHAIN_DIR)/" \
          LIBCC="$(subst libgcc.a,libgcc_initial.a,$(shell $(TARGET_CC) -print-libgcc-file-name))"

Note the call to $(shell ...) every time a compilation process that uses those compilation flags is started. In this case, it 's not that bad, because
those flags are only used once to run another GNU Make instance, but in some other compilation scenarios that can be very inefficient.

You would normally use MUSL_MAKEOPTS := (simply expanded variable) instead of just "=" (recursively expanded variable), in order to evaluate that
variable only once. The trouble is, the cross-compiler (TARGET_CC) that this variable wants to run is only available after a certain makefile recipe
has executed. Even if it were available, you would not want to run expensive shell commands to define a variable that is actually never used if the
user did not specify a makefile target that needs it.

I will rephrase the question just in case it is not clear. I want GNU Make to run recipe A, and then use the result of recipe A to compute the value
of a makefile variable, but only once. Other recipes that follow should use that value without evaluating the variable definition again.

It is kind of the reverse of "Target-specific Variable Values". A recipe defines a value that is valid for all other dependants that run afterwards.

Is that possible with GNU Make?

If not, what is the best way to handle the situation above?

Thanks in advance,
   rdiez

Reply | Threaded
Open this post in threaded view
|

Re: Evaluating a variable only once after a recipe has run

Kaz Kylheku (gmake)
On 2020-04-18 05:55, R. Diez wrote:

> Even if it were available, you
> would not want to run expensive shell commands to define a variable
> that is actually never used if the user did not specify a makefile
> target that needs it.
>
> I will rephrase the question just in case it is not clear. I want GNU
> Make to run recipe A, and then use the result of recipe A to compute
> the value of a makefile variable, but only once. Other recipes that
> follow should use that value without evaluating the variable
> definition again.

That's typically the job of a configure script. A configure
script can rely on special targets in the Makefile for doing some of
its probing. It deposits its results in an include makefile
like "config.make".

Instead of trying to uses the execution of a target recipe to try to
calculate this optimized variable (which isn't how make works,
and will likely be fruitless) you can calculate it using a conditional
expression which looks for the presence of that target in
$(MAKECMDGOALS).

   THAT_VAR := $(if $(filter that-target,$(MAKECMDGOALS)),$(shell ...))

In fact MAKECMDGOALS can even be tested with ifeq/ifneq to conditionally
include makefile text:

   ifneq($(filter that-target,$(MAKECMDGOALS)),)
   # chunk of makefile syntax here
   THAT_VAR := $(subst ... $(shell ...))
   endif




Reply | Threaded
Open this post in threaded view
|

Re: Evaluating a variable only once after a recipe has run

R. Diez
First of all, thanks for the quick response.

> That's typically the job of a configure script. A configure
> script can rely on special targets in the Makefile for doing some of
> its probing. It deposits its results in an include makefile
> like "config.make".

I am afraid that this does not really cut it, at least in big, real-life projects.

That actually means that all such projects need a previous step in another scripting language, like a configuration script. But in this particular
example with OpenWrt, building a cross-compilation toolchain is not the first step, but it is already midway in the makefile. So it is not like a
top-level script could run the makefile for some special targets to do the probing. It needs to run half of the makefile to get there.

The OpenWrt build system downloads and builds many other tools, including tools for the host which do not depend on the cross-compilation target. If
we could break the build system into 2 separate steps, then you could no longer take full advantage of parallelism: the 1st step must complete before
the 2nd step.

It is a shame that the variable evaluation I need is not possible in GNU Make.


> Instead of trying to uses the execution of a target recipe to try to
> calculate this optimized variable (which isn't how make works,
> and will likely be fruitless) you can calculate it using a conditional
> expression which looks for the presence of that target in $(MAKECMDGOALS).
>
>    THAT_VAR := $(if $(filter that-target,$(MAKECMDGOALS)),$(shell ...))
>
> In fact MAKECMDGOALS can even be tested with ifeq/ifneq to conditionally
> include makefile text:
>
>    ifneq($(filter that-target,$(MAKECMDGOALS)),)
>    # chunk of makefile syntax here
>    THAT_VAR := $(subst ... $(shell ...))
>    endif


That does not cut it either, I'm afraid. Say that the makefile has already run the cross-compiler recipe. The next invocation may not request it as a
goal any more. The variable is not actually needed for the cross-compiler recipe, but for all targets that depend on it. The variable's definition
uses the cross-compiler, but any other target later one may use that variable.

I would have to filter for any target that ultimately depends on the cross-compiler target, and that is not really feasible.

Best regards,
   rdiez


Reply | Threaded
Open this post in threaded view
|

Re: Evaluating a variable only once after a recipe has run

Kaz Kylheku (gmake)
On 2020-04-18 06:52, R. Diez wrote:

>> Instead of trying to uses the execution of a target recipe to try to
>> calculate this optimized variable (which isn't how make works,
>> and will likely be fruitless) you can calculate it using a conditional
>> expression which looks for the presence of that target in
>> $(MAKECMDGOALS).
>>
>>    THAT_VAR := $(if $(filter that-target,$(MAKECMDGOALS)),$(shell
>> ...))
>>
>> In fact MAKECMDGOALS can even be tested with ifeq/ifneq to
>> conditionally
>> include makefile text:
>>
>>    ifneq($(filter that-target,$(MAKECMDGOALS)),)
>>    # chunk of makefile syntax here
>>    THAT_VAR := $(subst ... $(shell ...))
>>    endif
>
>
> That does not cut it either, I'm afraid. Say that the makefile has
> already run the cross-compiler recipe. The next invocation may not
> request it as a goal any more.

Even if it is requested as a goal, it cannot work that way, because
the variable is calculated before any recipes are run.

The whole cross-compiler recipe will have to be handled by the logic
which also sets the variable.

Perhaps something like this:

    ifneq ($(MAKECMDGOALS),cross-toolchain) # prevent runaway recursion
      ifneq ($(filter that-target,$(MAKECMDGOALS)),)
        THAT_VAR := $(shell "make cross-toolchain; /path/to/tool --arg")
      endif
    endif

It's fairly gross that the Makefile rule base will be read twice
though due to the recursion, since the whole point of this exercise
is to optimize something. The optimization is still obtained for
targets which don't need the variable, but those which do are
impacted.

In this situation I would rather farm out the cross toolchain
build into a script, or a smaller Makefile or something. Or use some
conditionals to trim the rule base when cross-toolchain is the
only target.

> The variable is not actually needed for
> the cross-compiler recipe, but for all targets that depend on it. The
> variable's definition uses the cross-compiler, but any other target
> later one may use that variable.

> I would have to filter for any target that ultimately depends on the
> cross-compiler target, and that is not really feasible.

Isn't it? Since we can just add more words to the filter, it doesn't
seem like a difficulty.

    ifneq($(filter a-target b-target c-target,$(MAKECMDGOALS))
    ABCVAR := ...
    endif

Do you think there will be more than (let's pick a number) 17 of these
targets,
in the foreseeable lifetime of the project?

If those targets are part of some extensible system like per-directory
include makefiles, that does get ugly. But even that is solvable,
because
all the per-directory include makfefiles can be included first. They can
declare their target via some variable like this:

    # local per-directory include file

    TARGETS_NEEDING_VAR += my_local_target

    my_local_target: prereq
       cmd $(THAT_VAR) ...

These all get included, before we do the filter in the top-level
Makefile logic:

    ifneq($(filter $(TARGETS_NEEDING_VAR),$(MAKECMDGOALS))
    THAT_VAR := ...
    endif

Given all the lists of targets that get manualy maintained in all kinds
of makefiles
it doesn't seem like a big deal. E.g. OBJS := definitions with hundreds
of entries and whatnot.

To catch errors (someone adds a new target and doesn't add it to the
filter),
we can add an else clause which defines the variable in such a way that
bad
syntax will almost certainly result.

     else
     THAT_VAR = ("
     endif

or some better idea to make the dependent recipes fail. Or else
--warn-undefined-variables could be of use, possibly.

In the end, this is just an optimization anyway. Possibly, a negative
test
might be possible: test for the exclusive presence of certain targets
which certainly do not need that variable and define it under all other
circumstances.

Maybe the targets can follow some naming pattern which can be exploited
to condense the filter expression.


Reply | Threaded
Open this post in threaded view
|

Re: Evaluating a variable only once after a recipe has run

Paul Smith-20
In reply to this post by R. Diez
On Sat, 2020-04-18 at 14:55 +0200, R. Diez wrote:
> I will rephrase the question just in case it is not clear. I want GNU
> Make to run recipe A, and then use the result of recipe A to compute
> the value of a makefile variable, but only once. Other recipes that
> follow should use that value without evaluating the variable
> definition again.

You may find this trick from my blog to be helpful:

http://make.mad-scientist.net/deferred-simple-variable-expansion/

Cheers!


Reply | Threaded
Open this post in threaded view
|

Re: Evaluating a variable only once after a recipe has run

R. Diez
In reply to this post by Kaz Kylheku (gmake)

 > You may find this trick from my blog to be helpful:
 >
 > http://make.mad-scientist.net/deferred-simple-variable-expansion/

Many thanks, that is exactly what I was looking for!

Reply | Threaded
Open this post in threaded view
|

Re: Evaluating a variable only once after a recipe has run

R. Diez
>  > You may find this trick from my blog to be helpful: >  >>  > http://make.mad-scientist.net/deferred-simple-variable-expansion/
Would it be possible to encapsulate it somehow, so that other people immediately know what is going on?

I mean something like this:


# This definition does this because of that...

define DeferredSimpleVariableExpansion =

   ... something complicated here like this ...

   OUTPUT = $(eval OUTPUT := $$(shell some-comand))$(OUTPUT)

endef


Then the makefile user only has to do this:


$(call DeferredSimpleVariableExpansion VARIABLE some shell command with some arguments)


The name of the function is enough of a hint, and if not, the function definition has some advanced explanation.

Thanks in advance,
   rdiez

Reply | Threaded
Open this post in threaded view
|

Re: Evaluating a variable only once after a recipe has run

Paul Smith-20
On Sat, 2020-04-18 at 19:34 +0200, R. Diez wrote:
> >   > You may find this trick from my blog to be helpful: >  >>  >
> > http://make.mad-scientist.net/deferred-simple-variable-expansion/
>
> Would it be possible to encapsulate it somehow, so that other people
> immediately know what is going on?

I don't see why not.  Do you think there is some reason it wouldn't be
possible?


Reply | Threaded
Open this post in threaded view
|

Re: Evaluating a variable only once after a recipe has run

R. Diez

> I don't see why not.  Do you think there is some reason it wouldn't be
> possible?

Hahaha. I was obviously trying to get you to write it. Well, maybe just the code, I can write the comments myself... O8-)

You know, it's rather tricky stuff, I did not quite understand the escaping issue, you already have a nice blog entry about that, and you know GNU
Make much better. Last time I wrote a define / endef I needed emergency holiday, especially because of the extra escaping...  Are all these excuses
enough???

All the best,
   rdiez

Reply | Threaded
Open this post in threaded view
|

Re: Evaluating a variable only once after a recipe has run

Paul Smith-20
On Sat, 2020-04-18 at 21:17 +0200, R. Diez wrote:
> > I don't see why not.  Do you think there is some reason it wouldn't
> > be possible?
>
> Hahaha. I was obviously trying to get you to write it. Well, maybe
> just the code, I can write the comments myself... O8-)

Oh.  Well, you should just ask :)

You need to escape all the "$" other than the ones you want to be
evaluated by the call (e.g., the arguments to the call).

Something like:

  Deferred = $1 = $$(eval $1 := $$$$(shell $2))$$($1)

Then:

  $(eval $(call Deferred,OUTPUT,some-command))

Defines OUTPUT as a deferred variable assigned to running some-command
in $(shell ...)

There are issues with this.  If the command you want to run has
unmatched ")" you have problems.  Also if you want use a literal '$' in
some-command you'll have problems getting enough escaping although it
can be done.

But for simple commands it should work.

Personally I'm not convinced the eval/call version is more readable but
YMMV.


Reply | Threaded
Open this post in threaded view
|

Re: Evaluating a variable only once after a recipe has run

Gnu - Make - Help mailing list
In reply to this post by R. Diez
 > Oh.  Well, you should just ask :)

Many thanks, you are a star.  ;-)


 > [...]
 > Deferred = $1 = $$(eval $1 := $$$$(shell $2))$$($1)

Is there a way to allow variable number of arguments? This is so that all of the following work:

   $(eval $(call Deferred,OUTPUT,some-command))
   $(eval $(call Deferred,OUTPUT,some-command arg1))
   $(eval $(call Deferred,OUTPUT,some-command arg1 arg2))

I could not see in the GNU Make documentation an indication about what happens if the function uses 2 arguments and you pass 3.


I have yet another question: why is $(eval $(call ...)) necessary? Is it not possible to write it so that the user only needs a $(call ) ?


 > There are issues with this.  If the command you want to run has
 > unmatched ")" you have problems.

I am guessing that escaping a ')' is not possible. But I can live with:

CLOSING_PARENTHESIS := )

   $(eval $(call Deferred,OUTPUT,some-command $(CLOSING_PARENTHESIS)))

Or will the nested $(eval) invocations break the usual trick with CLOSING_PARENTHESIS ?


 > Also if you want use a literal
 >'$' in some-command you'll have problems getting enough
 > escaping although it can be done.

You mean that $$ is not enough in this example?

   $(eval $(call Deferred,OUTPUT,some-command arg1 $$ arg2))


 > Personally I'm not convinced the eval/call version is more
 > readable but YMMV.

I believe that this is important. You have written "Whoa! What is going on here?" in your blog. I would like to avoid that very effect every time
somebody sees the trick inside a makefile. Having a function with an informative name avoids most questions. If there are any left, the function is
the single point where you can find all documentation and warnings about itr.

I would like to put together in the comments of that function a few examples of how to use it, what pitfalls there are with escaping, and yet more
examples about how to work-around those escaping pitfalls. Or maybe with examples of what is not possible. Then I would have a function that you can
use 95 % of the time without having to think too much about it.

Thanks again,
   rdiez