Flag --no-builtin-rules etc. just for this makefile (no flag inheritance)

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Flag --no-builtin-rules etc. just for this makefile (no flag inheritance)

R. Diez
Hi all:

Over the years, I have become used to specifying the following flags when running makefiles:

1) --no-builtin-variables  or manually-written makefiles.

That flag also disables all implicit rules.

The reasons are:

a) Makefiles run much faster if they do not have to check implicit rules.

If implicit rules are active, GNU make seems to look for many different possible source file types when trying to find a suitable rule for each target
file. That can trigger many stat syscalls per target file, dramatically slowing down makefiles for large projects.


b) I usually want files to get built in a specific way. If I make a mistake, I want to know straight away, instead of wondering why another compiler
has been automatically chosen.


2) Automake-generated makefiles need built-in variables, but they seem to run fine with --no-builtin-rules , which also helps run the makefile faster.


3) --warn-undefined-variables , because it helps catch bugs when writing and debugging makefiles.


In fact, I believe that the GNU Make documentation should encourage using those flags more explicitly.

There is a catch, though: those flags are inherited over environment variable MAKEFLAGS. So I have to manually filter them out when calling submakefiles.

I believe that this inheritance is the main reason why other build systems are always so much faster than GNU Make in practice. For example, the
OpenWrt build system is a complex makefile. Assuming it could run with --no-builtin-rules , all submakefiles would fail. Therefore, in practice, you
cannot easily use --no-builtin-rules in complex makefiles. Keep in mind that you need some inheritance, at least for the job server file descriptors.

If you are the author of a makefile, you may design it to run well with --no-builtin-variables and with --warn-undefined-variables . In fact, you may
want to run it always with those flags. But you do not want to pass them to submakefiles, just in case.

Therefore, it would be nice if you could specify those flags as file-level directives, like this:

.NO_BUILT_IN_RULES:
.NO_BUILT_IN_VARIABLES:
.WARN_UNDEFINED_VARIABLES:

Or even better:
.ERROR_UNDEFINED_VARIABLES:

Special targets .NOTPARALLEL and .ONESHELL do not inherit either, do they?

You cannot do that with GNU Make at the moment for --no-builtin-variables and --warn-undefined-variables, or can you?

Regards,
   rdiez


Reply | Threaded
Open this post in threaded view
|

Re: Flag --no-builtin-rules etc. just for this makefile (no flag inheritance)

Kaz Kylheku (gmake)
On 2020-04-18 06:27, R. Diez wrote:
> There is a catch, though: those flags are inherited over environment
> variable MAKEFLAGS. So I have to manually filter them out when calling
> submakefiles.

MAKEFLAGS is actually the solution. You can *add* to it and the flags
take effect.

> Therefore, it would be nice if you could specify those flags as
> file-level directives, like this:
>
> .NO_BUILT_IN_RULES:
> .NO_BUILT_IN_VARIABLES:
> .WARN_UNDEFINED_VARIABLES:

Thus:

   MAKEFLAGS += --no-builtin-rules ...

You do have to keep filtering that out for sub-makes, though.

Another idea is: instead of filtering MAKEFLAGS, save it and
restore it:

    SAVED_MAKEFLAGS := $(MAKEFLAGS)

Then in recursive recipes do MAKEFLAGS=$(SAVED_MAKEFLAGS) or whatever.


Reply | Threaded
Open this post in threaded view
|

Re: Flag --no-builtin-rules etc. just for this makefile (no flag inheritance)

R. Diez

> MAKEFLAGS is actually the solution. You can *add* to it and the flags take effect.

Thanks for the trick.

Unfortunately, the filtering is still a big issue in practice. If you take a look at OpenWrt's makefile, you will see that it is a huge beast with
many contributors. The same is true for other projects.

But even if you have a smaller project, it is a pain having to remember every time that you need to filter/restore your MAKEFLAGS for every sub-make.

It would be much better if GNU Make could do that automatically, like it does for settings like .ONESHELL .

Regards,
   rdiez

Reply | Threaded
Open this post in threaded view
|

Re: Flag --no-builtin-rules etc. just for this makefile (no flag inheritance)

Gnu - Make - Help mailing list
In reply to this post by R. Diez

 > You do have to keep filtering that out for sub-makes, though.
 >
 > Another idea is: instead of filtering MAKEFLAGS, save it
 > and restore it:
 >
 >   SAVED_MAKEFLAGS := $(MAKEFLAGS)
 >
 > Then in recursive recipes do MAKEFLAGS=$(SAVED_MAKEFLAGS) or whatever.


I tried to implement your suggestion, but I ran into the following problem.

The value of MAKEFLAGS at the beginning of my makefile is:

rRw --warn-undefined-variables

However, the value of MAKEFLAGS at the beginning of a recipe is:

rRw -j5 -Orecurse --jobserver-auth=3,4 --warn-undefined-variables

So I guess that GNU Make is injecting the jobserver flags into MAKEFLAGS before running each rule.

If I save the value of MAKEFLAGS at the beginning, like you suggested, and then use it in the recipe, I will lose the jobserver flags, right?

Thanks in advance,
   rdiez

Reply | Threaded
Open this post in threaded view
|

Re: Flag --no-builtin-rules etc. just for this makefile (no flag inheritance)

R. Diez
In reply to this post by R. Diez
Hallo Paul:

I have been investigating Kaz Kylheku's suggestion about adding flags to MAKEFLAGS inside the makefile.

I believe that adding --no-builtin-variables has no effect. It is probably too late to do that inside the makefile.

That means that option '--no-builtin-rules' does not get automatically enabled either.

I think that GNU Make's documentation should mention the fact that --no-builtin-variables does not work in this scenario. I would be even better if
GNU Make issued a warning when the makefile does that.

Using GNU Make is very time-consuming because it is full of limitations and quirks. Every little help counts.

Best regards,
   rdiez

Reply | Threaded
Open this post in threaded view
|

Re: Flag --no-builtin-rules etc. just for this makefile (no flag inheritance)

Kaz Kylheku (gmake)
On 2020-04-20 03:04, R. Diez wrote:

> Hallo Paul:
>
> I have been investigating Kaz Kylheku's suggestion about adding flags
> to MAKEFLAGS inside the makefile.
>
> I believe that adding --no-builtin-variables has no effect. It is
> probably too late to do that inside the makefile.
>
> That means that option '--no-builtin-rules' does not get automatically
> enabled either.

It can be verified with a minimal Makefile that "make --print-data-base"
shows an absence of built-in rules if MAKEFLAGS += --no-builtin-rules is
present, and likewise that --no-builtin-variables causes the variables
to disappear.

However, environment variables do not disappear. If CC or whatever
is coming down as an environment variable, then of course it stays.

About this other business of calculating a variable inside a recipe
to be used by dependent recipe, the path of least resistance would
be to make it a shell variable.

    base-recipe:
      ...
      heavy-program --option > .base-recipe-var

    dependent-recipe: base-recipe
      VAR=$$(cat .base-recipe-var) ; command $$VAR; ... ; ...

The dependent recipe now just has to read a file rather than execute
heavy-program.

If the datum is small enough, it could be stored in a symlink. Symlinks
whose targets are small enough are stored entirely in the inode
structure
by some Unix-like file systems.

    base-recipe:
      ...
      ln -sf (heavy-program --show-important-path) .base-recipe-symlink

If the datum is a path, that symlink could be used directly:

    dependent-recipe: base-recipe
      IMPORTANT_PATH=.base-recipe-symlink ; command $$IMPORTANT_PATH ;
...

Or it could be resolved, if the commands need the original item:

    dependent-recipe: base-recipe
      IMPORTANT_PATH=$(readlink .base-recipe-symlink) ; command
$$IMPORTANT_PATH ; ...




Reply | Threaded
Open this post in threaded view
|

Re: Flag --no-builtin-rules etc. just for this makefile (no flag inheritance)

R. Diez

> It can be verified with a minimal Makefile that "make --print-data-base"
> shows an absence of built-in rules if MAKEFLAGS += --no-builtin-rules is
> present, and likewise that --no-builtin-variables causes the variables
> to disappear.
 > [...]

Well, let's verify it then:

Contents of file "makefile":

   MAKEFLAGS += --no-builtin-variables

   $(info CC: $(CC))

Commands I used to test this scenario:

   $ make --version
   GNU Make 4.3
   Built for x86_64-pc-linux-gnu

   $ echo "The value of CC is: $CC"
   The value of CC is:

   $ make --file makefile
   CC: cc
   make: *** No targets. Stop.

   $ make --file makefile  --no-builtin-variables
   CC:
   make: *** No targets. Stop.

So it looks like "MAKEFLAGS += --no-builtin-variables" is not working,  is it?

Note that --no-builtin-rules does work. I meant that adding --no-builtin-variables to MAKEFLAGS does not automatically enable --no-builtin-rules, as
it does when --no-builtin-variables is specified on the command line (as documented).

Regards,
   rdiez

Reply | Threaded
Open this post in threaded view
|

Re: Flag --no-builtin-rules etc. just for this makefile (no flag inheritance)

Kaz Kylheku (gmake)
On 2020-04-20 09:18, R. Diez wrote:

>> It can be verified with a minimal Makefile that "make
>> --print-data-base"
>> shows an absence of built-in rules if MAKEFLAGS += --no-builtin-rules
>> is
>> present, and likewise that --no-builtin-variables causes the variables
>> to disappear.
>> [...]
>
> Well, let's verify it then:
>
> Contents of file "makefile":
>
>   MAKEFLAGS += --no-builtin-variables
>
>   $(info CC: $(CC))
>
> Commands I used to test this scenario:
>
>   $ make --version
>   GNU Make 4.3
>   Built for x86_64-pc-linux-gnu
>
>   $ echo "The value of CC is: $CC"
>   The value of CC is:
>
>   $ make --file makefile
>   CC: cc
>   make: *** No targets. Stop.
>
>   $ make --file makefile  --no-builtin-variables
>   CC:
>   make: *** No targets. Stop.
>
> So it looks like "MAKEFLAGS += --no-builtin-variables" is not working,  
> is it?

I see that in spite of the variables being scrubbedd from the
--print-data-base
view, $(CC) continues to evaluate to cc at the top-level.

This can be seen in the same invocation:

   $ cat mkfile
   MAKEFLAGS += --no-builtin-variables

   $(info CC is defined as: $(CC))


Test:

   $ make --print-data-base -f mkfile | grep CC
   make: *** No targets.  Stop.
   CC is defined as: cc

Now True Scotsman's brand industrial-strength no-builtin-variables:

   $ make --print-data-base --no-builtin-variables -f mkfile | grep CC
   make: *** No targets.  Stop.
   CC is defined as:

I think this has to do with the order of expansions. When the Makefile
is being read, CC is still defined, and so $(CC) expands. Then before
GNU Make actually starts processing the rule base, at that time it
honors the requests that have appeared in MAKEFLAGS.

In other words, it doesn't instantaneously react to the MAKEFLAGS +=
update.

If we change the makefile to this:

   MAKEFLAGS += --no-builtin-variables --no-builtin-rules

   $(info CC is defined as: $(CC))

   .PHONY: all
   all:
        echo [in rule] CC is defined as: $(CC)

Then:

   $ make -f  mkfile
   CC is defined as: cc
   echo [in rule] CC is defined as:
   [in rule] CC is defined as:

This means we can rely on --no-builtin-variables in MAKEFLAGS to work as
long
as we avoid top-level capture via the := operator.

   MAKEFLAGS += --no-builtin-variables --no-builtin-rules

   HARD_CAPTURE_CC := $(CC)

   SOFT_CAPTURE_CC = $(CC)

   $(info CC is defined as: $(CC))

   .PHONY: all
   all:
        @echo [in rule] CC is defined as: $(CC)
        @echo [in rule] SOFT_CAPTURE_CC is defined as: $(SOFT_CAPTURE_CC)
        @echo [in rule] HARD_CAPTURE_CC is defined as: $(HARD_CAPTURE_CC)

Run:

   $ make -f  mkfile
   CC is defined as: cc
   [in rule] CC is defined as:
   [in rule] SOFT_CAPTURE_CC is defined as:
   [in rule] HARD_CAPTURE_CC is defined as: cc



Incidentally, what does POSIX have to say about this? POSIX defines
MAKEFLAGS
and specifies its semantics as an environment variable being defined
before
make executes.  And it says this: "The result of setting MAKEFLAGS in
the
Makefile is unspecified."

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/make.html