How to debug my Makefile

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

How to debug my Makefile

Stefan Monnier
My `make` complains that

    make: execvp: /bin/sh: Argument list too long
    make: execvp: /bin/sh: Argument list too long

but I'm having trouble figuring out which command's argument list is
too long.  I tried "make -d" but it doesn't really help:

    % LANG=C make -d
    GNU Make 4.2.1
    Built for i686-pc-linux-gnu
    Copyright (C) 1988-2016 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.
    Reading makefiles...
    Reading makefile 'GNUmakefile'...
    make: execvp: /bin/sh: Argument list too long
    make: execvp: /bin/sh: Argument list too long
    Updating makefiles....
     Considering target file 'GNUmakefile'.
      Looking for an implicit rule for 'GNUmakefile'.
      Trying pattern rule with stem 'GNUmakefile'.
      Trying implicit prerequisite 'GNUmakefile.o'.
      Trying pattern rule with stem 'GNUmakefile'.
      Trying implicit prerequisite 'GNUmakefile.c'.
      Trying pattern rule with stem 'GNUmakefile'.
      Trying implicit prerequisite 'GNUmakefile.cc'.
      Trying pattern rule with stem 'GNUmakefile'.
      Trying implicit prerequisite 'GNUmakefile.C'.
      Trying pattern rule with stem 'GNUmakefile'.
      Trying implicit prerequisite 'GNUmakefile.cpp'.
      Trying pattern rule with stem 'GNUmakefile'.
      Trying implicit prerequisite 'GNUmakefile.p'.
      Trying pattern rule with stem 'GNUmakefile'.
      Trying implicit prerequisite 'GNUmakefile.f'.
      Trying pattern rule with stem 'GNUmakefile'.
    [...]

As you can see, the error seems to be signaled "early".
How could I track it down?

Looking at the make file I see one $(shell ..) invocation which is the
most likely culprit:

    define SET-diff
    $(shell echo "$(1)" "$(2)" "$(2)" | tr ' ' '\n' | sort | uniq -u)
    endef

which I use for example in:

    extra_els := $(call SET-diff, $(els), $(patsubst %.elc, %.el, $(current_elcs)))

where `els` and `current_elcs` hold lists of file names (and there are
more than 1000 files).  Is there some way to compute this set-difference
without going through the shell, or passing the content of those
variables via stdin?


        Stefan


_______________________________________________
Help-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-make
Reply | Threaded
Open this post in threaded view
|

Re: How to debug my Makefile

Basile Starynkevitch


On 07/20/2018 03:39 AM, Stefan Monnier wrote:
> My `make` complains that
>
>      make: execvp: /bin/sh: Argument list too long
>      make: execvp: /bin/sh: Argument list too long
>
> but I'm having trouble figuring out which command's argument list is
> too long.  I tried "make -d" but it doesn't really help:

You might consider using make --trace and remake -x.

See http://bashdb.sourceforge.net/remake/

With remake you probably would be able to understand what is happening.

Cheers

--
Basile STARYNKEVITCH   == http://starynkevitch.net/Basile
opinions are mine only - les opinions sont seulement miennes
Bourg La Reine, France


_______________________________________________
Help-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-make
Reply | Threaded
Open this post in threaded view
|

How to debug my Makefile

thutt
In reply to this post by Stefan Monnier
Stefan Monnier writes:
 > My `make` complains that
 >
 >     make: execvp: /bin/sh: Argument list too long
 >     make: execvp: /bin/sh: Argument list too long
 >
 > but I'm having trouble figuring out which command's argument list is
 > too long.  I tried "make -d" but it doesn't really help:
 >
 >     % LANG=C make -d

 <snip>

 > As you can see, the error seems to be signaled "early".
 > How could I track it down?
 >


 If 'make -n' doesn't point out the long command line, then your
 diagnosis that it's the use of $(shell) is probably the culprit.


 > Looking at the make file I see one $(shell ..) invocation which is the
 > most likely culprit:
 >
 >     define SET-diff
 >     $(shell echo "$(1)" "$(2)" "$(2)" | tr ' ' '\n' | sort | uniq -u)
 >     endef

 Could you use a standard Make rule to generate an output file, and
 then use redirection to provide the output file to any programs.

 Within Gnu Make, it's pretty easy to $(sort) sort a list, but
 de-duplicating it is a bit tougher.  It could be done fairly easily
 with the associative arrays or sets in the Gnu Make Standard Library:

   https://gmsl.sourceforge.io/

_______________________________________________
Help-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-make
Reply | Threaded
Open this post in threaded view
|

Re: How to debug my Makefile

Paul Smith-20
On Fri, 2018-07-20 at 06:22 -0700, [hidden email] wrote:
>  Within Gnu Make, it's pretty easy to $(sort) sort a list, but
>  de-duplicating it is a bit tougher.

GNU make's sort also uniquifies...

https://www.gnu.org/software/make/manual/html_node/Text-Functions.html

$(sort list)
    Sorts the words of list in lexical order, **removing duplicate words.**
    The output is a list of words separated by single spaces.

_______________________________________________
Help-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-make
Reply | Threaded
Open this post in threaded view
|

Re: How to debug my Makefile

Stefan Monnier
In reply to this post by thutt
>  > Looking at the make file I see one $(shell ..) invocation which is the
>  > most likely culprit:
>  >
>  >     define SET-diff
>  >     $(shell echo "$(1)" "$(2)" "$(2)" | tr ' ' '\n' | sort | uniq -u)
>  >     endef
>
>  Could you use a standard Make rule to generate an output file,

How could I generate an output file via a Make rule when the desired
content of the file is in a Make variable that's too big for the
command line?

>  and then use redirection to provide the output file to any programs.

SET-diff is used to build Make variables which are then used as "lists
of targets", so I'd have to use a nested Make invocation :-(

>  Within Gnu Make, it's pretty easy to $(sort) sort a list, but
>  de-duplicating it is a bit tougher.

I need a set-difference, not a deduplication.

>  It could be done fairly easily
>  with the associative arrays or sets in the Gnu Make Standard Library:
>    https://gmsl.sourceforge.io/

Wow, heroic!
I'll see how/if I can make use of it, thanks,


        Stefan

_______________________________________________
Help-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-make
Reply | Threaded
Open this post in threaded view
|

Re: How to debug my Makefile

Paul Smith-20
On Fri, 2018-07-20 at 13:27 -0400, Stefan Monnier wrote:
> How could I generate an output file via a Make rule when the desired
> content of the file is in a Make variable that's too big for the
> command line?

If you are using GNU make 4.0 or better you can use the $(file ...)
function to write to a file (there is no limit, other than memory, to
the size of a variable in GNU make and so no limit to the amount of
content you can add to a file with the $(file ...) function), then
invoke a shell script on that file instead of using "echo ...".

https://www.gnu.org/software/make/manual/html_node/File-Function.html


_______________________________________________
Help-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-make
Reply | Threaded
Open this post in threaded view
|

Re: How to debug my Makefile

Stefan Monnier
>> How could I generate an output file via a Make rule when the desired
>> content of the file is in a Make variable that's too big for the
>> command line?
>
> If you are using GNU make 4.0 or better you can use the $(file ...)
> function to write to a file (there is no limit, other than memory, to
> the size of a variable in GNU make and so no limit to the amount of
> content you can add to a file with the $(file ...) function), then
> invoke a shell script on that file instead of using "echo ...".
>
> https://www.gnu.org/software/make/manual/html_node/File-Function.html

Aha!  Thanks, that might do the trick!


        Stefan


_______________________________________________
Help-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-make
Reply | Threaded
Open this post in threaded view
|

Re: How to debug my Makefile

thutt
In reply to this post by Stefan Monnier
Stefan Monnier writes:
 > >  > Looking at the make file I see one $(shell ..) invocation which is the
 > >  > most likely culprit:
 > >  >
 > >  >     define SET-diff
 > >  >     $(shell echo "$(1)" "$(2)" "$(2)" | tr ' ' '\n' | sort | uniq -u)
 > >  >     endef
 > >
 > >  Could you use a standard Make rule to generate an output file,
 >
 > How could I generate an output file via a Make rule when the desired
 > content of the file is in a Make variable that's too big for the
 > command line?

 You can use the 'foreach' function to split each individual file into
 a seaprate output, thereby avoiding a too-long command line.  In the
 example below, I construct a long list of files using 'wildcard', and
 then output them one-by-one to a file in /tmp.

.PHONY: default
/tmp/headers:
        [ ! -e $@ ] || rm $@;
        @$(foreach fname,$(wildcard /usr/include/*.h),echo "file: $(fname)" >>$@;)

 > >  It could be done fairly easily
 > >  with the associative arrays or sets in the Gnu Make Standard Library:

 <snip, URL -- avoid silly 'protection' provided by corp network>

 >
 > Wow, heroic!
 > I'll see how/if I can make use of it, thanks,
 >

 The author of the GMSL has a pretty good (IMHO) book on obscure (but
 useful) things you can do with Gnu Make.  It's called Gnu Make Unleashed.

--
A containerized AI has launched a pingflood siege on my microservice.

_______________________________________________
Help-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-make