GNU make 4.2 (and 4.2.1) failing due to length of command-line

classic Classic list List threaded Threaded
22 messages Options
12
Reply | Threaded
Open this post in threaded view
|

GNU make 4.2 (and 4.2.1) failing due to length of command-line

Adrian Muresan

For more complete info, go to here

http://stackoverflow.com/questions/37932984/gnu-make-4-2-and-4-2-1-on-windows-bug


I am making this target only


deploy_marketProperties: 
echo Copying application data... && $(MKDIR) "C:/Users/User1/Desktop/A/Proj/src/../lib/armle-v7/release/marketProperties" && $(COPY_DIR) $(wildcard C:/Users/User1/Desktop/A/Proj/src/subProj/marketProperties/*) "C:/Users/User1/Desktop/A/Proj/src/../lib/armle-v7/release/marketProperties"

Using following command
make deploy_marketProperties  --debug=vjm

On GNU make 3.81, it works fine. But on make 4.2 and 4.2.1, I get an error:

cp: target `C' is not a directory

The problem appears to be the length of the command. If I delete some JSON files, it works fine.

Here's output of make 3.81. Note: there's actually 264 JSON files but I only show a few here.
===============================================================================================================================================
GNU Make 3.81
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for Windows32
find_and_set_shell path search set default_shell = C:/QNX650/host/win32/x86/usr/bin/sh.exe
Reading makefiles...
Reading makefile `Makefile'...
Updating goal targets....
Considering target file `deploy_marketProperties'.
 File `deploy_marketProperties' does not exist.
 Finished prerequisites of target file `deploy_marketProperties'.
Must remake target `deploy_marketProperties'.
echo Copying application data... && mkdir -p "C:/Users/User/Desktop/A/Project/src/../lib/armle-v7/release/marketProperties" && cp -f -R C:/Users/User/Desktop/A/Project/src/SubProj/marketProperties/2blank.json C:/Users/User/Desktop/A/Project/src/SubProj/marketProperties/3blank.json C:/Users/User/Desktop/A/Project/src/SubProj/marketProperties/7_blank.json C:/Users/User/Desktop/A/Project/src/SubProj/marketProperties/aa.json  "C:/Users/User/Desktop/A/Project/src/../lib/armle-v7/release/marketProperties"
Main thread handle = 0x000000ac
Copying application data...
Successfully remade target file `deploy_marketProperties'.
===============================================================================================================================================


Output of make 4.2 (it's the same as 4.2.1).

NOTE: I did not put ALL 264 JSON files.

===============================================================================================================================================

GNU Make 4.2.1
Built for Windows32
Copyright (C) 1988-2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
find_and_set_shell() path search set default_shell = C:/QNX650/host/win32/x86/usr/bin/sh.exe
Reading makefiles...
Reading makefile 'Makefile'...
Updating makefiles....
Updating goal targets....
Considering target file 'deploy_marketProperties'.
 File 'deploy_marketProperties' does not exist.
 Finished prerequisites of target file 'deploy_marketProperties'.
Must remake target 'deploy_marketProperties'.
C:/Users/User1/Desktop/A/Project/src/SubProj/marketProperties/2blank.json C:/Users/Use1r/Desktop/A/Project/src/SubProj/marketProperties/3blank.json C:/Users/User1/Desktop/A/Project/src/SubProj/marketProperties/7_blank.json  
Main thread handle = 0000007C
Copying application data...
cp: target `C:/Users/User1/Des' is not a directory
make_msvc.net2003: *** [Makefile:384: deploy_marketProperties] Error 1

===============================================================================================================================================


It's pretty obvious what's happening: the line is "too long" and it's being cut-off but this only happens in 4.2/4.2.1 but not 3.81


The shell that's being used is:


C:\Users\User1\Desktop\A\Project\bld\armle-v7\release\subProj>where sh
C:\QNX650\host\win32\x86\usr\bin\sh.exe

C:\Users\User1\Desktop\A\Project\bld\armle-v7\release\subProj>sh --help
GNU bash, version 3.1.17(1)-release-(i686-pc-msys)
Usage:  sh [GNU long option] [option] ...
        sh [GNU long option] [option] script-file ...
GNU long options:
        --debug
        --debugger
        --dump-po-strings
        --dump-strings
        --help
        --init-file
        --login
        --noediting
        --noprofile
        --norc
        --posix
        --protected
        --rcfile
        --restricted
        --verbose
        --version
        --wordexp
Shell options:
        -irsD or -c command or -O shopt_option          (invocation only)
        -abefhkmnptuvxBCHP or -o option
Type `sh -c "help set"' for more information about shell options.
Type `sh -c help' for more information about shell builtin commands.
Use the `bashbug' command to report bugs.


_______________________________________________
Make-w32 mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/make-w32
Reply | Threaded
Open this post in threaded view
|

Re: GNU make 4.2 (and 4.2.1) failing due to length of command-line

Michael Stahl-6
On 22.06.2016 00:53, Adrian Muresan wrote:
> On GNU make 3.81, it works fine. But on make 4.2 and 4.2.1, I get an error:
>
> cp: target `C' is not a directory
>
> The problem appears to be the length of the command. If I delete some
> JSON files, it works fine.

FWIW i've noticed that with make 4.0/4.1 built for Win32, invoking the
Cygwin bash shell as C:/cygwin/bin/sh.exe, the command lines are
silently truncated at ~8K characters - this did not happen when we used
GNU make 3.82 built for Cygwin.

i haven't investigated where exactly that happens, the documented Win32
CreateProcess maximum command line length is 32k characters.

https://msdn.microsoft.com/en-us/library/windows/desktop/ms682425(v=vs.85).aspx

... our build system has a few more "response files" now :)



_______________________________________________
Make-w32 mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/make-w32
Reply | Threaded
Open this post in threaded view
|

Re: GNU make 4.2 (and 4.2.1) failing due to length of command-line

Eli Zaretskii
In reply to this post by Adrian Muresan
> From: Adrian Muresan <[hidden email]>
> Date: Tue, 21 Jun 2016 22:53:50 +0000
>
> For more complete info, go to here
>
> http://stackoverflow.com/questions/37932984/gnu-make-4-2-and-4-2-1-on-windows-bug

That discussion is full of misinformation, so let's ignore it, OK?

> I am making this target only
>
> deploy_marketProperties:
> echo Copying application data... && $(MKDIR)
> "C:/Users/User1/Desktop/A/Proj/src/../lib/armle-v7/release/marketProperties" && $(COPY_DIR) $(wildcard
> C:/Users/User1/Desktop/A/Proj/src/subProj/marketProperties/*)
> "C:/Users/User1/Desktop/A/Proj/src/../lib/armle-v7/release/marketProperties"
>
> Using following command
> make deploy_marketProperties --debug=vjm
>
> On GNU make 3.81, it works fine. But on make 4.2 and 4.2.1, I get an error:
>
> cp: target `C' is not a directory
>
> The problem appears to be the length of the command. If I delete some JSON files, it works fine.

I cannot reproduce this.  in general, Make on Windows uses APIs that
support command lines up to 32K bytes.  I just successfully ran a
Makefile that invoked a command with 23K bytes (which is more than
your 18K, as reported on Stackoverflow), with no problems.

So I think some other factor is at work here.  The first suspect is
the way you built Make 4.2.  Could you please try the binaries
available from this site:

  https://sourceforge.net/projects/ezwinports/files/?source=navbar

(I'm guessing you don't need Guile support in Make, so the
"without-guile" binary zip is for you.)  This is the binary I used in
the experiment described above.  The command line I tried was this:

  echo 123456789 123456789 123456789 123456789 123456789 ...

etc., up to 23K characters.  Also, since you seem to be using the MSYS
Bash, I used it as well (I have MSYS installed here).  I don't know
what kind of Bash build you got from the QNX toolkit, but at least the
output of "sh --help" I get here is identical to what you show, so
there's hope it's the same or similar binary.

So please try the above-mentioned binary distribution of Make 4.2, and
please try the simple command I used.  If that works, then I suggest
to try your real target with the same binary of Make.

_______________________________________________
Make-w32 mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/make-w32
Reply | Threaded
Open this post in threaded view
|

Re: GNU make 4.2 (and 4.2.1) failing due to length of command-line

Eli Zaretskii
In reply to this post by Michael Stahl-6
> From: Michael Stahl <[hidden email]>
> Date: Wed, 22 Jun 2016 13:58:41 +0200
>
> On 22.06.2016 00:53, Adrian Muresan wrote:
> > On GNU make 3.81, it works fine. But on make 4.2 and 4.2.1, I get an error:
> >
> > cp: target `C' is not a directory
> >
> > The problem appears to be the length of the command. If I delete some
> > JSON files, it works fine.
>
> FWIW i've noticed that with make 4.0/4.1 built for Win32, invoking the
> Cygwin bash shell as C:/cygwin/bin/sh.exe, the command lines are
> silently truncated at ~8K characters - this did not happen when we used
> GNU make 3.82 built for Cygwin.

You are talking about the Cygwin build of Make, yes?  That's a
different build from the one in the OP's report.

And yes, the native Windows build of Make supports command lines up to
32K bytes, except when it invokes cmd.exe, which itself is limited to
4K bytes.

_______________________________________________
Make-w32 mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/make-w32
Reply | Threaded
Open this post in threaded view
|

Re: GNU make 4.2 (and 4.2.1) failing due to length of command-line

Michael Stahl-6
On 22.06.2016 17:26, Eli Zaretskii wrote:

>> From: Michael Stahl <[hidden email]>
>> Date: Wed, 22 Jun 2016 13:58:41 +0200
>>
>> On 22.06.2016 00:53, Adrian Muresan wrote:
>>> On GNU make 3.81, it works fine. But on make 4.2 and 4.2.1, I get an error:
>>>
>>> cp: target `C' is not a directory
>>>
>>> The problem appears to be the length of the command. If I delete some
>>> JSON files, it works fine.
>>
>> FWIW i've noticed that with make 4.0/4.1 built for Win32, invoking the
>> Cygwin bash shell as C:/cygwin/bin/sh.exe, the command lines are
>> silently truncated at ~8K characters - this did not happen when we used
>> GNU make 3.82 built for Cygwin.
>
> You are talking about the Cygwin build of Make, yes?  That's a
> different build from the one in the OP's report.

no, Win32 make - i noticed while adapting the LibreOffice build system
from Cygwin make to Win32 make a couple years ago (which gave a large
speed improvement, see [1] for details) that some commands that used to
work before failed with Win32 make because the command lines were truncated.

> And yes, the native Windows build of Make supports command lines up to
> 32K bytes, except when it invokes cmd.exe, which itself is limited to
> 4K bytes.

i think i looked at it in a debugger and saw that the command line was
indeed passed to CreateProcess un-truncated but i forget the details, it
was some years ago, i don't think i found out where exactly the
truncation happened.

[1]
http://permalink.gmane.org/gmane.comp.documentfoundation.libreoffice.devel/59339



_______________________________________________
Make-w32 mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/make-w32
Reply | Threaded
Open this post in threaded view
|

Re: GNU make 4.2 (and 4.2.1) failing due to length of command-line

Eli Zaretskii
> From: Michael Stahl <[hidden email]>
> Date: Wed, 22 Jun 2016 17:43:44 +0200
>
> >> FWIW i've noticed that with make 4.0/4.1 built for Win32, invoking the
> >> Cygwin bash shell as C:/cygwin/bin/sh.exe, the command lines are
> >> silently truncated at ~8K characters - this did not happen when we used
> >> GNU make 3.82 built for Cygwin.
> >
> > You are talking about the Cygwin build of Make, yes?  That's a
> > different build from the one in the OP's report.
>
> no, Win32 make - i noticed while adapting the LibreOffice build system
> from Cygwin make to Win32 make a couple years ago (which gave a large
> speed improvement, see [1] for details) that some commands that used to
> work before failed with Win32 make because the command lines were truncated.

That certainly doesn't happen here -- but I don't use Cygwin programs
at all.

_______________________________________________
Make-w32 mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/make-w32
Reply | Threaded
Open this post in threaded view
|

Re: GNU make 4.2 (and 4.2.1) failing due to length of command-line

Eli Zaretskii
In reply to this post by Eli Zaretskii
> From: Adrian Muresan <[hidden email]>
> CC: "[hidden email]" <[hidden email]>
> Date: Wed, 22 Jun 2016 17:01:22 +0000
>
> I downloaded
> https://sourceforge.net/projects/ezwinports/files/make-4.2-without-guile-w32-bin.zip/download
>
> I ran the command
> C:\Users\mureadr\Desktop\A\HMI_FORGF\bld\armle-v7\release\fordhmi>"C:\Users\mureadr\Downloads\make-4.2-without-guile-w32-bin\bin\make.exe" deploy_marketProperties --debug=vjm --print-data-base > 42-SForge.err 2>&1
>
> And I still got the same error (see attachment).
> This time, I kept the entire output so you can reproduce it exactly.

Thanks.  Indeed, I see the command truncated at 16384 bytes,
i.e. exactly 16KB.  This seems to be the limitation of sh.exe itself,
because if I replace the 1st "echo" on the command line with
"echo.exe", and remove all the && and the quotes (which force Make to
invoke the command through the shell), then I see the full untruncated
command line in the output.  Which means invoking echo.exe directly
with such long command lines does work, and the truncation is not
Make's fault.

So my guess is that your make 3.81 used a disk file to run the shell
command as a script, whereas the default build of Make 4.2 invokes the
shell directly.  You could try rebuilding Make 4.2 with
BATCH_MODE_ONLY_SHELL defined, see the end of the file config.h.  Or
maybe you could find a better port of Bash that doesn't suffer from
this limitation.

_______________________________________________
Make-w32 mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/make-w32
Reply | Threaded
Open this post in threaded view
|

Re: GNU make 4.2 (and 4.2.1) failing due to length of command-line

Adrian Muresan
In reply to this post by Eli Zaretskii
Adrian Muresan has shared OneDrive files with you. To view them, click the links below.

This is an interesting data-point.

I used make 3.81 directly from SourceForge and it also failed with same error.


C:\Users\mureadr\Desktop\A\HMI_FORGF\bld\armle-v7\release\fordhmi>"C:\GnuWin32\bin\make.exe" deploy_marketProperties --debug=vjm --print-data-base > 381-SForge.err 2>&1
See 381-SForge.err

But when I used the 3.81 that comes with the QNX SDK, it works fine

C:\Users\mureadr\Desktop\A\HMI_FORGF\bld\armle-v7\release\fordhmi>where make
C:\QNX650\host\win32\x86\usr\bin\make.exe

C:\Users\mureadr\Desktop\A\HMI_FORGF\bld\armle-v7\release\fordhmi>make --ver
GNU Make 3.81
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for Windows32

C:\Users\mureadr\Desktop\A\HMI_FORGF\bld\armle-v7\release\fordhmi>make deploy_marketProperties --debug=vjm --print-data-base > 381-QNX.err 2>&1
See 381-QNX.err




From: Adrian Muresan <[hidden email]>
Sent: Wednesday, June 22, 2016 1:01 PM
To: Eli Zaretskii
Cc: [hidden email]
Subject: Re: GNU make 4.2 (and 4.2.1) failing due to length of command-line
 
The way that I built make 4.2.1 (and 4.2) is:
1) download http://ftp.gnu.org/gnu/make/make-4.2.1.tar.gz and http://ftp.gnu.org/gnu/make/make-4.2.tar.gz
2) open .sln file in VS 2008
3) a conversion process occurs because .sln is from an earlier version of
4) build release version

---------------------------
Checksum information for make 4.2.1 release executable
---------------------------
Name: make_msvc.net2003.exe
Size: 318464 bytes (0 MB)


CRC32: 92E7648E


CRC64: 9A025110B0589AD5


SHA256: 70479CE11C92ECB45AEDAEB2CBE3EFA96990A5804774C1D70E67135E404CCC08


SHA1: 7CF3919C78357084FE0D7C65BADC9BF41E4C6C8F


BLAKE2sp: 0A0A543305D67DC66DD823943B42489B507CE74FDD2EE1506039A76E6E9642B1


---------------------------
OK   
---------------------------



---------------------------
Checksum information for make 4.2 release executable
---------------------------
Name: make_msvc.net2003.exe
Size: 318976 bytes (0 MB)


CRC32: C98E40B9


CRC64: 1CDD9590939C9B97


SHA256: 980C00EAF8ED3CB58D56B4CBFF061C38C28A6E351B8963EE689DDCDC7C5837F5


SHA1: C84C705747C02CCEAF749CC92BF97B16EA5148C1


BLAKE2sp: 7B511AA39FC52732BF691B7552C1EF6BDBD97A01B36FE4A4685A1E54295A6108


---------------------------
OK   
---------------------------



I downloaded
https://sourceforge.net/projects/ezwinports/files/make-4.2-without-guile-w32-bin.zip/download

I ran the command
C:\Users\mureadr\Desktop\A\HMI_FORGF\bld\armle-v7\release\fordhmi>"C:\Users\mureadr\Downloads\make-4.2-without-guile-w32-bin\bin\make.exe" deploy_marketProperties --debug=vjm --print-data-base > 42-SForge.err 2>&1

And I still got the same error (see attachment).
This time, I kept the entire output so you can reproduce it exactly.

My sh --help
C:\Users\mureadr\Desktop\A\HMI_FORGF\bld\armle-v7\release\fordhmi>where sh
C:\QNX650\host\win32\x86\usr\bin\sh.exe

C:\Users\mureadr\Desktop\A\HMI_FORGF\bld\armle-v7\release\fordhmi>sh --help
GNU bash, version 3.1.17(1)-release-(i686-pc-msys)
Usage:  sh [GNU long option] [option] ...
        sh [GNU long option] [option] script-file ...
GNU long options:
        --debug
        --debugger
        --dump-po-strings
        --dump-strings
        --help
        --init-file
        --login
        --noediting
        --noprofile
        --norc
        --posix
        --protected
        --rcfile
        --restricted
        --verbose
        --version
        --wordexp
Shell options:
        -irsD or -c command or -O shopt_option          (invocation only)
        -abefhkmnptuvxBCHP or -o option
Type `sh -c "help set"' for more information about shell options.
Type `sh -c help' for more information about shell builtin commands.
Use the `bashbug' command to report bugs.



From: Eli Zaretskii <[hidden email]>
Sent: Wednesday, June 22, 2016 10:48 AM
To: Adrian Muresan
Cc: [hidden email]
Subject: Re: GNU make 4.2 (and 4.2.1) failing due to length of command-line
   
> From: Adrian Muresan <[hidden email]>
> Date: Tue, 21 Jun 2016 22:53:50 +0000
>
> For more complete info, go to here
>
http://stackoverflow.com/questions/37932984/gnu-make-4-2-and-4-2-1-on-windows-bug



GNU make 4.2 (and 4.2.1) on windows truncates line in recipe
stackoverflow.com
I want to move to make 4.2 so I can have the ability to do parallel jobs on Windows using --jobs=N. make 4.2 is not available in binary form from SourceForge, only 3.81 So I had to build it from


That discussion is full of misinformation, so let's ignore it, OK?

> I am making this target only
>
> deploy_marketProperties:
> echo Copying application data... && $(MKDIR)
> "C:/Users/User1/Desktop/A/Proj/src/../lib/armle-v7/release/marketProperties" && $(COPY_DIR) $(wildcard
> C:/Users/User1/Desktop/A/Proj/src/subProj/marketProperties/*)
> "C:/Users/User1/Desktop/A/Proj/src/../lib/armle-v7/release/marketProperties"
>
> Using following command
> make deploy_marketProperties --debug=vjm
>
> On GNU make 3.81, it works fine. But on make 4.2 and 4.2.1, I get an error:
>
> cp: target `C' is not a directory
>
> The problem appears to be the length of the command. If I delete some JSON files, it works fine.

I cannot reproduce this.  in general, Make on Windows uses APIs that
support command lines up to 32K bytes.  I just successfully ran a
Makefile that invoked a command with 23K bytes (which is more than
your 18K, as reported on Stackoverflow), with no problems.

So I think some other factor is at work here.  The first suspect is
the way you built Make 4.2.  Could you please try the binaries
available from this site:

  https://sourceforge.net/projects/ezwinports/files/?source=navbar

(I'm guessing you don't need Guile support in Make, so the
"without-guile" binary zip is for you.)  This is the binary I used in
the experiment described above.  The command line I tried was this:

  echo 123456789 123456789 123456789 123456789 123456789 ...

etc., up to 23K characters.  Also, since you seem to be using the MSYS
Bash, I used it as well (I have MSYS installed here).  I don't know
what kind of Bash build you got from the QNX toolkit, but at least the
output of "sh --help" I get here is identical to what you show, so
there's hope it's the same or similar binary.

So please try the above-mentioned binary distribution of Make 4.2, and
please try the simple command I used.  If that works, then I suggest
to try your real target with the same binary of Make.
   

_______________________________________________
Make-w32 mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/make-w32
Reply | Threaded
Open this post in threaded view
|

Re: GNU make 4.2 (and 4.2.1) failing due to length of command-line

Eli Zaretskii
> From: Adrian Muresan <[hidden email]>
> CC: "[hidden email]" <[hidden email]>
> Date: Wed, 22 Jun 2016 17:44:33 +0000
>
> This is an interesting data-point.
>
> I used make 3.81 directly from SourceForge and it also failed with same error.
>
> C:\Users\mureadr\Desktop\A\HMI_FORGF\bld\armle-v7\release\fordhmi>"C:\GnuWin32\bin\make.exe"
> deploy_marketProperties --debug=vjm --print-data-base > 381-SForge.err 2>&1
> See 381-SForge.err
>
> But when I used the 3.81 that comes with the QNX SDK, it works fine
>
> C:\Users\mureadr\Desktop\A\HMI_FORGF\bld\armle-v7\release\fordhmi>where make
> C:\QNX650\host\win32\x86\usr\bin\make.exe

I guess they've made some changes in Make when they built it.  Do they
provide the sources or diffs relative to upstream release?  They
should, according to GPL.

_______________________________________________
Make-w32 mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/make-w32
Reply | Threaded
Open this post in threaded view
|

Re: GNU make 4.2 (and 4.2.1) failing due to length of command-line

Paul Smith-20
In reply to this post by Adrian Muresan
Hi Adrian; we try to avoid top-posting on mailing lists: instead of
including the entire email you're replying to and then writing your
response at the top, please try to include only the parts of the
message that are relevant for your reply and intersperse your response
appropriately.

The complete thread is archived on the mailing list server, so there's
no need to including the entire thing again in every message, and even
today there are many people who have limited or costly internet access.

Thanks!

On Wed, 2016-06-22 at 17:44 +0000, Adrian Muresan wrote:

> This is an interesting data-point.
> I used make 3.81 directly from SourceForge and it also failed with
> same error.
>
> C:\Users\mureadr\Desktop\A\HMI_FORGF\bld\armle-
> v7\release\fordhmi>"C:\GnuWin32\bin\make.exe" deploy_marketProperties
> --debug=vjm --print-data-base > 381-SForge.err 2>&1
> See 381-SForge.err
>
> But when I used the 3.81 that comes with the QNX SDK, it works fine

So clearly the QNX version of GNU make is not a simple build using all
the default configuration.  Since GNU make is under the GPL, the QNX
folks must provide any changes they made to it on request.

My suspicion is that, as Eli suggested, they have enabled the
BATCH_MODE_ONLY_SHELL parameter in their config.h.  You should clear
your build, make that change to your config.h to enable that, rebuild,
and see if that version behaves as you expect.

_______________________________________________
Make-w32 mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/make-w32
Reply | Threaded
Open this post in threaded view
|

Re: GNU make 4.2 (and 4.2.1) failing due to length of command-line

Adrian Muresan
In reply to this post by Eli Zaretskii

Should I submit a bug report to the GNU make site?




From: Eli Zaretskii <[hidden email]>
Sent: Wednesday, June 22, 2016 1:55 PM
To: Adrian Muresan
Cc: [hidden email]
Subject: Re: GNU make 4.2 (and 4.2.1) failing due to length of command-line
 
> From: Adrian Muresan <[hidden email]>
> CC: "[hidden email]" <[hidden email]>
> Date: Wed, 22 Jun 2016 17:44:33 +0000
>
> This is an interesting data-point.
>
> I used make 3.81 directly from SourceForge and it also failed with same error.
>
> C:\Users\mureadr\Desktop\A\HMI_FORGF\bld\armle-v7\release\fordhmi>"C:\GnuWin32\bin\make.exe"
> deploy_marketProperties --debug=vjm --print-data-base > 381-SForge.err 2>&1
> See 381-SForge.err
>
> But when I used the 3.81 that comes with the QNX SDK, it works fine
>
> C:\Users\mureadr\Desktop\A\HMI_FORGF\bld\armle-v7\release\fordhmi>where make
> C:\QNX650\host\win32\x86\usr\bin\make.exe

I guess they've made some changes in Make when they built it.  Do they
provide the sources or diffs relative to upstream release?  They
should, according to GPL.

_______________________________________________
Make-w32 mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/make-w32
Reply | Threaded
Open this post in threaded view
|

Re: GNU make 4.2 (and 4.2.1) failing due to length of command-line

Eli Zaretskii
> From: Adrian Muresan <[hidden email]>
> CC: "[hidden email]" <[hidden email]>
> Date: Wed, 22 Jun 2016 17:57:25 +0000
>
> Should I submit a bug report to the GNU make site?

Bug report about what?  The limitations of the MSYS port of Bash are
not Make's responsibility.

_______________________________________________
Make-w32 mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/make-w32
Reply | Threaded
Open this post in threaded view
|

Re: GNU make 4.2 (and 4.2.1) failing due to length of command-line

Paul Smith-20
In reply to this post by Adrian Muresan
On Wed, 2016-06-22 at 17:57 +0000, Adrian Muresan wrote:
> Should I submit a bug report to the GNU make site?

As Eli's tests, and even your output, show, GNU make is invoking the
command with the complete command line, so there's no issue with GNU
make.  It's the command we're invoking which is somehow truncating the
command line, not GNU make.

It's possible that the already-suggested BATCH_MODE_ONLY_SHELL setting
will help, but this is a workaround for commands which can't accept
very large command lines.

So based on the information we have so far there's no bug in GNU make
and no need to submit a bug report.

_______________________________________________
Make-w32 mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/make-w32
Reply | Threaded
Open this post in threaded view
|

Re: GNU make 4.2 (and 4.2.1) failing due to length of command-line

Adrian Muresan
I ran using GNU make 3.81 from two different sources:

once using the binary form sourceforge
And
once using the binary from QNX.

QNX gave no error and SourceForge gave an error (the same error as 4.2, 4.2.1 I built and the 4.0+ binary version Eli told me to get from SourceForge).

Doesn't this prove there's a bug? As Eli said, QNX obviously patched something since everything else, even the shell used, was the same.

Sent from my iPhone

> On Jun 22, 2016, at 2:38 PM, Paul Smith <[hidden email]> wrote:
>
>>
>
> As Eli's tests, and even your output, show, GNU make is invoking the
> command with the complete command line, so there's no issue with GNU
> make.  It's the command we're invoking which is somehow truncating the
> command line, not GNU make.
>
> It's possible that the already-suggested BATCH_MODE_ONLY_SHELL setting
> will help, but this is a workaround for commands which can't accept
> very large command lines.
>
> So based on the information we have so far there's no bug in GNU make
> and no need to submit a bug report.

_______________________________________________
Make-w32 mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/make-w32
Reply | Threaded
Open this post in threaded view
|

Re: GNU make 4.2 (and 4.2.1) failing due to length of command-line

Paul Smith-20
On Wed, 2016-06-22 at 18:48 +0000, Adrian Muresan wrote:

> I ran using GNU make 3.81 from two different sources:
>
> once using the binary form sourceforge 
> And 
> once using the binary from QNX. 
>
> QNX gave no error and SourceForge gave an error (the same error as
> 4.2, 4.2.1 I built and the 4.0+ binary version Eli told me to get
> from SourceForge). 
>
> Doesn't this prove there's a bug? As Eli said, QNX obviously patched
> something since everything else, even the shell used, was the same.

It proves that the QNX version of GNU make did something to work around
the deficiencies of the shell.  That's not necessarily a bug in GNU
make: it's not clear that adding that workaround, whatever it was, as a
permanent solution into GNU make is the right thing to do.

A better answer would be to fix the tool (the shell) so that it didn't
have this deficiency so that no workaround was required.

In any event, there's nothing else to be done until you try a 4.2.1
build configured with the BATCH_MODE_ONLY_SHELL setting and see if that
works, and report back to us.

If it does, then the problem is solved.

If it doesn't, then you'll need to contact QNX and find out what
changes they made to GNU make.  Once we know that we can decide what
the next step might be.

_______________________________________________
Make-w32 mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/make-w32
Reply | Threaded
Open this post in threaded view
|

Re: GNU make 4.2 (and 4.2.1) failing due to length of command-line

Eli Zaretskii
In reply to this post by Adrian Muresan
> From: Adrian Muresan <[hidden email]>
> CC: Eli Zaretskii <[hidden email]>, "[hidden email]" <[hidden email]>
> Date: Wed, 22 Jun 2016 18:48:35 +0000
>
> I ran using GNU make 3.81 from two different sources:
>
> once using the binary form sourceforge
> And
> once using the binary from QNX.
>
> QNX gave no error and SourceForge gave an error (the same error as 4.2, 4.2.1 I built and the 4.0+ binary version Eli told me to get from SourceForge).
>
> Doesn't this prove there's a bug? As Eli said, QNX obviously patched something since everything else, even the shell used, was the same.

The bug, or rather, the limitation is not in Make, it's in the shell
you use.  I suggested how to build Make in a way that might work
around this limitation, and I'm guessing that the QNX folks built
their make with that option as well.  Of course, it would be best to
know what changes they made, so that you could make the same changes
in the newer Make.

However, all this does not change the basic fact that the limitation
is in Bash.  Make, on its side, invokes Bash with the full command
line, but Bash only reads the first 16KB of it.  How is it a bug in
Make?  Make doesn't have a goal of catering to limitations of programs
invoked from Make, certainly not the limitations of MSYS Bash, which
is not even a simple native Win32 application.  GNU Make development
cannot be responsible for limitations of other programs out there,
it's unreasonable to expect it to do that.

_______________________________________________
Make-w32 mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/make-w32
Reply | Threaded
Open this post in threaded view
|

Re: GNU make 4.2 (and 4.2.1) failing due to length of command-line

Paul Smith-20
In reply to this post by Paul Smith-20
On Wed, 2016-06-22 at 19:47 +0000, Adrian Muresan wrote:

> In config.h, I uncommented the line
> /* #define BATCH_MODE_ONLY_SHELL 1 */
>
> this affects job.c
>
>   static const char *sh_cmds_sh[] =
>     { "cd", "eval", "exec", "exit", "login", "logout", "set",
> "umask", "wait",
>       "while", "for", "case", "if", ":", ".", "break", "continue",
> "export",
>       "read", "readonly", "shift", "times", "trap", "switch", "test",
> #ifdef BATCH_MODE_ONLY_SHELL
>       "echo",
> #endif
>       0 };

I can see where you might think this makes a difference, but really
this is not relevant.  These are used to decide whether or not to use
the "fast path" or require a shell to run the recipe.  Your script is
complex enough that make will always decide to run the shell so having
"echo" in this list won't make any difference.

The thing that is relevant to your situation is in main.c:

    if (!unixy_shell && sh_found
        && (strstr (default_shell, "sh") || strstr (default_shell, "SH")))
      {
        unixy_shell = 1;
        batch_mode_shell = 0;
      }

  #ifdef BATCH_MODE_ONLY_SHELL
    batch_mode_shell = 1;
  #endif

Here we first decide that we have a UNIX-like shell (because make finds
your sh.exe); if that's true then we disable batch_mode_shell because a
UNIX shell should be sufficiently powerful to not need it.

Setting that config.h option causes batch_mode_shell to be set to 1
(true) always, in all situations.

That change is what's working around your broken shell.

_______________________________________________
Make-w32 mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/make-w32
Reply | Threaded
Open this post in threaded view
|

Re: GNU make 4.2 (and 4.2.1) failing due to length of command-line

Adrian Muresan
Does batch_mode_shell = 1 mean that it always uses the sh.exe instead of Windows.cmd?

That would make sense why it's now working. 
The limitation on cmd.exe is 8192


But I have around 18k characters. 

Sent from my iPhone

On Jun 22, 2016, at 3:57 PM, Paul Smith <[hidden email]> wrote:

On Wed, 2016-06-22 at 19:47 +0000, Adrian Muresan wrote:
In config.h, I uncommented the line
/* #define BATCH_MODE_ONLY_SHELL 1 */

this affects job.c

  static const char *sh_cmds_sh[] =
    { "cd", "eval", "exec", "exit", "login", "logout", "set",
"umask", "wait",
      "while", "for", "case", "if", ":", ".", "break", "continue",
"export",
      "read", "readonly", "shift", "times", "trap", "switch", "test",
#ifdef BATCH_MODE_ONLY_SHELL
      "echo",
#endif
      0 };

I can see where you might think this makes a difference, but really
this is not relevant.  These are used to decide whether or not to use
the "fast path" or require a shell to run the recipe.  Your script is
complex enough that make will always decide to run the shell so having
"echo" in this list won't make any difference.

The thing that is relevant to your situation is in main.c:

    if (!unixy_shell && sh_found
        && (strstr (default_shell, "sh") || strstr (default_shell, "SH")))
      {
        unixy_shell = 1;
       batch_mode_shell = 0;
      }

 #ifdef BATCH_MODE_ONLY_SHELL
    batch_mode_shell = 1;
 #endif

Here we first decide that we have a UNIX-like shell (because make finds
your sh.exe); if that's true then we disable batch_mode_shell because a
UNIX shell should be sufficiently powerful to not need it.

Setting that config.h option causes batch_mode_shell to be set to 1
(true) always, in all situations.

That change is what's working around your broken shell.

_______________________________________________
Make-w32 mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/make-w32
Reply | Threaded
Open this post in threaded view
|

Re: GNU make 4.2 (and 4.2.1) failing due to length of command-line

Paul Smith-20
On Wed, 2016-06-22 at 20:36 +0000, Adrian Muresan wrote:
> Does batch_mode_shell = 1 mean that it always uses the sh.exe instead
> of Windows.cmd?

No.  It's clear that make is not using Windows command.com, because the
script you're running is a POSIX shell script, not a command.com batch
script.  If you tried to enter those commands into your command.com
prompt you'd get a syntax error.

What that flag means (as I understand it: I'm not that familiar with
this aspect of Windows support) is that make will never try to invoke
the shell directly passing the recipe to be run on the command line.

Instead it is being forced to always write the recipe to a temporary
file ("batch file") on your disk and invoke the shell such that it runs
the recipe in the temporary file.

Apparently your shell is able to read and execute very long command
lines from a file, which it cannot successfully process when passed in
as arguments, even though these command line arguments are not
exceeding Windows limits.

_______________________________________________
Make-w32 mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/make-w32
Reply | Threaded
Open this post in threaded view
|

Re: GNU make 4.2 (and 4.2.1) failing due to length of command-line

Eli Zaretskii
> From: Paul Smith <[hidden email]>
> Cc: Eli Zaretskii <[hidden email]>, "[hidden email]" <[hidden email]>
> Date: Wed, 22 Jun 2016 16:54:35 -0400
>
> On Wed, 2016-06-22 at 20:36 +0000, Adrian Muresan wrote:
> > Does batch_mode_shell = 1 mean that it always uses the sh.exe instead
> > of Windows.cmd?
>
> No.  It's clear that make is not using Windows command.com, because the
> script you're running is a POSIX shell script, not a command.com batch
> script.  If you tried to enter those commands into your command.com
> prompt you'd get a syntax error.
>
> What that flag means (as I understand it: I'm not that familiar with
> this aspect of Windows support) is that make will never try to invoke
> the shell directly passing the recipe to be run on the command line.
>
> Instead it is being forced to always write the recipe to a temporary
> file ("batch file") on your disk and invoke the shell such that it runs
> the recipe in the temporary file.

Yes, that's true.  As an aside, the Windows shell is cmd.exe, not
command.com, and Make uses cmd.exe if either (a) the Makefile requires
that with the "SHELL =" command, or (b) it cannot find any sh.exe on
PATH.

_______________________________________________
Make-w32 mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/make-w32
12