[bug #41246] Allow to switch shell batch mode at runtime instead of build time

classic Classic list List threaded Threaded
43 messages Options
123
Reply | Threaded
Open this post in threaded view
|

[bug #41246] Allow to switch shell batch mode at runtime instead of build time

anonymous
URL:
  <http://savannah.gnu.org/bugs/?41246>

                 Summary: Allow to switch shell batch mode at runtime instead
of build time
                 Project: make
            Submitted by: None
            Submitted on: Thu 16 Jan 2014 05:58:28 AM UTC
                Severity: 3 - Normal
              Item Group: None
                  Status: None
                 Privacy: Public
             Assigned to: None
             Open/Closed: Open
         Discussion Lock: Any
       Component Version: SCM
        Operating System: MS Windows
           Fixed Release: None
           Triage Status: None

    _______________________________________________________

Details:

I've been successfully using a pristine build of make 4.0 on windows to build
Firefox for a few weeks now, after avoiding the sh.exe issues by using single
quotes instead of double quotes, but we unfortunately have some other nasty
makefiles that are *very* hard to fixup this way without spending a huge
amount of time rewriting it, and i would very much like to be able to just
make that one makefile use a pristine build of make 4.0 as if it were built
with BATCH_MODE_ONLY_SHELL defined.

The attached patch allows us to special-case that specific Makefile.



    _______________________________________________________

File Attachments:


-------------------------------------------------------
Date: Thu 16 Jan 2014 05:58:28 AM UTC  Name: diff  Size: 1kB   By: None
Proposed patch
<http://savannah.gnu.org/bugs/download.php?file_id=30286>

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?41246>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

[bug #41246] Allow to switch shell batch mode at runtime instead of build time

anonymous
Follow-up Comment #1, bug #41246 (project make):

I happen to have not been logged in when I submitted, but this bug and patch
are mine :)

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?41246>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

[bug #41246] Allow to switch shell batch mode at runtime instead of build time

anonymous
Follow-up Comment #2, bug #41246 (project make):

The changes seem simple enough and non-controversial.  However, this
introduces a new special target, .BATCH_MODE_SHELL (why not
.BATCH_MODE_ONLY_SHELL, btw?), which should at least be documented in the
manual.

Paul, do you agree with adding a new target?


    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?41246>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

[bug #41246] Allow to switch shell batch mode at runtime instead of build time

anonymous
Follow-up Comment #3, bug #41246 (project make):

Not being familiar with the various different options for shell behavior on
Windows, can someone describe for me what BATCH_MODE_ONLY_SHELL does?  Does is
overlap with ONESHELL?  If not is it a feature that might be useful to people
on non-Windows systems?  I could read the code but a short primer would be
nice :-)

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?41246>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

[bug #41246] Allow to switch shell batch mode at runtime instead of build time

anonymous
Follow-up Comment #4, bug #41246 (project make):

> why not .BATCH_MODE_ONLY_SHELL, btw?

Apart from making it the same as the macro, the "only" feels too much for a
target name. I don't care that much for the name, though, your pick.

> which should at least be documented in the manual.

Would you mind doing this for me?

> can someone describe for me what BATCH_MODE_ONLY_SHELL does? Does is overlap
with ONESHELL?

In essence, this is like ONESHELL, except it writes the script to a file to
then run "sh thatfile" instead of doing sh -c "line1nline2nline3". This is
useful because msys bash on windows doesn't handle double quotes properly when
run with sh -c. I don't think non-Windows systems would need this as a
feature.

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?41246>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

[bug #41246] Allow to switch shell batch mode at runtime instead of build time

anonymous
Follow-up Comment #5, bug #41246 (project make):

I'm not sure I like the idea of this being a special target in this way.  It's
also possible that something like this COULD be used on UNIX systems; there
are limits to the size of command lines that can be invoked.  On newer systems
like Linux that limit is pretty large (I think it's 16K or so on Linux) but it
exists, and on other systems it's much less.

There's long been requests for "@-file" support embedded in make although
exactly what form that would take hasn't been defined clearly IIRC.

Let me ask this: would it be possible to simply use the existing ONESHELL
capability and just say that on Windows it writes the script to a file and
invokes the shell on it?

The downside of this is that ONESHELL actually behaves differently (although
often this doesn't matter).

Also, if the "normal" behavior doesn't handle something as basic as quoting
properly, why is BATCH_MODE_ONLY_SHELL not the default?  Is there a downside
to it?

Or alternatively, if we detect that the script has content that might be
problematic (quotes) we could automatically use the batch mode.

I guess I'd just prefer things to work right automatically, without
recompiling make or adding new pseudo targets.  Is there a reason this is
tricky or difficult?

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?41246>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time

David Boyce-3
On Sat, Jan 18, 2014 at 4:17 PM, Paul D. Smith <[hidden email]> wrote:
> There are limits to the size of command lines that can be invoked.  On newer systems
> like Linux that limit is pretty large (I think it's 16K or so on Linux) but it
> exists, and on other systems it's much less.

It's actually quite a bit bigger than that. On RHEL6 (2.6.32 kernel),
"getconf ARG_MAX" reports 2621440. This, minus the size of the current
environment, is the number of bytes available for command lines. It's
conceivable that bash imposes its own limit due to fixed buffer sizes;
I haven't checked.

David

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

[bug #41246] Allow to switch shell batch mode at runtime instead of build time

anonymous
In reply to this post by anonymous
Follow-up Comment #6, bug #41246 (project make):

The command line size limit is in the order of several megabytes nowadays on
Linux. It used to be much less, though. Arguably, if this was a wanted
feature, you would likely have heard people complain when the command line
size limit was lower.

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?41246>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

[bug #41246] Allow to switch shell batch mode at runtime instead of build time

anonymous
Follow-up Comment #7, bug #41246 (project make):

I just realized I said something very wrong: BATCH_MODE_ONLY_SHELL is not all
all like ONESHELL. It still runs a shell per recipe line. Sorry for the
confusion.

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?41246>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

[bug #41246] Allow to switch shell batch mode at runtime instead of build time

anonymous
Follow-up Comment #8, bug #41246 (project make):

I understood that difference; that's why I said "The downside of this is that
ONESHELL actually behaves differently".  However often this doesn't matter so
much, particularly on Windows where the rules about what operations impact the
parent's environment are not so strict as on POSIX systems (IIRC).  I was
wondering if it was "close enough" to be acceptable.

In any event, I'd really prefer this be made to "just work" without special
compile options or pseudo targets needed.  Is there a reason why that's not
feasible?

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?41246>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

[bug #41246] Allow to switch shell batch mode at runtime instead of build time

anonymous
Follow-up Comment #9, bug #41246 (project make):

Also we do sometimes see people complaining about this, where they can't
invoke commands because the command line is too long.  There are various
tricks out there for "chunking" make variable values containing lots and lots
of words into smaller groupings.  It has to be done in make itself,
obviously.

However, I will admit it's been a while since I've noticed a complaint about
this.  Either people have gotten used to it or it's become less of a problem.

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?41246>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

[bug #41246] Allow to switch shell batch mode at runtime instead of build time

anonymous
Follow-up Comment #10, bug #41246 (project make):

This is a different approach to the problem, as suggested by Paul: this
triggers batch mode shell when there are double quotes in the recipe. Note
this doesn't quite work around all the problems that the previous patch did
solve when adding .BATCH_MODE_SHELL to the problematic makefile. There are
still issues with backslashes, which I suspect are due to bad escaping when
running with sh -c. Or discrepancy between sh -c and batch mode which leads to
that particular Makefile miraculously working in batch mode. I'll need to do
more testing.

(file #30424)
    _______________________________________________________

Additional Item Attachment:

File name: diff                           Size:2 KB


    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?41246>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time

Eli Zaretskii
> Date: Fri, 31 Jan 2014 09:14:32 +0000
> From: Mike Hommey <[hidden email]>
>
> This is a different approach to the problem, as suggested by Paul: this
> triggers batch mode shell when there are double quotes in the recipe.

This cannot be automatic, as we have deliberately made the double
quotes not special for Windows shells.  So please make this
conditional on some Make option (sorry, I didn't yet have time to look
at the patch).

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

[bug #41246] Allow to switch shell batch mode at runtime instead of build time

anonymous
In reply to this post by anonymous
Follow-up Comment #11, bug #41246 (project make):

Hi Mike,

Sorry to hijack this thread a little.

I work on the MSYS2 project and I'd be keen to fix "sh.exe -c" to behave
correctly. If you are interested in this, could you please email me at
[hidden email] with a testcase?

FWIW I agree that being able to punt the shell commands into a single script
executed via sh.exe is a good feature to have. I'd also like to be able to
keep them around afterwards for inspection or to re-run them, like GCC's
--save-temps argument.

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?41246>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time

Mike Hommey
In reply to this post by Eli Zaretskii
On Fri, Jan 31, 2014 at 11:37:47AM +0200, Eli Zaretskii wrote:
> > Date: Fri, 31 Jan 2014 09:14:32 +0000
> > From: Mike Hommey <[hidden email]>
> >
> > This is a different approach to the problem, as suggested by Paul: this
> > triggers batch mode shell when there are double quotes in the recipe.
>
> This cannot be automatic, as we have deliberately made the double
> quotes not special for Windows shells.

You mean cmd.exe? The patch only uses batch mode shell when there are
double quotes *and* the shell is unixy (so, not cmd.exe)

> So please make this
> conditional on some Make option (sorry, I didn't yet have time to look
> at the patch).

An option as command line argument, or as a special target as the
original patch did?

Mike

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time

Eli Zaretskii
> Date: Fri, 31 Jan 2014 22:15:56 +0900
> From: Mike Hommey <[hidden email]>
> Cc: [hidden email], [hidden email], [hidden email],
> [hidden email]
>
> On Fri, Jan 31, 2014 at 11:37:47AM +0200, Eli Zaretskii wrote:
> > > Date: Fri, 31 Jan 2014 09:14:32 +0000
> > > From: Mike Hommey <[hidden email]>
> > >
> > > This is a different approach to the problem, as suggested by Paul: this
> > > triggers batch mode shell when there are double quotes in the recipe.
> >
> > This cannot be automatic, as we have deliberately made the double
> > quotes not special for Windows shells.
>
> You mean cmd.exe? The patch only uses batch mode shell when there are
> double quotes *and* the shell is unixy (so, not cmd.exe)

That's better, but still too subtly to be the default, I think.
Having some essentially random character trigger very different
behavior is bad mojo for a program in general.

> > So please make this
> > conditional on some Make option (sorry, I didn't yet have time to look
> > at the patch).
>
> An option as command line argument, or as a special target as the
> original patch did?

The former, as Paul objected to the latter.

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time

Paul Smith-20
On Fri, 2014-01-31 at 16:55 +0200, Eli Zaretskii wrote:
> > An option as command line argument, or as a special target as the
> > original patch did?
>
> The former, as Paul objected to the latter.

I didn't object, per se.  I just prefer my tools to DTRT in all cases
without me having to use any magical incantations to make it happen :-).

My question, or _challenge_ if you like, is whether we can find a way to
know, without any hints from the user, whether a given command line will
work properly for /bin/sh -c in Windows, or if it needs to use a batch
file.

It seems, from my lofty vantige point of knowing virtually nothing about
it, that this is a solveable problem.  If we can try to write down (in
text) the rules for when we would or would not be able to use /bin/sh
-c, I would think it would quickly become clear if we can solve it or
not.

As for whether we use a command line option or makefile target, I think
probably the makefile target is probably more appropriate.  In general I
like to use command line options to control user preferences which do
not have direct impacts on the build, so if you ran the same build with
different flags it would still work, just with different behaviors.

But for things which will cause the makefile to not work properly if not
specified, those things should be embedded into the makefile itself: I
don't want it to be the case that you always have to remember to give a
particular command line option or your build will break!


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time

Eli Zaretskii
> From: Paul Smith <[hidden email]>
> Cc: Mike Hommey <[hidden email]>, [hidden email],
> [hidden email], [hidden email]
> Date: Fri, 31 Jan 2014 10:59:13 -0500
>
> My question, or _challenge_ if you like, is whether we can find a way to
> know, without any hints from the user, whether a given command line will
> work properly for /bin/sh -c in Windows, or if it needs to use a batch
> file.

Mike, could you please post some details about your use case(s) that
required this?  I'm not sure why would quotes not work with a Unixy
shell.

Perhaps this is once again something specific to the MSYS conversion
of file names?

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time

Mike Hommey
On Sat, Feb 01, 2014 at 11:29:19AM +0200, Eli Zaretskii wrote:

> > From: Paul Smith <[hidden email]>
> > Cc: Mike Hommey <[hidden email]>, [hidden email],
> > [hidden email], [hidden email]
> > Date: Fri, 31 Jan 2014 10:59:13 -0500
> >
> > My question, or _challenge_ if you like, is whether we can find a way to
> > know, without any hints from the user, whether a given command line will
> > work properly for /bin/sh -c in Windows, or if it needs to use a batch
> > file.
>
> Mike, could you please post some details about your use case(s) that
> required this?  I'm not sure why would quotes not work with a Unixy
> shell.
>
> Perhaps this is once again something specific to the MSYS conversion
> of file names?

I'm not sure why it happens, but here's how it manifests:

$ cat <<EOF > foo.mk
foo:
        $(LS) "c:\\Program Files"
EOF
$ make -f foo.mk LS=$(which ls)
C:/mozilla-build/msys/bin/ls.exe "c:\\Program Files"
C:/mozilla-build/msys/bin/ls.exe: c:\Program: No such file or directory
C:/mozilla-build/msys/bin/ls.exe: Files: No such file or directory
foo.mk:2: recipe for target 'foo' failed
make: *** [foo] Error 2

This happens for various absolute-path versions of LS, but not when
LS=ls. This is something I mentioned in bug 40241 comment 12 and 13.

Another issue is with backslashes in paths.

For example:
$ cat <<EOF > foo.mk
foo:
        grep foo < foo\\bar
EOF

(Note the < is just there to trigger sh -c)

This executes sh -c "grep foo < foo\\bar", which fails with:
  /usr/bin/sh: foobar: No such file or directory
But in batch mode shell, this executes sh makennnnn.sh with a content of
  grep foo < foo\\bar
and that fails with:
  makennnnn.sh: line 1: foo\bar: No such file or directory

(Note how in one case the backslash is eaten and not in the other case)

When < is not there and make executes grep directly, it fails with:
  grep: foo\bar: No such file or directory

So there is a lack of consistency in how backslashes are treated when
make uses sh -c or not.

Note the second patch to bug 41246 solves the first issue but not the
second. The first patch, with .BATCH_MODE_SHELL added to the makefile
"fixes" both issues.

Mike

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time

Eli Zaretskii
> Date: Tue, 4 Feb 2014 13:34:31 +0900
> From: Mike Hommey <[hidden email]>
> Cc: [hidden email], [hidden email], [hidden email]
>
> On Sat, Feb 01, 2014 at 11:29:19AM +0200, Eli Zaretskii wrote:
> > > From: Paul Smith <[hidden email]>
> > > Cc: Mike Hommey <[hidden email]>, [hidden email],
> > > [hidden email], [hidden email]
> > > Date: Fri, 31 Jan 2014 10:59:13 -0500
> > >
> > > My question, or _challenge_ if you like, is whether we can find a way to
> > > know, without any hints from the user, whether a given command line will
> > > work properly for /bin/sh -c in Windows, or if it needs to use a batch
> > > file.
> >
> > Mike, could you please post some details about your use case(s) that
> > required this?  I'm not sure why would quotes not work with a Unixy
> > shell.
> >
> > Perhaps this is once again something specific to the MSYS conversion
> > of file names?
>
> I'm not sure why it happens, but here's how it manifests:
>
> $ cat <<EOF > foo.mk
> foo:
>         $(LS) "c:\\Program Files"
> EOF
> $ make -f foo.mk LS=$(which ls)
> C:/mozilla-build/msys/bin/ls.exe "c:\\Program Files"
> C:/mozilla-build/msys/bin/ls.exe: c:\Program: No such file or directory
> C:/mozilla-build/msys/bin/ls.exe: Files: No such file or directory
> foo.mk:2: recipe for target 'foo' failed
> make: *** [foo] Error 2

So you are using the MSYS Bash and MSYS ls.exe from the native build
of Make, right?  Because your recipe succeeds for me both with cmd.exe
and with a native build of a Unixy shell used as a shell by Make.

> This happens for various absolute-path versions of LS, but not when
> LS=ls. This is something I mentioned in bug 40241 comment 12 and 13.
>
> Another issue is with backslashes in paths.
>
> For example:
> $ cat <<EOF > foo.mk
> foo:
>         grep foo < foo\\bar
> EOF
>
> (Note the < is just there to trigger sh -c)
>
> This executes sh -c "grep foo < foo\\bar", which fails with:
>   /usr/bin/sh: foobar: No such file or directory
> But in batch mode shell, this executes sh makennnnn.sh with a content of
>   grep foo < foo\\bar
> and that fails with:
>   makennnnn.sh: line 1: foo\bar: No such file or directory
>
> (Note how in one case the backslash is eaten and not in the other case)

Why are you using backslashes in file names when your shell is a Unixy
shell?  That makes little sense to me, and I don't see why Make on
Windows should support such use.  Unixy shells are supposed to get
Unixy file names with forward slashes, including in redirection.

> When < is not there and make executes grep directly, it fails with:
>   grep: foo\bar: No such file or directory

Probably because that grep is an MSYS grep, right?

In general, if you are using MSYS shell and commands, you are supposed
to invoke them from the MSYS build of Make.  Otherwise, there can be
no guarantee that commands aimed at native tools will work with MSYS
programs, which are Posix programs with a twist.

I'm willing to support MSYS tools invoked from a native Make, but only
up to a point, and these examples go way beyond that point, sorry.

If I'm confused, and the same problems happen when MSYS is not
involved, please show such examples.

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
123