# [bug #41246] Allow to switch shell batch mode at runtime instead of build time

43 messages
123
Open this post in threaded view
|

## [bug #41246] Allow to switch shell batch mode at runtime instead of build time

 URL:                    Summary: Allow to switch shell batch mode at runtime instead of build time                  Project: make             Submitted by: None             Submitted on: Thu 16 Jan 2014 05:58:28 AM UTC                 Severity: 3 - Normal               Item Group: None                   Status: None                  Privacy: Public              Assigned to: None              Open/Closed: Open          Discussion Lock: Any        Component Version: SCM         Operating System: MS Windows            Fixed Release: None            Triage Status: None     _______________________________________________________ Details: I've been successfully using a pristine build of make 4.0 on windows to build Firefox for a few weeks now, after avoiding the sh.exe issues by using single quotes instead of double quotes, but we unfortunately have some other nasty makefiles that are *very* hard to fixup this way without spending a huge amount of time rewriting it, and i would very much like to be able to just make that one makefile use a pristine build of make 4.0 as if it were built with BATCH_MODE_ONLY_SHELL defined. The attached patch allows us to special-case that specific Makefile.     _______________________________________________________ File Attachments: ------------------------------------------------------- Date: Thu 16 Jan 2014 05:58:28 AM UTC  Name: diff  Size: 1kB   By: None Proposed patch     _______________________________________________________ Reply to this item at:   _______________________________________________   Message sent via/by Savannah   http://savannah.gnu.org/_______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make
Open this post in threaded view
|

## [bug #41246] Allow to switch shell batch mode at runtime instead of build time

 Follow-up Comment #1, bug #41246 (project make): I happen to have not been logged in when I submitted, but this bug and patch are mine :)     _______________________________________________________ Reply to this item at:   _______________________________________________   Message sent via/by Savannah   http://savannah.gnu.org/_______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make
Open this post in threaded view
|

## [bug #41246] Allow to switch shell batch mode at runtime instead of build time

 Follow-up Comment #2, bug #41246 (project make): The changes seem simple enough and non-controversial.  However, this introduces a new special target, .BATCH_MODE_SHELL (why not .BATCH_MODE_ONLY_SHELL, btw?), which should at least be documented in the manual. Paul, do you agree with adding a new target?     _______________________________________________________ Reply to this item at:   _______________________________________________   Message sent via/by Savannah   http://savannah.gnu.org/_______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make
Open this post in threaded view
|

## [bug #41246] Allow to switch shell batch mode at runtime instead of build time

 Follow-up Comment #3, bug #41246 (project make): Not being familiar with the various different options for shell behavior on Windows, can someone describe for me what BATCH_MODE_ONLY_SHELL does?  Does is overlap with ONESHELL?  If not is it a feature that might be useful to people on non-Windows systems?  I could read the code but a short primer would be nice :-)     _______________________________________________________ Reply to this item at:   _______________________________________________   Message sent via/by Savannah   http://savannah.gnu.org/_______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make
Open this post in threaded view
|

## [bug #41246] Allow to switch shell batch mode at runtime instead of build time

 Follow-up Comment #4, bug #41246 (project make): > why not .BATCH_MODE_ONLY_SHELL, btw? Apart from making it the same as the macro, the "only" feels too much for a target name. I don't care that much for the name, though, your pick. > which should at least be documented in the manual. Would you mind doing this for me? > can someone describe for me what BATCH_MODE_ONLY_SHELL does? Does is overlap with ONESHELL? In essence, this is like ONESHELL, except it writes the script to a file to then run "sh thatfile" instead of doing sh -c "line1nline2nline3". This is useful because msys bash on windows doesn't handle double quotes properly when run with sh -c. I don't think non-Windows systems would need this as a feature.     _______________________________________________________ Reply to this item at:   _______________________________________________   Message sent via/by Savannah   http://savannah.gnu.org/_______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make
Open this post in threaded view
|

## [bug #41246] Allow to switch shell batch mode at runtime instead of build time

 Follow-up Comment #5, bug #41246 (project make): I'm not sure I like the idea of this being a special target in this way.  It's also possible that something like this COULD be used on UNIX systems; there are limits to the size of command lines that can be invoked.  On newer systems like Linux that limit is pretty large (I think it's 16K or so on Linux) but it exists, and on other systems it's much less. There's long been requests for "@-file" support embedded in make although exactly what form that would take hasn't been defined clearly IIRC. Let me ask this: would it be possible to simply use the existing ONESHELL capability and just say that on Windows it writes the script to a file and invokes the shell on it? The downside of this is that ONESHELL actually behaves differently (although often this doesn't matter). Also, if the "normal" behavior doesn't handle something as basic as quoting properly, why is BATCH_MODE_ONLY_SHELL not the default?  Is there a downside to it? Or alternatively, if we detect that the script has content that might be problematic (quotes) we could automatically use the batch mode. I guess I'd just prefer things to work right automatically, without recompiling make or adding new pseudo targets.  Is there a reason this is tricky or difficult?     _______________________________________________________ Reply to this item at:   _______________________________________________   Message sent via/by Savannah   http://savannah.gnu.org/_______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make
Open this post in threaded view
|

## Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time

 On Sat, Jan 18, 2014 at 4:17 PM, Paul D. Smith <[hidden email]> wrote: > There are limits to the size of command lines that can be invoked.  On newer systems > like Linux that limit is pretty large (I think it's 16K or so on Linux) but it > exists, and on other systems it's much less. It's actually quite a bit bigger than that. On RHEL6 (2.6.32 kernel), "getconf ARG_MAX" reports 2621440. This, minus the size of the current environment, is the number of bytes available for command lines. It's conceivable that bash imposes its own limit due to fixed buffer sizes; I haven't checked. David _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make
Open this post in threaded view
|

## [bug #41246] Allow to switch shell batch mode at runtime instead of build time

 In reply to this post by anonymous Follow-up Comment #6, bug #41246 (project make): The command line size limit is in the order of several megabytes nowadays on Linux. It used to be much less, though. Arguably, if this was a wanted feature, you would likely have heard people complain when the command line size limit was lower.     _______________________________________________________ Reply to this item at:   _______________________________________________   Message sent via/by Savannah   http://savannah.gnu.org/_______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make
Open this post in threaded view
|

## [bug #41246] Allow to switch shell batch mode at runtime instead of build time

 Follow-up Comment #7, bug #41246 (project make): I just realized I said something very wrong: BATCH_MODE_ONLY_SHELL is not all all like ONESHELL. It still runs a shell per recipe line. Sorry for the confusion.     _______________________________________________________ Reply to this item at:   _______________________________________________   Message sent via/by Savannah   http://savannah.gnu.org/_______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make
Open this post in threaded view
|

## [bug #41246] Allow to switch shell batch mode at runtime instead of build time

 Follow-up Comment #8, bug #41246 (project make): I understood that difference; that's why I said "The downside of this is that ONESHELL actually behaves differently".  However often this doesn't matter so much, particularly on Windows where the rules about what operations impact the parent's environment are not so strict as on POSIX systems (IIRC).  I was wondering if it was "close enough" to be acceptable. In any event, I'd really prefer this be made to "just work" without special compile options or pseudo targets needed.  Is there a reason why that's not feasible?     _______________________________________________________ Reply to this item at:   _______________________________________________   Message sent via/by Savannah   http://savannah.gnu.org/_______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make
Open this post in threaded view
|

## [bug #41246] Allow to switch shell batch mode at runtime instead of build time

 Follow-up Comment #9, bug #41246 (project make): Also we do sometimes see people complaining about this, where they can't invoke commands because the command line is too long.  There are various tricks out there for "chunking" make variable values containing lots and lots of words into smaller groupings.  It has to be done in make itself, obviously. However, I will admit it's been a while since I've noticed a complaint about this.  Either people have gotten used to it or it's become less of a problem.     _______________________________________________________ Reply to this item at:   _______________________________________________   Message sent via/by Savannah   http://savannah.gnu.org/_______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make
Open this post in threaded view
|

## [bug #41246] Allow to switch shell batch mode at runtime instead of build time

 Follow-up Comment #10, bug #41246 (project make): This is a different approach to the problem, as suggested by Paul: this triggers batch mode shell when there are double quotes in the recipe. Note this doesn't quite work around all the problems that the previous patch did solve when adding .BATCH_MODE_SHELL to the problematic makefile. There are still issues with backslashes, which I suspect are due to bad escaping when running with sh -c. Or discrepancy between sh -c and batch mode which leads to that particular Makefile miraculously working in batch mode. I'll need to do more testing. (file #30424)     _______________________________________________________ Additional Item Attachment: File name: diff                           Size:2 KB     _______________________________________________________ Reply to this item at:   _______________________________________________   Message sent via/by Savannah   http://savannah.gnu.org/_______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make
Open this post in threaded view
|

## Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time

 > Date: Fri, 31 Jan 2014 09:14:32 +0000 > From: Mike Hommey <[hidden email]> > > This is a different approach to the problem, as suggested by Paul: this > triggers batch mode shell when there are double quotes in the recipe. This cannot be automatic, as we have deliberately made the double quotes not special for Windows shells.  So please make this conditional on some Make option (sorry, I didn't yet have time to look at the patch). _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make
Open this post in threaded view
|

## [bug #41246] Allow to switch shell batch mode at runtime instead of build time

 In reply to this post by anonymous Follow-up Comment #11, bug #41246 (project make): Hi Mike, Sorry to hijack this thread a little. I work on the MSYS2 project and I'd be keen to fix "sh.exe -c" to behave correctly. If you are interested in this, could you please email me at [hidden email] with a testcase? FWIW I agree that being able to punt the shell commands into a single script executed via sh.exe is a good feature to have. I'd also like to be able to keep them around afterwards for inspection or to re-run them, like GCC's --save-temps argument.     _______________________________________________________ Reply to this item at:   _______________________________________________   Message sent via/by Savannah   http://savannah.gnu.org/_______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make
Open this post in threaded view
|

## Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time

 In reply to this post by Eli Zaretskii On Fri, Jan 31, 2014 at 11:37:47AM +0200, Eli Zaretskii wrote: > > Date: Fri, 31 Jan 2014 09:14:32 +0000 > > From: Mike Hommey <[hidden email]> > > > > This is a different approach to the problem, as suggested by Paul: this > > triggers batch mode shell when there are double quotes in the recipe. > > This cannot be automatic, as we have deliberately made the double > quotes not special for Windows shells. You mean cmd.exe? The patch only uses batch mode shell when there are double quotes *and* the shell is unixy (so, not cmd.exe) > So please make this > conditional on some Make option (sorry, I didn't yet have time to look > at the patch). An option as command line argument, or as a special target as the original patch did? Mike _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make
Open this post in threaded view
|

## Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time

 > Date: Fri, 31 Jan 2014 22:15:56 +0900 > From: Mike Hommey <[hidden email]> > Cc: [hidden email], [hidden email], [hidden email], > [hidden email] > > On Fri, Jan 31, 2014 at 11:37:47AM +0200, Eli Zaretskii wrote: > > > Date: Fri, 31 Jan 2014 09:14:32 +0000 > > > From: Mike Hommey <[hidden email]> > > > > > > This is a different approach to the problem, as suggested by Paul: this > > > triggers batch mode shell when there are double quotes in the recipe. > > > > This cannot be automatic, as we have deliberately made the double > > quotes not special for Windows shells. > > You mean cmd.exe? The patch only uses batch mode shell when there are > double quotes *and* the shell is unixy (so, not cmd.exe) That's better, but still too subtly to be the default, I think. Having some essentially random character trigger very different behavior is bad mojo for a program in general. > > So please make this > > conditional on some Make option (sorry, I didn't yet have time to look > > at the patch). > > An option as command line argument, or as a special target as the > original patch did? The former, as Paul objected to the latter. _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make
Open this post in threaded view
|

## Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time

 On Fri, 2014-01-31 at 16:55 +0200, Eli Zaretskii wrote: > > An option as command line argument, or as a special target as the > > original patch did? > > The former, as Paul objected to the latter. I didn't object, per se.  I just prefer my tools to DTRT in all cases without me having to use any magical incantations to make it happen :-). My question, or _challenge_ if you like, is whether we can find a way to know, without any hints from the user, whether a given command line will work properly for /bin/sh -c in Windows, or if it needs to use a batch file. It seems, from my lofty vantige point of knowing virtually nothing about it, that this is a solveable problem.  If we can try to write down (in text) the rules for when we would or would not be able to use /bin/sh -c, I would think it would quickly become clear if we can solve it or not. As for whether we use a command line option or makefile target, I think probably the makefile target is probably more appropriate.  In general I like to use command line options to control user preferences which do not have direct impacts on the build, so if you ran the same build with different flags it would still work, just with different behaviors. But for things which will cause the makefile to not work properly if not specified, those things should be embedded into the makefile itself: I don't want it to be the case that you always have to remember to give a particular command line option or your build will break! _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make
Open this post in threaded view
|

## Re: [bug #41246] Allow to switch shell batch mode at runtime instead of build time

 > From: Paul Smith <[hidden email]> > Cc: Mike Hommey <[hidden email]>, [hidden email], > [hidden email], [hidden email] > Date: Fri, 31 Jan 2014 10:59:13 -0500 > > My question, or _challenge_ if you like, is whether we can find a way to > know, without any hints from the user, whether a given command line will > work properly for /bin/sh -c in Windows, or if it needs to use a batch > file. Mike, could you please post some details about your use case(s) that required this?  I'm not sure why would quotes not work with a Unixy shell. Perhaps this is once again something specific to the MSYS conversion of file names? _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make