[bug #56701] Do not allow -j without a number

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

[bug #56701] Do not allow -j without a number

Paul D. Smith
URL:
  <https://savannah.gnu.org/bugs/?56701>

                 Summary: Do not allow -j without a number
                 Project: make
            Submitted by: None
            Submitted on: Wed 31 Jul 2019 02:55:34 PM UTC
                Severity: 3 - Normal
              Item Group: Bug
                  Status: None
                 Privacy: Public
             Assigned to: None
             Open/Closed: Open
         Discussion Lock: Any
       Component Version: 4.2.1
        Operating System: POSIX-Based
           Fixed Release: None
           Triage Status: None

    _______________________________________________________

Details:

If there is no number with -j, make uses infinite resources, compiling the
Linux kernel stops to out of memory error and causes Xfce desktop to reboot
login via the Linux OOM killer.




    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?56701>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

[bug #56701] Do not allow -j without a number

Paul D. Smith
Update of bug #56701 (project make):

              Item Group:                     Bug => Enhancement            

    _______________________________________________________

Follow-up Comment #1:

Actually -j without a number is useful: it's used in conjunction with the -l
option to allow parallelism to be limited by system load rather than an
explicit number of outstanding jobs.

I implemented a change which requires the -l option to be provided if -j is
given without an argument, else you get an error.

However that's a large backward-compatibility change so I'm not sure about it.
 Just as an example, I had to modify quite a number of tests in the GNU make
regression test suite after making this change.  Of course, it's quite
reasonable to say that the usages in the regression test suite are not
appropriate to "real world" usages.

I'll need to think about this.  If anyone has opinions on whether this would
be a good change and/or how much breakage it would cause please let me know.

I'm changing this to an enhancement because the current behavior is (a)
documented, (b) useful, and (c) how make has worked for 30+ years.  The
question is can we find a way to avoid the downsides, and is the cost in
backward-compatibiity worth it.

    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?56701>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

[bug #56701] Do not allow -j without a number

Paul D. Smith
Follow-up Comment #2, bug #56701 (project make):

Let's try to fix this issue and avoid introducing any regressions.

For example, if -j is given without an argument and no -l option is provided,
behave as if -j -l $(nproc) was specified.

    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?56701>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: [bug #56701] Do not allow -j without a number

Gnu - Make - Bugs mailing list
Personally I would have closed this as Not a Bug since it has been
clearly documented for 30+ years. Users are responsible for reading
the docs and learning how their tools work.

Dmitry V. Levin wrote:
> Follow-up Comment #2, bug #56701 (project make):
>
> Let's try to fix this issue and avoid introducing any regressions.
>
> For example, if -j is given without an argument and no -l option is provided,
> behave as if -j -l $(nproc) was specified.

--
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

[bug #56701] Do not allow -j without a number

Paul D. Smith
In reply to this post by Paul D. Smith
Follow-up Comment #3, bug #56701 (project make):

First, "$(nproc)" is not easy to obtain in a portable manner; this has been
asked many times before.

Second, I'm not sure if using -l $(nproc) is the right way to handle this.
It's still a change in behavior, just one that's more obscure.

Let's still think and discuss :)

    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?56701>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

[bug #56701] Do not allow -j without a number

Paul D. Smith
Follow-up Comment #4, bug #56701 (project make):

FWIW, nproc(1) is part of GNU coreutils, it is based on nproc module from
gnulib, so the way it works is probably as much portable as one can get.

If GNU make was using gnulib directly, it would cost just a single invocation
of num_processors.

    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?56701>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: [bug #56701] Do not allow -j without a number

David Boyce-3
My vote is to leave -j alone. Everyone knows it shouldn't be used that way in production, especially on shared host, but I've been known to use -j alone when testing out the parallel-safety of a makefile or in various other ad-hoc test scenarios. I could even imagine it being used deliberately as a stress test.

What might make more sense to me would be introducing a new syntax like -j#2 which would evaluate to 2 * $(nproc), with caveats to previous $(nproc) discussion.

On Mon, Aug 26, 2019 at 8:04 AM Dmitry V. Levin <[hidden email]> wrote:
Follow-up Comment #4, bug #56701 (project make):

FWIW, nproc(1) is part of GNU coreutils, it is based on nproc module from
gnulib, so the way it works is probably as much portable as one can get.

If GNU make was using gnulib directly, it would cost just a single invocation
of num_processors.

    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?56701>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: [bug #56701] Do not allow -j without a number

Paul Smith-20
On Mon, 2019-08-26 at 10:32 -0700, David Boyce wrote:
> My vote is to leave -j alone. Everyone knows it shouldn't be used
> that way in production, especially on shared host, but I've been
> known to use -j alone when testing out the parallel-safety of a
> makefile or in various other ad-hoc test scenarios. I could even
> imagine it being used deliberately as a stress test.

If it weren't so devastating if you got it wrong, and so simple to get
wrong especially for newbies, I would definitely not even entertain
changing it.

But it's kind of sucky to have an innocuous typo bring down your entire
session, or worse.

Maybe, though, Dmitry's idea of changing the default of the -l option
would be a better way forward than failing.  This would likely have far
fewer noticeable consequences to users and setting it sufficiently high
probably wouldn't bother anyone while still limiting damage.

> What might make more sense to me would be introducing a new syntax
> like -j#2 which would evaluate to 2 * $(nproc), with caveats to
> previous $(nproc) discussion.

I've been toying with the idea of adding -j +N / -j -N which would mean
"number of processers [-+] N".  Adding * (and /?) could also be done.
This could even take a float to allow something like -j *1.5.

Regarding nproc: GNU make *does* use gnulib and it seems that gnulib
nproc may support Windows as well so that's at least a good baseline.

GNU make also supports much more esoteric systems (particularly VMS)
which gnulib doesn't even take a stab at.  I guess we could disable
these features on systems without an nproc equivalent.


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: [bug #56701] Do not allow -j without a number

Dennis Clarke
In reply to this post by David Boyce-3
On 8/26/19 1:32 PM, David Boyce wrote:
> My vote is to leave -j alone. Everyone knows it shouldn't be used that
> way in production...

Please allow me to follow up here and agree. Common sense should prevail
and in the true sense of UNIX and Linux philosophy a tool does one thing
and does that one thing well. It should need to hand hold the user with
common sense.



--
Dennis Clarke
RISC-V/SPARC/PPC/ARM/CISC
UNIX and Linux spoken
GreyBeard and suspenders optional

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: [bug #56701] Do not allow -j without a number

Dennis Clarke
On 8/26/19 1:46 PM, Dennis Clarke wrote:
> On 8/26/19 1:32 PM, David Boyce wrote:
>> My vote is to leave -j alone. Everyone knows it shouldn't be used that
>> way in production...
>
> Please allow me to follow up here and agree. Common sense should prevail
> and in the true sense of UNIX and Linux philosophy a tool does one thing
> and does that one thing well. It should need to hand hold the user with
> common sense.

Should NOT need to hand hold the user.

Also gnulib is not what I call portable. So if the sysconf/getconf calls
are not around to give us _NPROCESSORS_ONLN or even _NPROCESSORS_CONF
then no way should some hack attempt be made to extract that data. No
promise the data is available at all in a portable manner anyways and
get_nprocs_conf should be entirely avoided.



--
Dennis Clarke
RISC-V/SPARC/PPC/ARM/CISC
UNIX and Linux spoken
GreyBeard and suspenders optional

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

[bug #56701] Do not allow -j without a number

Paul D. Smith
In reply to this post by Paul D. Smith
Follow-up Comment #5, bug #56701 (project make):

Well gnulib is not what I call portable. So if the sysconf/getconf calls
are not around to give us _NPROCESSORS_ONLN or even _NPROCESSORS_CONF
then no way should some hack attempt be made to extract that data. No
promise the data is available at all in a portable manner anyways and
get_nprocs_conf should be entirely avoided.

The "defacto" standard which has worked for the past three decades
should not be messed with unless there exist some really amazingly
solid arguments.  As for a system exhausting all resources due to a
badly specified set of parameters to "make" one could argue that any
user can do worse damage with "rm -rf *" and that is the users issue
entirely.  Giving the user a hug should not be required here.

Dennis


    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?56701>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: [bug #56701] Do not allow -j without a number

Robert Pluim
In reply to this post by Paul D. Smith
>>>>> On Mon, 26 Aug 2019 08:35:15 -0400 (EDT), "Paul D. Smith" <[hidden email]> said:

    Paul> Update of bug #56701 (project make):
    Paul>               Item Group:                     Bug => Enhancement            

    Paul>     _______________________________________________________

    Paul> Follow-up Comment #1:

    Paul> Actually -j without a number is useful: it's used in conjunction with the -l
    Paul> option to allow parallelism to be limited by system load rather than an
    Paul> explicit number of outstanding jobs.

    Paul> I implemented a change which requires the -l option to be provided if -j is
    Paul> given without an argument, else you get an error.

    Paul> However that's a large backward-compatibility change so I'm not sure about it.
    Paul>  Just as an example, I had to modify quite a number of tests in the GNU make
    Paul> regression test suite after making this change.  Of course, it's quite
    Paul> reasonable to say that the usages in the regression test suite are not
    Paul> appropriate to "real world" usages.

    Paul> I'll need to think about this.  If anyone has opinions on whether this would
    Paul> be a good change and/or how much breakage it would cause please let me know.

    Paul> I'm changing this to an enhancement because the current behavior is (a)
    Paul> documented, (b) useful, and (c) how make has worked for 30+ years.  The
    Paul> question is can we find a way to avoid the downsides, and is the cost in
    Paul> backward-compatibiity worth it.

Please donʼt do this. When I type 'make -j', I mean it. If I get it
wrong and kill my box, thatʼs on me, make should not be handholding
here.

Robert

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make