[bug #49844] 'make -j' without explicit process count sometimes doesn't parallelize

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[bug #49844] 'make -j' without explicit process count sometimes doesn't parallelize

Robert Morell
URL:
  <http://savannah.gnu.org/bugs/?49844>

                 Summary: 'make -j' without explicit process count sometimes
doesn't parallelize
                 Project: make
            Submitted by: dkogan
            Submitted on: Wed 14 Dec 2016 07:02:15 PM GMT
                Severity: 3 - Normal
              Item Group: None
                  Status: None
                 Privacy: Public
             Assigned to: None
             Open/Closed: Open
         Discussion Lock: Any
       Component Version: None
        Operating System: None
           Fixed Release: None
           Triage Status: None

    _______________________________________________________

Details:

Hi. I have this trivial Makefile:

%:
        echo Starting $@ || true
        timeout 5 bash -c 'while true; do true; done' || true
        echo Done $@ || true

It's meant to spin the cpu for 5 seconds for each target being built. I can
ask for 5 simultaneous jobs with something like 'seq 1000 | xargs -n1000 make
-j5' and this command should always redline 5 CPUs. If I give 'make -j'
instead of 'make -j5' I would expect it to redline ALL my CPUs, but I see it
not do any parallelization at all instead. If the Makefile contains normal
(not pattern) rules, then it works how I would expect.




    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?49844>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[bug #49844] 'make -j' without explicit process count sometimes doesn't parallelize

Robert Morell
Follow-up Comment #1, bug #49844 (project make):

Hello.

There is a bug in your trivial example:

1) 'seq 1000 | xargs -n1000 make -j5'

expands to

'make -j5 1 2 3 4 5 6 7 ...'

and

2) 'seq 1000 | xargs -n1000 make -j'

expands to

'make -j 1 2 3 4 5 6 7 ...'

As we can see, in second example '1' after '-j' make interprets as number of
jobs, so "it not do any parallelization at all".


    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?49844>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [bug #49844] 'make -j' without explicit process count sometimes doesn't parallelize

Sven C. Dack
Hello,

this isn't a bug, but it's how the -j option works.

 From the manual:

        -j [jobs], --jobs[=jobs]
             Specifies the number of jobs (commands) to run
simultaneously.  If
             there is more than one -j option, the last one is
effective.   If
             the  -j  option  is given without an argument, make will
not limit
             the number of jobs that can run simultaneously.

Specifying -j1 or -j 1 are the same here.

Your problem is that you are using numbers as target names. So it's not
a bug, because you are literally telling make to run only a single job.

You either have to restrict the number of jobs by giving an explicit
count or by limiting it with a load average ( -l option) or use
non-numerical make targets such as "t1 t2 t3 ..." or simply add another
flag after -j to make.

 From my own experience is it generally better to limit the number of
jobs to what is reasonable or otherwise risk setting off a fork bomb - a
run-away process which spawns too many process too fast and brings the
system to a halt.

If it really needs to be unlimited then using non-numerical targets is
what you can do:

$ seq 1000 | sed 's,^,t,' | xargs -n1000 make -j

which results in:

$ make -j t1 t2 t3 ...

Or if it needs to be unlimited and it has to be numerical arguments can
you simply do this:

$ seq 1000 | xargs -n1000 make -j -C .

which results in:

$ make -j -C . 1 2 3 ...

The -C option tells make to change the directory to the current
directory and avoids the target 1 to be treated as an optional argument
to -j.

Cheers



On 21/06/17 20:39, Michael Builov wrote:

> Follow-up Comment #1, bug #49844 (project make):
>
> Hello.
>
> There is a bug in your trivial example:
>
> 1) 'seq 1000 | xargs -n1000 make -j5'
>
> expands to
>
> 'make -j5 1 2 3 4 5 6 7 ...'
>
> and
>
> 2) 'seq 1000 | xargs -n1000 make -j'
>
> expands to
>
> 'make -j 1 2 3 4 5 6 7 ...'
>
> As we can see, in second example '1' after '-j' make interprets as number of
> jobs, so "it not do any parallelization at all".
>
>
>      _______________________________________________________
>
> Reply to this item at:
>
>    <http://savannah.gnu.org/bugs/?49844>
>
> _______________________________________________
>    Message sent via/by Savannah
>    http://savannah.gnu.org/
>
>
> _______________________________________________
> Bug-make mailing list
> [hidden email]
> https://lists.gnu.org/mailman/listinfo/bug-make
>


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [bug #49844] 'make -j' without explicit process count sometimes doesn't parallelize

Paul Smith-20
On Thu, 2017-06-22 at 13:01 +0100, Sven C. Dack wrote:
> You either have to restrict the number of jobs by giving an explicit
> count or by limiting it with a load average ( -l option) or use
> non-numerical make targets such as "t1 t2 t3 ..." or simply add another
> flag after -j to make.

> $ seq 1000 | xargs -n1000 make -j -C .
>
> which results in:
>
> $ make -j -C . 1 2 3 ...

GNU make, as with all GNU tools (and with all well-formed POSIX
commands) accepts the "--" option to mean "everything after this is not
an option even if it looks like one".

So, the simplest solution is to use:

  seq 1000 | xargs -n1000 make j --

I don't really understand why you use the pipe to xargs.  Wouldn't it be
simpler to just say:

  make -j -- $(seq 1000)

?

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[bug #49844] 'make -j' without explicit process count sometimes doesn't parallelize

Robert Morell
In reply to this post by Robert Morell
Update of bug #49844 (project make):

                  Status:                    None => Not A Bug              
             Open/Closed:                    Open => Closed                

    _______________________________________________________

Follow-up Comment #2:

Even simpler is to just use "--" (and what's the goal of using xargs?):

+verbose+
  make -j5 -- $(seq 1000)

  make -j -- $(seq 1000)
-verbose-

If that ends up not solving your problem please add a new comment and I'll
re-open this issue.

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?49844>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [bug #49844] 'make -j' without explicit process count sometimes doesn't parallelize

Sven C. Dack
In reply to this post by Paul Smith-20
Hello Paul,

I was thinking of "--", but didn't test it, because it isn't being
mentioned in the man page nor in the info page.

Cheers



On 22/06/17 13:10, Paul Smith wrote:

> On Thu, 2017-06-22 at 13:01 +0100, Sven C. Dack wrote:
>> You either have to restrict the number of jobs by giving an explicit
>> count or by limiting it with a load average ( -l option) or use
>> non-numerical make targets such as "t1 t2 t3 ..." or simply add another
>> flag after -j to make.
>> $ seq 1000 | xargs -n1000 make -j -C .
>>
>> which results in:
>>
>> $ make -j -C . 1 2 3 ...
> GNU make, as with all GNU tools (and with all well-formed POSIX
> commands) accepts the "--" option to mean "everything after this is not
> an option even if it looks like one".
>
> So, the simplest solution is to use:
>
>    seq 1000 | xargs -n1000 make j --
>
> I don't really understand why you use the pipe to xargs.  Wouldn't it be
> simpler to just say:
>
>    make -j -- $(seq 1000)
>
> ?
>
> _______________________________________________
> Bug-make mailing list
> [hidden email]
> https://lists.gnu.org/mailman/listinfo/bug-make


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [bug #49844] 'make -j' without explicit process count sometimes doesn't parallelize

Dima Kogan
In reply to this post by Robert Morell

Paul D. Smith <[hidden email]> writes:

> Even simpler is to just use "--" (and what's the goal of using xargs?):
>
> +verbose+
>   make -j5 -- $(seq 1000)
>
>   make -j -- $(seq 1000)
> -verbose-

Hi. Yes, the above is what I intended, and clearly this isn't a bug.
Thanks for looking at it, and sorry for the noise.

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Loading...