[bug #57178] Job server not invoked in initial pass

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[bug #57178] Job server not invoked in initial pass

anonymous
URL:
  <https://savannah.gnu.org/bugs/?57178>

                 Summary: Job server not invoked in initial pass
                 Project: make
            Submitted by: paulberg
            Submitted on: Wed 06 Nov 2019 11:07:42 AM UTC
                Severity: 3 - Normal
              Item Group: Bug
                  Status: None
                 Privacy: Public
             Assigned to: None
             Open/Closed: Open
         Discussion Lock: Any
       Component Version: 4.2.1
        Operating System: Any
           Fixed Release: None
           Triage Status: None

    _______________________________________________________

Details:

In a recipe, a call to a sub-make process using the $(MAKE) macro coordinates
with the job server to maintain a maximum number of parallel jobs as
instructed by the -j parameter. It is expected that any use of this macro
would do the same, but in the initial expansion, use of this macro does not
coordinate and is restricted to -j1.

To Repro use 2 files (below) Makefile and test.mk. test.mk called with -j10
will return 10 normally (some race conditions possible). However, if Makefile
is called with -j10, it invokes test.mk as a sub-make and test.mk will report
that only -j1 was invoked, indicating that the job server is not being
utilized.

Expected: This test when invoked with -j n should report n (or close to n,
race conditions permitting) since the job server should coordinate the number
of available jobs and the sub process should parallelize accordingly.

FILE1 Makefile:

$(info $(shell $(MAKE) -f test.mk all))

all:
        @:

FILE2 test.mk:

all: | parallel ; @echo $(JOB_COUNT)

parallel: .parallel ; @$(eval JOB_COUNT := $(shell sort -n $< | tail -n 1))
.parallel: FORCE ; @$(MAKE) -f test.mk --no-print-directory par 2>/dev/null
>$@ || true
FORCE:

to_n = $(words $2) $(if $(filter-out $1,$(words x $2)),$(call to_n,$1,x $2))

PAR_COUNT :=
par: $(addprefix par-,$(call to_n,32))
par-%: ; @$(eval PAR_COUNT += x)@echo $(words $(PAR_COUNT)) && sleep 1 &&
false





    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?57178>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/


Reply | Threaded
Open this post in threaded view
|

[bug #57178] Job server not invoked in initial pass

anonymous
Follow-up Comment #1, bug #57178 (project make):

Looks like MAKEFLAGS var is not set at this point. Possibly related?

    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?57178>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/


Reply | Threaded
Open this post in threaded view
|

[bug #57178] Allow $(shell $(MAKE) ...) to participate in jobserver

anonymous
Update of bug #57178 (project make):

              Item Group:                     Bug => Enhancement            
                 Summary: Job server not invoked in initial pass => Allow
$(shell $(MAKE) ...) to participate in jobserver

    _______________________________________________________

Follow-up Comment #2:

This issue isn't really about "the first pass", it's about the shell function.
 Even if you were to use the shell function (outside of a recipe) in the
second pass, for example via secondary expansion, you'd still see this same
behavior.

It doesn't have anything to do with the setting of MAKEFLAGS.

Make disables the jobserver when it determines that the process it's about to
invoke is not a sub-make.  This is needed because some programs do not expect
to have extra file descriptors available and may even corrupt the jobserver
pipe.

In recipes make detects this by looking for the '+' recipe prefix or else
detecting $(MAKE) or ${MAKE} in the recipe.

However, this detection is never done for the shell function: we always
disable jobserver information even if the text of the shell function contains
$(MAKE) or ${MAKE}.

This could be resolved.  However, I do urge you to consider a different method
for solving your real problem, rather than running a subshell inside a shell
function.  There are a lot of issues with this (for example, error handling
just to name one).

    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?57178>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/


Reply | Threaded
Open this post in threaded view
|

[bug #57178] Allow $(shell $(MAKE) ...) to participate in jobserver

anonymous
Follow-up Comment #3, bug #57178 (project make):

That makes sense. My use case was not critical, it just appeared to be odd
behavior. Perhaps it should be added to the documentation?

My use case was that I needed a third party component built which utilized
make prior before my makefile's macros could be fully resolved and I was
trying to 1) avoid a full second pass restart and 2) respect the jobserver
flags passed in when calling make on the 3rd party component. An unrelated
dependency ended up requiring a full restart anyway so I ended up tucking the
3rd party component in that first pass as a standard submake, so this
currently isn't affecting me.

I can see an argument for adding this behavior, but it is clear now that it is
intended and would be a feature request, not a bug. I do think a blub in the
docs would be helpful for those who come after me though and I'd agree this
can be resolved.

    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?57178>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/