"unparallelizing" a recursive make

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

"unparallelizing" a recursive make

Darin Johnson
I've been playing around with distcc to see how useful it might be in our
cygwin environment.  but I'm seeing too much parallelism with regards
to recursive sub-makes.  The problem is that the outputs from different
subdirectories get mixed together, which can be confusing when trying
to see what's going on.  Another problem is that if the build may end
after an error in one submake, yet another submake continues to
build in the backround even after returning to the command line.

What I'd like to be able to easily do is have all the makes serialized
while allowing parallelism within each individual subdirectory.  There's
plenty of parallelism to go around even when limiting it this way.
I could go back to the standard for-loop method as one option, or
I could add a "MAKEFLAGS += -j4" inside the makefile rather than on
the command line, but I'm hoping for a cleaner solution if anyone
has suggestions.

Thanks,
Darin Johnson


_______________________________________________
Help-make mailing list
[hidden email]
http://lists.gnu.org/mailman/listinfo/help-make
Reply | Threaded
Open this post in threaded view
|

Re: "unparallelizing" a recursive make

Dan Kegel-2
Darin Johnson wrote:
> I've been playing around with distcc to see how useful it might be in our
> cygwin environment.  but I'm seeing too much parallelism with regards
> to recursive sub-makes.  The problem is that the outputs from different
> subdirectories get mixed together, which can be confusing when trying
> to see what's going on.

That's par for the course.  If you run into errors,
just run without distcc to get a clean log.

(Or do what I did: change your project's .c -> .o
rule to redirect stdout and stderr to .log.
Then do a find and stitch together all the .log files
later.  It's annoying and time-consuming, but
for some projects it's worth the trouble.)


> Another problem is that if the build may end
> after an error in one submake, yet another submake continues to
> build in the backround even after returning to the command line.

That sounds like a classic cygwin problem;
I've seen that happen without distcc.

> What I'd like to be able to easily do is have all the makes serialized
> while allowing parallelism within each individual subdirectory.  There's
> plenty of parallelism to go around even when limiting it this way.
> I could go back to the standard for-loop method as one option, or
> I could add a "MAKEFLAGS += -j4" inside the makefile rather than on
> the command line, but I'm hoping for a cleaner solution if anyone
> has suggestions.

Well, if you ask me, the right solution is to rewrite
your project's build system to get
rid of the recursive sub-makes.  As we all know,
"recursive make is considered harmful".
- Dan

--
Trying to get a job as a c++ developer?  See http://kegel.com/academy/getting-hired.html


_______________________________________________
Help-make mailing list
[hidden email]
http://lists.gnu.org/mailman/listinfo/help-make
Reply | Threaded
Open this post in threaded view
|

Re: "unparallelizing" a recursive make

Darin Johnson
In reply to this post by Darin Johnson
> Well, if you ask me, the right solution is to rewrite
> your project's build system to get
> rid of the recursive sub-makes.  As we all know,
> "recursive make is considered harmful".

Which may be true.  But on cygwin and a fast computer, there is
a noticeable start-up delay on one sub-directory where I used the
non-recursive method.  If the entire project was done this way I
could imagine a startup delay of a minute.  Maybe that's not so
important, but I think it's nicer to the users to not have the delay.

(The sub-projects are pretty much independent of each other as well,
and everything is layed out to be easy to convert to non-recursive
approach if I needed to)




_______________________________________________
Help-make mailing list
[hidden email]
http://lists.gnu.org/mailman/listinfo/help-make
Reply | Threaded
Open this post in threaded view
|

Re: "unparallelizing" a recursive make

Mike Shal
On 6/12/05, Darin Johnson <[hidden email]> wrote:
> Which may be true.  But on cygwin and a fast computer, there is
> a noticeable start-up delay on one sub-directory where I used the
> non-recursive method.  If the entire project was done this way I
> could imagine a startup delay of a minute.  Maybe that's not so
> important, but I think it's nicer to the users to not have the delay.
>
> (The sub-projects are pretty much independent of each other as well,
> and everything is layed out to be easy to convert to non-recursive
> approach if I needed to)

What size delay are you talking about for the single directory? Are
you using a/any/several $(shell) commands to get build parameters?
I've found that's one of the biggest slowdowns for a make - recursive
or non. (Though I have limited experience with Cygwin, if that's any
different)

Also, you seem to imply that by using a non-recursive make, that you
must always do a top-level build. This is not necessarily true - I
generally use a wrapper script that searches up the tree for where to
start the build, and passes in the directory of where the script was
launched. The non-recursive make can then build everything from the
top level, but only load the rules from the execution level (and its
subdirectories).

Of course, if you skip loading any information in this way you might
have an incomplete build. But, if your directories are as independent
as you say, you wouldn't lose any flexibility by using a non-recursive
make.

-Mike


_______________________________________________
Help-make mailing list
[hidden email]
http://lists.gnu.org/mailman/listinfo/help-make
Reply | Threaded
Open this post in threaded view
|

Re: "unparallelizing" a recursive make

Darin Johnson
In reply to this post by Darin Johnson
> What size delay are you talking about for the single directory? Are
> you using a/any/several $(shell) commands to get build parameters?
> I've found that's one of the biggest slowdowns for a make - recursive
> or non. (Though I have limited experience with Cygwin, if that's any
> different)

There are not $(shell) commands.  The thing that takes time is in
loading up all the dependency .d files.   I sped this up a lot by
removing the builtin rules, and now it takes only a second to figure
out that nothing needs building.

My earlier estimates are off though.  I was just scaling up based
on the number of subdirectories to be built.  But after counting everything,
it turns out the directory I tested with accounts for nearly 1/4th of the
total .o files, so that's not a fair calculation.   Maybe it'll take only 5 seconds
to startup because of .d files.



_______________________________________________
Help-make mailing list
[hidden email]
http://lists.gnu.org/mailman/listinfo/help-make