Parallel build dependency locking

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Parallel build dependency locking

brlcad
Hello,

I've been working on adding a "fast" compilation target to a relatively
large project that uses the GNU Build System and have been running into
a problem that I have not been able to overcome.  The intent is to
build all of the object files in a given directory before performing
any linking similar to what is done when there is only one product.  
This project has several What I am seeing is that usually the last
objects end up getting compiled twice on a dual processor machine when
using -j2 (hence 4 times for an ltcompile).  Very rarely, this may also
occur for other object targets too and seems to be some sort of race
condition where the two -j threads end up compiling the same file.

I've narrowed down the problem to a relatively simple set of make rules
that exhibit the problem and I'm hoping there's some light someone
might be able to shine on me to get this to work.  It's not evident
whether or not this is a parallel make bug, but it is at least
unexpected.

In a Makefile.fast I have the following:

.PHONY : fast fast-local fast-recursive

fast: fast-local $(FAST_OBJECTS) fast-recursive all

fast-recursive:
        @list='$(SUBDIRS)'; for subdir in $$list; do \
                test "$$subdir" = . || (cd $$subdir && $(MAKE) $(AM_MAKEFLAGS) fast);
\
        done

fast-local:

Then in the automake'd Makefile.am I simply add the object lists (for
this example, it's simplified to just a single libtool library being
built):

include $(top_srcdir)/Makefile.fast
FAST_OBJECTS = $(libbn_la_OBJECTS)

The resultant Makefile looks correct and I don't see how it's even any
different for the libbn.la target, but the end result certainly is:

...
FAST_OBJECTS = $(libbn_la_OBJECTS)
libbn_la_DEPENDENCIES =
am__objects_1 = anim.lo asize.lo axis.lo bn_tcl.lo complex.lo const.lo \
         font.lo fortran.lo list.lo marker.lo mat.lo msr.lo noise.lo \
         number.lo plane.lo plot3.lo poly.lo qmath.lo rand.lo scale.lo \
         sphmap.lo symbol.lo tabdata.lo tplot.lo vectfont.lo vector.lo \
         vert_tree.lo wavelet.lo
am_libbn_la_OBJECTS = $(am__objects_1) vers.lo
libbn_la_OBJECTS = $(am_libbn_la_OBJECTS)
...
libbn.la: $(libbn_la_OBJECTS) $(libbn_la_DEPENDENCIES)
         $(LINK) -rpath $(libdir) $(libbn_la_LDFLAGS)
$(libbn_la_OBJECTS) $(libbn_la_LIBADD) $(LIBS)
...
.c.lo:
         $(LTCOMPILE) -c -o $@ `test -f '$<' || echo '$(srcdir)/'`$<
...
vers.c: $(SRCS)
         /bin/sh $(top_srcdir)/sh/vers.sh bn_version "The BRL-CAD
Numerical Computation Library" > $@


So that all looks good and peachy to me.  The fast target rules get
called when I do a 'make -j2 fast' and will walk the hierarchy
properly, and the object files are built as I'd hope.  The problem
seems to occur, though, that multiple .c.lo's are occuring
simultaneously under certain circumstances for the fast target yet not
for the libbn.la target:

[~/brlcad/src/libbn]$ make -j2 fast
/bin/sh ../../libtool --mode=compile gcc -DHAVE_CONFIG_H -I. -I.
-I../../include   -I/usr/X11R6/include -I../../include  -pipe
-fno-strict-aliasing -fno-common -fPIC -g -c -o anim.lo `test -f
'anim.c' || echo './'`anim.c
/bin/sh ../../libtool --mode=compile gcc -DHAVE_CONFIG_H -I. -I.
-I../../include   -I/usr/X11R6/include -I../../include  -pipe
-fno-strict-aliasing -fno-common -fPIC -g -c -o asize.lo `test -f
'asize.c' || echo './'`asize.c
mkdir .libs
  gcc -DHAVE_CONFIG_H -I. -I. -I../../include -I/usr/X11R6/include
-I../../include -pipe -fno-strict-aliasing -fno-common -fPIC -g -c
anim.c  -fno-common -DPIC -o .libs/anim.o
  gcc -DHAVE_CONFIG_H -I. -I. -I../../include -I/usr/X11R6/include
-I../../include -pipe -fno-strict-aliasing -fno-common -fPIC -g -c
asize.c  -fno-common -DPIC -o .libs/asize.o
  gcc -DHAVE_CONFIG_H -I. -I. -I../../include -I/usr/X11R6/include
-I../../include -pipe -fno-strict-aliasing -fno-common -fPIC -g -c
asize.c -o asize.o >/dev/null 2>&1
  gcc -DHAVE_CONFIG_H -I. -I. -I../../include -I/usr/X11R6/include
-I../../include -pipe -fno-strict-aliasing -fno-common -fPIC -g -c
anim.c -o anim.o >/dev/null 2>&1
...
/bin/sh ../../libtool --mode=compile gcc -DHAVE_CONFIG_H -I. -I.
-I../../include   -I/usr/X11R6/include -I../../include  -pipe
-fno-strict-aliasing -fno-common -fPIC -g -c -o wavelet.lo `test -f
'wavelet.c' || echo './'`wavelet.c
  gcc -DHAVE_CONFIG_H -I. -I. -I../../include -I/usr/X11R6/include
-I../../include -pipe -fno-strict-aliasing -fno-common -fPIC -g -c
vert_tree.c -o vert_tree.o >/dev/null 2>&1
  gcc -DHAVE_CONFIG_H -I. -I. -I../../include -I/usr/X11R6/include
-I../../include -pipe -fno-strict-aliasing -fno-common -fPIC -g -c
wavelet.c  -fno-common -DPIC -o .libs/wavelet.o
/bin/sh ../../sh/vers.sh bn_version "The BRL-CAD Numerical Computation
Library" > vers.c
make  all-am
/bin/sh ../../libtool --mode=compile gcc -DHAVE_CONFIG_H -I. -I.
-I../../include   -I/usr/X11R6/include -I../../include  -pipe
-fno-strict-aliasing -fno-common -fPIC -g -c -o wavelet.lo `test -f
'wavelet.c' || echo './'`wavelet.c
  gcc -DHAVE_CONFIG_H -I. -I. -I../../include -I/usr/X11R6/include
-I../../include -pipe -fno-strict-aliasing -fno-common -fPIC -g -c
wavelet.c -o wavelet.o >/dev/null 2>&1
  gcc -DHAVE_CONFIG_H -I. -I. -I../../include -I/usr/X11R6/include
-I../../include -pipe -fno-strict-aliasing -fno-common -fPIC -g -c
wavelet.c  -fno-common -DPIC -o .libs/wavelet.o
  gcc -DHAVE_CONFIG_H -I. -I. -I../../include -I/usr/X11R6/include
-I../../include -pipe -fno-strict-aliasing -fno-common -fPIC -g -c
wavelet.c -o wavelet.o >/dev/null 2>&1
/bin/sh ../../libtool --mode=compile gcc -DHAVE_CONFIG_H -I. -I.
-I../../include   -I/usr/X11R6/include -I../../include  -pipe
-fno-strict-aliasing -fno-common -fPIC -g -c -o vers.lo `test -f
'vers.c' || echo './'`vers.c
  gcc -DHAVE_CONFIG_H -I. -I. -I../../include -I/usr/X11R6/include
-I../../include -pipe -fno-strict-aliasing -fno-common -fPIC -g -c
vers.c  -fno-common -DPIC -o .libs/vers.o
  gcc -DHAVE_CONFIG_H -I. -I. -I../../include -I/usr/X11R6/include
-I../../include -pipe -fno-strict-aliasing -fno-common -fPIC -g -c
vers.c -o vers.o >/dev/null 2>&1
/bin/sh ../../libtool --mode=compile gcc -DHAVE_CONFIG_H -I. -I.
-I../../include   -I/usr/X11R6/include -I../../include  -pipe
-fno-strict-aliasing -fno-common -fPIC -g -c -o vers.lo `test -f
'vers.c' || echo './'`vers.c
/bin/sh ../../libtool --mode=link gcc  -pipe -fno-strict-aliasing
-fno-common -fPIC -g  -L/usr/X11R6/lib -L/usr/local/lib -pipe
-fno-strict-aliasing -fno-common -fPIC -Wl,-search_paths_first
-mdynamic-no-pic -g -o libbn.la -rpath /usr/brlcad/lib -version-info
19:1 anim.lo asize.lo axis.lo bn_tcl.lo complex.lo const.lo font.lo
fortran.lo list.lo marker.lo mat.lo msr.lo noise.lo number.lo plane.lo
plot3.lo poly.lo qmath.lo rand.lo scale.lo sphmap.lo symbol.lo
tabdata.lo tplot.lo vectfont.lo vector.lo vert_tree.lo wavelet.lo
vers.lo ../../src/libbu/libbu.la ../../src/other/libtcl/libtcl8.4.la  
-lm
  gcc -DHAVE_CONFIG_H -I. -I. -I../../include -I/usr/X11R6/include
-I../../include -pipe -fno-strict-aliasing -fno-common -fPIC -g -c
vers.c  -fno-common -DPIC -o .libs/vers.o
  gcc -DHAVE_CONFIG_H -I. -I. -I../../include -I/usr/X11R6/include
-I../../include -pipe -fno-strict-aliasing -fno-common -fPIC -g -c
vers.c -o vers.o >/dev/null 2>&1
libtool: link: `wavelet.lo' is not a valid libtool object
make[1]: *** [libbn.la] Error 1
make: *** [all] Error 2

What's hopefully apparent in that is that wavelet.lo and vers.lo are
getting ltcompiled twice there at the end.  The wavelet.lo is invalid
as it's written to by both thread simultaneously and one clobbers a
part of the file.  Why two started in the first place seems to be the
issue.

Compared to wavelet.c, vers.c is very trivial so I can see how a thread
working on vers.lo would finish before the wavelet.lo target's thread
gets a chance to even start writing.  That would only make sense,
though, if the OBJECTS are not being properly locked by make on a
parallel build -- and it doesn't seem to explain why the libbn.la
target with the same target dependancies doesn't exhibit the same
problem.  Any advice, suggestions, hints, or slaps on my forehead?

Cheers!
Sean



_______________________________________________
Help-make mailing list
[hidden email]
http://lists.gnu.org/mailman/listinfo/help-make
Reply | Threaded
Open this post in threaded view
|

Re: Parallel build dependency locking

Boris Kolpackov
First off, there are no threads; GNU make issues (certain) commands in
parallel when run with -j flag.

Christopher Sean Morrison <[hidden email]> writes:


> fast: fast-local $(FAST_OBJECTS) fast-recursive all
>
> fast-recursive:
> @list='$(SUBDIRS)'; for subdir in $$list; do \
> test "$$subdir" = . || (cd $$subdir && $(MAKE) $(AM_MAKEFLAGS) fast);
> \
> done

Perhaps you are building the same object file from two concurrent make
invocations. One is current and the other is the result of the
fast-recursive rule?



> Compared to wavelet.c, vers.c is very trivial so I can see how a thread
> working on vers.lo would finish before the wavelet.lo target's thread
> gets a chance to even start writing.


> That would only make sense,
> though, if the OBJECTS are not being properly locked by make on a
> parallel build

GNU make doesn't do any locking, AFAICT.


> -- and it doesn't seem to explain why the libbn.la
> target with the same target dependancies doesn't exhibit the same
> problem.

Perhaps it's because fast-recursive rule is not built?


hth,
-boris




_______________________________________________
Help-make mailing list
[hidden email]
http://lists.gnu.org/mailman/listinfo/help-make
Reply | Threaded
Open this post in threaded view
|

Re: Parallel build dependency locking

brlcad

Boris,

Thanks for the reply!  One of your comments pushed me into the right
direction; I believe I have it working now and maybe even understand
what was happening that caused problems.  I simplified the fast target
rule to just simply build without recursing since the recursion was not
strictly relevant to that particular library example (since I was
already fully recursed with no remaining SUBDIRS).  That said, it does
seem like a possible contention bug or perhaps just undesirable
behavior.  I simplified the 'fast' target rule to just:

fast: $(FAST_OBJECTS) all

and the problem still remained.  Yet when I removed the 'all'
dependency, the problem went away.  Looking at the automake-generated
'all' dependency, the problem becomes more clear:

all:
        $(MAKE) $(AM_MAKEFLAGS) all-am

So there is the second invocation of make coming into play that hits up
the 'all-am' rule, which eventually includes the same objects included
in FAST_OBJECTS.  So when the first process is working on the last
couple FAST_OBJECTS objects, the second one continues to the 'all'
rule, causes make to reinvoke, and ends up picking up where the build
seemingly left off at which ultimately clobbers the first.  Sounds like
it's fairly safe to presume that there's no concurrent make file/job
locking across to submakes (e.g. some file-based semaphore locking,
shared memory locking, etc).  Is this a bug/feature request worth
reporting?

The fix/workaround for this particular example was rather simple.  The
second invocation can be avoided by using 'all-am' as the 'fast' rule
dependency instead of 'all'.  Once I did that, the sub-make is skipped.
  The entire recursion and fast build subsequently works great.  For our
particular software suite which includes over 400 binaries and a couple
dozen libraries, this saves quite a bit of time (wish the automake
folks provided such a facility automatically).  Thanks again.

Cheers,
Sean


On Jun 8, 2005, at 12:15 PM, Boris Kolpackov wrote:

> First off, there are no threads; GNU make issues (certain) commands in
> parallel when run with -j flag.
>
> Christopher Sean Morrison <[hidden email]> writes:
>
>> fast: fast-local $(FAST_OBJECTS) fast-recursive all
>>
>> fast-recursive:
>> @list='$(SUBDIRS)'; for subdir in $$list; do \
>> test "$$subdir" = . || (cd $$subdir && $(MAKE) $(AM_MAKEFLAGS)
>> fast);
>> \
>> done
>
> Perhaps you are building the same object file from two concurrent make
> invocations. One is current and the other is the result of the
> fast-recursive rule?
>
>> Compared to wavelet.c, vers.c is very trivial so I can see how a
>> thread
>> working on vers.lo would finish before the wavelet.lo target's thread
>> gets a chance to even start writing.
>
>> That would only make sense,
>> though, if the OBJECTS are not being properly locked by make on a
>> parallel build
>
> GNU make doesn't do any locking, AFAICT.
>
>> -- and it doesn't seem to explain why the libbn.la
>> target with the same target dependancies doesn't exhibit the same
>> problem.
>
> Perhaps it's because fast-recursive rule is not built?
>
>
> hth,
> -boris



_______________________________________________
Help-make mailing list
[hidden email]
http://lists.gnu.org/mailman/listinfo/help-make
Reply | Threaded
Open this post in threaded view
|

Re: Parallel build dependency locking

Boris Kolpackov
Christopher Sean Morrison <[hidden email]> writes:

> Sounds like
> it's fairly safe to presume that there's no concurrent make file/job
> locking across to submakes (e.g. some file-based semaphore locking,
> shared memory locking, etc).  Is this a bug/feature request worth
> reporting?

No. ;-)

Seriously, if you are building the same target more than once then
it's a bug in your makefile and you normally wouldn't want make to
hide it from you.

-boris



_______________________________________________
Help-make mailing list
[hidden email]
http://lists.gnu.org/mailman/listinfo/help-make
Reply | Threaded
Open this post in threaded view
|

Re: Parallel build dependency locking

brlcad
>> Sounds like
>> it's fairly safe to presume that there's no concurrent make file/job
>> locking across to submakes (e.g. some file-based semaphore locking,
>> shared memory locking, etc).  Is this a bug/feature request worth
>> reporting?
>
> No. ;-)
>
> Seriously, if you are building the same target more than once then
> it's a bug in your makefile and you normally wouldn't want make to
> hide it from you.

That may very well be true, but in this particular instance, the
workaround/solution to use the all-am did nothing to change which
targets were reached.  It merely avoided a secondary invocation of make
(that did the exact same thing, reached the same targets/rules in
parallel, etc).

There is already support for processing multiple targets simultaneously
albeit through some form of locking to ensure that each
dependency/target is being handled by only one of the -j workers (in
that process).  That is the simple nature of a -j parallel build.  All
I'm suggesting is a means to extend that same locking mechanism on
dependancies being processed by child, sub-process, or otherwise
concurrent makes also.

Basically, that would be a means to share the current processing
environment across concurrent makes.  It should only impact the -j
parallel builds if done correctly, allowing the workers that have
completed with one dependency to progress to the next set of rules
while remaining coherent with respect to each other's progress.  This
would be a significant boost on my larger SMP systems where a -j32 or
-j200 or larger is not unheard of -- with proper locking, you could
even start to think about structuring together makefiles that perform
simultaneous multi-directory builds (e.g. build all your libraries in
parallel before building your binaries (also in parallel)) or more
complicated rule resolutions.  Heck, one could even invoke make
directly N times (without -j and with proper external locking) and have
it successfully build in parallel instead of having the various
processes stomp on each other's results (quite useless).

It's a feature that I've heard from others as well that work on larger
SMP systems with make-based projects.  In general, it's just requests
for better support for parallel builds in general and the
aforementioned suggestion would be an overall improvement.  For now,
I've hacked around it enough so that almost our entire package will now
compile and link in parallel, significantly reducing our compile times
-- so I'm quite happy.  ;-)  The only thing that I didn't attempt was
processing multiple independent directories simultaneously.. I'll save
that for another rainy day.

Cheers!
Sean






_______________________________________________
Help-make mailing list
[hidden email]
http://lists.gnu.org/mailman/listinfo/help-make
Reply | Threaded
Open this post in threaded view
|

Re: Parallel build dependency locking

John Graham-Cumming
On Thu, 2005-06-09 at 19:11 -0400, Christopher Sean Morrison wrote:

> That may very well be true, but in this particular instance, the
> workaround/solution to use the all-am did nothing to change which
> targets were reached.  It merely avoided a secondary invocation of make
> (that did the exact same thing, reached the same targets/rules in
> parallel, etc).
>
> There is already support for processing multiple targets simultaneously
> albeit through some form of locking to ensure that each
> dependency/target is being handled by only one of the -j workers (in
> that process).  That is the simple nature of a -j parallel build.  All
> I'm suggesting is a means to extend that same locking mechanism on
> dependancies being processed by child, sub-process, or otherwise
> concurrent makes also.

(Disclaimer: I am not a GNU Make developer, but I spend all my time
dealing with parallel builds in my day job at Electric Cloud.)

What you are suggesting is, no doubt, possible but you are asking for
locking on a target across independent make processes.  Clearly, one
could implement locking of that form (although doing it on every
platform that GNU Make supports might be tricky).

However, I think it's the wrong solution to your problem.  It appears
that what you want to do is speed up your build, and you've chosen to do
that by running a parallel build.   If you want to do that then there
are a couple of necessities: you need to make sure that there are no
missing dependencies and that nothing is built twice.

Neither seem, to me, to be GNU Make's responsibility, and implementing
locking would just mean that you allow your build to make targets more
than once.  Isn't the right solution (if your goal is speed) to only
build things once?

John.




_______________________________________________
Help-make mailing list
[hidden email]
http://lists.gnu.org/mailman/listinfo/help-make
Reply | Threaded
Open this post in threaded view
|

Re: Parallel build dependency locking

brlcad
On Jun 9, 2005, at 8:01 PM, John Graham-Cumming wrote:

> On Thu, 2005-06-09 at 19:11 -0400, Christopher Sean Morrison wrote:
>>
>> There is already support for processing multiple targets
>> simultaneously
>> albeit through some form of locking to ensure that each
>> dependency/target is being handled by only one of the -j workers (in
>> that process).  That is the simple nature of a -j parallel build.  All
>> I'm suggesting is a means to extend that same locking mechanism on
>> dependancies being processed by child, sub-process, or otherwise
>> concurrent makes also.
> [snip]
> What you are suggesting is, no doubt, possible but you are asking for
> locking on a target across independent make processes.  Clearly, one
> could implement locking of that form (although doing it on every
> platform that GNU Make supports might be tricky).

That is indeed what I'd be interested in having.  Another alternative
might be to consider the sub-process makes as not being independent,
sharing the exec knowledge of their parent via some mechanism
(environment vars, process id arguments, etc).  It would seem more
simple to just handle the more general independent process case, but
considering only sub-process makes and making them dependent or
otherwise state sharing would even be beneficial.

> However, I think it's the wrong solution to your problem.  It appears
> that what you want to do is speed up your build, and you've chosen to
> do
> that by running a parallel build.   If you want to do that then there
> are a couple of necessities: you need to make sure that there are no
> missing dependencies and that nothing is built twice.

In the cases I have been involved with, those two concerns are
generally not a problem to say the least.  Most (including the one in
question) are 'gnu build system' projects where the dependencies are
already in place being described via automake and libtool declarations.
  With those declarations in place, the build proceeds just fine with
proper dependency ordering and nothing builds twice.  The
double-compile problem that I was working through was clearly
undesirable of course and I'd probably even go so far as to say
building twice is simply 'wrong'.  Mind you, I'm referring to the
actual compilation of the source to an object file, not reaching the
Makefile rule and realizing there's nothing to do (e.g. making target
binary foo depend on binary bar's objects and bar itself -- when
processing foo's dependencies, bar's objects are listed explicitly and
indirectly through bar's rule).

The 'problem' is perhaps just a limitation of the make logic being
exported by automake and presumptions on possible target dependancies
(e.g. SUBDIRS, bin_PROGRAMS, lib_LTPROGRAMS, etc are always traversed
in order one at a time).  Since it processes each program/library in
order, if there are a lot of 'em (hundreds as there are in my case),
parallel build performance is crippled.  It ends up compiling an
object, linking a binary, compiling an object, linking a binary over
and over effectively serializing the build process.  All of the objects
are actually independent (and specified as such via the automake rules)
so they can go in parallel, as are all the binaries so they could all
link in parallel so long as the corresponding objects are done.  
Minimally, I now know that it 'can' be done, since I just worked in my
own make traversal logic that does just that -- it now compiles all
object files in parallel and then links all binaries together in
parallel.

That said, I was only able to do that by avoiding that separate make
process that automake or autoconf injected on the 'all' target.  While
I do heavily concede that it's probably more a design limitation of the
automake/autoconf-generated Makefiles since they could just as easily
provide a similar 'fast' make path like I wrote, I believe that the
cross-process dependency locking would allow much greater potential for
more effective parallel builds without requiring too much hoop-jumping
and customization on the Makefile writer.  It would also produce the
exact same end result with no detriment to other parallel or
non-parallel builds (that come to mind).

It should indeed only 'build things once'.  I'm suggesting extending
the same logic that exists with the -j option across make instances.  
So if make -j2 works, so would running make& twice and it should be
possible to make it execute with practically identical behavior.

Sean



_______________________________________________
Help-make mailing list
[hidden email]
http://lists.gnu.org/mailman/listinfo/help-make
Reply | Threaded
Open this post in threaded view
|

Re: Parallel build dependency locking

John Graham-Cumming
On Thu, 2005-06-09 at 21:39 -0400, Christopher Sean Morrison wrote:
> That is indeed what I'd be interested in having.  Another alternative
> might be to consider the sub-process makes as not being independent,
> sharing the exec knowledge of their parent via some mechanism
> (environment vars, process id arguments, etc).  It would seem more
> simple to just handle the more general independent process case, but
> considering only sub-process makes and making them dependent or
> otherwise state sharing would even be beneficial.

That's kind of what we do at Electric Cloud, we merge all the sub-makes
into one giant make and have visibility over everything.  It's highly
non-trivial to implement that :-)

John.
--
John Graham-Cumming

Home: http://www.jgc.org/
Work: http://www.electric-cloud.com/
POPFile: http://getpopfile.org/
GNU Make Standard Library: http://gmsl.sf.net/




_______________________________________________
Help-make mailing list
[hidden email]
http://lists.gnu.org/mailman/listinfo/help-make
Reply | Threaded
Open this post in threaded view
|

Re: Parallel build dependency locking

Paul Smith-20
In reply to this post by brlcad
%% Christopher Sean Morrison <[hidden email]> writes:

  csm> On Jun 9, 2005, at 8:01 PM, John Graham-Cumming wrote:
  >> On Thu, 2005-06-09 at 19:11 -0400, Christopher Sean Morrison wrote:

  >>> There is already support for processing multiple targets
  >>> simultaneously albeit through some form of locking to ensure that
  >>> each dependency/target is being handled by only one of the -j
  >>> workers (in that process).  That is the simple nature of a -j
  >>> parallel build.  All I'm suggesting is a means to extend that same
  >>> locking mechanism on dependancies being processed by child,
  >>> sub-process, or otherwise concurrent makes also.

I agree with Boris and John here.  What you're asking for is not only
very difficult, but not necessary.  A well-formed makefile environment
will have exactly one makefile controlling any given target, and that
one makefile will be invoked once by one instance of make (or at least,
one at a time), never more than once at the same time.

If your makefile environment is well-formed, then the kind of locking
etc. you are looking for is completely unnecessary as far as I can
tell.  I don't see the point in expending huge amounts of time and
effort to better support what is, basically, bad makefile design.

  csm> That is indeed what I'd be interested in having.  Another
  csm> alternative might be to consider the sub-process makes as not
  csm> being independent, sharing the exec knowledge of their parent via
  csm> some mechanism (environment vars, process id arguments, etc).  It
  csm> would seem more simple to just handle the more general
  csm> independent process case, but considering only sub-process makes
  csm> and making them dependent or otherwise state sharing would even
  csm> be beneficial.

If you want this model you should just use inclusive (non-recursive)
makefile structures and be done with it, IMO.  As above, I don't see the
point in creating a very complex way for multiple instances of make to
share dependency graph information, detailed runtime information,
etc. when all you need to do is have one make instance read in both
makefiles to get the same effect.

  csm> The 'problem' is perhaps just a limitation of the make logic
  csm> being exported by automake and presumptions on possible target
  csm> dependancies (e.g. SUBDIRS, bin_PROGRAMS, lib_LTPROGRAMS, etc are
  csm> always traversed in order one at a time).

As far as I can see this is true of SUBDIRS (which is, of course,
unfortunate), but it isn't true of programs, libraries, etc.  They are
all listed as prerequisites of the all-am: target, and as such they are
eligible to be built in parallel, unless there are further prerequisite
statements made.

  csm> Since it processes each program/library in order, if there are a
  csm> lot of 'em (hundreds as there are in my case), parallel build
  csm> performance is crippled.  It ends up compiling an object, linking
  csm> a binary, compiling an object, linking a binary over and over
  csm> effectively serializing the build process.

I don't know what you mean by this.  Make does NOT require a sibling
prerequisite to be completely built before it moves to the next one
(again, assuming no other prerequisite relationships).  I must be
misunderstanding the problem you're seeing.  Consider this makefile:

  .PHONY: all
  all: one.x two.x three.x four.x five.x six.x

  %.x : %.y ; @echo cp $< $@; sleep 1

  one.y two.y three.y four.y five.y six.y: ; @echo touch $@; sleep 1

This simulates a build such as you describe, where lots of little
programs are built each from one object file.  If I run this without -j,
I get the serial behavior you'd expect:

    $ make
    touch one.y
    cp one.y one.x
    touch two.y
    cp two.y two.x
    touch three.y
    cp three.y three.x
    touch four.y
    cp four.y four.x
    touch five.y
    cp five.y five.x
    touch six.y
    cp six.y six.x

And, if I run it with -j6 I get the parallel behavior you'd expect:

    $ make -j6
    touch one.y
    touch two.y
    touch four.y
    touch five.y
    touch six.y
    touch three.y
    cp one.y one.x
    cp two.y two.x
    cp four.y four.x
    cp five.y five.x
    cp six.y six.x
    cp three.y three.x

Soooo... I'm missing something :-).

--
-------------------------------------------------------------------------------
 Paul D. Smith <[hidden email]>          Find some GNU make tips at:
 http://www.gnu.org                      http://make.paulandlesley.org
 "Please remain calm...I may be mad, but I am a professional." --Mad Scientist


_______________________________________________
Help-make mailing list
[hidden email]
http://lists.gnu.org/mailman/listinfo/help-make