Building some targets serially, others in parallel

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Building some targets serially, others in parallel

dan soucy
Hi list.

I find myself in a situation where I want the makefile to know not to build
certain targets in parallel, even though they do not actually depend on each
other.

To explain:

Suppose we have three executables to build: `server`, `client`, `demo`. Each
of these has an associated source file.

There are many other source files -- maybe a dozen. They will be compiled to
object files which are depended on by all three executable files.

Normally, the quick solution is to add one target for each object file and
tell `make` to invoke the compiler on each in a separate process, using as
many processors as are available.

But -- in this case, the design of the compiler is such that telling the
compiler to build all required object files in parallel is much faster than
invoking the compiler many times. This is due to a (relatively) slow start-up
and also the re-use of data between source files (such as types).

So instead, we have something like this:

```
server client demo: $(OBJECTS)
     $(COMPILER) -j 4 $@ -o $<
```

Running `make -j 4 server client demo` with the object files not built will
result in the compiler being invoked thrice, and each will begin by compiling
the same object files. The result is that this actually takes longer than
running `make` serially.

However, if the object files are already built, then `server`, `client`, and
`demo` can be built faster in parallel.

For the curious, the compiler is GHC and the code is Haskell.

Reply | Threaded
Open this post in threaded view
|

Re: Building some targets serially, others in parallel

Kaz Kylheku (gmake)
On 2020-08-22 13:46, dan soucy wrote:
takes longer than running `make` serially.
>
> However, if the object files are already built, then `server`,
> `client`, and
> `demo` can be built faster in parallel.

I don't see the big problem here other than the need to represent the
shared object files as if they are a single artifact.

The server, client and demo targets have to depend on some sort of file
entity
with a time stamp which, if it is up to date, indicates that the objects
are ready to go.

Note that this would happen naturally if we were, say, putting C .o
files
into a .a archive, and then linking the archive:

    libcommon.a: x.o y.o z.o

    server: server.o libcommon.a

    client: client.o libcommon.a

If there is no archive, we can make a fake timestamp file that serves in
its place.

    .PHONY: all
    all: server client

    timestamp: $(SOURCES)
        compiler $? # invoke compiler on all sources newer than timestamp
        touch $@    # create or refresh stamp

    server: server.src timestamp   # server.src is the server source file
        compiler $< -o $(OBJECTS)

    client: client.src timestamp
        compiler $< -o $(OBJECTS)

server and client are done in parallel. They both depend on timestamp,
which is updated once.

If the timestamp doesn't exist, then $? expands to all the $(SOURCES).

Otherwise, it expands to just that subset of them which are newer, so
you get incremental rebuilds. E.g. if $(SOURCES) contains 17 files but
we touched just two, then those two are fed to the compiler to make
the corresponding object files.

Then the server and client rules are triggered.

Don't forget to have your "clean" target remove the timestamp,
and give that file a better name.

Reply | Threaded
Open this post in threaded view
|

Re: Building some targets serially, others in parallel

pacalet
In reply to this post by dan soucy
On 22/08/2020 22:46, dan soucy wrote:

> Hi list.
>
> I find myself in a situation where I want the makefile to know not to build
> certain targets in parallel, even though they do not actually depend on
> each
> other.
> To explain:
> Suppose we have three executables to build: `server`, `client`, `demo`.
> Each
> of these has an associated source file.
>
> There are many other source files -- maybe a dozen. They will be
> compiled to
> object files which are depended on by all three executable files.
>
> Normally, the quick solution is to add one target for each object file
> and tell `make` to invoke the compiler on each in a separate process,
> using as many processors as are available.
>
> But -- in this case, the design of the compiler is such that telling the
> compiler to build all required object files in parallel is much faster than
> invoking the compiler many times. This is due to a (relatively) slow
> start-up and also the re-use of data between source files (such as types).
>
> So instead, we have something like this:
>
> ```
> server client demo: $(OBJECTS)
>     $(COMPILER) -j 4 $@ -o $< ```
>
> Running `make -j 4 server client demo` with the object files not built
> will result in the compiler being invoked thrice, and each will begin by
> compiling the same object files.

I do not understand why the compiler would "begin by compiling the same
object files". Normally, one of the main make features is to avoid doing
useless things. If the same source file gets compiled several times
there must be a problem with your Makefile. For instance, instead of
having rules where the object file itself is the target, you have rules
where the target is something else.

Moreover, your recipe looks a bit strange: the way you use automatic
variables suggests that you are compiling the target ($@) and output the
first prerequisite ($<). Are you sure your recipe is correct?

> The result is that this actually takes
> longer than running `make` serially.
>
> However, if the object files are already built, then `server`, `client`,
> and
> `demo` can be built faster in parallel.
>
> For the curious, the compiler is GHC and the code is Haskell.

GNU make 4.3 introduces the grouped targets (`&:` separator). So, if you
want to build all your object files with one single execution of a
recipe, and if you use GNU make 4.3, you could try:

$(OBJECTS) &: $(SOURCES)
    $(COMPILER) -j4 $^ ...

This tells make that all object files are built by a single execution of
the recipe. One last improvement would be to build OBJECTS (and SOURCES)
such that they contain only outdated object files.
--
Renaud Pacalet
Télécom Paris
Campus SophiaTech
450 Route des Chappes, CS 50193
06904 Biot Sophia Antipolis cedex, FRANCE
Tel : +33 (0) 4 9300 8402
Web : http://www.telecom-paris.fr/