Make and Clusters

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Make and Clusters

David Delahaye
Hello,

I wonder how the make command is compatible with clusters. I know the
"-j" option, which allows us to run several jobs simultaneously and uses
the several cores of a given processor. But if we have several nodes of
computation of a cluster, is the command "make -j" able to run over
these nodes (transparently for the user)? I know there are specific
commands to run jobs on a cluster, I would like to know how "make -j" is
compatible with these commands. Thanks in advance for your help.

Best regards,

David Delahaye.

--

David Delahaye

Professor
Université de Montpellier
LIRMM (UMR 5506)
[hidden email]
Phone: (+33) (0)4 67 41 86 01
http://www.lirmm.fr/~delahaye/ <http://www.lirmm.fr/%7Edelahaye/>


Université de Montpellier • LIRMM UMR 5506 • CC477
161 rue Ada • 34095 Montpellier Cedex 5 • France

_______________________________________________
Help-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-make
Reply | Threaded
Open this post in threaded view
|

Re: Make and Clusters

Alberto Luaces
David Delahaye writes:

> Hello,
>
> I wonder how the make command is compatible with clusters. I know the
> "-j" option, which allows us to run several jobs simultaneously and
> uses the several cores of a given processor. But if we have several
> nodes of computation of a cluster, is the command "make -j" able to
> run over these nodes (transparently for the user)? I know there are
> specific commands to run jobs on a cluster, I would like to know how
> "make -j" is compatible with these commands. Thanks in advance for
> your help.
>

Hi, David,

For those building problems among several machines, you usually use
distcc+make (http://distcc.org).

--
Alberto


_______________________________________________
Help-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-make
Reply | Threaded
Open this post in threaded view
|

Re: Make and Clusters

Paul Smith-20
In reply to this post by David Delahaye
On Tue, 2017-10-10 at 15:53 +0200, David Delahaye wrote:
> I wonder how the make command is compatible with clusters. I know the
> "-j" option, which allows us to run several jobs simultaneously and
> uses the several cores of a given processor. But if we have several
> nodes of computation of a cluster, is the command "make -j" able to
> run over these nodes (transparently for the user)? I know there are
> specific commands to run jobs on a cluster, I would like to know how
> "make -j" is compatible with these commands.

GNU make doesn't have any built-in support for building on clusters. 
To do so it would have to accept all sorts of other arguments which
defined what machines were in the cluster, what authentication methods
are needed to access the cluster, etc. for all different sorts of
cluster management software, including plain ssh.  Clearly (from the
GNU make manual :)) it doesn't provide any of that.

However, if you have cluster management software you can just change
the commands invoked by make to run those commands in a cluster
instead.  There are many ways to do it: for one, you can write a
wrapper script around your compiler and set the make variable to run
that wrapper instead.  Your wrapper script will send the compilation
job to the cluster, and stay alive until the job has been completed so
it can tell make it's done and what the result was (success/failure).

You can also use other facilities such as distcc http://distcc.org/ to
distribute compilation: these are often designed to work well with GNU
make's -j option.

_______________________________________________
Help-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-make
Reply | Threaded
Open this post in threaded view
|

RE: Make and Clusters

Cook, Malcolm-2
If your cluster is Open Grid Engine, another option is qmake - http://gridscheduler.sourceforge.net/htmlman/htmlman1/qmake.html - which

" is based on gmake (GNU make), version  3.78.1.
" All Sun Grid Engine options valid with  qsub(1)  or  qrsh(1) can  be  specified with qmake
"The syntax of qmake makefiles corresponds to  gmake  and  is described in the "GNU Make Manual".

Alas, to my knowledge this project has not kept up with developments in GNU make.  

~[hidden email]

 > -----Original Message-----
 > From: Help-make [mailto:help-make-bounces+mec=[hidden email]]
 > On Behalf Of Paul Smith
 > Sent: Tuesday, October 10, 2017 9:28 AM
 > To: [hidden email]
 > Subject: Re: Make and Clusters
 >
 > On Tue, 2017-10-10 at 15:53 +0200, David Delahaye wrote:
 > > I wonder how the make command is compatible with clusters. I know the
 > > "-j" option, which allows us to run several jobs simultaneously and
 > > uses the several cores of a given processor. But if we have several
 > > nodes of computation of a cluster, is the command "make -j" able to
 > > run over these nodes (transparently for the user)? I know there are
 > > specific commands to run jobs on a cluster, I would like to know how
 > > "make -j" is compatible with these commands.
 >
 > GNU make doesn't have any built-in support for building on clusters.
 > To do so it would have to accept all sorts of other arguments which
 > defined what machines were in the cluster, what authentication methods
 > are needed to access the cluster, etc. for all different sorts of
 > cluster management software, including plain ssh.  Clearly (from the
 > GNU make manual :)) it doesn't provide any of that.
 >
 > However, if you have cluster management software you can just change
 > the commands invoked by make to run those commands in a cluster
 > instead.  There are many ways to do it: for one, you can write a
 > wrapper script around your compiler and set the make variable to run
 > that wrapper instead.  Your wrapper script will send the compilation
 > job to the cluster, and stay alive until the job has been completed so
 > it can tell make it's done and what the result was (success/failure).
 >
 > You can also use other facilities such as distcc http://distcc.org/ to
 > distribute compilation: these are often designed to work well with GNU
 > make's -j option.
 >
 > _______________________________________________
 > Help-make mailing list
 > [hidden email]
 > https://lists.gnu.org/mailman/listinfo/help-make
_______________________________________________
Help-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-make
Reply | Threaded
Open this post in threaded view
|

Re: Make and Clusters

Martin d'Anjou-5
In reply to this post by David Delahaye
Hi David,

I explored this problem a while ago. I captured my musings in a blog:
https://ninjaverification.wordpress.com/2008/10/21/overriding-gnu-make-shell-variable-for-massive-parallel-make/

In the end the idea is to have make call out to a job scheduler as part
of the target recipe. This is the method I use and it works really well.

One problem I still have though is that if one of the parallel jobs
aborts and compromises the whole build, make will wait for all other
in-flight jobs to finish before itself returning. At the time I had
manually patched my local copy of GNU Make, but the patch no longer
applies and I have not gotten back to it. Here is a reference in case
you find it useful:
http://savannah.gnu.org/bugs/?41781

Best,
Martin

On 17-10-10 09:53 AM, David Delahaye wrote:

> Hello,
>
> I wonder how the make command is compatible with clusters. I know the
> "-j" option, which allows us to run several jobs simultaneously and
> uses the several cores of a given processor. But if we have several
> nodes of computation of a cluster, is the command "make -j" able to
> run over these nodes (transparently for the user)? I know there are
> specific commands to run jobs on a cluster, I would like to know how
> "make -j" is compatible with these commands. Thanks in advance for
> your help.
>
> Best regards,
>
> David Delahaye.
>


_______________________________________________
Help-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-make
Reply | Threaded
Open this post in threaded view
|

Re: Make and Clusters

Tim Murphy-4
Hi,

Firstly I must warn you that I am an interested party since I do consulting
related to this product but I did use it purely as a customer for several
years.

Electric Accelerator is pretty much the pièce de résistance for running GNU
Make and Ninja makefiles on clusters and there is a free evaluation version.

It solves several of your problems:
1)  Making sure the files you need (including tools and source) are
available to every machine in your cluster.  You never have to manage this
or debug whether all your machines have been kept uptodate or provide NFS
mounts everywhere or manage NIS authentication or any of that stuff.
2) It spots problems in your makefiles where tasks are run in the wrong
order - this usually happens because there isn't enough dependency
information in the makefiles.
3) There is a tool called Electric Insight which lets you visualise the
build across your cluster and the analysis tools and reporting on this are
incredible for helping to fix performance or correctness problems.

http://electric-cloud.com/products/electricaccelerator/

Scroll down to the "Use it for free" link.   There are a few limitations to
the free version but they don't generally matter too much if you're not
supporting 100s of users and if you find it very useful you can now get
team licenses priced so that you don't have to be a huge company to afford
it which is a welcome new change.

Regards,

Tim




On 11 October 2017 at 03:15, Martin d'Anjou <[hidden email]>
wrote:

> Hi David,
>
> I explored this problem a while ago. I captured my musings in a blog:
> https://ninjaverification.wordpress.com/2008/10/21/
> overriding-gnu-make-shell-variable-for-massive-parallel-make/
>
> In the end the idea is to have make call out to a job scheduler as part
> of the target recipe. This is the method I use and it works really well.
>
> One problem I still have though is that if one of the parallel jobs
> aborts and compromises the whole build, make will wait for all other
> in-flight jobs to finish before itself returning. At the time I had
> manually patched my local copy of GNU Make, but the patch no longer
> applies and I have not gotten back to it. Here is a reference in case
> you find it useful:
> http://savannah.gnu.org/bugs/?41781
>
> Best,
> Martin
>
> On 17-10-10 09:53 AM, David Delahaye wrote:
> > Hello,
> >
> > I wonder how the make command is compatible with clusters. I know the
> > "-j" option, which allows us to run several jobs simultaneously and
> > uses the several cores of a given processor. But if we have several
> > nodes of computation of a cluster, is the command "make -j" able to
> > run over these nodes (transparently for the user)? I know there are
> > specific commands to run jobs on a cluster, I would like to know how
> > "make -j" is compatible with these commands. Thanks in advance for
> > your help.
> >
> > Best regards,
> >
> > David Delahaye.
> >
>
>
> _______________________________________________
> Help-make mailing list
> [hidden email]
> https://lists.gnu.org/mailman/listinfo/help-make
>
_______________________________________________
Help-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-make