Pattern rules with % matching empty string?

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Pattern rules with % matching empty string?

Glen Huang-2
Hi,

I’m trying to do something like this:

remote-bin-host%: bin
        scp $< host$*:$<
        touch $@

to update files in remote hosts.

It works for remote-bin-host1, remote-bin-host2, etc, but not remote-bin-host. It seems % won’t match empty string.

I tried mixing static and pattern rule like this to make it work:

remote-bin-host remote-bin-host%: bin
        scp $< host$*:$<
        touch $@

And make complains such mixture is deprecated.

Is there a clean way to write it without splitting the rule and duplicating most of the receipt?

Thanks in advance, and hope everyone is staying strong in this crisis.

BTW, is updating remote files like this a bad idea? My use case is that bin is a web service, and once built, should be run on a remote dev machine to be tested.
Reply | Threaded
Open this post in threaded view
|

Re: Pattern rules with % matching empty string?

Philip Guenther-2
On Thu, Apr 23, 2020 at 10:55 PM Glen Huang <[hidden email]> wrote:

> I’m trying to do something like this:
>
> remote-bin-host%: bin
>         scp $< host$*:$<
>         touch $@
>
> to update files in remote hosts.
>
> It works for remote-bin-host1, remote-bin-host2, etc, but not
> remote-bin-host. It seems % won’t match empty string.
>

This is the documented behavior of pattern rules.  To quote the info pages:

10.5 Defining and Redefining Pattern Rules
==========================================

You define an implicit rule by writing a "pattern rule".  A pattern
rule looks like an ordinary rule, except that its target contains the
character `%' (exactly one of them).  The target is considered a
pattern for matching file names; the `%' can match any nonempty
substring, while other characters match only themselves.


Note: "nonempty"


I tried mixing static and pattern rule like this to make it work:
>
> remote-bin-host remote-bin-host%: bin
>         scp $< host$*:$<
>         touch $@
>
> And make complains such mixture is deprecated.
>

Right, because pattern rules mean with multiple targets have a magic
meaning, making what they mean when mixed with non-pattern targets
ambiguous.


Is there a clean way to write it without splitting the rule and duplicating
> most of the receipt?
>

Put the commands for the rule in a variable and have two rules, one pattern
and one not, that both use the variable as their entire recipe.


Philip Guenther
Reply | Threaded
Open this post in threaded view
|

Re: Pattern rules with % matching empty string?

Paul Smith-20
In reply to this post by Glen Huang-2
On Fri, 2020-04-24 at 12:15 +0800, Glen Huang wrote:
> remote-bin-host%: bin
>         scp $< host$*:$<
>         touch $@
>
> to update files in remote hosts.
>
> It works for remote-bin-host1, remote-bin-host2, etc, but not remote-
> bin-host. It seems % won’t match empty string.

Another option is to match one extra character:

remote-bin-hos%: bin
        ...

Of course this also matches targets like remote-bin-hosX etc.  If you
can't live with that then your only option is to declare both rules and
either duplicate the recipe or put the recipe in a variable and use
that in both rules, as Philip mentioned.


Reply | Threaded
Open this post in threaded view
|

Re: Pattern rules with % matching empty string?

Kaz Kylheku (gmake)
On 2020-04-24 14:39, Paul Smith wrote:

> On Fri, 2020-04-24 at 12:15 +0800, Glen Huang wrote:
>> remote-bin-host%: bin
>>         scp $< host$*:$<
>>         touch $@
>>
>> to update files in remote hosts.
>>
>> It works for remote-bin-host1, remote-bin-host2, etc, but not remote-
>> bin-host. It seems % won’t match empty string.
>
> Another option is to match one extra character:
>
> remote-bin-hos%: bin
>         ...

There is a third option: choose your host naming scheme so
that all the names have a suffix.

And a fourth option: keep the host names inconsistent
(they are probably hard to change) but enumerate them
with consistently named stamp files (easy to control).

Use computed variable names to associate the two together:

   remote-bin-host%: bin
      scp $< $(host_$*):$<
      touch $@

Note the nesting $(host_$*).  The stem such as 1, 2, 3
is first interpolated into this to produce $(host_1),
$(host_2) and so on. These are variables, which get expanded
again. Make allows computed variable names without eval
having to be used. This effectively gives us an associative
data structure in which variables are keys. We somewhere
populate that:

   host_1 := alpha.localdomain
   host_2 := beta.othersite
   ...

Thus if the stamp file remote-bin-host1 is out of date
with respect to bin, then bin is scp-ed out to
alpha.localdomain:bin. Then the stamp file is touched.

Similarly for remote-bin-host2, bin is copied out to
beta.othersite.

Idea: instead of just touching the stamp file, we can write
the host name into it:

       echo $(host_$*) > $@

Then if someone wants a quick reminder what host is in
the stamp file, the answer is inside.


Reply | Threaded
Open this post in threaded view
|

Re: Pattern rules with % matching empty string?

Kaz Kylheku (gmake)
On 2020-04-24 21:56, Kaz Kylheku (gmake) wrote:
> And a fourth option: keep the host names inconsistent
> (they are probably hard to change) but enumerate them
> with consistently named stamp files (easy to control).
>
> Use computed variable names to associate the two together:
>
>   remote-bin-host%: bin
>      scp $< $(host_$*):$<
>      touch $@


With computed variable names, you can fake out a data
structure in which it looks like keys are matched to records
that have fields.

Then you can specify a database of hosts records, such
that for each one you can give a stamp file name, host
name and any other property, like using a different SSH
key or whatever.

Please study this Makefile:

   host_1.name := foo
   host_1.stamp := foo.txt

   host_2.name := bar
   host_2.stamp := bar.txt

   host_list := 1 2

   define host_bin_template =
   $$(host_$1.stamp): bin
    echo scp $$< $$(host_$1.name):$$<
    touch $$@
   endef

   $(foreach n,$(host_list),$(eval $(call host_bin_template,$n)))

If we run "make foo.txt", this happens:

   echo scp bin foo:bin
   scp bin foo:bin
   touch foo.txt

Make has deduced through the structure we set up and the generated rules
that foo.txt is a stamp file associated with host foo.

This doesn't use pattern rules at all, but procedural rule expansion. We
have a list of hosts 1 2.
The foreach loop steps over this list, and substitutes it as parameter
$1 into the host_bin_template, which is evaluated via $(eval ...) as a
piece of Makefile syntax. That syntax creates a rule, exactly as if we
had typed this in:

   $(host_1.stamp): bin
    echo scp $< $(host_1.name):$<
    touch $@

   $(host_2.stamp): bin
    echo scp $< $(host_2.name):$<
    touch $@

Of course, these variables expand, and so the effect of the for loop and
eval is that this is generated:

   foo.txt: bin
    echo scp $< foo:$<
    touch $@

   bar.txt: bin
    echo scp $< bar:$<
    touch $@


To add, a third rule, we just create new variables:

   host_3.name := saturn.localdomain
   host_3.stamp := saturn.stamp

Update the list:

   host_list = 1 2 3


Done. Now we have a third rule where saturn.stamp is a target,
and updating it copies the bin file to saturn.localdomain.

With a little more hacking, we can probably (and ironically)
re-create the "rdist" utility. :)

The advantage of pattern rules is that they avoid instantiation bloat.
A pattern rule can potentially match just, say, one of potentially
thousands of targets, which is more efficient than generating thousands
of rules with an eval. And it can match new things in the filesystem
that don't require any new definitions in the Makefile.

We lost some of the advantage of the pattern rule when we turned
to computed variables for assistance; manual definitions had to be made
to satisfy the computed variable references.



Reply | Threaded
Open this post in threaded view
|

Re: Pattern rules with % matching empty string?

Glen Huang
Thanks for the detailed examples.

Using variables to instantiate all hosts sounds overkill. I just want the pattern rule to also apply when the stem would be empty, and all but one host has a number suffix in their names.

I end up making all hosts contain the number suffix. Given the solutions provided, it feels like the cleanest one.

> On Apr 25, 2020, at 1:29 PM, Kaz Kylheku (gmake) <[hidden email]> wrote:
>
> On 2020-04-24 21:56, Kaz Kylheku (gmake) wrote:
>> And a fourth option: keep the host names inconsistent
>> (they are probably hard to change) but enumerate them
>> with consistently named stamp files (easy to control).
>> Use computed variable names to associate the two together:
>>  remote-bin-host%: bin
>>     scp $< $(host_$*):$<
>>     touch $@
>
>
> With computed variable names, you can fake out a data
> structure in which it looks like keys are matched to records
> that have fields.
>
> Then you can specify a database of hosts records, such
> that for each one you can give a stamp file name, host
> name and any other property, like using a different SSH
> key or whatever.
>
> Please study this Makefile:
>
>  host_1.name := foo
>  host_1.stamp := foo.txt
>
>  host_2.name := bar
>  host_2.stamp := bar.txt
>
>  host_list := 1 2
>
>  define host_bin_template =
>  $$(host_$1.stamp): bin
>   echo scp $$< $$(host_$1.name):$$<
>   touch $$@
>  endef
>
>  $(foreach n,$(host_list),$(eval $(call host_bin_template,$n)))
>
> If we run "make foo.txt", this happens:
>
>  echo scp bin foo:bin
>  scp bin foo:bin
>  touch foo.txt
>
> Make has deduced through the structure we set up and the generated rules that foo.txt is a stamp file associated with host foo.
>
> This doesn't use pattern rules at all, but procedural rule expansion. We have a list of hosts 1 2.
> The foreach loop steps over this list, and substitutes it as parameter $1 into the host_bin_template, which is evaluated via $(eval ...) as a piece of Makefile syntax. That syntax creates a rule, exactly as if we had typed this in:
>
>  $(host_1.stamp): bin
>   echo scp $< $(host_1.name):$<
>   touch $@
>
>  $(host_2.stamp): bin
>   echo scp $< $(host_2.name):$<
>   touch $@
>
> Of course, these variables expand, and so the effect of the for loop and eval is that this is generated:
>
>  foo.txt: bin
>   echo scp $< foo:$<
>   touch $@
>
>  bar.txt: bin
>   echo scp $< bar:$<
>   touch $@
>
>
> To add, a third rule, we just create new variables:
>
>  host_3.name := saturn.localdomain
>  host_3.stamp := saturn.stamp
>
> Update the list:
>
>  host_list = 1 2 3
>
>
> Done. Now we have a third rule where saturn.stamp is a target,
> and updating it copies the bin file to saturn.localdomain.
>
> With a little more hacking, we can probably (and ironically)
> re-create the "rdist" utility. :)
>
> The advantage of pattern rules is that they avoid instantiation bloat.
> A pattern rule can potentially match just, say, one of potentially
> thousands of targets, which is more efficient than generating thousands
> of rules with an eval. And it can match new things in the filesystem
> that don't require any new definitions in the Makefile.
>
> We lost some of the advantage of the pattern rule when we turned
> to computed variables for assistance; manual definitions had to be made
> to satisfy the computed variable references.
>
>