[bug #51309] Determination of a file list from a single folder without changing the working directory

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[bug #51309] Determination of a file list from a single folder without changing the working directory

Robert Morell
URL:
  <http://savannah.gnu.org/bugs/?51309>

                 Summary: Determination of a file list from a single folder
without changing the working directory
                 Project: make
            Submitted by: elfring
            Submitted on: Sun 25 Jun 2017 07:53:27 PM CEST
                Severity: 3 - Normal
              Item Group: Enhancement
                  Status: None
                 Privacy: Public
             Assigned to: None
             Open/Closed: Open
         Discussion Lock: Any
       Component Version: 4.2.1
        Operating System: Any
           Fixed Release: None
           Triage Status: None

    _______________________________________________________

Details:

The function “wildcard”
<https://www.gnu.org/software/make/manual/html_node/File-Name-Functions.html#index-wildcard-2>
is supported by the make software.
Corresponding file patterns work also in combination with directory names.
Such combinations are occasionally useful. The query result will contain them
as usual.

I came along the need to query a single folder without changing the working
directory
<https://github.com/elfring/Coccinelle-20160205/blob/replace_suffix_rules_by_pattern_rules1/modules_without_mli.make#L7>
so that only file names (without folder names) will be in the result.
Can another software extension become useful for the function library to make
such data processing more efficient?




    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?51309>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [bug #51309] Determination of a file list from a single folder without changing the working directory

Henrik Carlqvist
> I came along the need to query a single folder without changing the
> working directory so that only file names (without folder names) will be
> in the result.

The easiest way to accomplish such a thing is probably to use the notdir
function:

MY_FILES=$(notdir $(wildcard somedir/*.ext))

See also the documentation of notdir at
https://www.gnu.org/software/make/manual/html_node/File-Name-Functions.html

regards Henrik

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [bug #51309] Determination of a file list from a single folder without changing the working directory

SF Markus Elfring
> MY_FILES=$(notdir $(wildcard somedir/*.ext))

I agree that this algorithm can work. But it does not fit to my imaginations
for efficient data processing.

If the desired directory on which a file filter should be applied
could be selected before, the file names would not contain references
to this directory and another function call is therefore unnecessary
to remove it from the returned list elements.

The difference can eventually better seen by an approach like the following.

~/Projekte> ls Bau/*txt
Bau/MOTD.txt
~/Projekte> cd Bau && ls *txt
MOTD.txt


I would like to achieve the second list result without using
the command “cd” before.

Regards,
Markus

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[bug #51309] Determination of a file list from a single folder without changing the working directory

Robert Morell
In reply to this post by Robert Morell
Update of bug #51309 (project make):

                  Status:                    None => Wont Fix              
             Open/Closed:                    Open => Closed                

    _______________________________________________________

Follow-up Comment #1:

The current capabilities (using notdir to remove the entire directory, or
using patsubst etc. to remove partial directories) is sufficient.  Nothing new
is needed here.

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?51309>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[bug #51309] Determination of a file list from a single folder without changing the working directory

Robert Morell
Follow-up Comment #2, bug #51309 (project make):

I disagree with the run time characteristics of the mentioned functionality
for my special use case.


~/Projekte> $(cd Bau && echo "echo $(ls *txt)")
MOTD.txt



Can a similar data processing operation be performed faster by advanced make
functions?

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?51309>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[bug #51309] Determination of a file list from a single folder without changing the working directory

Robert Morell
Follow-up Comment #3, bug #51309 (project make):

There's no question that


FOO := $(notdir $(wildcard Bau/*.txt))


is more efficient than:


~/Projekte> $(cd Bau && echo "echo $(ls *txt)")


because the latter involves starting a whole new shell (two, technically)
while the former is handled entirely within the make process with no fork/exec
operation required.

If you can show actual results where adding the notdir makes a significant
performance difference in a makefile environment we can reconsider, but I'm
not interested in adding new features duplicating existing capabilities
without proof that there's a real need.

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?51309>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[bug #51309] Determination of a file list from a single folder without changing the working directory

Robert Morell
Follow-up Comment #4, bug #51309 (project make):

I repeat my argument from 26. June:
https://lists.gnu.org/archive/html/bug-make/2017-06/msg00198.html

If the desired directory on which a file filter should be applied could be
selected before, the file names would not contain references to this directory
and another function call is therefore unnecessary to remove it from the
returned list elements.

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?51309>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[bug #51309] Determination of a file list from a single folder without changing the working directory

Robert Morell
Follow-up Comment #5, bug #51309 (project make):

I understand what you are suggesting.

But I don't see any need to add new capabilities for this when the existing
features of GNU make can already give these results.

The only reasons for it that I can see would be (a) you prefer to do it this
way rather than the currently supported method of post-processing the wildcard
results, or (b) there's a significant performance benefit to avoiding the
extra processing.  For (a), I'm not inclined to create new capabilities just
for this reason.  For (b), I seriously doubt that there will be any measurable
performance difference in a real makefile.

If you can show an example where there _is_ a significant performance
difference then we can discuss that.

To be clear, my attitudes on makefile syntax are that (1) it's already complex
and there are already a lot of features, complexities, and "gotchas", (2)
adding new features in a backward- and POSIX-compatible way is difficult due
to the relatively free-form syntax of makefiles, and (3) features combine in
ways that are multiplicative, not additive, which greatly increases the
potential for incorrect interactions, bugs, and the testing needed to support
them.

Thus, my initial answer to adding new capabilities is almost always going to
be "no".  It requires a clear improvement in power (not just syntactic sugar)
and a reasonable syntax that makes sense and doesn't break lots of existing
makefiles to clear that hurdle and get to "maybe" or "yes".

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?51309>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[bug #51309] Determination of a file list from a single folder without changing the working directory

Robert Morell
Follow-up Comment #6, bug #51309 (project make):

A theoretical argument is not what Paul meant by "actual results".  Have you
measured the time spent adding and removing the directory names in some
realistic case?  If not, then please do not advocate for making Make bigger..

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?51309>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[bug #51309] Determination of a file list from a single folder without changing the working directory

Robert Morell
Follow-up Comment #7, bug #51309 (project make):

> (a) you prefer to do it this way rather than the currently supported method
of post-processing the wildcard results, …

Yes.


>(b) there's a significant performance benefit to avoiding the extra
processing.

I find that the difference can be eventually measured.

We have got different opinions about the circumstances when it would be
significant enough.


> … adding new features in a backward- and POSIX-compatible way is difficult
due …

How many software developers would dare to extend corresponding programming
interfaces so that they can be better reused by the make function library?

I am curious if this story could ever turn into a more promising “maybe”
thing.

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?51309>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[bug #51309] Determination of a file list from a single folder without changing the working directory

Robert Morell
Follow-up Comment #8, bug #51309 (project make):

If, for example, you can show an example of notdir adding significantly to the
amount of time the make program takes.

To test this I created a makefile like this:


foo = ...

all: ; : $(words $(foo))


where the "..." represents 6000 paths hardcoded into the makefile.  I then ran
"time make":


$ time make
: 6000

real    0m0.004s
user    0m0.000s
sys     0m0.000s


Then I modified the makefile to add a notdir like this:


foo = $(notdir ...)

all: ; : $(words $(foo))


where the "..." is the same as last time.  I got this result:


$ time make
: 6000

real    0m0.004s
user    0m0.000s
sys     0m0.000s


Same time.  This means that the amount of time needed to run notdir on 6000
paths is less than 1/1000th of a second.  From this I conclude that the
performance impact of an additional notdir function is negligible and there's
no performance advantage to avoiding it.

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?51309>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[bug #51309] Determination of a file list from a single folder without changing the working directory

Robert Morell
Follow-up Comment #9, bug #51309 (project make):

I suggested also a comparison of two algorithms
<https://en.wikipedia.org/wiki/Computational_complexity_theory>.

* A simple time measurement can also show test results for which you might not
be looking for.
* The usage of data structures like list and trees has got also different run
time characteristics. Each of them can have a corresponding application
domain.
* Which test environments would you find representative?


> From this I conclude that the performance impact of an additional notdir
function is negligible and there's no performance advantage to avoiding it.

It might matter more under other circumstances. (The current software can be
fast enough to some degree.)

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?51309>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [bug #51309] Determination of a file list from a single folder without changing the working directory

Henrik Carlqvist-2
On Sun,  2 Jul 2017 16:14:31 -0400 (EDT)
Markus Elfring <[hidden email]> wrote:
> It might matter more under other circumstances.

Are you able to provide any example proving how it matters under some more
or less rare circumstances?

> (The current software can be fast enough to some degree.)

Or do you find the current software good enough and no changes needed?

regards Henrik

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [bug #51309] Determination of a file list from a single folder without changing the working directory

SF Markus Elfring
> Are you able to provide any example proving how it matters under some more
> or less rare circumstances?

I guess not at the moment.


> Or do you find the current software good enough

I dared to point some change possibilities out recently.


> and no changes needed?

It depends on your views for software developments which your prefer.

I imagine that algorithm comparisons can lead to related improvements,
can't they?

Regards,
Markus

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[bug #51309] Determination of a file list from a single folder without changing the working directory

Robert Morell
In reply to this post by Robert Morell
Follow-up Comment #10, bug #51309 (project make):

Would you like to compare run time characteristics a bit more with an approach
similar to “$(shell find ~/Projekte/Bau/OCamlbuild -name '*.txt' -printf "%f
")”?

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?51309>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Loading...