Memory leaks in Make

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Memory leaks in Make

Jeffrey Walton-3
Hi Everyone,

My apologies if I raised this issue already. I did not see it in the
archives at https://lists.gnu.org/archive/html/bug-make/2019-04/index.html
or https://lists.gnu.org/archive/html/bug-make/2019-05/index.html .

I'm trying to test libidn with Asan, which is used for
internationalized domain names. IDN also uses Gnulib and Gnulib is
resisting testing. I can build everything with -fsanitize=address, but
Gnulib test runners are doing something such that libasan.so is not
being loaded properly. It results in all tests runners failing.

The work-around is to LD_PRELOAD libasan.so. Unfortunately for me,
that means I have to LD_PRELOAD on the invocation of make. So,
running:

    LD_PRELOAD=/lib64/libasan.so.5 make check

Results in:

=================================================================
==19205==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 56 byte(s) in 1 object(s) allocated from:
    #0 0x7feb1d6e1c08 in __interceptor_malloc (/lib64/libasan.so.5+0xefc08)
    #1 0x55f2ff6d2778 in xmalloc (/usr/bin/make+0x1d778)

Indirect leak of 33 byte(s) in 1 object(s) allocated from:
    #0 0x7feb1d62de60 in strdup (/lib64/libasan.so.5+0x3be60)
    #1 0x55f2ff6d29ec in xstrdup (/usr/bin/make+0x1d9ec)

SUMMARY: AddressSanitizer: 89 byte(s) leaked in 2 allocation(s).

It looks like Make's memory leaks are killing the build process long
before the test runners execute.

It would be nice if Make cleaned up its resources before exiting. The
leaks are affecting other components, and they have halted Asan
testing for this unit. LD_PRELOAD was my last chance to test this
library, and I don't seem to have any other alternatives.

I'm working on Fedora 29 with Make 4.2.1, but I have a feeling
platform and versions do not matter.

Jeff

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: Memory leaks in Make

Martin Dorey-2
> It would be nice if Make cleaned up its resources before exiting.

Would it?  Linking all those blocks back on to free lists and coalescing adjacent ones doesn't take a long time but it takes more time than just tearing down the address space.  It can be a reasonable economy for short-lived processes not to clean up before exiting.  Recursive make might be considered harmful but it's still a common use-case with such short-lived processes.

> It looks like Make's memory leaks are killing the build process long before the test runners execute.

I presume you have some compelling evidence for that and I appreciate not being deluged with large attachments... but the trouble with what you have shared is that 89 bytes leaked in total seems like it wouldn't be a problem for even the smallest system.


From: Bug-make <bug-make-bounces+martin.dorey=[hidden email]> on behalf of Jeffrey Walton <[hidden email]>
Sent: Tuesday, May 14, 2019 19:19
To: [hidden email]
Subject: Memory leaks in Make
 
***** EXTERNAL EMAIL *****

Hi Everyone,

My apologies if I raised this issue already. I did not see it in the
archives at https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.gnu.org%2Farchive%2Fhtml%2Fbug-make%2F2019-04%2Findex.html&amp;data=01%7C01%7CMartin.Dorey%40hitachivantara.com%7C5aed86365df54e159bc208d6d8dbb855%7C18791e1761594f52a8d4de814ca8284a%7C0&amp;sdata=Bg4sCZ%2FBcfzAgwvnxbWrE2Qt82DogNnIiitUo4A3FF8%3D&amp;reserved=0
or https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.gnu.org%2Farchive%2Fhtml%2Fbug-make%2F2019-05%2Findex.html&amp;data=01%7C01%7CMartin.Dorey%40hitachivantara.com%7C5aed86365df54e159bc208d6d8dbb855%7C18791e1761594f52a8d4de814ca8284a%7C0&amp;sdata=sn%2FvutWMzNn4KQRviRbEfRXFIw%2FycXmekgPjrzKcG8A%3D&amp;reserved=0 .

I'm trying to test libidn with Asan, which is used for
internationalized domain names. IDN also uses Gnulib and Gnulib is
resisting testing. I can build everything with -fsanitize=address, but
Gnulib test runners are doing something such that libasan.so is not
being loaded properly. It results in all tests runners failing.

The work-around is to LD_PRELOAD libasan.so. Unfortunately for me,
that means I have to LD_PRELOAD on the invocation of make. So,
running:

LD_PRELOAD=/lib64/libasan.so.5 make check

Results in:

=================================================================
==19205==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 56 byte(s) in 1 object(s) allocated from:
#0 0x7feb1d6e1c08 in __interceptor_malloc (/lib64/libasan.so.5+0xefc08)
#1 0x55f2ff6d2778 in xmalloc (/usr/bin/make+0x1d778)

Indirect leak of 33 byte(s) in 1 object(s) allocated from:
#0 0x7feb1d62de60 in strdup (/lib64/libasan.so.5+0x3be60)
#1 0x55f2ff6d29ec in xstrdup (/usr/bin/make+0x1d9ec)

SUMMARY: AddressSanitizer: 89 byte(s) leaked in 2 allocation(s).

It looks like Make's memory leaks are killing the build process long
before the test runners execute.

It would be nice if Make cleaned up its resources before exiting. The
leaks are affecting other components, and they have halted Asan
testing for this unit. LD_PRELOAD was my last chance to test this
library, and I don't seem to have any other alternatives.

I'm working on Fedora 29 with Make 4.2.1, but I have a feeling
platform and versions do not matter.

Jeff

_______________________________________________
Bug-make mailing list
[hidden email]
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.gnu.org%2Fmailman%2Flistinfo%2Fbug-make&amp;data=01%7C01%7CMartin.Dorey%40hitachivantara.com%7C5aed86365df54e159bc208d6d8dbb855%7C18791e1761594f52a8d4de814ca8284a%7C0&amp;sdata=NOkh6fIatCa01kfLzYjb7YpwBnYkhtXErJtxPNyGBn4%3D&amp;reserved=0

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: Memory leaks in Make

Jeffrey Walton-3
On Tue, May 14, 2019 at 10:40 PM Martin Dorey
<[hidden email]> wrote:
>
> > It would be nice if Make cleaned up its resources before exiting.
>
> Would it?  Linking all those blocks back on to free lists and coalescing adjacent ones doesn't take a long time but it takes more time than just tearing down the address space.  It can be a reasonable economy for short-lived processes not to clean up before exiting.  Recursive make might be considered harmful but it's still a common use-case with such short-lived processes.

Interesting you bring up economic models for this. I've given that
quite a bit of thought, and even researched the numbers.

  * 25 million programmers in the world
  * 6 - 12 % are C and C++, depending on the source
  * 3 million QA folks in the world

That means there's a potential pool of (25M+3M) * 0.09 = 2.5 million
folks. Suppose 1/1000 perform acceptance testing, security testing and
evaluation, etc. That means about 3000 folks are experiencing this
problem.

I spent 5 days trying to test IDN and IDN2. It was not just the
LD_PRELOAD hail mary. It included a lot of other troubleshooting and
tinkering trying to get IDN and IDN2 to test.

Assume it takes the Make project 3 days to make the necessary changes
to avoid the leak. 3 days is 24 man-hours, and that beats the work I
put in by 16 hours. Assume the other 3000 folks try and give up after
a day. The world has spent 24000 man-hours on a problem that could
have been cleared in 3 days.

Which makes more sense to you? The Make project spending 3 days to fix
the problem, or the world servicing your technological debt at the
cost of 24,000 man hours or 11 man-years?

> > It looks like Make's memory leaks are killing the build process long before the test runners execute.
>
> I presume you have some compelling evidence for that and I appreciate not being deluged with large attachments...

Yes, I have the build results that starts with 'make check', and end with:

    gmake: *** [Makefile:1495: check-recursive] Error 1
    Failed to test IDN

Not one test was run because the tools failed.

> but the trouble with what you have shared is that 89 bytes leaked in total seems like it wouldn't be a problem for even the smallest system.

It taints results of other components. It does not matter if it is a 1
byte leak.

Jeff

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make