gmake and ccache conspiring together in creating gremlins

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

gmake and ccache conspiring together in creating gremlins

Sam Varshavchik
I've been chasing down non-deterministic, random build fails. make
randomly blows up with

write error: stdout

A lot of stracing, experimentation, and pains, determined that this is
due to a sequence of the following events:

1) how make uses the job server file descriptor

2) an intentional leak of the standard error file descriptor in ccache

3) LTO linking results in linker recursively invoking make (that was a
new one to me)

4) A rather nuanced behavior of the Linux PTY driver that can be best
explained with the following short demo:

#include <sys/types.h>
#include <fcntl.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/stat.h>

int main()
{
        struct stat state_buf;

        for (int i=0; i<8; ++i)
        {
                if (fstat(i, &state_buf) == 0)
                {
                        printf("%d: %d/%d\n", i,
                               (int)state_buf.st_dev,
                               (int)state_buf.st_ino);
                }
        }

        printf("NON BLOCKING: %s\n", fcntl(0, F_GETFL) & O_NONBLOCK ? "y":"n");

        fcntl(2, F_SETFL, O_NONBLOCK);

        printf("NON BLOCKING: %s\n", fcntl(0, F_GETFL) & O_NONBLOCK ? "y":"n");
        return 0;
}

I'm sshed into my server, and the output of the above is, for me:

0: 25/3
1: 25/3
2: 25/3
NON BLOCKING: n
NON BLOCKING: y

My stdin, stdout, and stderr is the same dev/ino, and setting non-blocking
mode one any one of them puts them all in non-blocking mode. I'm setting non-
blocking on standard error, and my standard input also becomes non-blocking.

After sifting through sources, and straces, this is what I see is happening.

Starting with make, and how it sets up its job file descriptors:

jobserver_parse_auth (const char *auth)
{
  /* Given the command-line parameter, parse it.  */
  if (sscanf (auth, "%d,%d", &job_fds[0], &job_fds[1]) != 2)

  // …

  /* Make sure our pipeline is valid, and (possibly) create a duplicate pipe,
     that will be closed in the SIGCHLD handler.  If this fails with EBADF,
     the parent has closed the pipe on us because it didn't think we were a
     submake.  If so, warn and default to -j1.  */

  if (!FD_OK (job_fds[0]) || !FD_OK (job_fds[1]) || make_job_rfd () < 0)
}

The TLDR of the above: make reads the job server's file descriptors
from the MAKEFLAGS environment variable, then checks here if they
actually exist. If they don't exist, make will create the job server
pipe. Important: by default they will be file descriptors 3 and 4.
This becomes a key player in this mystery, a little bit later.

When make spawns a child job (other than a recursive make) the job
server file descriptors get closed (marked O_CLOEXEC before the actual
execve), but MAKEFLAGS remains in the environment. They remain open
for a recursive make call, and the recursive instance picks up the
ball from here.

The problem with this approach is that if, somehow, a child process
runs make, and there happens to be a pair of file descriptors here,
make will assume that they're the job server file descriptors.

Another thing that make does is that it sets one of the file
descriptors to non-blocking mode. This is in jobserver_setup:

  /* When using pselect() we want the read to be non-blocking.  */
  set_blocking (job_fds[0], 0);

Now we get to ccache which, amongst things, does the following, in its
ccache.cpp:

// Make a copy of stderr that will not be cached, so things like distcc can
// send networking errors to it.
static void
set_up_uncached_err()
{
  int uncached_fd =
    dup(STDERR_FILENO); // The file descriptor is intentionally leaked.
  if (uncached_fd == -1) {
    log("dup(2) failed: {}", strerror(errno));
    throw Failure(Statistic::internal_error);
  }

  Util::setenv("UNCACHED_ERR_FD", fmt::format("{}", uncached_fd));
}

TLDR: it intentionally leaks a file descriptor. A basic standalone
strace of a standalone cache invocation showed ccache leaking file
descriptor 2 to 3. Now, somewhere else I was also getting file
descriptor 4, I haven't tracked down where, exactly. But it was there,

The final ingredient to this perfect storm, is LTO linking. I have
verified that -flto=auto -ffat-lto-objects results in ld running make.
If you compile helloworld, or something, with -flto=auto
-ffat-lto-objects, and link it:

strace -s 256 -o z -f gcc -flto=auto -ffat-lto-objects -o t t.o

strace will show that, somehow, make gets involved:

123302 execve("/usr/bin/make", ["make", "-f", "/tmp/ccyRcNmv.mk", "-j32",
"all"], 0x234ed40 /* 47 vars */ <unfinished …>

And now, let's go back to what make is doing, here:

static void
close_stdout (void)
{
  int prev_fail = ferror (stdout);
  int fclose_fail = fclose (stdout);

  if (prev_fail || fclose_fail)
    {
      if (fclose_fail)
        perror_with_name (_("write error: stdout"), "");
      else
        O (error, NILF, _("write error: stdout"));
      exit (MAKE_TROUBLE);
    }
}

So, the grand total, this is what was happening when I was building a
large hairball, using make, distcc, cache, and when using LTO:

1) The original instance of make came up, and set up a job server on
file descriptors 3 and 4, and put them into MAKEFLAGS.

2) This was a large build, in recursive directories, with recursive
make invocations.

3) A link command was executed, invoking ccache. Before exec-ing it,
make set CLOEXEC on both file descriptors 3 and 4, and ccache did not
inherit them.

4) ccache duped file descriptor 2 to 3, and invoked distcc, which then
invoked the real gcc to finally do the LTO link. At some point along
the line something got leaked to file descriptor 4. That's the only
piece of the puzzle I've yet to track down, but it didn't seem like it
was very important to figure outwhere it came from.

5) gcc invoked the linker, which invoking make again for LTO.

6) make reads MAKEFLAGS, and sees that it should have job server file
descriptors. make sees something on file descriptors 3 and 4, thinks its the
original job server file descriptors (spoiler: they aren't).

7) this clusterfark wasn't fatal enough to blow up this link. It succeeded,
but:

 set_blocking (job_fds[0], 0);

put the fake job file descriptor in non-blocking mode. This file
descriptor was really the original terminal standard error, courtesy
of ccache. Now, both standard input, output, and error, are in
non-blocking mode.

8) My original build was -j 40. They rolled merrily along, spewing
tons of output, a good chunk of it was from make itself. One of make's
write()s to standard output was EAGAIN-ed. This resulted in the
original "write error: stdout" that prompted me to burn almost a month
chasing this down.

So, now what? Both ccache, and the linker are contributing to this
situation. ccache leaks the file descriptor (to distcc apparently?).
The linker invokes make for its own use, but it doesn't sanitize the
environment. make closes the job server file descriptors, but not
removing them from MAKEFLAGS.

In the meantime, -Orecursive seems to be a temporary bandaid. I think
it works around this frankenbug.

Reply | Threaded
Open this post in threaded view
|

Re: gmake and ccache conspiring together in creating gremlins

Edward Welbourne-3
Hi Sam,

Thanks for a delightfully illuminating analysis.
I hope you enjoyed the sleuthing, even if it did cost you a month !

> The TLDR of the above: make reads the job server's file descriptors
> from the MAKEFLAGS environment variable, then checks here if they
> actually exist. If they don't exist, make will create the job server
> pipe. Important: by default they will be file descriptors 3 and 4.
> This becomes a key player in this mystery, a little bit later.
>
> When make spawns a child job (other than a recursive make) the job
> server file descriptors get closed (marked O_CLOEXEC before the actual
> execve), but MAKEFLAGS remains in the environment.

Sounds to me like that's a bug: when the descriptors are closed, the
part of MAKEFLAGS that claims they're make's jobserver file descriptors
should be removed, since that's when the claim stops being true.

        Eddy.

Reply | Threaded
Open this post in threaded view
|

Re: gmake and ccache conspiring together in creating gremlins

Gnu - Make - Bugs mailing list
On Mon, Feb 8, 2021 at 12:36 PM Edward Welbourne <[hidden email]> wrote:
> Sounds to me like that's a bug: when the descriptors are closed, the
> part of MAKEFLAGS that claims they're make's jobserver file descriptors
> should be removed, since that's when the claim stops being true.

make uses posix_spawn by default to create children.
posix_spawn makes it difficult to modify env per child.
As a workaround the user can have the recipe remove (or modify) MAKEFLAGS
E.g.

%.o: %.c ; unset MAKEFLAGS && $(CC)  $(CFLAGS) -o $@ -c $<

regards, Dmitry

Reply | Threaded
Open this post in threaded view
|

Re: gmake and ccache conspiring together in creating gremlins

Gnu - Make - Bugs mailing list
On Mon, Feb 8, 2021 at 12:51 PM Dmitry Goncharov
<[hidden email]> wrote:

>
> On Mon, Feb 8, 2021 at 12:36 PM Edward Welbourne <[hidden email]> wrote:
> > Sounds to me like that's a bug: when the descriptors are closed, the
> > part of MAKEFLAGS that claims they're make's jobserver file descriptors
> > should be removed, since that's when the claim stops being true.
>
> make uses posix_spawn by default to create children.
> posix_spawn makes it difficult to modify env per child.
> As a workaround the user can have the recipe remove (or modify) MAKEFLAGS
> E.g.
>
> %.o: %.c ; unset MAKEFLAGS && $(CC)  $(CFLAGS) -o $@ -c $<

Oops. Forgot that posix_spawn takes an envp parameter.
Yes, worth fixing.

regards, Dmitry

Reply | Threaded
Open this post in threaded view
|

Re: gmake and ccache conspiring together in creating gremlins

Paul Smith-20
In reply to this post by Edward Welbourne-3
On Mon, 2021-02-08 at 10:43 +0000, Edward Welbourne wrote:
> Sounds to me like that's a bug: when the descriptors are closed, the
> part of MAKEFLAGS that claims they're make's jobserver file
> descriptors should be removed, since that's when the claim stops
> being true.

I believe there have been other similar issues reported recently.

Certainly fixing MAKEFLAGS when we run without jobserver available is
something that could be done.

There is a loss of debugging information if we make this change: today
make can detect if it was invoked in a way that _should_ expect to
receive a jobserver context, but _didn't_ receive that context.  That
is, if make sees that jobserver-auth is set but it can't open the
jobserver pipes it can warn the user that most likely there's a problem
in their environment or with the setup of their makefiles.

Without this warning there's no way to know when this situation occurs.
 It's easy to create a situation where every sub-make will create its
own completely unique jobserver domain.  So you start the top make with
-j4 and run 4 sub-makes; if you do it wrong then each of 4 sub-makes
could create a new jobserver domain, and now you're running 16 jobs in
parallel instead of 4... there's no way for make to warn you about this
situation.


Another option that I'm considering is moving away from anonymous pipes
and switching to either named pipes or named semaphores instead (I'm
not sure if one or the other is preferred WRT portability).  If I did
that then all of this hullabaloo around open/closed file descriptors,
inherited FDs opened blocking vs. non-blocking, and "passing through"
jobserver access across non-make boundaries would go away.

I liked the original implementation for these reasons:
 * It is very generic: pretty much every system supports simple pipes.
    However, in the end only POSIX systems are using anonymous pipes
   anyway (Windows jobserver already uses Windows named semaphores).
 * It is very easy to manage: using named pipes means that make is
   creating context on the filesystem that it needs to manage and clean
   up, which it otherwise never does; we need to worry about
   permissions, etc.  Anonymous pipes just go away magically.
 * It is very safe: it's not possible for any other process to access
   the pipe and mess up the jobserver count, unless it was invoked by
   make in a "sub-make context".

But, it is difficult to use in some subtle ways as we've seen.

A change would also mean that the format of the --jobserver-auth flag
would change: if the value provided were the current 2 numbers then the
old-school anonymous pipe process would be used.  If it were a path,
then we'd assume it was a named pipe (or named semaphore).

Other tools like LTO etc. that look for jobserver-auth would,
hopefully, be able to manage this.  I tried to be clear about the
accepted formats and behaviors in the GNU make documentation; hopefully
developers are handling incorrect formats properly.


Reply | Threaded
Open this post in threaded view
|

Re: gmake and ccache conspiring together in creating gremlins

Sam Varshavchik
On Mon, Feb 8, 2021 at 2:38 PM Paul Smith <[hidden email]> wrote:

>
> On Mon, 2021-02-08 at 10:43 +0000, Edward Welbourne wrote:
> > Sounds to me like that's a bug: when the descriptors are closed, the
> > part of MAKEFLAGS that claims they're make's jobserver file
> > descriptors should be removed, since that's when the claim stops
> > being true.
>
> I believe there have been other similar issues reported recently.
>
> Certainly fixing MAKEFLAGS when we run without jobserver available is
> something that could be done.
>
> There is a loss of debugging information if we make this change: today
> make can detect if it was invoked in a way that _should_ expect to
> receive a jobserver context, but _didn't_ receive that context.  That
> is, if make sees that jobserver-auth is set but it can't open the
> jobserver pipes it can warn the user that most likely there's a problem
> in their environment or with the setup of their makefiles.
>
> Without this warning there's no way to know when this situation occurs.
>  It's easy to create a situation where every sub-make will create its
> own completely unique jobserver domain.  So you start the top make with
> -j4 and run 4 sub-makes; if you do it wrong then each of 4 sub-makes
> could create a new jobserver domain, and now you're running 16 jobs in
> parallel instead of 4... there's no way for make to warn you about this
> situation.

One thought occurred to me. Specifically: when make executes what it
believes to be something other than a recursive invocation of $(MAKE),
and it closes the job server pipe file descriptors for that, it can
also:

1) Add an additional parameter to MAKEFLAGS, let's call it
"--no-jobserver", and perhaps remove the --jobserver-auth parameter
completely. It might be easier just to append something there, instead
of surgically removing this.

2) Make checks for a --no-jobserver in MAKEFLAGS when it starts. If
it's there it does NOT attempt to validate the file descriptors that
are given in --jobserver-auth (if this parameter is preserved). It's a
given that they're not there:

  if (!FD_OK (job_fds[0]) || !FD_OK (job_fds[1]) || make_job_rfd () < 0)

Don't even do that. What happens right now a warning message gets
printed and make runs without a job server. This change should have
the same result, print the warning but skip the FD_OK tests.

This will result in the same warning, but it should avoid triggering
the bug that I found.

However that might cause a minor regression in LTO linking. I think
that this prevents the LTO linker's internal invocation of make from
finding that it can attach to the original make process's job server.

From sifting through strace dumps, I see that a linker-invoked make
gets its own -j flag. It appears that the linker is courteous enough
to count how many CPUs it has and use it to construct its own -j flag.

How about this, safe approach: once --no-jobserver is there it stays
there, and gets propagated to all recursively invoked makes. If an
invoke make finds that it has both a --no-jobserver and a -j flag,
it'll warn and refuse to create its own job server, and then proceed
executing one command at a time.

This prevents an arithmetic proliferation of job worker processes if
the original job server's file descriptors get lost. Currently
recursively-invoked makes will find, and attach themselves to, an
existing job server. This is nice; but this is vulnerable to an edge
case that I think I'm hitting: a false positive involving a leaked
file descriptor. This change encourages fixing whatever's causing make
to fail to detect a recursive invocation.

Reply | Threaded
Open this post in threaded view
|

Re: gmake and ccache conspiring together in creating gremlins

Edward Welbourne-3
In reply to this post by Paul Smith-20
Paul Smith (8 February 2021 20:38) wrote:
> There is a loss of debugging information if we make this change: today
> make can detect if it was invoked in a way that _should_ expect to
> receive a jobserver context, but _didn't_ receive that context.  That
> is, if make sees that jobserver-auth is set but it can't open the
> jobserver pipes it can warn the user that most likely there's a
> problem in their environment or with the setup of their makefiles.

Rather than removing the jobserver-auth data, you could amend the
MAKEFLAGS to includ jobserver-auth data with plainly invalid fds,
e.g. -1, as the two fds, to make clear that we're in a context where
jobserver-auth could beneficially have been propagated but wasn't.  The
main thing is just to *not* claim that file descriptors you've closed
are available to access as the jobserver.  That doesn't preclude leaving
it evident to [grand-)*children that this has happened,

        Eddy.

Reply | Threaded
Open this post in threaded view
|

Re: gmake and ccache conspiring together in creating gremlins

Gnu - Make - Bugs mailing list
On Tue, Feb 9, 2021 at 5:31 AM Edward Welbourne <[hidden email]> wrote:

> Rather than removing the jobserver-auth data, you could amend the
> MAKEFLAGS to includ jobserver-auth data with plainly invalid fds,

i like jobserver-auth data with plainly invalid fds, because it lets
older binaries fail on parsing jobserver-auth
and print some (hopefully useful) error message.
E.g. make itself would print
"internal error: invalid --jobserver-auth string"


> e.g. -1, as the two fds

The program could read this -1 to (for example) uint16_t and interpret
as 65535 and fail to notice the parent is not giving any fd.
i'd rather provide no fd at all. E.g.
--jobserver-auth=

regards, Dmitry


>
>         Eddy.
>

Reply | Threaded
Open this post in threaded view
|

Re: gmake and ccache conspiring together in creating gremlins

David Boyce-3
> The program could read this -1 to (for example) uint16_t and interpret as 65535 and fail to notice the parent is not giving any fd.

File descriptors have been of int type since Unix was designed (at least) and -1 is documented as the invalid descriptor. E.g. the open() system call and every other system call returning an fd returns -1 on error and always has. The idea that any program would read something explicitly documented as a file descriptor into an unsigned type seems effectively inconceivable.

David

On Tue, Feb 9, 2021 at 7:36 AM Dmitry Goncharov via Bug reports and discussion for GNU make <[hidden email]> wrote:
On Tue, Feb 9, 2021 at 5:31 AM Edward Welbourne <[hidden email]> wrote:

> Rather than removing the jobserver-auth data, you could amend the
> MAKEFLAGS to includ jobserver-auth data with plainly invalid fds,

i like jobserver-auth data with plainly invalid fds, because it lets
older binaries fail on parsing jobserver-auth
and print some (hopefully useful) error message.
E.g. make itself would print
"internal error: invalid --jobserver-auth string"


> e.g. -1, as the two fds

The program could read this -1 to (for example) uint16_t and interpret
as 65535 and fail to notice the parent is not giving any fd.
i'd rather provide no fd at all. E.g.
--jobserver-auth=

regards, Dmitry


>
>         Eddy.
>