ENOEXEC from exec*() functions...?

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

ENOEXEC from exec*() functions...?

Paul Smith-20
While looking into cleanups around fork/exec I ran into some confusing
code in GNU make.  This code dates back to the first version checked
into source control by Roland in 1992 (so who knows how far it really
goes back).

When an exec() fails, GNU make looks at the errno code and if the code
is ENOEXEC it retries the exec but this time giving the command to the
shell to run.  So for example if the command was:

  ./foo bar baz

it will attempt to rerun the exec() with:

  /bin/sh ./foo bar baz

I can't find a way to exercise this code path.

If the command being invoked doesn't have the executable bit set (e.g.,
I use "touch ./foo" with the above) then exec() fails with errno set to
EPERM not ENOEXEC, and if I make the script executable but without a #!
line at the top then exec() runs it in a shell without returning
ENOEXEC.

The GNU/Linux man page doesn't appear to allow this (a script that
doesn't start with #!) or at least doesn't document it as valid, but it
does work.  It lists ENOEXEC errno code as meaning:

  ENOEXEC
         An  executable  is  not in a recognized format, is for the wrong
         architecture, or has some other format error that means it  can‐
         not be executed.

Which doesn't sound like something that would be helped by re-running
as a shell script.  Maybe this is a feature of GNU/Linux and other
systems use ENOEXEC when there's no #! line?

Maybe some folks out there using less common systems know the answer to
that.

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: ENOEXEC from exec*() functions...?

Andreas Schwab
On Jul 30 2018, Paul Smith <[hidden email]> wrote:

> Which doesn't sound like something that would be helped by re-running
> as a shell script.  Maybe this is a feature of GNU/Linux and other
> systems use ENOEXEC when there's no #! line?

http://pubs.opengroup.org/onlinepubs/9699919799/functions/execve.html

    There are two distinct ways in which the contents of the process
    image file may cause the execution to fail, distinguished by the
    setting of errno to either [ENOEXEC] or [EINVAL] (see the ERRORS
    section). In the cases where the other members of the exec family of
    functions would fail and set errno to [ENOEXEC], the execlp() and
    execvp() functions shall execute a command interpreter and the
    environment of the executed command shall be as if the process
    invoked the sh utility using execl() as follows:

    execl(<shell path>, arg0, file, arg1, ..., (char *)0);

Andreas.

--
Andreas Schwab, SUSE Labs, [hidden email]
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: ENOEXEC from exec*() functions...?

Paul Smith-20
On Mon, 2018-07-30 at 13:16 +0200, Andreas Schwab wrote:

> On Jul 30 2018, Paul Smith <[hidden email]> wrote:
> > Which doesn't sound like something that would be helped by re-
> > running as a shell script.  Maybe this is a feature of GNU/Linux
> > and other systems use ENOEXEC when there's no #! line?
>
> http://pubs.opengroup.org/onlinepubs/9699919799/functions/execve.html
>
>     There are two distinct ways in which the contents of the process
>     image file may cause the execution to fail, distinguished by the
>     setting of errno to either [ENOEXEC] or [EINVAL] (see the ERRORS
>     section). In the cases where the other members of the exec family of
>     functions would fail and set errno to [ENOEXEC], the execlp() and
>     execvp() functions shall execute a command interpreter and the
>     environment of the executed command shall be as if the process
>     invoked the sh utility using execl() as follows:
>
>     execl(<shell path>, arg0, file, arg1, ..., (char *)0);

Aha.  GNU make is using execvp() so that explains that.  Maybe the
original code was using a different form of exec().

So it sounds like this code in GNU make is redundant, assuming a POSIX-
compliant implementation of execvp().

I wasn't able to find any similar text in the posix_spawn() document
although posix_spawnp() does appear to behave the same way as execvp()
on GNU/Linux (not surprising since I believe posix_spawn() is
implemented in terms of fork/exec there).

Thanks Andreas!

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: ENOEXEC from exec*() functions...?

Eli Zaretskii
In reply to this post by Paul Smith-20
> From: Paul Smith <[hidden email]>
> Date: Mon, 30 Jul 2018 06:47:38 -0400
>
> I can't find a way to exercise this code path.
>
> If the command being invoked doesn't have the executable bit set (e.g.,
> I use "touch ./foo" with the above) then exec() fails with errno set to
> EPERM not ENOEXEC, and if I make the script executable but without a #!
> line at the top then exec() runs it in a shell without returning
> ENOEXEC.
>
> The GNU/Linux man page doesn't appear to allow this (a script that
> doesn't start with #!) or at least doesn't document it as valid, but it
> does work.  It lists ENOEXEC errno code as meaning:
>
>   ENOEXEC
>          An  executable  is  not in a recognized format, is for the wrong
>          architecture, or has some other format error that means it  can‐
>          not be executed.
>
> Which doesn't sound like something that would be helped by re-running
> as a shell script.  Maybe this is a feature of GNU/Linux and other
> systems use ENOEXEC when there's no #! line?

But in GNU Make, SHELL can be set to anything, including a command
that runs some executables which the Unix kernel and the Unix shell
don't recognize.  Maybe that code tries to cater to this situation?
AFAIU, such a situation will not be resolved by execvp's fallback to
the shell, because I presume execvp will call the standard shell,
right?

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: ENOEXEC from exec*() functions...?

Paul Smith-20
On Mon, 2018-07-30 at 17:29 +0300, Eli Zaretskii wrote:

> > Which doesn't sound like something that would be helped by re-
> > running
> > as a shell script.  Maybe this is a feature of GNU/Linux and other
> > systems use ENOEXEC when there's no #! line?
>
> But in GNU Make, SHELL can be set to anything, including a command
> that runs some executables which the Unix kernel and the Unix shell
> don't recognize.  Maybe that code tries to cater to this situation?
> AFAIU, such a situation will not be resolved by execvp's fallback to
> the shell, because I presume execvp will call the standard shell,
> right?

Well, this code won't help with that.

It will run "/bin/sh foo bar" and the execvp() call will succeed and
the process will be replaced by the shell.  If "foo" is not a shell
script then the shell will still try to run it and fail with some sort
of syntax error or something.  That will be a very different error than
execvp() returning ENOEXEC.

The only way you'd get ENOEXEC here is if, I suppose, execvp() couldn't
find a shell at all.  Even then you probably just get ENOENT (I didn't
hide /bin/sh on my system to test this :)) which is what you'd get for
any other non-existent program.

As far as I can tell the only way execvp() can return ENOEXEC is if you
try to run a 64bit binary on a 32bit system, or a Windows binary on a
GNU/Linux system, or something like that: something where the kernel
can't even load the program.

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: ENOEXEC from exec*() functions...?

Martin Dorey-2
Sorry to reanimate this but I think I've run into a regression in 4.2.92 over 4.2.1 that's probably related to this old email thread.  Bug or email, bug or email... email:

martind@swiftboat:~/playpen/make-2019-10-08$ cat > Makefile
all: ; ./dodgy
martind@swiftboat:~/playpen/make-2019-10-08$ cat > dodgy
true
martind@swiftboat:~/playpen/make-2019-10-08$ chmod +x dodgy
martind@swiftboat:~/playpen/make-2019-10-08$ make
./dodgy
make: *** [Makefile:1: all] Error 127
martind@swiftboat:~/playpen/make-2019-10-08$ /usr/bin/make
./dodgy
martind@swiftboat:~/playpen/make-2019-10-08$ strace -f make 2>&1 | grep execve
execve("/usr/local/bin/make", ["make"], [/* 209 vars */]) = 0
[pid 21223] execve("./dodgy", ["./dodgy"], [/* 212 vars */]) = -1 ENOEXEC (Exec format error)
martind@swiftboat:~/playpen/make-2019-10-08$ strace -f /usr/bin/make 2>&1 | grep execve
execve("/usr/bin/make", ["/usr/bin/make"], [/* 209 vars */]) = 0
[pid 21247] execve("./dodgy", ["./dodgy"], [/* 212 vars */]) = -1 ENOEXEC (Exec format error)
[pid 21247] execve("/bin/sh", ["/bin/sh", "./dodgy"], [/* 212 vars */]) = 0
martind@swiftboat:~/playpen/make-2019-10-08$

"make" is 4.2.92 (today's git), where /usr/bin/make is actually 4.0, but 4.2.1 behaves the same.


From: Bug-make <bug-make-bounces+martin.dorey=[hidden email]> on behalf of Paul Smith <[hidden email]>
Sent: Monday, July 30, 2018 07:56
To: Eli Zaretskii <[hidden email]>
Cc: [hidden email] <[hidden email]>
Subject: Re: ENOEXEC from exec*() functions...?
 
On Mon, 2018-07-30 at 17:29 +0300, Eli Zaretskii wrote:
> > Which doesn't sound like something that would be helped by re-
> > running
> > as a shell script.  Maybe this is a feature of GNU/Linux and other
> > systems use ENOEXEC when there's no #! line?
>
> But in GNU Make, SHELL can be set to anything, including a command
> that runs some executables which the Unix kernel and the Unix shell
> don't recognize.  Maybe that code tries to cater to this situation?
> AFAIU, such a situation will not be resolved by execvp's fallback to
> the shell, because I presume execvp will call the standard shell,
> right?

Well, this code won't help with that.

It will run "/bin/sh foo bar" and the execvp() call will succeed and
the process will be replaced by the shell.  If "foo" is not a shell
script then the shell will still try to run it and fail with some sort
of syntax error or something.  That will be a very different error than
execvp() returning ENOEXEC.

The only way you'd get ENOEXEC here is if, I suppose, execvp() couldn't
find a shell at all.  Even then you probably just get ENOENT (I didn't
hide /bin/sh on my system to test this :)) which is what you'd get for
any other non-existent program.

As far as I can tell the only way execvp() can return ENOEXEC is if you
try to run a 64bit binary on a 32bit system, or a Windows binary on a
GNU/Linux system, or something like that: something where the kernel
can't even load the program.

_______________________________________________
Bug-make mailing list
[hidden email]
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.gnu.org%2Fmailman%2Flistinfo%2Fbug-make&amp;data=01%7C01%7CMartin.Dorey%40hitachivantara.com%7C07a28247064f4b95485508d5f62cae39%7C18791e1761594f52a8d4de814ca8284a%7C0&amp;sdata=%2Bb5jGO7LiqcJFE34LjHbW7LPiSn1nTjdyV8gyf6BAfw%3D&amp;reserved=0

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: ENOEXEC from exec*() functions...?

Martin Dorey-2
Rereading the rest of the thread, I see I should have included:

martind@swiftboat:~$ uname -a
Linux swiftboat 3.16.0-9-amd64 #1 SMP Debian 3.16.68-2 (2019-06-17) x86_64 GNU/Linux
martind@swiftboat:~$ dpkg --status libc6:amd64
...
Version: 2.19-18+deb8u10

That's Debian Jessie.  ltrace shows it's using posix_spawn for me.  The test case works when running the same make-4.2.92 binary on a newer Linux release, eg Debian Stretch:

martind@scoot:~/playpen/make-2019-10-08$ uname -a
Linux scoot 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1+deb9u2 (2019-05-13) x86_64 GNU/Linux
martind@scoot:~/playpen/make-2019-10-08$ dpkg --status libc6:amd64
...
Version: 2.24-11+deb9u4


From: Martin Dorey <[hidden email]>
Sent: Tuesday, October 8, 2019 12:00
To: Eli Zaretskii <[hidden email]>; [hidden email] <[hidden email]>
Cc: [hidden email] <[hidden email]>
Subject: Re: ENOEXEC from exec*() functions...?
 
Sorry to reanimate this but I think I've run into a regression in 4.2.92 over 4.2.1 that's probably related to this old email thread.  Bug or email, bug or email... email:

martind@swiftboat:~/playpen/make-2019-10-08$ cat > Makefile
all: ; ./dodgy
martind@swiftboat:~/playpen/make-2019-10-08$ cat > dodgy
true
martind@swiftboat:~/playpen/make-2019-10-08$ chmod +x dodgy
martind@swiftboat:~/playpen/make-2019-10-08$ make
./dodgy
make: *** [Makefile:1: all] Error 127
martind@swiftboat:~/playpen/make-2019-10-08$ /usr/bin/make
./dodgy
martind@swiftboat:~/playpen/make-2019-10-08$ strace -f make 2>&1 | grep execve
execve("/usr/local/bin/make", ["make"], [/* 209 vars */]) = 0
[pid 21223] execve("./dodgy", ["./dodgy"], [/* 212 vars */]) = -1 ENOEXEC (Exec format error)
martind@swiftboat:~/playpen/make-2019-10-08$ strace -f /usr/bin/make 2>&1 | grep execve
execve("/usr/bin/make", ["/usr/bin/make"], [/* 209 vars */]) = 0
[pid 21247] execve("./dodgy", ["./dodgy"], [/* 212 vars */]) = -1 ENOEXEC (Exec format error)
[pid 21247] execve("/bin/sh", ["/bin/sh", "./dodgy"], [/* 212 vars */]) = 0
martind@swiftboat:~/playpen/make-2019-10-08$

"make" is 4.2.92 (today's git), where /usr/bin/make is actually 4.0, but 4.2.1 behaves the same.


From: Bug-make <bug-make-bounces+martin.dorey=[hidden email]> on behalf of Paul Smith <[hidden email]>
Sent: Monday, July 30, 2018 07:56
To: Eli Zaretskii <[hidden email]>
Cc: [hidden email] <[hidden email]>
Subject: Re: ENOEXEC from exec*() functions...?
 
On Mon, 2018-07-30 at 17:29 +0300, Eli Zaretskii wrote:
> > Which doesn't sound like something that would be helped by re-
> > running
> > as a shell script.  Maybe this is a feature of GNU/Linux and other
> > systems use ENOEXEC when there's no #! line?
>
> But in GNU Make, SHELL can be set to anything, including a command
> that runs some executables which the Unix kernel and the Unix shell
> don't recognize.  Maybe that code tries to cater to this situation?
> AFAIU, such a situation will not be resolved by execvp's fallback to
> the shell, because I presume execvp will call the standard shell,
> right?

Well, this code won't help with that.

It will run "/bin/sh foo bar" and the execvp() call will succeed and
the process will be replaced by the shell.  If "foo" is not a shell
script then the shell will still try to run it and fail with some sort
of syntax error or something.  That will be a very different error than
execvp() returning ENOEXEC.

The only way you'd get ENOEXEC here is if, I suppose, execvp() couldn't
find a shell at all.  Even then you probably just get ENOENT (I didn't
hide /bin/sh on my system to test this :)) which is what you'd get for
any other non-existent program.

As far as I can tell the only way execvp() can return ENOEXEC is if you
try to run a 64bit binary on a 32bit system, or a Windows binary on a
GNU/Linux system, or something like that: something where the kernel
can't even load the program.

_______________________________________________
Bug-make mailing list
[hidden email]
https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.gnu.org%2Fmailman%2Flistinfo%2Fbug-make&amp;data=01%7C01%7CMartin.Dorey%40hitachivantara.com%7C07a28247064f4b95485508d5f62cae39%7C18791e1761594f52a8d4de814ca8284a%7C0&amp;sdata=%2Bb5jGO7LiqcJFE34LjHbW7LPiSn1nTjdyV8gyf6BAfw%3D&amp;reserved=0

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: ENOEXEC from exec*() functions...?

Paul Smith-20
In reply to this post by Martin Dorey-2
On Tue, 2019-10-08 at 19:00 +0000, Martin Dorey wrote:

> Sorry to reanimate this but I think I've run into a regression in 4.2.92
> over 4.2.1 that's probably related to this old email thread.  Bug or
> email, bug or email... email:
>
> martind@swiftboat:~/playpen/make-2019-10-08$ cat > Makefile
> all: ; ./dodgy
> martind@swiftboat:~/playpen/make-2019-10-08$ cat > dodgy
> true
> martind@swiftboat:~/playpen/make-2019-10-08$ chmod +x dodgy
> martind@swiftboat:~/playpen/make-2019-10-08$ make ./dodgy
> make: *** [Makefile:1: all] Error 127
>
> "make" is 4.2.92 (today's git), where /usr/bin/make is actually 4.0, but
> 4.2.1 behaves the same.

Some changes were added for this to allow the posix_spawn detection to try
to determine whether it works properly; please try make 4.2.93 and see if
you get the right behavior.

It worked for me on GNU/Linux and MacOS.


Reply | Threaded
Open this post in threaded view
|

Re: ENOEXEC from exec*() functions...?

Martin Dorey-2
please try make 4.2.93 and see if you get the right behavior.

Works for me.

Off topic for this thread and I don't want to stand in the way of progress towards pervasive use of a better C, but it was another notch more difficult to build, back in the Debian Jessie / gcc-4.9 era, thanks to some new for loop initial declarations in file.c and rule.c, which weren't legal in -std=gnu90, the compiler's default dialect.


From: Paul Smith <[hidden email]>
Sent: Friday, January 3, 2020 09:16
To: Martin Dorey <[hidden email]>; Eli Zaretskii <[hidden email]>
Cc: [hidden email] <[hidden email]>
Subject: Re: ENOEXEC from exec*() functions...?
 
***** EXTERNAL EMAIL *****

On Tue, 2019-10-08 at 19:00 +0000, Martin Dorey wrote:
> Sorry to reanimate this but I think I've run into a regression in 4.2.92
> over 4.2.1 that's probably related to this old email thread.  Bug or
> email, bug or email... email:
>
> martind@swiftboat:~/playpen/make-2019-10-08$ cat > Makefile
> all: ; ./dodgy
> martind@swiftboat:~/playpen/make-2019-10-08$ cat > dodgy
> true
> martind@swiftboat:~/playpen/make-2019-10-08$ chmod +x dodgy
> martind@swiftboat:~/playpen/make-2019-10-08$ make ./dodgy
> make: *** [Makefile:1: all] Error 127
>
> "make" is 4.2.92 (today's git), where /usr/bin/make is actually 4.0, but
> 4.2.1 behaves the same.

Some changes were added for this to allow the posix_spawn detection to try
to determine whether it works properly; please try make 4.2.93 and see if
you get the right behavior.

It worked for me on GNU/Linux and MacOS.

Reply | Threaded
Open this post in threaded view
|

Re: ENOEXEC from exec*() functions...?

Paul Smith-20
On Fri, 2020-01-03 at 22:51 +0000, Martin Dorey wrote:
> thanks to some new for loop initial declarations in file.c and rule.c,
> which weren't legal in -std=gnu90, the compiler's default dialect.

Hrm.  I had thought that was legal in C90 but I guess I was wrong :(.


Reply | Threaded
Open this post in threaded view
|

Re: ENOEXEC from exec*() functions...?

Eli Zaretskii
> From: Paul Smith <[hidden email]>
> Cc: "[hidden email]" <[hidden email]>
> Date: Fri, 03 Jan 2020 17:56:55 -0500
>
> On Fri, 2020-01-03 at 22:51 +0000, Martin Dorey wrote:
> > thanks to some new for loop initial declarations in file.c and rule.c,
> > which weren't legal in -std=gnu90, the compiler's default dialect.
>
> Hrm.  I had thought that was legal in C90 but I guess I was wrong :(.

No, I think that was introduced by C99.