Shell function and LD_LIBRARY_PATH

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Shell function and LD_LIBRARY_PATH

Mohammad Akhlaghi
Dear GNU Make developers,

I recently installed the newly released Bash 5.0 (accompanied by a new
libreadline library) in a user-specific directory. The host operating
system (on a server, where I don't have root access) uses an older
version of Bash.

Through Make's `SHELL' variable, I have asked Make to use the updated
version of Bash. In Make, I have also exported `LD_LIBRARY_PATH', so my
Bash can find the proper version of libreadline to link with. This
exportation of LD_LIBRARY_PATH is done very early in the Makefile and is
set with `:=':

export LD_LIBRARY_PATH := /home/user/.local/lib

This works nicely for all the shell calls within recipes. However, when
I use the $(shell ...) function, I get a linking error (shown below) by
my new Bash for every use of the $(shell ...) function.

/home/user/.local/bin/bash: error while loading shared libraries:
libreadline.so.8: cannot open shared object file: No such file or directory

Reverting back to the system's Bash, I tried $(shell printenv) as well
as a call to `printenv' in a recipe. I noticed that while the other
important environment variables like `PATH', `LDFLAGS' and etc are in
both, LD_LIBRARY_PATH is not present in value of $(shell printenv).

Since I couldn't find any mention of this in in the manual, I wanted to
see if this removal of LD_LIBRARY_PATH from the $(shell) function is
intentional or if its a bug?

I am using Make 4.2.90 (recently bootstrapped
https://git.savannah.gnu.org/cgit/make.git ).

Cheers,
Mohammad

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: Shell function and LD_LIBRARY_PATH

Paul Smith-20
On Sun, 2019-01-20 at 19:32 +0000, Mohammad Akhlaghi wrote:
> Since I couldn't find any mention of this in in the manual, I wanted to
> see if this removal of LD_LIBRARY_PATH from the $(shell) function is
> intentional or if its a bug?

It is not the case that LD_LIBRARY_PATH is removed by the $(shell...)
function.

What is true is that variables marked as exported in the makefile are
not exported to the $(shell...) function: they are only exported to
recipes.  This is a VERY long-standing behavior (basically, GNU make
has always worked that way since the export directive was added in the
early 1990's).

See for example: https://savannah.gnu.org/bugs/?10593

Usually the solution is to add the variable assignment into the shell
function invocation, but since you need this variable to start the
shell itself that's not possible here.

However:

> In Make, I have also exported `LD_LIBRARY_PATH', so my Bash can find
> the proper version of libreadline to link with.

this is not really right.  Setting LD_LIBRARY_PATH is a bad idea in
general.  It can wreak all sorts of havoc because it applies to  all
programs not just one program.

If a program must have LD_LIBRARY_PATH the right way to handle it is to
use a shell script wrapper for that binary that sets up the environment
then exec's it.  For example:

  mv bash bash-real
  cat >bash <<'EOF'
  #!/bin/sh
  export LD_LIBRARY_PATH=${LD_LIBRARY_PATH:+$LD_LIBRARY_PATH:}/home/user/.local/lib
  exec bash-real "$@"
  EOF
  chmod 755 bash

Now running bash should "work".  While this is great for SOME programs,
even this is really not a good idea for a shell because the entire
point of shell is to invoke other programs, and this passes
LD_LIBRARY_PATH to all of them.

What you should do is configure bash's runtime lookup path to look in
the right place by default using RPATH at link time, so you don't need
to set LD_LIBRARY_PATH in the first place.


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: Shell function and LD_LIBRARY_PATH

Martin Dorey-2

>> on a server, where I don't have root access


Ouch.

> using RPATH at link time

Debian has a package called chrpath, supplying an executable of the same name which lets you change the RPATH later.  That might be easier than doing battle with the underlying bash package's build rules.  The man page points out an unfortunate constraint:

"The new path must be shorter or the same length as the current path."

So it's probably no use, but I'll mention it just in case.

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: Shell function and LD_LIBRARY_PATH

Paul Smith-20
On Sun, 2019-01-20 at 21:18 +0000, Martin Dorey wrote:
> >> on a server, where I don't have root access
>
> Ouch.

Why ouch?  He's installing his own bash so he can modify his own
version of bash.  He just can't install it as the system version or add
libraries to the default runtime library search directories.

If you configure the bash build with --prefix=$HOME/.local that should
be enough to have it search $HOME/.local/lib for shared libraries by
default.

You just can't configure it with some --prefix then try to install it
somewhere different.

> > using RPATH at link time
>
> Debian has a package called chrpath, supplying an executable of the
> same name which lets you change the RPATH later.

Better is patchelf which allows you to make all sorts of changes to ELF
binaries, including showing, setting, etc. RPATH, without any length
restrictions:

http://manpages.ubuntu.com/manpages/cosmic/man1/patchelf.1.html


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: Shell function and LD_LIBRARY_PATH

Martin Dorey-2

Why ouch?


Because it's unpleasant and wasteful to work in an environment where you're not trusted.


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: Shell function and LD_LIBRARY_PATH

Paul Smith-20
On Sun, 2019-01-20 at 21:57 +0000, Martin Dorey wrote:
> > Why ouch?
>
> Because it's unpleasant and wasteful to work in an environment where
> you're not trusted.

Well, it might not be work-related.

I can ssh into my ISP server using my local account so I can muck
around and set things up, but I don't have root access to it, and I
wouldn't expect to (or, I would be OK with having root because I trust
myself but I wouldn't want other people to have root access to that
server :-p).

For that matter I have an account on the GNU servers but again, no root
access.  In the early days of the internet anyone with login access to
the GNU servers could become root (these are not the FTP etc. servers
mind you, but still).  After some sad, but predictable, problems RMS
was reluctantly convinced that those systems had to be locked down.

Anyway, this is getting off-topic for sure ... :).


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: Shell function and LD_LIBRARY_PATH

Mohammad Akhlaghi
Thank you very much for the prompt replies, they were very useful.

In the end (for the time being!), I am using patchelf. I couldn't find a
way to configure Bash with RPATH (I still haven't stopped searching, but
I need to progress for now).

It is interesting that exported variables are not set in `$(shell ...)'.
But I am a little confused, because I also export `PATH' and `CPPFLAGS'
and many other basic flags, but I could see them in `$(shell printenv)'!

Your points about a generic setting of LD_LIBRARY_PATH in the Makefile
is indeed correct, and I completely understand it. My usage scenario is
a little non-standard/different.

In summary, it is a project aiming to make a fully reproducible paper:
trying to tackle the reproducibility crisis in the sciences. A complete
description of it is available here:

https://gitlab.com/makhlaghi/reproducible-paper/blob/pipeline/README-pipeline.md

As you see, the whole thing (even the analysis of a research project) is
managed in Make.

Since many scientific projects (at least in astrophysics: my field) use
large clusters or super computers for processing very large datasets, I
cannot assume root access.

In summary it builds almost all the necessary tools, does the analysis
using those tools, and creates the final LaTeX-compiled PDF paper.

Since many scientists use Mac OS (unfortunately!), it also needs to work
on those operating systems. So it just doesn't build the C compiler and
linker, but builds almost everything on top of that (its like a
mini-package-manager!).

I hope this shows why its necessary to fully re-write PATH,
LD_LIBRARY_PATH and other basic environment variables within Make, so
the processing for each project is as independent of the host operating
system as possible (and thus reproducible, or fully under control on
different systems).

By the way, in case you have any thoughts/points regarding this
"reproducible paper" project, they would be most highly appreciated and
welcome.

Thanks again for all the great points,
Cheers,
Mohammad

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: Shell function and LD_LIBRARY_PATH

Dennis Clarke
On 1/20/19 8:47 PM, Mohammad Akhlaghi wrote:
> Thank you very much for the prompt replies, they were very useful.
>
> In the end (for the time being!), I am using patchelf. I couldn't find a
> way to configure Bash with RPATH (I still haven't stopped searching, but
> I need to progress for now).


I have not followed the entire thread and I am certain that this has
been looked at but doesn't -Wl,-rpath=/usr/local/lib do the necessary
passing to the ld stage?

Dennis


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make