Edit: I resolved this issue, the solution is below.
I am building a code in a shared computing cluster dedicated for scientific computing, thus I can only control files in my home folder. Although I am using fftw as an example, I would like to understand the specific reason, why my attempt to setup LD_LIBRARY_PATH does not work.
I build the fftw and fftw_mpi libraries in my home folder like this
./configure --prefix=$HOME/install/fftw --enable-mpi --enable-shared
make install
It builds fine, but in install/fftw/lib, I find that the freshly built libfftw3_mpi.so links to wrong version of fftw library.
$ ldd libfftw3_mpi.so |grep fftw
libfftw3.so.3 => /usr/lib64/libfftw3.so.3 (0x00007f7df0979000)
If I now try to set the LD_LIBRARY_PATH correctly pointing to this directory, it still prefers the wrong library:
$ export LD_LIBRARY_PATH=$HOME/install/fftw/lib
$ ldd libfftw3_mpi.so |grep fftw
libfftw3.so.3 => /usr/lib64/libfftw3.so.3 (0x00007f32b4794000)
Only if I explicitly use LD_PRELOAD, I can override this behavior. I don't think LD_PRELOAD is a proper solution though.
$ export LD_PRELOAD=$HOME/install/fftw/lib/libfftw3.so.3
$ ldd libfftw3_mpi.so |grep fftw
$HOME/install/fftw/lib/libfftw3.so.3 (0x00007f5ca3d14000)
Here is what I would have expecting, a small test done in Ubuntu desktop, where I installed fftw to /usr/lib first, and then override this search path with LD_LIBRARY_PATH.
$ export LD_LIBRARY_PATH=
$ ldd q0test_mpi |grep fftw3
libfftw3.so.3 => /usr/lib/x86_64-linux-gnu/libfftw3.so.3
$ export LD_LIBRARY_PATH=$HOME/install/fftw-3.3.4/lib
$ ldd q0test_mpi |grep fftw3
libfftw3.so.3 => $HOME/install/fftw-3.3.4/lib/libfftw3.so.3
In short: Why is libfft3_mpi library still finding the wrong dynamic fftw3 library? Where is this searchpath hard coded in a such way that it is prioritized over LD_LIBARY_PATH? Why is this is not the case in another computer?
I am using intel compilers 13.1.2, mkl 11.0.4.183 and openmpi 1.6.2 if this matters.
Edit: Thanks for all the answers. With help of those, we were able to isolate the problem to RPATH, and from there, the cluster support was able to figure out the problem. I accepted the first answer, but both answers were good.
The reason, why this was so hard to figure out, is that we did not know that the compilers were actually wrapper scripts, adding things to compiler command line. Here a part of a reply from the support:
[The] compilation goes through our compiler wrapper. We do RPATH-ing
by default as it helps most users in correctly running their jobs
without loading LD-LIBRARY_PATH etc. However we exclude certain
library paths from default RPATH which includes /lib, /lib64 /proj
/home etc. Earlier the /usr/lib64 was not excluded by mistake
(mostly). Now we have added that path in the exclusion list.
From http://man7.org/linux/man-pages/man8/ld.so.8.html
When resolving shared object dependencies, the dynamic linker first
inspects each dependency string to see if it contains a slash (this
can occur if a shared object pathname containing slashes was
specified at link time). If a slash is found, then the dependency
string is interpreted as a (relative or absolute) pathname, and the
shared object is loaded using that pathname.
If a shared object dependency does not contain a slash, then it is
searched for in the following order:
o (ELF only) Using the directories specified in the DT_RPATH dynamic
section attribute of the binary if present and DT_RUNPATH
attribute does not exist. Use of DT_RPATH is deprecated.
o Using the environment variable LD_LIBRARY_PATH. Except if the
executable is a set-user-ID/set-group-ID binary, in which case it
is ignored.
o (ELF only) Using the directories specified in the DT_RUNPATH
dynamic section attribute of the binary if present.
o From the cache file /etc/ld.so.cache, which contains a compiled
list of candidate shared objects previously found in the augmented
library path. If, however, the binary was linked with the -z
nodeflib linker option, shared objects in the default paths are
skipped. Shared objects installed in hardware capability
directories (see below) are preferred to other shared objects.
o In the default path /lib, and then /usr/lib. (On some 64-bit
archiectures, the default paths for 64-bit shared objects are
/lib64, and then /usr/lib64.) If the binary was linked with the
-z nodeflib linker option, this step is skipped.
with readelf readelf -d libfftw3_mpi.so you can check if your lib contains such a attribute in the dynamic section.
with export LD_DEBUG=libs you can debug the search path used to find your libs
with chrpath -r<new_path> <executable> the rpath can be changed
I see two possible reasons for this.
First, libfftw3_mpi.so may be linked with /usr/lib64/ as RPATH. In that case, providing LD_LIBRARY_PATH will have no effect. To check if it is your case, run readelf -d libfftw3_mpi.so | grep RPATH and see if it has /usr/lib64/ as a library path. If it does, use chrpath utility to change or remove it.
Alternatively, you may be running a system that does not support LD_LIBRARY_PATH at all (like HP-UX).
Related
I am new to Linux. I see in my CMakeLists.txt the following;
target_link_libraries(app wiringPi
serializer
iothub_client
iothub_client_mqtt_transport
umqtt
aziotsharedutil
ssl
crypto
curl
pthread
m
ssl
crypto)
My question/understanding is; by doing this
Are we telling CMake tool to build wiringPi, serializer etc. and link with app(which is an executable created in my code)?
Where are all these libraries located?
When I do ldconfig -p | grep <libraryname>, for the above libraries
sometimes I find outputs like the following and sometimes nothing, why is that?
Is target_link_libraries smart enough to look for libraries under
the sub-directories too? I mean I see that some are just there under
user/lib and some are one more level under such as
/usr/lib/arm-linux-gnueabihf
pi#raspberrypi:~ $ ldconfig -p | grep curl
libcurl.so.4 (libc6,hard-float) => /usr/lib/arm-linux-gnueabihf/libcurl.so.4
libcurl-gnutls.so.4 (libc6,hard-float) => /usr/lib/arm-linux-gnueabihf/libcurl-gnutls.so.4
pi#raspberrypi:~ $ ldconfig -p | grep wiringPi
libwiringPiDev.so (libc6,hard-float) => /usr/local/lib/libwiringPiDev.so
libwiringPiDev.so (libc6,hard-float) => /usr/lib/libwiringPiDev.so
libwiringPi.so (libc6,hard-float) => /usr/local/lib/libwiringPi.so
libwiringPi.so (libc6,hard-float) => /usr/lib/libwiringPi.so
Are we telling CMake tool to build wiringPi, serializer etc. and link with app(which is an executable created in my code)?
Not exactly. That command tells CMake that the libraries wiringPi, serializer, etc. must be linked to target 'app' during the link stage for that target. It says nothing about building the libraries themselves, and usually they are expected to already be available rather than built.
Where are all these libraries located?
It can vary. The compiler has a default list of directories in which it looks for libraries. Other CMake commands can add link options that add directories to that list.
When I do ldconfig -p | grep <libraryname>, for the above libraries sometimes I find outputs like the following and sometimes
nothing, why is that?
ldconfig reports on libraries known to the dynamic linker. This is distinct from the linker that runs at compile time, and the directories and libraries the two know about are not necessarily the same. Reasons why ldconfig might not list a given library include:
The library is not installed.
Only a static version of the library is installed.
The library is not in any of the locations that the dynamic linker checks by default (additional directories can be specified when a program is launched, in at least two different ways).
Is target_link_libraries smart enough to look for libraries under the sub-directories too? I mean I see that some are just there under user/lib and some are one more level under such as /usr/lib/arm-linux-gnueabihf
This is not a function of CMake, but rather of the chosen toolchain and its configuration (on Linux, that's often the GNU toolchain, featuring GCC). It is usually safe to assume that the toolchain uses all the right standard library directories by default. CMake sometimes can successfully be instructed to search for specific libraries in other likely places, too, but target_link_libraries is not part of that.
When you pass to target_link_libraries a plain name (not a path) which is not a target, CMake just transforms this name into the linker flag. E.g. on Linux this is flag
-l<library-name>
So questions about searching the library you may address directly to the linker - CMake is out of the game here.
So I got several shared libraries that I am trying to permanently install on my Ubuntu system but I am having some difficulty with it.
I want to install the libraries and the headers in a separate folder under /usr/local/lib and /usr/local/include (for example a folder named agony) so it would be clean and removing them would just require that I delete those folders. so it looks something like this:
/usr/local/lib/agony/libbtiGPIO.so
/usr/local/lib/agony/libbtiDSP.so
...
/usr/local/include/agony/GPIO.h
/usr/local/include/agony/DSP.h
...
And I added a file here /etc/ld.so.conf.d/agony.conf which include a line describing the path to the library folder:
$ cat /etc/ld.so.conf.d/agony.conf
/usr/local/lib/agony
and I perform sudo ldconfig to update the library database.
So to double check if the library is found I do ldconfig -p | grep bti* and
I see the following result:
$ ldconfig -p | grep bti
...
libbtiGPIO.so (libc6,x86-64) => /usr/local/lib/agony/libbtiGPIO.so
libbtiDSP.so (libc6,x86-64) => /usr/local/lib/agony/libbtiDSP.so
...
At this point I should be able to use the libraries without specifying the library path. But When I attempt to compile an application without providing the library path (-L) it fails. However, when I supply gcc with the library path ex:
gcc source.c -L /usr/local/lib/agony output -lbtiGPIO -lbtiDSP
it works!!
I don't want to use LD_LIBRARY_PATH environment variable because this library is going to be used everywhere on the system and I don't want other compilers to worry about providing LD_LIBRARY_PATH.
What am I doing wrong here?
At this point I should be able to use the libraries without specifying the library path
Here lies the confusion.
You have built your shared library libbtiGPIO.so (just sticking with that one),
placed it in /usr/local/lib/agony, and updated the ldconfig database accordingly.
The effect of that is when you run a program that has been linked with libbtiGPIO
then the dynamic linker (/lib/x86_64-linux-gnu/ld-2.21.so, or similar) will know where to look
to load that library into the process and you will not need to tell it by setting an LD_LIBRARY_PATH in the environment.
However, you haven't done anything that affects the list of default library
search directories that are hardwired into your build of gcc, that it passes to
the linker (/usr/bin/ld) when you link a program with libbtiGPIO in the first place.
That list of default search directories is what you will find if your do a verbose
build of your program - gcc -v ... - and then pick out the value of LIBRARY_PATH
from the output, e.g.
LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/5/:\
/usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/:\
/usr/lib/gcc/x86_64-linux-gnu/5/../../../../lib/:\
/lib/x86_64-linux-gnu/:\
/lib/../lib/:\
/usr/lib/x86_64-linux-gnu/:\
/usr/lib/../lib/:\
/usr/lib/gcc/x86_64-linux-gnu/5/../../../:\
/lib/:\
/usr/lib
/usr/local/lib/agony is not one of those and to make it one of those you
would have to build gcc from source yourself. Hence, in order to link your
program with libbtiGPIO you still need to tell ld where to find it with
-L/usr/local/lib/agony -lbtiGPIO.
man, you misunderstand the procedure of complier and link.
First, libbtiGPIO.so is a shared link library not a static link library. it is important to know those difference .
Then you need to know something else. changing ld.so.conf.d/*.conf and run sudo ldconfig, it affects the procedure of link. in other words, if you don't add agony.conf and sudo ldconfig, you will receive a error when you run ./a.out rather than gcc source.c -L ...., the gcc command can run successfully even thougth you don't ldconfig.
Finally,if you don't pollute the LD_LIBRARY_PATH environment variable, you have to add -L ... options in your gcc command. What'more, if you don't want to input too many words in your shell frequently, you can learn to use Makefile.
I built a binary using premake (gmake) that links dynamically to another. When I then tried to run the binary, it complained that it can't find the dynamic library.
ldd on the binary and of course the dynamic library is => Not Found!
Of course I can export LD_LIBRARY_PATH=<path of the dynamic library> but I don't want that.
I would like that the binary to work out of the box, on different machines (assuming the dynamic library location doesn't change of course)
1- How do people do this? Do they set RPATH all the time through come linker flags?
From what I gathered, RUNPATH can be over-ridden by LD_LIBRARY_PATH but that's not the case for RPATH.
There's the -rpath and --enable-new-dtags options that will instruct gcc (or the linker to be more precise) to set both RUNPATH and RPATH to the same value, but that's not what I want really, and I can't even see the point of that.
2- What is the point of that?
3- Am I missing something? how can I set RUN_PATH only, so that in general the dependencies are found automatically (in RUN_PATH) unless instructed to search a specific path using LD_LIBRARY_PATH first.
on my laptop the paths are probably different from yours, but the "-d" option should do it.
Usage: c:/strawberry/c/bin/../lib/gcc/x86_64-w64-mingw32/4.7.3/../../../../x86_64-w64-mingw32/bin/ld.exe [options] file...
Options:
-d, -dc, -dp Force common symbols to be defined
"to set both RUNPATH and RPATH to the same value"
That is only for backward compatibility purposes for old ld.sos which did not yet know RUNPATH.
New ld.sos according to its man page:
Using the directories specified in the DT_RPATH dynamic section
attribute of the binary if present and DT_RUNPATH attribute does not
exist. Use of DT_RPATH is deprecated.
So using both RUNPATH and RPATH is in fact the best option of all and it behaves as if only has been set.
Moreover when I try it on Fedora 36 x86_64 it even no longer sets RPATH:
$ :|gcc -fPIC -shared -Wl,-rpath,$PWD,--enable-new-dtags -x c -;readelf
-Wd a.out|grep -i path 0x000000000000001d (RUNPATH) Library runpath: [/tmp]
Also Fedora 36 started to even default to this behavior (--enable-new-dtags).
I have a directory with the following contents:
bin/busybox
lib/ld-linux.so.2
lib/libc.so.6
and when I invoke:
chroot . bin/busybox sh
it fails with the following:
/bin/busybox: error while loading shared libraries: libc.so.6: cannot open shared object file: No such file or directory
When I move lib/libc.so.6 to usr/lib, it works fine.
Why is libc required to be in /usr/lib? When I invoke:
objcdump -p bin/busybox | grep NEEDED
I get:
NEEDED libc.so.6
So I thought, as only the soname of the library is used without slashes etc. the loaded will be able to find it in the standard folders, which is /lib and /usr/lib. Apparently, this is not the case.
To make matters even more confusing, ld-linux.so.2 seems to have to be in /lib because when it is moved to /usr/lib, chroot fails with:
chroot: failed to run command '/bin/busybox': No such file or directory
which I learned is actually an error that the loader cannot be found, not the busybox binary.
Is the issue with libc.so.2 distro specific? If this is important, I'm using Arch Linux.
The location of the loader (typically something like /lib/ld-linux.so) is hard-coded in the binary. There's no search process for this component — if it cannot be found, the binary won't run at all.
(The exact path depends on what libc and architecture you're using. It's at /lib64/ld-linux-x86-64.so.2 for glibc on x86_64, for instance.)
The locations that will be searched for dynamic libraries are configurable in /etc/ld.so.conf. If you don't have that file in the chroot, though, some of the standard paths may not be configured!
I have an old executable that's scheduled for the scrap heap, but it's not there yet. It relies on some libs that have been removed from my environment, but I have some stub libs someplace where it works fine. Id like to point this executable to these stub libs. Yes, i could set LD_LIBRARY_PATH, but this executable is called from many scripts, and many users and I'd love to fix it in one spot.
I don't have source for this, and would be hard to get it. I was thinking - can I edit this file, using an ELF aware editor, and add a simple PATH to rpath to have it hit the new libs? Is this possible, or once you create an ELF binary, you fix things to locations and they can't be moved?
There is a more universal tool than chrpath called patchelf. It was originally created for use in making packages for Nix and NixOS (packaging system and a GNU/Linux distribution).
In case there is no rpath in a binary (here called rdsamp), chrpath fails:
chrpath -r '$ORIGIN/../lib64' rdsamp
rdsamp: no rpath or runpath tag found.
On the other hand,
patchelf --set-rpath '$ORIGIN/../lib64' rdsamp
succeeds just fine.
There is a tool called chrpath which can do this - it's probably available in your distribution's packages.
Just like #user7610 said, the right way to go is the patchelf tool.
But, I feel that I can give a more comprehensive answer, covering all the commands one needs to do exactly that.
For a comprehensive article on the subject, click here
First of all, many developers talk about RPATH, but they actually mean RUNPATH. These are two different optional dynamic sections, and the loader handles them very differently. You can read more about the difference between them in the link I mentioned before.
For now, just remember:
If RUNPATH is set, RPATH is ignored
RPATH is deprecated and should be avoided
RUNPATH is preferred because it can be overridden by LD_LIBRARY_PATH
See the current R[UN]PATH
readelf -d <path-to-elf> | egrep "RPATH|RUNPATH"
Clear the R[UN]PATH
patchelf --remove-rpath <path-to-elf>
Notes:
Removes both RPATH and RUNPATH
Add values to R[UN]PATH
patchelf [--force-rpath] --set-rpath "<desired-rpath>" <path-to-elf>
Notes:
<desired-path> is a colon separated directories list, e.g: /my/libs:/my/other/libs
If you specify --force-rpath, sets RPATH, otherwise sets RUNPATH
This worked for me, replacing XORIGIN with $ORIGIN.
chrpath -r '\$\ORIGIN/../lib64' httpd