How are shared library dependency paths determined on Linux? - linux

When I run ldd against a shared library such as libphp5.so I see that it has a dependency on libmysqlclient.so.16:
$ ldd ./libphp5.so
libmysqlclient.so.16 => /usr/lib/mysql/libmysqlclient.so.16
[other dependencies snipped out]
Are these dependency filenames and paths (/usr/lib/mysql/libmysqlclient.so.16) baked into the shared library binary? Or is this path determined by some other means such as via /etc/ld.so.conf.d/mysql-i386.conf, which incidentally contains:
/usr/lib/mysql/
One other thing is puzzling me:
There is a shared library I have that I compile from source. This has a dependency on libmysqlclient_r. The gcc compiler switches to produce this this library look like:
gcc -shared -L/usr/lib/mysql -lmysqlclient_r [+various other switches]
When I do ldd mylib.so I see:
libmysqlclient_r.so.16 => /usr/lib/mysql/libmysqlclient_r.so.16 (0x0055c000)
However in the /usr/lib/mysql directory I see:
-rwxr-xr-x. libmysqlclient_r.so -> libmysqlclient_r.so.16.0.0
lrwxrwxrwx. libmysqlclient_r.so.16 -> libmysqlclient_r.so.16.0.0
-rwxr-xr-x. libmysqlclient_r.so.16.0.0
lrwxrwxrwx. libmysqlclient.so -> libmysqlclient.so.16.0.0
lrwxrwxrwx. libmysqlclient.so.16 -> libmysqlclient.so.16.0.0
-rwxr-xr-x. libmysqlclient.so.16.0.0
libmysqlclient_r.so is a symbolic link to libmysqlclient_r.so.16.0.0, so why does ldd show the dependency as libmysqlclient_r.so.16. Is there some magic I'm missing here?
Having been a Windows dev for many years I'm a bit new to gcc and development on Linux.
My Linux distribution is CentOS 6.0 x86-32bit.

You can see which paths are coming from where by running
LD_DEBUG=libs ldd ./libphp5.so
Are these dependency filenames and paths (/usr/lib/mysql/libmysqlclient.so.16) baked into the shared library binary?
The filename almost certainly is. The path usually isn't. You can see what is baked into the binary with
readelf -d ./libphp5.so
Look for (NEEDED) and (RPATH) entries.
Also give man ld.so a read. There are many factors that affect how dynamic loader searches for shared libraries: ld.so.conf, LD_LIBRARY_PATH, whether the executable is suid or not, how glibc was configured, which -rpath settings were given at link time, etc. etc.

Are these dependency filenames and paths (/usr/lib/mysql/libmysqlclient.so.16) baked into the shared library binary?
Yes, they can be and often are. The keyword here is -rpath. However, ld.conf also has its say. The whole system is quite complex, unfortunately.

Related

Understanding target_link_libraries

I am new to Linux. I see in my CMakeLists.txt the following;
target_link_libraries(app wiringPi
serializer
iothub_client
iothub_client_mqtt_transport
umqtt
aziotsharedutil
ssl
crypto
curl
pthread
m
ssl
crypto)
My question/understanding is; by doing this
Are we telling CMake tool to build wiringPi, serializer etc. and link with app(which is an executable created in my code)?
Where are all these libraries located?
When I do ldconfig -p | grep <libraryname>, for the above libraries
sometimes I find outputs like the following and sometimes nothing, why is that?
Is target_link_libraries smart enough to look for libraries under
the sub-directories too? I mean I see that some are just there under
user/lib and some are one more level under such as
/usr/lib/arm-linux-gnueabihf
pi#raspberrypi:~ $ ldconfig -p | grep curl
libcurl.so.4 (libc6,hard-float) => /usr/lib/arm-linux-gnueabihf/libcurl.so.4
libcurl-gnutls.so.4 (libc6,hard-float) => /usr/lib/arm-linux-gnueabihf/libcurl-gnutls.so.4
pi#raspberrypi:~ $ ldconfig -p | grep wiringPi
libwiringPiDev.so (libc6,hard-float) => /usr/local/lib/libwiringPiDev.so
libwiringPiDev.so (libc6,hard-float) => /usr/lib/libwiringPiDev.so
libwiringPi.so (libc6,hard-float) => /usr/local/lib/libwiringPi.so
libwiringPi.so (libc6,hard-float) => /usr/lib/libwiringPi.so
Are we telling CMake tool to build wiringPi, serializer etc. and link with app(which is an executable created in my code)?
Not exactly. That command tells CMake that the libraries wiringPi, serializer, etc. must be linked to target 'app' during the link stage for that target. It says nothing about building the libraries themselves, and usually they are expected to already be available rather than built.
Where are all these libraries located?
It can vary. The compiler has a default list of directories in which it looks for libraries. Other CMake commands can add link options that add directories to that list.
When I do ldconfig -p | grep <libraryname>, for the above libraries sometimes I find outputs like the following and sometimes
nothing, why is that?
ldconfig reports on libraries known to the dynamic linker. This is distinct from the linker that runs at compile time, and the directories and libraries the two know about are not necessarily the same. Reasons why ldconfig might not list a given library include:
The library is not installed.
Only a static version of the library is installed.
The library is not in any of the locations that the dynamic linker checks by default (additional directories can be specified when a program is launched, in at least two different ways).
Is target_link_libraries smart enough to look for libraries under the sub-directories too? I mean I see that some are just there under user/lib and some are one more level under such as /usr/lib/arm-linux-gnueabihf
This is not a function of CMake, but rather of the chosen toolchain and its configuration (on Linux, that's often the GNU toolchain, featuring GCC). It is usually safe to assume that the toolchain uses all the right standard library directories by default. CMake sometimes can successfully be instructed to search for specific libraries in other likely places, too, but target_link_libraries is not part of that.
When you pass to target_link_libraries a plain name (not a path) which is not a target, CMake just transforms this name into the linker flag. E.g. on Linux this is flag
-l<library-name>
So questions about searching the library you may address directly to the linker - CMake is out of the game here.

Why I cannot override search path of dynamic libraries with LD_LIBRARY_PATH?

Edit: I resolved this issue, the solution is below.
I am building a code in a shared computing cluster dedicated for scientific computing, thus I can only control files in my home folder. Although I am using fftw as an example, I would like to understand the specific reason, why my attempt to setup LD_LIBRARY_PATH does not work.
I build the fftw and fftw_mpi libraries in my home folder like this
./configure --prefix=$HOME/install/fftw --enable-mpi --enable-shared
make install
It builds fine, but in install/fftw/lib, I find that the freshly built libfftw3_mpi.so links to wrong version of fftw library.
$ ldd libfftw3_mpi.so |grep fftw
libfftw3.so.3 => /usr/lib64/libfftw3.so.3 (0x00007f7df0979000)
If I now try to set the LD_LIBRARY_PATH correctly pointing to this directory, it still prefers the wrong library:
$ export LD_LIBRARY_PATH=$HOME/install/fftw/lib
$ ldd libfftw3_mpi.so |grep fftw
libfftw3.so.3 => /usr/lib64/libfftw3.so.3 (0x00007f32b4794000)
Only if I explicitly use LD_PRELOAD, I can override this behavior. I don't think LD_PRELOAD is a proper solution though.
$ export LD_PRELOAD=$HOME/install/fftw/lib/libfftw3.so.3
$ ldd libfftw3_mpi.so |grep fftw
$HOME/install/fftw/lib/libfftw3.so.3 (0x00007f5ca3d14000)
Here is what I would have expecting, a small test done in Ubuntu desktop, where I installed fftw to /usr/lib first, and then override this search path with LD_LIBRARY_PATH.
$ export LD_LIBRARY_PATH=
$ ldd q0test_mpi |grep fftw3
libfftw3.so.3 => /usr/lib/x86_64-linux-gnu/libfftw3.so.3
$ export LD_LIBRARY_PATH=$HOME/install/fftw-3.3.4/lib
$ ldd q0test_mpi |grep fftw3
libfftw3.so.3 => $HOME/install/fftw-3.3.4/lib/libfftw3.so.3
In short: Why is libfft3_mpi library still finding the wrong dynamic fftw3 library? Where is this searchpath hard coded in a such way that it is prioritized over LD_LIBARY_PATH? Why is this is not the case in another computer?
I am using intel compilers 13.1.2, mkl 11.0.4.183 and openmpi 1.6.2 if this matters.
Edit: Thanks for all the answers. With help of those, we were able to isolate the problem to RPATH, and from there, the cluster support was able to figure out the problem. I accepted the first answer, but both answers were good.
The reason, why this was so hard to figure out, is that we did not know that the compilers were actually wrapper scripts, adding things to compiler command line. Here a part of a reply from the support:
[The] compilation goes through our compiler wrapper. We do RPATH-ing
by default as it helps most users in correctly running their jobs
without loading LD-LIBRARY_PATH etc. However we exclude certain
library paths from default RPATH which includes /lib, /lib64 /proj
/home etc. Earlier the /usr/lib64 was not excluded by mistake
(mostly). Now we have added that path in the exclusion list.
From http://man7.org/linux/man-pages/man8/ld.so.8.html
When resolving shared object dependencies, the dynamic linker first
inspects each dependency string to see if it contains a slash (this
can occur if a shared object pathname containing slashes was
specified at link time). If a slash is found, then the dependency
string is interpreted as a (relative or absolute) pathname, and the
shared object is loaded using that pathname.
If a shared object dependency does not contain a slash, then it is
searched for in the following order:
o (ELF only) Using the directories specified in the DT_RPATH dynamic
section attribute of the binary if present and DT_RUNPATH
attribute does not exist. Use of DT_RPATH is deprecated.
o Using the environment variable LD_LIBRARY_PATH. Except if the
executable is a set-user-ID/set-group-ID binary, in which case it
is ignored.
o (ELF only) Using the directories specified in the DT_RUNPATH
dynamic section attribute of the binary if present.
o From the cache file /etc/ld.so.cache, which contains a compiled
list of candidate shared objects previously found in the augmented
library path. If, however, the binary was linked with the -z
nodeflib linker option, shared objects in the default paths are
skipped. Shared objects installed in hardware capability
directories (see below) are preferred to other shared objects.
o In the default path /lib, and then /usr/lib. (On some 64-bit
archiectures, the default paths for 64-bit shared objects are
/lib64, and then /usr/lib64.) If the binary was linked with the
-z nodeflib linker option, this step is skipped.
with readelf readelf -d libfftw3_mpi.so you can check if your lib contains such a attribute in the dynamic section.
with export LD_DEBUG=libs you can debug the search path used to find your libs
with chrpath -r<new_path> <executable> the rpath can be changed
I see two possible reasons for this.
First, libfftw3_mpi.so may be linked with /usr/lib64/ as RPATH. In that case, providing LD_LIBRARY_PATH will have no effect. To check if it is your case, run readelf -d libfftw3_mpi.so | grep RPATH and see if it has /usr/lib64/ as a library path. If it does, use chrpath utility to change or remove it.
Alternatively, you may be running a system that does not support LD_LIBRARY_PATH at all (like HP-UX).

Is libc.so.2 required to be located in /usr/lib?

I have a directory with the following contents:
bin/busybox
lib/ld-linux.so.2
lib/libc.so.6
and when I invoke:
chroot . bin/busybox sh
it fails with the following:
/bin/busybox: error while loading shared libraries: libc.so.6: cannot open shared object file: No such file or directory
When I move lib/libc.so.6 to usr/lib, it works fine.
Why is libc required to be in /usr/lib? When I invoke:
objcdump -p bin/busybox | grep NEEDED
I get:
NEEDED libc.so.6
So I thought, as only the soname of the library is used without slashes etc. the loaded will be able to find it in the standard folders, which is /lib and /usr/lib. Apparently, this is not the case.
To make matters even more confusing, ld-linux.so.2 seems to have to be in /lib because when it is moved to /usr/lib, chroot fails with:
chroot: failed to run command '/bin/busybox': No such file or directory
which I learned is actually an error that the loader cannot be found, not the busybox binary.
Is the issue with libc.so.2 distro specific? If this is important, I'm using Arch Linux.
The location of the loader (typically something like /lib/ld-linux.so) is hard-coded in the binary. There's no search process for this component — if it cannot be found, the binary won't run at all.
(The exact path depends on what libc and architecture you're using. It's at /lib64/ld-linux-x86-64.so.2 for glibc on x86_64, for instance.)
The locations that will be searched for dynamic libraries are configurable in /etc/ld.so.conf. If you don't have that file in the chroot, though, some of the standard paths may not be configured!

Qt cross-compile Linux -> MacOS cannot link to Qt libs

I'm trying to link a Qt program I have cross-compiled for a MacOS target on a Linux host. I followed the instructions found here (https://github.com/Tatsh/xchain) to succesfully build gcc, and I can compile and link both simple test programs as on that page to produce executables which run fine on MacOS.
I have built my own Qt from source using this gcc, and I can compile my Qt program without a hitch. The problem comes at the linking stage, when the object files of my program should be linked to the Qt libs, it almost seems as if the Qt libs can't be found at all, or are ignored because they are somehow incompatible.
The object files of my program and the Qt libs I built both seem to be ok:
$ file MainWindow.o
MainWindow.o: Mach-O object i386
$ file /usr/local/myqt/mac32/Qt-4.8.2/lib/libQtCore.dylib.4.8.2
/usr/local/myqt/mac32/Qt-4.8.2/lib/libQtCore.dylib.4.8.2: Mach-O dynamically linked shared library i386
The g++ invocation goes like this:
$ i686-apple-darwin10-g++ -o myapp.app/Contents/MacOS/myapp
main.o MainWindow.o ...
-L/usr/local/myqt/mac32/Qt-4.8.2/lib -lQtCore -lQtGui ...
Of course there are many more object files and many more Qt libs, I have left them out for brevity.
The errors I get are typical of simply not adding an object or lib to the command line, for example:
...
"QMainWindow::event(QEvent*)", referenced from:
vtable for MainWindowin moc_MainWindow.o
"QDir::~QDir()", referenced from:
_main in main.o
ld: symbol(s) not found
collect2: ld returned 1 exit status
and it seems that basically every Qt lib is not being found.
I have also tried specifying the lib files directly instead of the -L/path and -lQtFoo combo, and I even tried renaming them from .dylib to .so :)
If I run g++ with -v to see the linker call, then also add -v to that, I can see that the paths being checked for libs look ok:
$ i686-apple-darwin10-ld64 -dynamic -arch i386 -headerpad_max_install_names
-macosx_version_min 10.4 -syslibroot /usr/i686-apple-darwin10
-weak_reference_mismatches non-weak -o myapp.app/Contents/MacOS/myapp
-lcrt1.o /usr/i686-apple-darwin10/SDKs/MacOSX10.6.sdk/usr/bin/../lib/gcc/i686-apple-darwin10/4.2.1/crt3.o
main.o MainWindow.o
...
-L/usr/local/myqt/mac32/Qt-4.8.2/lib -lQtCore -lQtGui ...
-v
#(#)PROGRAM:ld64 PROJECT:odcctools-622.3od16
Library search paths:
/usr/local/myqt/mac32/Qt-4.8.2/lib
/usr/i686-apple-darwin10/usr/X11/lib
/usr/local/myqt/mac32/Qt-4.8.2/lib
/usr/i686-apple-darwin10/usr/lib/i686-apple-darwin10/4.2.1
/usr/i686-apple-darwin10/usr/lib
/usr/i686-apple-darwin10/SDKs/MacOSX10.6.sdk/usr/lib/gcc/i686-apple-darwin10/4.2.1
/usr/i686-apple-darwin10/SDKs/MacOSX10.6.sdk/usr/lib/gcc
/usr/i686-apple-darwin10/usr/lib/gcc/i686-apple-darwin10/4.2.1
/usr/i686-apple-darwin10/SDKs/MacOSX10.6.sdk/usr/i686-apple-darwin10/lib
/usr/i686-apple-darwin10/SDKs/MacOSX10.6.sdk/usr/i686-apple-darwin10/lib
/usr/i686-apple-darwin10/usr/lib
/usr/local/lib
Framework search paths:
/usr/i686-apple-darwin10/Library/Frameworks/
/usr/i686-apple-darwin10/System/Library/Frameworks/
The lib location and setup looks ok:
$ ls -lh /usr/local/myqt/mac32/Qt-4.8.2/lib/libQtCore.dylib*
lrwxrwxrwx. 1 root root 21 Aug 7 05:22 /usr/local/myqt/mac32/Qt-4.8.2/lib/libQtCore.dylib -> libQtCore.dylib.4.8.2
lrwxrwxrwx. 1 root root 21 Aug 7 05:22 /usr/local/myqt/mac32/Qt-4.8.2/lib/libQtCore.dylib.4 -> libQtCore.dylib.4.8.2
lrwxrwxrwx. 1 root root 21 Aug 7 05:22 /usr/local/myqt/mac32/Qt-4.8.2/lib/libQtCore.dylib.4.8 -> libQtCore.dylib.4.8.2
-rwxr-xr-x. 1 root root 2.7M Aug 7 05:22 /usr/local/myqt/mac32/Qt-4.8.2/lib/libQtCore.dylib.4.8.2
The lib contents also seem to be ok, but I am not too familiar with this part... actually I just discovered 'nm' on another question here just now.
$ i686-apple-darwin10-nm /usr/local/myqt/mac32/Qt-4.8.2/lib/libQtGui.dylib.4.8.2 | grep event
U __ZN16QCoreApplication5eventEP6QEvent
U __ZN16QEventTransition5eventEP6QEvent
U __ZN16QEventTransition9eventTestEP6QEvent
U __ZN17QVariantAnimation5eventEP6QEvent
U __ZN19QAbstractTransition5eventEP6QEvent
U __ZN23QCoreApplicationPrivate15eventDispatcherE
U __ZN7QObject11eventFilterEPS_P6QEvent
U __ZN7QObject5eventEP6QEvent
I am also not too familiar with 'name mangling', but I get the feeling it may have something to do with this.
It has been quite a long journey so far with this whole cross-compiling thing, so it sucks to be this close to the end and hit something that really stumps me this much - any advice would be greatly appreciated!
** EDIT ***
I found that if I take out the -dynamic flag for the ld64 call, different function names are shown as missing. For example the missing reference to "QMainWindow::event(QEvent*)" mentioned above becomes:
"__ZN11QMainWindow5eventEP6QEvent", referenced from:
__ZTV10MainWindow in moc_MainWindow.o
And now I can see that these names are indeed, NOT found in the Qt libs I built (configuration issue...?). This didn't enlighten me much as to what should be done to fix the problem, but perhaps it might give somebody else a clue.
I don't have a full answer for you, but I can tell you that the 'U' in your nm output means the symbol is not defined in that executable (or library in this case) but is expected to be defined in an externally linked library, suggesting that something may actually be wrong with that library.
As a side note, you can use the -C argument to nm to have it demangle your C++ symbols.
I managed to get past this problem, although without totally understanding the details of it. Originally I had these two lines in the mkspecs file and although they did not cause any errors when building Qt, commenting them out fixed the problem:
QMAKE_RANLIB = i686-apple-darwin10-ranlib
QMAKE_STRIP = i686-apple-darwin10-strip
I had set these to the cross-compiler executables like this thinking it was necessary, but it seems that everything in the tool-chain is taken care of simply by specifying the correct gcc and g++.

Linking 3rd party shared libraries without soname, linker name

Installing liboost-dev on Debian Squeeze gives me several libraries like /usr/lib/libboost_thread.so.1.42.0, but no libboost_thread.so. Now I can't link using the -l flag of gcc / ld because the names don't end in .so.
I notice that /usr/lib has plenty of other libraries of the form libfoo.so.N without a libfoo.so, so this isn't peculiar to Boost. I ended up adding libboost_thread.so.1 and libboost_thread.so symlinks links by hand. (The man page for ldconfig suggests it will add the links, but it didn't do anything).
Everything works fine, but it feels dirty. What should I have done?
use some more specific linker option I haven't found yet (at the cost of making my makefiles depend on a specific version number).
just add the symlinks by hand (at the risk of subverting package management).
some other Debian 'right way' to do it.
You installed the run-time package libboost-thread1.42.0 but the development package libboost-thread-dev (or even the catch-all package libboost-all-dev.
Once you have the corresponding -dev package, linking will work. That is a general feature of most Linux distribution---you almost never want to mess with the symlinks by hand.
Edit: In response to your comment:
edd#max:~$ ls -l /usr/lib/libboost_thread.*
-rw-r--r-- 1 root root 176324 2010-10-21 00:56 /usr/lib/libboost_thread.a
lrwxrwxrwx 1 root root 25 2011-05-14 10:17 /usr/lib/libboost_thread.so -> \
libboost_thread.so.1.42.0
-rw-r--r-- 1 root root 88824 2010-10-21 00:56 /usr/lib/libboost_thread.so.1.42.0
edd#max:~$ dpkg -S /usr/lib/libboost_thread.so
libboost-thread1.42-dev: /usr/lib/libboost_thread.so
edd#max:~$
Clearly the package management system created the links, and owns them.
Dirk's answer is correct about the general principle but there seems to be an extra trap for the unwary with the boost packaging.
Normally the headers and the library symlink are in the same package, so you get the library symlink without thinking about it. However with boost "libboost<version>-dev" provides the headers but "libboost-<lib><version>-dev" provides the shared library symlink. So if you only install the former you get stuff compiling but not linking.

Resources