How do I diagnose why a shared library is not found? - linux

Question

I am trying to run an app which is linked against libboost_system.so
$ ./app
app: error while loading shared libraries: libboost_system.so.1.63.0: cannot open shared object file: \
No such file or directory
If I run ldd on the app, it shows it can't find the library:
$ ldd ./app
linux-vdso.so.1 (0x00007ffdedb94000)
libboost_system.so.1.63.0 => not found
libpcap.so.1 => /lib64/libpcap.so.1 (0x00007f19a1a7b000)
librt.so.1 => /lib64/librt.so.1 (0x00007f19a1a70000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f19a1a69000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f19a1879000)
libm.so.6 => /lib64/libm.so.6 (0x00007f19a1733000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f19a1716000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f19a16f4000)
libc.so.6 => /lib64/libc.so.6 (0x00007f19a152a000)
/lib64/ld-linux-x86-64.so.2 (0x00007f19a1ae5000)
The library is present on my system:
$ locate libboost_system.so.1.63.0
/usr/local/lib/libboost_system.so.1.63.0
$ ls -la /usr/local/lib/libboost_system*
-rw-r--r--. 1 root root 47014 Jul 2 16:57 /usr/local/lib/libboost_system.a
lrwxrwxrwx. 1 root root 25 Jul 2 16:57 /usr/local/lib/libboost_system.so -> libboost_system.so.1.63.0
-rwxr-xr-x. 1 root root 19816 Jul 2 16:57 /usr/local/lib/libboost_system.so.1.63.0
ld is configured with /usr/local/lib:
$ ld --verbose | grep SEARCH_DIR | sed 's/; /\n/g'
SEARCH_DIR("=/usr/x86_64-redhat-linux/lib64")
SEARCH_DIR("=/usr/lib64")
SEARCH_DIR("=/usr/local/lib64")
SEARCH_DIR("=/lib64")
SEARCH_DIR("=/usr/x86_64-redhat-linux/lib")
SEARCH_DIR("=/usr/local/lib") <----- here
SEARCH_DIR("=/lib")
SEARCH_DIR("=/usr/lib")
I've also run ldconfig to refresh the cache, but that doesn't help.
I tried searching for RPATH or RUNPATH with readelf, neither are specified:
$ readelf -d ./app | grep -i path
< no results >
If I explicitly set LD_LIBARY_PATH then the library is found:
$ LD_LIBRARY_PATH=/usr/local/lib ldd ./app
linux-vdso.so.1 (0x00007fffa9a5d000)
libboost_system.so.1.63.0 => /usr/local/lib/libboost_system.so.1.63.0 (0x00007f5fff664000)
libpcap.so.1 => /lib64/libpcap.so.1 (0x00007f5fff5fc000)
librt.so.1 => /lib64/librt.so.1 (0x00007f5fff5f1000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f5fff5ea000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f5fff3fa000)
libm.so.6 => /lib64/libm.so.6 (0x00007f5fff2b4000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f5fff297000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f5fff275000)
libc.so.6 => /lib64/libc.so.6 (0x00007f5fff0ab000)
/lib64/ld-linux-x86-64.so.2 (0x00007f5fff66b000)
Why can't ld find libboost_system.so.1.63.0 without specifying LD_LIBRARY_PATH?

Answer 1

You are looking at the "wrong" linker. ld is the static linker and isn't what's used by at runtime to locate the shared libraries.
The dynamic linker, ld.so, is what locates shared libraries.
The dynamic linker does look at the LD_LIBRARY_PATH and that's why it works if you set it.
If you want to update the search locations for dynamic linker (and not want to set LD_LIBRARY_PATH), you can create a file in /etc/ld.so.conf.d/ and update /etc/ld.so.cache with ldconfig.

Answer 2

The diagnostic is activated by LD_DEBUG (cf. man ld.so):
LD_DEBUG=libs ./a.out
It shows the search paths that the loader tried: ($LD_LIBRARY_PATH, R[UN]PATH from the app or library's ELF, ld.so.cache).
Its output is a little long to copy here, but would look like this:
265: find library=libboost_system.so.1.63.0 [0]; searching
265: search path= [...] (LD_LIBRARY_PATH)
[trying many files...]
265: search path= [...] (RUNPATH from file ./app)
[still trying many files...]
265: search cache=/etc/ld.so.cache
[still trying many files...]
In your case, you'd probably have seen that /usr/local/lib was not in the cache. Depending on the distribution, it may look by default in /usr/local/lib/x86_64-linux-gnu or /usr/local/lib64 or none at all.