Why is my Linux application pulling in the wrong .so library? - linux

Question

I have an application I'm building that's using the NetCDF C++ library, and NetCDF is pulling in the HDF-4 libary. However, it's pulling in the wrong HDF-4 library.
Here's how my app is linked:
/apps1/intel/bin/icpc -gxx-name=/apps1/gcc-4.5.0/bin/g++ -shared -o lib/libMyCustom.so
-Llib -L/apps1/boost-1.48.0/lib -Wl,-rpath=/apps1/boost-1.48.0/lib
-L/apps1/gdal-1.8.0-jasper/lib -Wl,-rpath=/apps1/gdal-1.8.0-jasper/lib
-L/new_apps1/hdf4/lib -Wl,-rpath=/new_apps1/hdf4/lib -L/new_apps1/netcdf/lib
-Wl,-rpath=/new_apps1/netcdf/lib -lboost_system -lboost_serialization
-lboost_date_time -lboost_thread -lgdal -ldf -lmfhdf -lnetcdf_c++
MyProj/obj/ProjUtility.o MyProj/obj/ProjMetadataException.o
MyProj/obj/ProjTimestampUtil.o
I have set my LD_LIBRARY_PATH very short:
LD_LIBRARY_PATH=/new_apps1/hdf4/lib:/new_apps1/hdf5/lib:
/apps1/intel/composerxe/lib/intel64:/apps1/gcc-4.5.0/lib64:/apps1/gcc-4.5.0/lib
And here is an excerpt from the ldd -v output:
libdf.so.0 => /new_apps1/hdf4/lib/libdf.so.0 (0x00002af5baabc000)
libmfhdf.so.0 => /new_apps1/hdf4/lib/libmfhdf.so.0 (0x00002af5bad61000)
libnetcdf_c++.so.5 => /new_apps1/netcdf/lib/libnetcdf_c++.so.5 (0x00002af5baf85000)
libhdf5.so.6 => /new_apps1/hdf5/lib/libhdf5.so.6 (0x00002af5bd1e7000)
libgif.so.4 => /usr/lib64/libgif.so.4 (0x0000003a6bc00000)
libpng12.so.0 => /usr/lib64/libpng12.so.0 (0x0000003a71000000)
libnetcdf.so.6 => /new_apps1/netcdf/lib/libnetcdf.so.6 (0x00002af5bd682000)
libhdf5_hl.so.6 => /new_apps1/hdf5/lib/libhdf5_hl.so.6 (0x00002af5be272000)
/new_apps1/hdf4/lib/libdf.so.0:
libc.so.6 (GLIBC_2.3) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
/new_apps1/hdf4/lib/libmfhdf.so.0:
libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
/new_apps1/netcdf/lib/libnetcdf_c++.so.5:
libgcc_s.so.1 (GCC_3.0) => /apps1/gcc-4.5.0/lib64/libgcc_s.so.1
libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
libstdc++.so.6 (CXXABI_1.3) => /apps1/gcc-4.5.0/lib64/libstdc++.so.6
libstdc++.so.6 (GLIBCXX_3.4) => /apps1/gcc-4.5.0/lib64/libstdc++.so.6
/new_apps1/hdf5/lib/libhdf5.so.6:
libm.so.6 (GLIBC_2.2.5) => /lib64/libm.so.6
libc.so.6 (GLIBC_2.3) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
/new_apps1/netcdf/lib/libnetcdf.so.6:
libm.so.6 (GLIBC_2.2.5) => /lib64/libm.so.6
libc.so.6 (GLIBC_2.3) => /lib64/libc.so.6
libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
/new_apps1/hdf5/lib/libhdf5_hl.so.6:
libc.so.6 (GLIBC_2.2.5) => /lib64/libc.so.6
So far, everything in the LD_LIBRARY_PATH, rpath, and ldd indicate that it's pointing to the HDF that I want to reference (/new_apps1/hdf4/lib/libmfhdf.so.0). But when I run, Valgrind is telling me that it's dying in the OLD HDF-4 library (which is probably why it's segfaulting), instead of the HDF-4 library I'm attempting to link against:
Invalid read of size 4
at 0x67CF765: NC_var_shape (in /apps1/hdf-4.2.6/lib/libmfhdf.so.0.0.0)
by 0x91327CA: nc_get_NC (v1hpg.c:1113)
by 0x91303C0: l3nc__open_mp (nc.c:1096)
by 0x915B279: nc3d__open_mp (dapdispatch3.c:336)
by 0x914A752: nc3d_open (ncdap3.c:94)
by 0x911F8A2: l4nc_open_file (nc4file.c:2338)
by 0x916A290: nc4d_open_file (ncdap4.c:122)
by 0x911CDDF: nc__open (nc4file.c:2407)
by 0x69E85F8: NcFile::NcFile(char const*, NcFile::FileMode, unsigned long*, unsigned long, NcFile::FileFormat) (netcdf.cpp:384)
by 0x710F0B8: getData(std::string const&) (ProjTimestampUtil.cc:593)
by 0x70E9BEA: (anonymous namespace)::parseOptions(int, char**) (ProjUtility.cc:190)
by 0x70EAAFB: main(int, char**) (ProjUtility.cc:243)
Address 0x1051 is not stack'd, malloc'd or (recently) free'd
Process terminating with default action of signal 11 (SIGSEGV)
Access not within mapped region at address 0x1051
at 0x67CF765: NC_var_shape (in /apps1/hdf-4.2.6/lib/libmfhdf.so.0.0.0)
by 0x91327CA: nc_get_NC (v1hpg.c:1113)
by 0x91303C0: l3nc__open_mp (nc.c:1096)
by 0x915B279: nc3d__open_mp (dapdispatch3.c:336)
by 0x914A752: nc3d_open (ncdap3.c:94)
by 0x911F8A2: l4nc_open_file (nc4file.c:2338)
by 0x916A290: nc4d_open_file (ncdap4.c:122)
by 0x911CDDF: nc__open (nc4file.c:2407)
by 0x69E85F8: NcFile::NcFile(char const*, NcFile::FileMode, unsigned long*, unsigned long, NcFile::FileFormat) (netcdf.cpp:384)
by 0x710F0B8: getData(std::string const&) (ProjTimestampUtil.cc:593)
by 0x70E9BEA: (anonymous namespace)::parseOptions(int, char**) (ProjUtility.cc:190)
by 0x70EAAFB: main(int, char**) (ProjUtility.cc:243)
Where else is my app getting path info when dynamically pulling in other libraries?

Answer 1

I'm not exactly sure of all the details of how -rpath and LD_LIBRARY_PATH work, and their precedence, but I did find some useful environment variables:
LD_DEBUG=all - This env variable turns on verbose dynamic linker debugging. Now doing an ldd on your app will spew output about the details of how all its dependencies find their dependencies.
LD_DEBUG_OUTPUT=<filename_prefix> - Used in conjunction with LD_DEBUG to specify output files to log the debugging info to.
The LD_DEBUG env variable helped me track down that /apps1/gdal-1.8.0-jasper/lib/libgdal.so.1 was compiled with an -rpath option that was pulling the old (wrong) versions of my libraries. It gave this helpful debug output:
search path=/pathXYZ/lib/tls/x86_64:/pathXYZ/lib/tls:/pathXYZ/lib/x86_64:
/pathABC/jasper/lib:/pathABC/hdf5/lib/tls/x86_64:/pathABC/hdf5/lib/tls:
/pathABC/hdf5/lib/x86_64:/pathABC/hdf5/lib:/pathABC/netcdf/lib/tls/x86_64:
/pathABC/netcdf/lib/tls:/pathABC/netcdf/lib/x86_64:/pathABC/netcdf/lib
(RPATH from file /apps1/gdal-1.8.0-jasper/lib/libgdal.so.1)
So the rpath of how the GDAL library was compiled seemed to be making an end-run around my LD_LIBRAR_PATH. Until I can get my lab team to rebuild libgdal correctly, I found this env var, which helped me load the "right" library versions that I wanted:
LD_PRELOAD=<path/to/libName.so> - Point this to the location of a library (or a space-separated list of libraries) that should be loaded before all others. See the ld.so man page.