So I've edited my /etc/ld.so.conf file and now it looks like this:
include /opt/intel/mkl/lib/intel64_lin
include /etc/ld.so.conf.d/*.conf
include /usr/local/lib
I've tried running sudo ldconfig -v, but in the output it does not seem to even attempt to load the MKL library directory. When I try to run a program using MKL it fails. However, if I add a symbolic link to the correct library in /usr/local/lib which gets added from /etc/ld.so.conf.d/libc.conf it works. Obviously this is an easy solution, but I'm trying to understand why my ldconfig fails?
You have to add library or directory itself:
/full/Path/to/library.so/or/directory to /etc/ld.so.conf file
You must remove include word before directory in your config file.
From man ldconfig:
The ldconfig utility is used to prepare a set of ``hints'' for use
by the
dynamic linker to facilitate quick lookup of shared libraries available
in multiple directories.
<...>
Files named on the command line are expected to contain directories to
scan for shared libraries. Each directory's pathname must start on a new
line. Blank lines and lines starting with the comment character `#' are
ignored. Filenames must conform to the lib*.so.[0-9] pattern in order to
be added to the hints file.
Related
So I got several shared libraries that I am trying to permanently install on my Ubuntu system but I am having some difficulty with it.
I want to install the libraries and the headers in a separate folder under /usr/local/lib and /usr/local/include (for example a folder named agony) so it would be clean and removing them would just require that I delete those folders. so it looks something like this:
/usr/local/lib/agony/libbtiGPIO.so
/usr/local/lib/agony/libbtiDSP.so
...
/usr/local/include/agony/GPIO.h
/usr/local/include/agony/DSP.h
...
And I added a file here /etc/ld.so.conf.d/agony.conf which include a line describing the path to the library folder:
$ cat /etc/ld.so.conf.d/agony.conf
/usr/local/lib/agony
and I perform sudo ldconfig to update the library database.
So to double check if the library is found I do ldconfig -p | grep bti* and
I see the following result:
$ ldconfig -p | grep bti
...
libbtiGPIO.so (libc6,x86-64) => /usr/local/lib/agony/libbtiGPIO.so
libbtiDSP.so (libc6,x86-64) => /usr/local/lib/agony/libbtiDSP.so
...
At this point I should be able to use the libraries without specifying the library path. But When I attempt to compile an application without providing the library path (-L) it fails. However, when I supply gcc with the library path ex:
gcc source.c -L /usr/local/lib/agony output -lbtiGPIO -lbtiDSP
it works!!
I don't want to use LD_LIBRARY_PATH environment variable because this library is going to be used everywhere on the system and I don't want other compilers to worry about providing LD_LIBRARY_PATH.
What am I doing wrong here?
At this point I should be able to use the libraries without specifying the library path
Here lies the confusion.
You have built your shared library libbtiGPIO.so (just sticking with that one),
placed it in /usr/local/lib/agony, and updated the ldconfig database accordingly.
The effect of that is when you run a program that has been linked with libbtiGPIO
then the dynamic linker (/lib/x86_64-linux-gnu/ld-2.21.so, or similar) will know where to look
to load that library into the process and you will not need to tell it by setting an LD_LIBRARY_PATH in the environment.
However, you haven't done anything that affects the list of default library
search directories that are hardwired into your build of gcc, that it passes to
the linker (/usr/bin/ld) when you link a program with libbtiGPIO in the first place.
That list of default search directories is what you will find if your do a verbose
build of your program - gcc -v ... - and then pick out the value of LIBRARY_PATH
from the output, e.g.
LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/5/:\
/usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/:\
/usr/lib/gcc/x86_64-linux-gnu/5/../../../../lib/:\
/lib/x86_64-linux-gnu/:\
/lib/../lib/:\
/usr/lib/x86_64-linux-gnu/:\
/usr/lib/../lib/:\
/usr/lib/gcc/x86_64-linux-gnu/5/../../../:\
/lib/:\
/usr/lib
/usr/local/lib/agony is not one of those and to make it one of those you
would have to build gcc from source yourself. Hence, in order to link your
program with libbtiGPIO you still need to tell ld where to find it with
-L/usr/local/lib/agony -lbtiGPIO.
man, you misunderstand the procedure of complier and link.
First, libbtiGPIO.so is a shared link library not a static link library. it is important to know those difference .
Then you need to know something else. changing ld.so.conf.d/*.conf and run sudo ldconfig, it affects the procedure of link. in other words, if you don't add agony.conf and sudo ldconfig, you will receive a error when you run ./a.out rather than gcc source.c -L ...., the gcc command can run successfully even thougth you don't ldconfig.
Finally,if you don't pollute the LD_LIBRARY_PATH environment variable, you have to add -L ... options in your gcc command. What'more, if you don't want to input too many words in your shell frequently, you can learn to use Makefile.
Edit: I resolved this issue, the solution is below.
I am building a code in a shared computing cluster dedicated for scientific computing, thus I can only control files in my home folder. Although I am using fftw as an example, I would like to understand the specific reason, why my attempt to setup LD_LIBRARY_PATH does not work.
I build the fftw and fftw_mpi libraries in my home folder like this
./configure --prefix=$HOME/install/fftw --enable-mpi --enable-shared
make install
It builds fine, but in install/fftw/lib, I find that the freshly built libfftw3_mpi.so links to wrong version of fftw library.
$ ldd libfftw3_mpi.so |grep fftw
libfftw3.so.3 => /usr/lib64/libfftw3.so.3 (0x00007f7df0979000)
If I now try to set the LD_LIBRARY_PATH correctly pointing to this directory, it still prefers the wrong library:
$ export LD_LIBRARY_PATH=$HOME/install/fftw/lib
$ ldd libfftw3_mpi.so |grep fftw
libfftw3.so.3 => /usr/lib64/libfftw3.so.3 (0x00007f32b4794000)
Only if I explicitly use LD_PRELOAD, I can override this behavior. I don't think LD_PRELOAD is a proper solution though.
$ export LD_PRELOAD=$HOME/install/fftw/lib/libfftw3.so.3
$ ldd libfftw3_mpi.so |grep fftw
$HOME/install/fftw/lib/libfftw3.so.3 (0x00007f5ca3d14000)
Here is what I would have expecting, a small test done in Ubuntu desktop, where I installed fftw to /usr/lib first, and then override this search path with LD_LIBRARY_PATH.
$ export LD_LIBRARY_PATH=
$ ldd q0test_mpi |grep fftw3
libfftw3.so.3 => /usr/lib/x86_64-linux-gnu/libfftw3.so.3
$ export LD_LIBRARY_PATH=$HOME/install/fftw-3.3.4/lib
$ ldd q0test_mpi |grep fftw3
libfftw3.so.3 => $HOME/install/fftw-3.3.4/lib/libfftw3.so.3
In short: Why is libfft3_mpi library still finding the wrong dynamic fftw3 library? Where is this searchpath hard coded in a such way that it is prioritized over LD_LIBARY_PATH? Why is this is not the case in another computer?
I am using intel compilers 13.1.2, mkl 11.0.4.183 and openmpi 1.6.2 if this matters.
Edit: Thanks for all the answers. With help of those, we were able to isolate the problem to RPATH, and from there, the cluster support was able to figure out the problem. I accepted the first answer, but both answers were good.
The reason, why this was so hard to figure out, is that we did not know that the compilers were actually wrapper scripts, adding things to compiler command line. Here a part of a reply from the support:
[The] compilation goes through our compiler wrapper. We do RPATH-ing
by default as it helps most users in correctly running their jobs
without loading LD-LIBRARY_PATH etc. However we exclude certain
library paths from default RPATH which includes /lib, /lib64 /proj
/home etc. Earlier the /usr/lib64 was not excluded by mistake
(mostly). Now we have added that path in the exclusion list.
From http://man7.org/linux/man-pages/man8/ld.so.8.html
When resolving shared object dependencies, the dynamic linker first
inspects each dependency string to see if it contains a slash (this
can occur if a shared object pathname containing slashes was
specified at link time). If a slash is found, then the dependency
string is interpreted as a (relative or absolute) pathname, and the
shared object is loaded using that pathname.
If a shared object dependency does not contain a slash, then it is
searched for in the following order:
o (ELF only) Using the directories specified in the DT_RPATH dynamic
section attribute of the binary if present and DT_RUNPATH
attribute does not exist. Use of DT_RPATH is deprecated.
o Using the environment variable LD_LIBRARY_PATH. Except if the
executable is a set-user-ID/set-group-ID binary, in which case it
is ignored.
o (ELF only) Using the directories specified in the DT_RUNPATH
dynamic section attribute of the binary if present.
o From the cache file /etc/ld.so.cache, which contains a compiled
list of candidate shared objects previously found in the augmented
library path. If, however, the binary was linked with the -z
nodeflib linker option, shared objects in the default paths are
skipped. Shared objects installed in hardware capability
directories (see below) are preferred to other shared objects.
o In the default path /lib, and then /usr/lib. (On some 64-bit
archiectures, the default paths for 64-bit shared objects are
/lib64, and then /usr/lib64.) If the binary was linked with the
-z nodeflib linker option, this step is skipped.
with readelf readelf -d libfftw3_mpi.so you can check if your lib contains such a attribute in the dynamic section.
with export LD_DEBUG=libs you can debug the search path used to find your libs
with chrpath -r<new_path> <executable> the rpath can be changed
I see two possible reasons for this.
First, libfftw3_mpi.so may be linked with /usr/lib64/ as RPATH. In that case, providing LD_LIBRARY_PATH will have no effect. To check if it is your case, run readelf -d libfftw3_mpi.so | grep RPATH and see if it has /usr/lib64/ as a library path. If it does, use chrpath utility to change or remove it.
Alternatively, you may be running a system that does not support LD_LIBRARY_PATH at all (like HP-UX).
On my UNIX machine I observed that the binaries are looking for the dependent shared libraries in '/lib' directory by default even though the '/lib' directory is not configured in 'PATH' and 'LD_LIBRARY_PATH' variables.
I the below see that the library 'libssl.so.4' is found from the '/lib' directory.
bash-3.00$ ldd openssl
/lib/libcwait.so (0x00f86000)
libssl.so.4 => /lib/libssl.so.4 (0x00408000)
My 'PATH' and 'LD_LIBRARY_PATH' are below:
bash-3.00$ echo $LD_LIBRARY_PATH
:/opt/oracle/product/11.2.0/client32/lib:
bash-3.00$ echo $PATH
/opt/pure/releases/purify.hp.2003a.06.15.FixPack.0214/cache/opt/star-ncf-prod/ep_patch/usr/lib:/usr/ccs/bin:/usr/bin:/usr/ucb:/etc:/bin:.:/opt/ccm71/bin:/opt/oracle/product/11.2.0/client32/bin:/opt/tools/bin:/usr/local/bin
Please let mw know if the binaries refer to '/lib' directory by default ?
Read ld.so(8), ldd(1) and dlopen(3) man pages and Drepper's paper: How To Write Shared Libraries
You'll see that "If a library dependency does not contain a slash, then it is searched
for" at last in /lib then /usr/lib
These two directories are builtin, i.e. wired in the code of the dynamic linker, e.g. of /lib64/ld-linux-x86-64.so.2 on my Debian system (I guess -but I am not sure- that on my Debian /lib64/ is also wired in).
In order for an executable to find the required libraries to link with during run time, one must configure the system so that the libraries can be found. Methods available: (Do at least one of the following)
Add library directories to be included during dynamic linking to the file /etc/ld.so.conf
Sample: /etc/ld.so.conf
/usr/X11R6/lib
/usr/lib
...
/usr/lib/sane
/usr/lib/mysql
/opt/lib
Add the library path to this file and then execute the command (as root) ldconfig to configure the linker run-time bindings.
You can use the "-f file-name" flag to reference another configuration file if you are developing for different environments.
See man page for command ldconfig
OR
Add specified directory to library cache: (as root)
ldconfig -n /opt/lib
Where /opt/lib is the directory containing your library libctest.so
(When developing and just adding your current directory: ldconfig -n . Link with -L.)
This will NOT permanently configure the system to include this directory. The information will be lost upon system reboot.
OR
Specify the environment variable LD_LIBRARY_PATH to point to the directory paths containing the shared object library. This will specify to the run time loader that the library paths will be used during execution to resolve dependencies.
(Linux/Solaris: LD_LIBRARY_PATH, SGI: LD_LIBRARYN32_PATH, AIX: LIBPATH, Mac OS X: DYLD_LIBRARY_PATH, HP-UX: SHLIB_PATH)
Example (bash shell): export LD_LIBRARY_PATH=/opt/lib:$LD_LIBRARY_PATH or add to your ~/.bashrc file:
`...
if [ -d /opt/lib ];
then
LD_LIBRARY_PATH=/opt/lib:$LD_LIBRARY_PATH
fi
...
export LD_LIBRARY_PATH`
This instructs the run time loader to look in the path described by the environment variable LD_LIBRARY_PATH, to resolve shared libraries. This will include the path /opt/lib.
Library paths used should conform to the "Linux Standard Base" directory structure.
I'm trying run a program(Snort) that uses libdnet but it fails to find it and outputs:
snort: error while loading shared libraries: libdnet.1: cannot open
shared object file: No such file or directory
Now I know that I should add the library by running ldconfig and putting path to the library in /etc/ld.so.conf. libdnet is located in /usr/local/lib so I don't have to modify ld.so.conf since it already covers that dirctory. So I ran the following commands and tracing the output, I noticed my library is not being loaded.
ldconfig -v
Apparently ldconfig only loads files that have .so somewhere in their names and libdnet.1 doesn't match the pattern.
I've built libdnet from source and installed it using ./configure; make; make install commands. I'd rather not install it using the package manager unless I have to. What should I do?
EDIT:
It says here that libraries should match the patter lib*.so* but I can't rename the library. I neither made it nor am I using it in my own app: if I rename it it will be loaded but I think Snort is looking for libdnet.1 not libdnet.so.1.
Found the answer here. The Solution was simple: make a copy that matches the pattern.
cp /usr/local/lib/libdnet.1.0.1 /usr/local/lib/libdnet.so.1.0.1
A less preferred alternative:
$ LD_LIBRARY_PATH=/usr/local/lib
$ export LD_LIBRARY_PATH
I have a core file generated on a remote system that I don't have direct access to. I also have local copies of the library files from the remote system, and the executable file for the crashing program.
I'd like to analyse this core dump in gdb.
For example:
gdb path/to/executable path/to/corefile
My libraries are in the current directory.
In the past I've seen debuggers implement this by supplying the option "-p ." or "-p /=."; so my question is:
How can I specify that libraries be loaded first from paths relative to my current directory when analysing a corefile in gdb?
Start gdb without specifying the executable or core file, then type the following commands:
set solib-absolute-prefix ./usr
file path/to/executable
core-file path/to/corefile
You will need to make sure to mirror your library path exactly from the target system. The above is meant for debugging targets that don't match your host, that is why it's important to replicate your root filesystem structure containing your libraries.
If you are remote debugging a server that is the same architecture and Linux/glibc version as your host, then you can do as fd suggested:
set solib-search-path <path>
If you are trying to override some of the libraries, but not all then you can copy the target library directory structure into a temporary place and use the solib-absolute-prefix solution described above.
I'm not sure this is possible at all within gdb but then I'm no expert.
However I can comment on the Linux dynamic linker. The following should print the path of all resolved shared libraries and the unresolved ones.
ldd path/to/executable
We need to know how your shared libraries were linked with your executable. To do this, use the following command:
readelf -d path/to/executable | grep RPATH
Should the command print nothing, the dynamic linker will use standard locations plus the LD_LIBRARY_PATH environment variable to find the shared libraries.
If the command prints some lines, the dynamic linker will ignore LD_LIBRARY_PATH and use the hardcoded rpaths instead.
If the listed rpaths are absolute, the only solution I know is to copy (or symlink) your libraries to the listed locations.
If the listed rpaths are relative, they will contain a $ORIGIN which will be replaced at run time by the path of the executable. Move either the executable or the libraries to match.
For further informations, you could start with:
man ld.so
I found this excerpt on developer.apple.com
set solib-search-path path
If this variable is set, path is a
colon-separated list of directories to
search for shared libraries.
solib-search-path' is used after
solib-absolute-prefix' fails to
locate the library, or if the path to
the library is relative instead of
absolute. If you want to use
solib-search-path' instead of
solib-absolute-prefix', be sure to
set `solib-absolute-prefix' to a
nonexistant directory to prevent GDB
from finding your host's libraries.
EDIT:
I don't think using the above setting prepends the directories I added, but it does seem to append them, so files missing from my current system are picked up in the paths I added. I guess setting the solib-absolute-prefix to something bogus and adding directories in the solib-search-path in the order I need might be a full solution.
You can also just set LD_PRELOAD to each of the libraries or LD_LIBRARY_PATH to the current directory when invoking gdb. This will only cause problems if gdb itself tries to use any of the libraries you're preloading.
One important note:
if you're doing a cross compiling and trying to debug with gdb, then
after you've done file ECECUTABLE_NAME if you see smth. like :
Using host libthread_db library "/lib/libthread_db.so.1"
then check whether you have libthread_db for your target system. I found a lot of similar problems on the web. Such problem cannot be solved just using "set solib-", you has to build libthread_db using your cross-compiler as well.