Following question How do applications resolve to different versions of shared libraries at run time?, I wondered how to specify on the link command line which version of the library to use?
Let's say I have
libmy.so.1.0
libmy.so.1 -> libmy.so.1.0
libmy.so.2.0
libmy.so.2 -> libmy.so.2.0
libmy.so -> libmy.so.2
The usual way to specify the library to link with the executable does not show the version to use. Furthermore, it is very likely that one wants to link with the most recent version. Thus the usual line works fine in most cases.
gcc app.o -lmy -o app
What is the command line to link app that should use version 1 of the library?
The linker is able to accept filenames too
gcc app.o -l:libmy.so.1 -o app
From man ld:
-l namespec
--library=namespec
Add the archive or object file specified by namespec to the list of files to link. This option may be used any number of times. If
namespec is of the form :filename, ld will search the library path for a file called filename, otherwise it will search the library
path for a file called libnamespec.a.
I noticed that older versions do not support it, so check man ld -l or --library option on your system.
You could also link to the file mentioning its full name
gcc app.o /mylibpath/libmy.so.1 -o app
Related
I have started looking into the concepts of linking libraries with exes and working in Linux machine. I'm struggling to understand the concept of linking so files with executables.
app:$(CC) $(CFLAGS) $(LDFLAGS) app.o app_dep.o -L . -ldynamic -Wl,-rpath . \
-o app
I'm trying to create an executable app with the above lines in makefile. I have to link it with a libdynamic.so file which exists in the working directory. So i used -L flag and -rpath to point to the directory and name of the so file. It worked and executable is created.
But when i tried to run the executable, it again complains that libdynamic.so: cannot open shared object file: No such file or directory.
Why do i need it since i linked the sharedlibrary during the creation of executable itself?
If the answer to the question is "Yes, it is required to point to the lib even it is linked". How could i point to the folder where it presents during the execution of binary?
One way i found is using LD_LIBRARY_PATH environment variable. Is there any other way to do without environment variable?
Thanks
Why do i need it since i linked the sharedlibrary during the creation of executable itself?
Because linking against a shared library does not embed that shared library into the executable. That is the main difference between linking with a shared versus archive library.
How could i point to the folder where it presents during the execution of binary?
There are no "folders" in UNIX. They are directories.
The correct link command to make an executable look in current directory is:
gcc -o app app.o ... -L. -ldynamic -Wl,-rpath=.
Note that in general it's a really bad idea to do so: your executable will or will not run depending on your current directory (or it may use different versions of libdynamic.so depending on which directory you are in).
Your link command should have worked if you invoked app in the directory with libdynamic.so present (i.e. if you invoked it like ./app), but not if you used /path/to/app (and were not in the same directory).
A significantly better approach is to do this:
gcc -o app app.o ... -L. -ldynamic -Wl,-rpath='$ORIGIN'
That tells the app binary to look for libdynamic.so in the same directory in which app is located, regardless of your current directory.
A cleaned up version of my compile command looks like gcc -o semantic_seg -Wl,-rpath,... -l:libnvrtc-5e8a26c9.so.10.1 ... and I have a dynamic library file named exactly libnvrtc-5e8a26c9.so.10.1 in the directory specified by the -rpath flag. The command succeeds.
When I go to run my program, it says semantic_seg: error while loading shared libraries: libnvrtc.so.10.1: cannot open shared object file: No such file or directory and when I do ldd it shows libnvrtc.so.10.1 => not found.
So the problem looks like the name of the shared library in the executable is not the same as the filename. Could that be true? Did the 5e8a26c9 part of the name somehow get stripped off?
Update: Creating a symbolic link libnvrtc-5e8a26c9.so.10.1 -> libnvrtc.so.10.1 allows the executable to run. But I'm still not sure the mechanism that causes this name modification to happen. It seems a bit magic.
Could that be true?
This is often true.
On ELF systems, the linker uses SONAME of the library (if it has one) and not its filename to record a runtime dependency.
Running readelf -d libnvrtc-5e8a26c9.so.10.1 | grep SONAME will likely show that in fact that library does have SONAME, and the value of SONAME is libnvrtc.so.10.1.
This used to be very convenient for external library versioning.
Update:
it allows you to link against a library which will be different than the one which will used at run time, but why would I ever want that?
Like I said, it's useful for external library versioning and ABI evolution.
Suppose you ship version 1.0 of libfoo.so. You provide libfoo.so as a symlink to libfoo.so.1, and you use libfoo.so.1 as SONAME.
Any program that links with gcc main.c -lfoo (note: no funny -l:libfoo.so.1 syntax required) will record libfoo.so.1 as its external dependency, and will use that name at runtime.
Time passes, and you are ready to ship version 2, which is not ABI-compatible.
You don't want to cause all your end-users to change their link line, and you don't want to break any existing binaries.
With the SONAME, this is trivial: new package will include libfoo.so.2 with SONAME of libfoo.so.2, and a symlink libfoo.so now pointing to libfoo.so.2.
Voila: both requirements are achieved: existing binaries continue to use libfoo.so.1, newly-linked binaries use libfoo.so.2, no Makefile changes required.
The problem I faced has been solved here:
Loading shared library in open-mpi/ mpi-run
I know not how, setting LD_LIBRARY_PATH or specifying -x LD_LIBRARY_PATH fixes the problem, when my installation itself specifies the necessary -L arguments. My installation is in ~/mpi/
I have also included my compile-link configs.
$ mpic++ -showme:version
mpic++: Open MPI 1.6.3 (Language: C++)
$ mpic++ -showme
g++ -I/home/vigneshwaren/mpi/include -pthread -L/home/vigneshwaren/mpi/lib
-lmpi_cxx -lmpi -ldl -lm -Wl,--export-dynamic -lrt -lnsl -lutil -lm -ldl
$ mpic++ -showme:libdirs
/home/vigneshwaren/mpi/lib
$ mpic++ -showme:libs
mpi_cxx mpi dl m rt nsl util m dl % Notice mpi_cxx here %
When I compiled with mpic++ <file> and ran with mpirun a.out I got a (shared library) linker error
error while loading shared libraries: libmpi_cxx.so.1:
cannot open shared object file: No such file or directory
The error has been fixed by setting LD_LIBRARY_PATH. The question is how and why? What am i missing? Why is LD_LIBRARY_PATH required when my installation looks just fine.
libdl, libm, librt, libnsl and libutil are all essential system-wide libraries and they come as part of the very basic OS installation. libmpi and libmpi_cxx are part of the Open MPI installation and in your case are located in a non-standard location that must be explicitly included in the linker search path LD_LIBRARY_PATH.
It is possible to modify the configuration of the Open MPI compiler wrappers and make them pass the -rpath option to the linker. -rpath takes a library path and appends its to a list, stored inside the executable file, which tells the runtime link editor (a.k.a. the dynamic linker) where to search for libraries before it consults the LD_LIBRARY_PATH variable. For example, in your case the following option would suffice:
-Wl,-rpath,/home/vigneshwaren/mpi/lib
This would embed the path to the Open MPI libraries inside the executable and it would not matter if that path is part of LD_LIBRARY_PATH at run time or not.
To make the corresponding wrapper add that option to the list of compiler flags, you would have to modify the mpiXX-wrapper-data.txt file (where XX is cc, c++, CC, f90, etc.), located in mpi/share/openmpi/. For example, to make mpicc pass the option, you would have to modify /home/vigneshwaren/mpi/share/openmpi/mpicc-wrapper-data.txt and add the following to the line that starts with linker_flags=:
linker_flags= ... -Wl,-rpath,${prefix}/lib
${prefix} is automatically expanded by the wrapper to the current Open MPI installation path.
In my case, I just simply appends
export LD_LIBRARY_PATH=/PATH_TO_openmpi-version/lib:$LD_LIBRARY_PATH
For example
export LD_LIBRARY_PATH=/usr/local/openmpi-1.8.1/lib:$LD_LIBRARY_PATH
into $HOME/.bashrc file and then source it to active again source $HOME/.bashrc.
I installed mpich 3.2 using the following command on Ubuntu.
sudo apt-get install mpich
When I tried to run the mpi process using mpiexec, I got the same error.
/home/node1/examples/.libs/lt-cpi: error while loading shared libraries: libmpi.so.0: cannot open shared object file: No such file or directory
Configuring LD_LIBRARY_PATH didn't fix my problem.
I did a search for the file 'libmpi.so.0' on my machine but couldn't find it. Took me some time to figure out that 'libmpi.so.0' file is named as 'libmpi.so' on my machine. So I renamed it to 'libmpi.so.0'.
It solved my problem!
If you are having the same problem and you installed the library through apt-get, then do the following.
The file 'libmpi.so' should be in the location '/usr/lib/'. Rename the file to 'libmpi.so.0'
mv /usr/lib/libmpi.so /usr/lib/libmpi.so.0
After that MPI jobs should run without any problem.
If 'libmpi.so' is not found in '/usr/lib', you can get its location using the following command.
whereis libmpi.so
first, run this command
$ sudo apt-get install libcr-dev
if still have this problem then configure LD_LIBRARY_PATH like this:
export LD_LIBRARY_PATH=/usr/local/mpich-3.2.1/lib:$LD_LIBRARY_PATH
then add it to ~/.bashrc before this line:
[ -z "$PS1" ] && return
Simply running
$ ldconfig
appears to me as a better way to solve the problem (taken from a comment on this question). In particular, since it avoids misuse of the LD_LIBRARY_PATH environment variable. See here and here, for why I believe it's misused to solve the problem at hand.
I learned the "Program Library HOWTO". It mention that using soname to manage the version like follow.
gcc -shared -fPIC -Wl,-soname,libfoo.so.1 -o libfoo.so.1.0.0 foo.c
ln -s libfoo.so.1.0.0 libfoo.so.1
ln -s libfoo.so.1 libfoo.so
And I get the information that if the soname is not set. it will be equal to libfoo.so.1.0.0 ,see the answer from here.
And I find that it also can work without soname , like following
gcc -shared -fPIC -o libfoo.so.1.0.0 foo.c
ln -s libfoo.so.1.0.0 libfoo.so.1
ln -s libfoo.so.1 libfoo.so
So I think that the only one useful point is that the soname option can tell you the version of the shared library when you use readelf -d libfoo.so command to check it.
What else can it do?
soname is used to indicate what binary api compatibility your library support.
SONAME is used at compilation time by linker to determine from the library file what actual target library version. gcc -lNAME will seek for libNAME.so link or file then capture its SONAME that will certainly be more specific ( ex libnuke.so links to libnuke.so.0.1.4 that contains SONAME libnuke.so.0 ).
At run time it will link with this is then set into ELF dynamic section NEEDED, then a library with this name ( or a link to it ) should exists.
At Run time SONAME is disregarded, so only the link or the file existence is enough.
Remark: SONAME is enforced only at link/build time and not at run time.
'SONAME' of library can be seen with 'objdump -p file |grep SONAME'.
'NEEDED' of binaries can be seen with 'objdump -p file |grep NEEDED'.
[EDIT] WARNING Following is a general remark, not the one deployed in linux. See at the end.
Let's assume you have a library with libnuke.so.1.2 name and you develop a new libnuke library :
if your new library is a fix from previous without api change, you should just keep same soname, increase the version of filename. ie file will be libnuke.so.1.2.1 but soname will still be libnuke.so.1.2.
if you have a new library that only added new function but didn't break functionality and is still compatible with previous you would like to use same soname than previous plus a new suffix like .1. ie file and soname will be libnuke.so.1.2.1. Any program linked with libnuke.1.2 will still work with that one. New programs linked with libnuke.1.2.1 will only work with that one ( until new subversion come like libnuke.1.2.1.1 ).
if your new library is not compatible with any libnuke : libnuke.so.2
if your new library is compatible with bare old version : libnuke.so.1.3 [ ie still compatible with libnuke.so.1 ]
[EDIT] to complete : linux case.
In linux real life SONAME as a specific form :
lib[NAME][API-VERSION].so.[major-version]
major-version is only one integer value that increase at each major library change.
API-VERSION is empty by default
ex libnuke.so.0
Then real filename include minor versions and subversions ex : libnuke.so.0.1.5
I think that not providing a soname is a bad practice since renaming of file will change its behavior.
You created a dynamic library named libx.1.0.0 in naming tradition libname.{a}.{b}.{c}
{a} stand for primary version, should changes when APIs changes(which making things incompatible).
{b} stand for sub version, should changes by adding APIs.
{c} stand for mirror version, should changes by bug fixing or optimizing
Now you are releasing libx.1.2.0, and you need to declare that libx.1.2.0 is compatible with libx.1.0.0 since just adding functions and people's executable would not crashes, just link it like in the old time by:
Setting libx.1.0.0 and libx.1.2.0 to have the same soname, for example libx.1
This's what soname does.
Here is an example supporting Johann Klasek's answer.
In short, SONAME is needed at run time.
At compilation time, only a linker name or real name is needed (e.g. g++ main.cpp -L. -ladd or g++ main.cpp -L. -l:libadd.so.1.1). The definitons of linker name and real name follow Program Library HOWTO: 3. Shared Libraries.
Source tree:
├── add.cpp
├── add.h
├── main.cpp
└── Makefile
Makefile:
SOURCE_FILE=add.cpp
# main.cpp includes `add.h`, whose implementation is `add.cpp`
MAIN_FILE=main.cpp
SONAME=libadd.so.1
REAL_NAME=libadd.so.1.1
LINKER_NAME=libadd.so
OUTPUT_FILE=a.out
all:
g++ -shared -fPIC -Wl,-soname,${SONAME} -o ${REAL_NAME} ${SOURCE_FILE}
ln -s ${REAL_NAME} ${LINKER_NAME}
g++ main.cpp -I. -L. -ladd -o ${OUTPUT_FILE}
# Same as `ldconfig -n .`, creates a symbolic link
ln -s ${REAL_NAME} ${SONAME}
#./a.out: error while loading shared libraries: libadd.so.1: cannot open
# shared object file: No such file or directory
LD_LIBRARY_PATH=. ./${OUTPUT_FILE}
clean:
rm ${SONAME} ${REAL_NAME} ${LINKER_NAME} ${OUTPUT_FILE}
Let's assume libA.so depends on libB.so, and they all in a directory(of course the directory cannot be found by dynamic linker). If you didn't set soname then dlopen doesn't work:
auto pB = dlopen("./libB.so", RTLD_LAZY | RTLD_GLOBAL);
auto pA = dlopen("./libA.so", RTLD_LAZY | RTLD_GLOBAL);
Because runtime linker cannot find libB.so, so pA is set to NULL.
In this case soname will save you from hell...
Another aspect:
At least on Linux the SONAME entry provides a hint for the runtime-linker system on how to create appropriate links in /lib, /lib64 etc.
Running the command ldconfig tries to create a symbolic link named with SONAME which is also taken into the run-time linker cache. The newest one of the libraries tagging the same SONAME wins the link-race.
If some software relies on the specific SONAME and you want to renew a library you have to provide this SONAME to get ldconfig stick on this new library (if ldconfig is used to rebuild the cache and the links). E.g. libssl.so.6 and libcrypto.so.6 are such cases.
I've a shell on a system without root privileges. I am trying to use a custom library for my new project and it cannot be installed onto the system because I don't have the root privilege. I'm building the library from source. Making the '.o' from the sources has been done. I've tried passing the '.o' file, generated after building the source, as the library argument (-l) to gcc , but gcc says file not found. Any possible workarounds for this?
Just pass the .o as an extra bit just like the rest of your program.
gcc <library.o> <yourprogram.o> -o <executable>
gcc -L/path/to/library/directory
LD_LIBRARY_PATH=/path/to/library/directory:$LD_LIBRARY_PATH ./a.out