I have built some shared libraries on Ubuntu Linux 16.0.2 from source.
They are 64-bit libs.
I manually copied them to /usr/local/lib.
I verified that the /usr/local/lib path is indeed in one of the .conf files that ld.so.conf includes.
I then ran: sudo ldconfig to update the cache.
But then when I try to run my executable which tries to dynamically load one of the .so files that I previously copied into /usr/local/lib using dlopen, it fails.
In my code, I have:
dlopen ("foobar.so", RTLD_LAZY);
Can anybody tell me what I am doing wrong?
The dynamic linker normally doesn't access the paths recursively included from /etc/ld.so.conf directly, but it uses a cache.
You can update the cache with
sudo ldconfig
See ldconfig(8) for more details.
for dlopen to work, there's no directory list of places to find the shared object. So doing dlopen("somefile", ...); probably won't work.
You don't need to use any path or to put the shared object (or to comply with naming conventions) to use a shared object via dlopen(3). That's only a requirement of the dynamic linker that loads and links all the shared libraries at launch time: linux-vdso.so.1 (in 64bit)
To test, just put the shared in your local dir and try to open it with it's basename, like you post.
For a system library, there are more requirements, like defining a soname for the library, which is used by the loader to load the library and to construct the cache database index, so if you have no idea of what I'm talking about, you will not be able to use automatic loading procedure. If you want to see if an executable file has all the libraries it needs and where the loader finds them out, just run ldd(1) with the executable as argument, and you'll se the dependencies for automatic loading and how the dynamic linker resolves the paths.
Related
A cleaned up version of my compile command looks like gcc -o semantic_seg -Wl,-rpath,... -l:libnvrtc-5e8a26c9.so.10.1 ... and I have a dynamic library file named exactly libnvrtc-5e8a26c9.so.10.1 in the directory specified by the -rpath flag. The command succeeds.
When I go to run my program, it says semantic_seg: error while loading shared libraries: libnvrtc.so.10.1: cannot open shared object file: No such file or directory and when I do ldd it shows libnvrtc.so.10.1 => not found.
So the problem looks like the name of the shared library in the executable is not the same as the filename. Could that be true? Did the 5e8a26c9 part of the name somehow get stripped off?
Update: Creating a symbolic link libnvrtc-5e8a26c9.so.10.1 -> libnvrtc.so.10.1 allows the executable to run. But I'm still not sure the mechanism that causes this name modification to happen. It seems a bit magic.
Could that be true?
This is often true.
On ELF systems, the linker uses SONAME of the library (if it has one) and not its filename to record a runtime dependency.
Running readelf -d libnvrtc-5e8a26c9.so.10.1 | grep SONAME will likely show that in fact that library does have SONAME, and the value of SONAME is libnvrtc.so.10.1.
This used to be very convenient for external library versioning.
Update:
it allows you to link against a library which will be different than the one which will used at run time, but why would I ever want that?
Like I said, it's useful for external library versioning and ABI evolution.
Suppose you ship version 1.0 of libfoo.so. You provide libfoo.so as a symlink to libfoo.so.1, and you use libfoo.so.1 as SONAME.
Any program that links with gcc main.c -lfoo (note: no funny -l:libfoo.so.1 syntax required) will record libfoo.so.1 as its external dependency, and will use that name at runtime.
Time passes, and you are ready to ship version 2, which is not ABI-compatible.
You don't want to cause all your end-users to change their link line, and you don't want to break any existing binaries.
With the SONAME, this is trivial: new package will include libfoo.so.2 with SONAME of libfoo.so.2, and a symlink libfoo.so now pointing to libfoo.so.2.
Voila: both requirements are achieved: existing binaries continue to use libfoo.so.1, newly-linked binaries use libfoo.so.2, no Makefile changes required.
I am currently working on a userland ELF File loader in C. LD_LIBRARY_PATH does not seem to be an option for me, as it does not seem to be set by default on my system(x86_64 openSUSE). What is the best way to get all of the directories where libraries are stored ?
/usr/lib64 and /lib64 for 64-bit binaries or /usr/lib and /lib for 32-bit binaries, than paths taken from /etc/ld.so.conf and included configs
From man ldconfig
ldconfig creates the necessary links and cache to the most recent shared libraries found in the directories specified on the command line, in the file /etc/ld.so.conf, and in the trusted directories, /lib and /usr/lib (on some 64-bit architectures such as x86-64, /lib and /usr/lib are the trusted directories for 32-bit libraries, while /lib64 and /usr/lib64 are used for 64-bit libraries).
The cache is used by the run-time linker, ld.so or ld-linux.so.
...
/etc/ld.so.conf File containing a list of directories, one per line, in which to search for libraries.
Note that this info is for openSUSE, other distros may use different paths.
LD_LIBRARY_PATH is the standard environment variable used for users to add and load their own libraries when they are unable or have no access to the system directories to install shared libraries.
There's a file which is normally read by ldconfig at boot time (it reads /etc/ld.so.conf to create a binary DBMsomewhat file /etc/ld.so.cache, with a hash table to quick access the paths to use when loading library shared objects, and that is used by the dynamic loader (there's only one such thing, as a kernel tool, so it is not dependant on which distribution you run, but just on the kernel version you use ---it has changed somewhat, but not as much as the kernel does---)
To know which sonames (soname is the common name used by a shared object to refer to an interface, which is what is required to warranty that a shared object will be compatible with the library) are being used by the dynamic loader, just run
ldconfig -p
and you'll get all the sonames registered, and the path to the library actually loaded for that soname.
If you want to know which libraries will be loaded by some specific executable by the dynamic loader, just execute this:
ldd your_executable
and It will print the sonames that executable needs and where in the system they are located.
What ldconfig(8) does, is to search al the directories included in the file /etc/ld.so.conf for shared object files, and look all the ones whose name matches the soname stored in the file, and include a reference to the file named for the soname found. Once the table is completed, the file /etc/ld.so.cache is created and used by /lib64/ld-linux-x86-64.so.2 which is the shared module in charge of user mode loading the rest of shared libraries used by a program.
There's no problem in having a local $HOME/lib directory to store your locally developed shared libraries, but as that directory will not be normally included in /etc/ld.so.conf, you'll need to create LD_LIBRARY_PATH=${HOME}/lib and be careful to export it, and never try to use it as root user, as for root user that env variable is disabled.
EDIT 1
By the way, if you need to load on demand a shared library (this is possibly what you need, probably), read about dlopen(3) and friend functions, as that is the method used by most programs to dynamically load modules you haven't heard about before compilation of main program. You'll need to load the module, look for the symbols you need (dlsym(3) or dlfunc(3)) store the references given by the module, and finally call them.
What I'd like to do is configure my CMakeLists file so that while building my project the linker uses a copy of a shared library (.so) that resides in my build tree to link the executable against but then does not set the rpath in the linked executable so that the system must provide the library when the loader requests it.
Specifically, I want to link against libOpenCL.so during build time on a build farm that doesn't have libOpenCL.so installed as a system library. To do this, libOpenCL.so is in the project build tree and referenced using an absolute path in the CMakeLists file. This absolute path is to ensure that if the system does happen to have libOpenCL.so installed then it is not used.
However, when running the final executable, CMake has added the absolute path to the rpath which stops the system version of libOpenCL.so being picked up by the library loader and used.
Seems simple but I can't quite figure it out.
Thanks!
I know this answer is super late. I faced the same requirement as yours.
Either we need is whitelist approach where we set CMAKE_BUILD_RPATH explicitly with what we need. Or we need a blacklist approach where we tell cmake, which RPATHs we don't want in the executable. Way to remove RPath from build tree is not documented yet: https://gitlab.kitware.com/cmake/cmake/issues/16825
The solution I took is:
Set RUNPATH instead of RPATH. You can achieve this by the statement:
SET(CMAKE_EXE_LINKER_FLAGS "-Wl,--enable-new-dtags")
When RUNPATH is present, RPATH is ignored.
RUNPATH - same as RPATH, but searched after LD_LIBRARY_PATH, supported only on most recent UNIX
Then I can achieve the overriding the library using the environment variable LD_LIBRARY_PATH.
According to the CMake Wiki this should not be a problem:
By default if you don't change any RPATH related settings, CMake will link the executables and shared libraries with full RPATH to all used libraries in the build tree. When installing, it will clear the RPATH of these targets so they are installed with an empty RPATH.
So you might try to simply install it?
My project includes a library and example projects for how to use it. I place the library in the "bin" folder along with all executable examples. I can run the example projects on the machine where they were compiled but when I try to run them on another machine I get:
./example: error while loading shared libraries: libMyLib.so: cannot
open shared object file: No such file or directory
This makes no sense since the library is in the same folder. What is causes it to ignore the library on other machines?
Just because the library is in the same directory as the executable doesn't mean it will look there for it. By default on linux, executables will only look in a limited set of directories, set by ldconfig and the LD_LIBRARY_PATH environment variable.
One trick that is very useful is to link your program with the extra linker option
-Wl,-rpath,'$ORIGIN'
which will cause the executable to also look in the directory the executable is in for shared objects.
You can usually set this by adding to your Makefile:
LDFLAGS := -Wl,-rpath,'$$ORIGIN'
Note the double-$ here -- make will interpret this as a make variable which expands to just $
The current directory is not necessarily a place where the dynamic linker will look for dynamic libraries. The directory where the executable is much less.
You might want to check ldconfig to see where it looks for them.
I created a shared library, libsslab_core.so.1.0.0 using gcc with proper options. I am pretty sure that the shared library works because I already linked it to another source code (I explicitly tells a compiler, gcc, the location of the library using -l option of the compiler).
After testing the library works, I tried to integrate the library into my Linux machine. I went to the /etc/ld.so.conf.d/ and added a file, sslab.conf. In the file I just typed the absolute path of the library, /opt/lib/sslab. Next, I executed ldconfig as a root to update the cache file of ldconfig. And I checked if the system finds the newly added library by typing ldconfig -p | grep libsslab. My Linux machine found the library, so I thought everything is finished.
However, when I try to compile a source code using the library, it gives me the following error:
/usr/bin/ld: cannot find -lsslab_core
When I move the library to /usr/local/lib, update the content of sslab.conf, and execute ldconfig as a root. I can use the shared library without any problems.
Do you have any ideas about the problem that I've come across on Ubuntu 12.04?
For your information, I refer to a document in TLDP to generate my own shared library. Here is the link: http://www.tldp.org/HOWTO/Program-Library-HOWTO/