Understanding target_link_libraries - linux

I am new to Linux. I see in my CMakeLists.txt the following;
target_link_libraries(app wiringPi
serializer
iothub_client
iothub_client_mqtt_transport
umqtt
aziotsharedutil
ssl
crypto
curl
pthread
m
ssl
crypto)
My question/understanding is; by doing this
Are we telling CMake tool to build wiringPi, serializer etc. and link with app(which is an executable created in my code)?
Where are all these libraries located?
When I do ldconfig -p | grep <libraryname>, for the above libraries
sometimes I find outputs like the following and sometimes nothing, why is that?
Is target_link_libraries smart enough to look for libraries under
the sub-directories too? I mean I see that some are just there under
user/lib and some are one more level under such as
/usr/lib/arm-linux-gnueabihf
pi#raspberrypi:~ $ ldconfig -p | grep curl
libcurl.so.4 (libc6,hard-float) => /usr/lib/arm-linux-gnueabihf/libcurl.so.4
libcurl-gnutls.so.4 (libc6,hard-float) => /usr/lib/arm-linux-gnueabihf/libcurl-gnutls.so.4
pi#raspberrypi:~ $ ldconfig -p | grep wiringPi
libwiringPiDev.so (libc6,hard-float) => /usr/local/lib/libwiringPiDev.so
libwiringPiDev.so (libc6,hard-float) => /usr/lib/libwiringPiDev.so
libwiringPi.so (libc6,hard-float) => /usr/local/lib/libwiringPi.so
libwiringPi.so (libc6,hard-float) => /usr/lib/libwiringPi.so

Are we telling CMake tool to build wiringPi, serializer etc. and link with app(which is an executable created in my code)?
Not exactly. That command tells CMake that the libraries wiringPi, serializer, etc. must be linked to target 'app' during the link stage for that target. It says nothing about building the libraries themselves, and usually they are expected to already be available rather than built.
Where are all these libraries located?
It can vary. The compiler has a default list of directories in which it looks for libraries. Other CMake commands can add link options that add directories to that list.
When I do ldconfig -p | grep <libraryname>, for the above libraries sometimes I find outputs like the following and sometimes
nothing, why is that?
ldconfig reports on libraries known to the dynamic linker. This is distinct from the linker that runs at compile time, and the directories and libraries the two know about are not necessarily the same. Reasons why ldconfig might not list a given library include:
The library is not installed.
Only a static version of the library is installed.
The library is not in any of the locations that the dynamic linker checks by default (additional directories can be specified when a program is launched, in at least two different ways).
Is target_link_libraries smart enough to look for libraries under the sub-directories too? I mean I see that some are just there under user/lib and some are one more level under such as /usr/lib/arm-linux-gnueabihf
This is not a function of CMake, but rather of the chosen toolchain and its configuration (on Linux, that's often the GNU toolchain, featuring GCC). It is usually safe to assume that the toolchain uses all the right standard library directories by default. CMake sometimes can successfully be instructed to search for specific libraries in other likely places, too, but target_link_libraries is not part of that.

When you pass to target_link_libraries a plain name (not a path) which is not a target, CMake just transforms this name into the linker flag. E.g. on Linux this is flag
-l<library-name>
So questions about searching the library you may address directly to the linker - CMake is out of the game here.

Related

shared library not found during compilation

So I got several shared libraries that I am trying to permanently install on my Ubuntu system but I am having some difficulty with it.
I want to install the libraries and the headers in a separate folder under /usr/local/lib and /usr/local/include (for example a folder named agony) so it would be clean and removing them would just require that I delete those folders. so it looks something like this:
/usr/local/lib/agony/libbtiGPIO.so
/usr/local/lib/agony/libbtiDSP.so
...
/usr/local/include/agony/GPIO.h
/usr/local/include/agony/DSP.h
...
And I added a file here /etc/ld.so.conf.d/agony.conf which include a line describing the path to the library folder:
$ cat /etc/ld.so.conf.d/agony.conf
/usr/local/lib/agony
and I perform sudo ldconfig to update the library database.
So to double check if the library is found I do ldconfig -p | grep bti* and
I see the following result:
$ ldconfig -p | grep bti
...
libbtiGPIO.so (libc6,x86-64) => /usr/local/lib/agony/libbtiGPIO.so
libbtiDSP.so (libc6,x86-64) => /usr/local/lib/agony/libbtiDSP.so
...
At this point I should be able to use the libraries without specifying the library path. But When I attempt to compile an application without providing the library path (-L) it fails. However, when I supply gcc with the library path ex:
gcc source.c -L /usr/local/lib/agony output -lbtiGPIO -lbtiDSP
it works!!
I don't want to use LD_LIBRARY_PATH environment variable because this library is going to be used everywhere on the system and I don't want other compilers to worry about providing LD_LIBRARY_PATH.
What am I doing wrong here?
At this point I should be able to use the libraries without specifying the library path
Here lies the confusion.
You have built your shared library libbtiGPIO.so (just sticking with that one),
placed it in /usr/local/lib/agony, and updated the ldconfig database accordingly.
The effect of that is when you run a program that has been linked with libbtiGPIO
then the dynamic linker (/lib/x86_64-linux-gnu/ld-2.21.so, or similar) will know where to look
to load that library into the process and you will not need to tell it by setting an LD_LIBRARY_PATH in the environment.
However, you haven't done anything that affects the list of default library
search directories that are hardwired into your build of gcc, that it passes to
the linker (/usr/bin/ld) when you link a program with libbtiGPIO in the first place.
That list of default search directories is what you will find if your do a verbose
build of your program - gcc -v ... - and then pick out the value of LIBRARY_PATH
from the output, e.g.
LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/5/:\
/usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/:\
/usr/lib/gcc/x86_64-linux-gnu/5/../../../../lib/:\
/lib/x86_64-linux-gnu/:\
/lib/../lib/:\
/usr/lib/x86_64-linux-gnu/:\
/usr/lib/../lib/:\
/usr/lib/gcc/x86_64-linux-gnu/5/../../../:\
/lib/:\
/usr/lib
/usr/local/lib/agony is not one of those and to make it one of those you
would have to build gcc from source yourself. Hence, in order to link your
program with libbtiGPIO you still need to tell ld where to find it with
-L/usr/local/lib/agony -lbtiGPIO.
man, you misunderstand the procedure of complier and link.
First, libbtiGPIO.so is a shared link library not a static link library. it is important to know those difference .
Then you need to know something else. changing ld.so.conf.d/*.conf and run sudo ldconfig, it affects the procedure of link. in other words, if you don't add agony.conf and sudo ldconfig, you will receive a error when you run ./a.out rather than gcc source.c -L ...., the gcc command can run successfully even thougth you don't ldconfig.
Finally,if you don't pollute the LD_LIBRARY_PATH environment variable, you have to add -L ... options in your gcc command. What'more, if you don't want to input too many words in your shell frequently, you can learn to use Makefile.

Why I cannot override search path of dynamic libraries with LD_LIBRARY_PATH?

Edit: I resolved this issue, the solution is below.
I am building a code in a shared computing cluster dedicated for scientific computing, thus I can only control files in my home folder. Although I am using fftw as an example, I would like to understand the specific reason, why my attempt to setup LD_LIBRARY_PATH does not work.
I build the fftw and fftw_mpi libraries in my home folder like this
./configure --prefix=$HOME/install/fftw --enable-mpi --enable-shared
make install
It builds fine, but in install/fftw/lib, I find that the freshly built libfftw3_mpi.so links to wrong version of fftw library.
$ ldd libfftw3_mpi.so |grep fftw
libfftw3.so.3 => /usr/lib64/libfftw3.so.3 (0x00007f7df0979000)
If I now try to set the LD_LIBRARY_PATH correctly pointing to this directory, it still prefers the wrong library:
$ export LD_LIBRARY_PATH=$HOME/install/fftw/lib
$ ldd libfftw3_mpi.so |grep fftw
libfftw3.so.3 => /usr/lib64/libfftw3.so.3 (0x00007f32b4794000)
Only if I explicitly use LD_PRELOAD, I can override this behavior. I don't think LD_PRELOAD is a proper solution though.
$ export LD_PRELOAD=$HOME/install/fftw/lib/libfftw3.so.3
$ ldd libfftw3_mpi.so |grep fftw
$HOME/install/fftw/lib/libfftw3.so.3 (0x00007f5ca3d14000)
Here is what I would have expecting, a small test done in Ubuntu desktop, where I installed fftw to /usr/lib first, and then override this search path with LD_LIBRARY_PATH.
$ export LD_LIBRARY_PATH=
$ ldd q0test_mpi |grep fftw3
libfftw3.so.3 => /usr/lib/x86_64-linux-gnu/libfftw3.so.3
$ export LD_LIBRARY_PATH=$HOME/install/fftw-3.3.4/lib
$ ldd q0test_mpi |grep fftw3
libfftw3.so.3 => $HOME/install/fftw-3.3.4/lib/libfftw3.so.3
In short: Why is libfft3_mpi library still finding the wrong dynamic fftw3 library? Where is this searchpath hard coded in a such way that it is prioritized over LD_LIBARY_PATH? Why is this is not the case in another computer?
I am using intel compilers 13.1.2, mkl 11.0.4.183 and openmpi 1.6.2 if this matters.
Edit: Thanks for all the answers. With help of those, we were able to isolate the problem to RPATH, and from there, the cluster support was able to figure out the problem. I accepted the first answer, but both answers were good.
The reason, why this was so hard to figure out, is that we did not know that the compilers were actually wrapper scripts, adding things to compiler command line. Here a part of a reply from the support:
[The] compilation goes through our compiler wrapper. We do RPATH-ing
by default as it helps most users in correctly running their jobs
without loading LD-LIBRARY_PATH etc. However we exclude certain
library paths from default RPATH which includes /lib, /lib64 /proj
/home etc. Earlier the /usr/lib64 was not excluded by mistake
(mostly). Now we have added that path in the exclusion list.
From http://man7.org/linux/man-pages/man8/ld.so.8.html
When resolving shared object dependencies, the dynamic linker first
inspects each dependency string to see if it contains a slash (this
can occur if a shared object pathname containing slashes was
specified at link time). If a slash is found, then the dependency
string is interpreted as a (relative or absolute) pathname, and the
shared object is loaded using that pathname.
If a shared object dependency does not contain a slash, then it is
searched for in the following order:
o (ELF only) Using the directories specified in the DT_RPATH dynamic
section attribute of the binary if present and DT_RUNPATH
attribute does not exist. Use of DT_RPATH is deprecated.
o Using the environment variable LD_LIBRARY_PATH. Except if the
executable is a set-user-ID/set-group-ID binary, in which case it
is ignored.
o (ELF only) Using the directories specified in the DT_RUNPATH
dynamic section attribute of the binary if present.
o From the cache file /etc/ld.so.cache, which contains a compiled
list of candidate shared objects previously found in the augmented
library path. If, however, the binary was linked with the -z
nodeflib linker option, shared objects in the default paths are
skipped. Shared objects installed in hardware capability
directories (see below) are preferred to other shared objects.
o In the default path /lib, and then /usr/lib. (On some 64-bit
archiectures, the default paths for 64-bit shared objects are
/lib64, and then /usr/lib64.) If the binary was linked with the
-z nodeflib linker option, this step is skipped.
with readelf readelf -d libfftw3_mpi.so you can check if your lib contains such a attribute in the dynamic section.
with export LD_DEBUG=libs you can debug the search path used to find your libs
with chrpath -r<new_path> <executable> the rpath can be changed
I see two possible reasons for this.
First, libfftw3_mpi.so may be linked with /usr/lib64/ as RPATH. In that case, providing LD_LIBRARY_PATH will have no effect. To check if it is your case, run readelf -d libfftw3_mpi.so | grep RPATH and see if it has /usr/lib64/ as a library path. If it does, use chrpath utility to change or remove it.
Alternatively, you may be running a system that does not support LD_LIBRARY_PATH at all (like HP-UX).

Build OpenSSL with RPATH?

I have Ubuntu 14.04. It came with openssl 1.0.1f. I want to install another openssl version (1.0.2) and I want to compile it by myself.
I configure it as follows:
LDFLAGS='-Wl,--export-dynamic -L/home/myhome/programs/openssl/i/lib
-L/home/myhome/programs/zlib/i/lib'
CPPFLAGS='-I/home/myhome/programs/openssl/i/include
-I/home/myhome/programs/zlib/i/include'
./config --prefix=/home/myhome/programs/openssl/i \
zlib-dynamic shared --with-zlib-lib=/home/myhome/programs/zlib/i/lib \
--with-zlib-include=/home/myhome/programs/zlib/i/include
make
make install
After install, when i check the binary with ldd openssl, and the result is:
...
libssl.so.1.0.0 => /home/myhome/programs/openssl/i/lib/libssl.so.1.0.0 (0x00007f91138c0000)
libcrypto.so.1.0.0 => /home/myhome/programs/openssl/i/lib/libcrypto.so.1.0.0 (0x00007f9113479000)
...
which looks fine. But when I check ldd libssl.so, the result is:
...
libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007fac70930000)
...
It still uses the system version of libcrypto. I tried different ways to
build, but result is always stays the same.
My question is how to configure the build in a way, that it can hardcode all binary and library dependencies of shared libraries without using LD_LIBRARY_PATH, or anything like that.
My question is how to configure the build in a way, that it can hardcode all binary and library dependencies of shared libraries without using LD_LIBRARY_PATH, or anything like that.
OpenSSL supports RPATH's out of the box for BSD targets (but not others). From Configure:
# Unlike other OSes (like Solaris, Linux, Tru64, IRIX) BSD run-time
# linkers (tested OpenBSD, NetBSD and FreeBSD) "demand" RPATH set on
# .so objects. Apparently application RPATH is not global and does
# not apply to .so linked with other .so. Problem manifests itself
# when libssl.so fails to load libcrypto.so. One can argue that we
# should engrave this into Makefile.shared rules or into BSD-* config
# lines above. Meanwhile let's try to be cautious and pass -rpath to
# linker only when --prefix is not /usr.
if ($target =~ /^BSD\-/)
{
$shared_ldflag.=" -Wl,-rpath,\$(LIBRPATH)" if ($prefix !~ m|^/usr[/]*$|);
}
The easiest way to do it for OpenSSL 1.0.2 appears to be add it as a CFLAG:
./config -Wl,-rpath=/usr/local/ssl/lib
The next easiest way to do it for OpenSSL 1.0.2 appears to be add a Configure line and hard code the rpath. For example, I am working on Debian x86_64. So I opened the file Configure in an editor, copied linux-x86_64, named it linux-x86_64-rpath, and made the following change to add the -rpath option:
"linux-x86_64-rpath", "gcc:-m64 -DL_ENDIAN -O3 -Wall -Wl,-rpath=/usr/local/ssl/lib::
-D_REENTRANT::-Wl,-rpath=/usr/local/ssl/lib -ldl:SIXTY_FOUR_BIT_LONG RC4_CHUNK DES_INT DES_UNROLL:
${x86_64_asm}:elf:dlfcn:linux-shared:-fPIC:-m64:.so.\$(SHLIB_MAJOR).\$(SHLIB_MINOR):::64",
Above, fields 2 and 6 were changed. They correspond to $cflag and $ldflag in OpenSSL's builds system.
Then, Configure with the new configuration:
$ ./Configure linux-x86_64-rpath shared no-ssl2 no-ssl3 no-comp \
--openssldir=/usr/local/ssl enable-ec_nistp_64_gcc_128
Finally, after make, verify the settings stuck:
$ readelf -d ./libssl.so | grep -i rpath
0x000000000000000f (RPATH) Library rpath: [/usr/local/ssl/lib]
$ readelf -d ./libcrypto.so | grep -i rpath
0x000000000000000f (RPATH) Library rpath: [/usr/local/ssl/lib]
$ readelf -d ./apps/openssl | grep -i rpath
0x000000000000000f (RPATH) Library rpath: [/usr/local/ssl/lib]
Once you perform make install, then ldd will produce expected results:
$ ldd /usr/local/ssl/lib/libssl.so
linux-vdso.so.1 => (0x00007ffceff6c000)
libcrypto.so.1.0.0 => /usr/local/ssl/lib/libcrypto.so.1.0.0 (0x00007ff5eff96000)
...
$ ldd /usr/local/ssl/bin/openssl
linux-vdso.so.1 => (0x00007ffc30d3a000)
libssl.so.1.0.0 => /usr/local/ssl/lib/libssl.so.1.0.0 (0x00007f9e8372e000)
libcrypto.so.1.0.0 => /usr/local/ssl/lib/libcrypto.so.1.0.0 (0x00007f9e832c0000)
...
OpenSSL has a Compilation and Installation on its wiki. This has now been added to the wiki at Compilation and Installation | Using RPATHs
It's 2019, and OpenSSL might have changed a little, so I'll describe how I solved this, on the odd chance someone else might find it useful (and in case I ever need to figure out this command line argument again for myself).
I wanted to build OpenSSL in a way that would cross-compile (using docker containers, because I'm dealing with freakishly old Linux kernels yet modern compilers), yet provide an install that did not depend upon absolute paths, as would be the case using rpath as I've seen described in jww's answer here.
I found I can run OpenSSL's Configure script in this way to achieve what I want (from a bash prompt):
./Configure linux-x86 zlib shared -Wl,-rpath=\\\$\$ORIGIN/../lib
This causes the generated Makefile to build the executables and the shared objects in a way that makes the loader look for dependencies first in "./../lib" (relative to the location of the executable or the shared object), then in the LD_LIBRARY_PATH, etc. That wacky combination of characters properly gets past the bash command line, the script, and the Makefile combinations to create the -rpath argument according to how the linker requires it ($ORIGIN/../lib).
(Obviously, choose the other options that make sense to you.. the key here is in the -Wl,-rpath=\\\$\$ORIGIN/../lib option).
So, if I called ./Configure with a prefix of '--prefix=/opt/spiffness', and later decided to rename 'spiffness' to 'guttersnipe', everything will still work correctly, since the paths are relative rather than absolute.
I have not tried passing the argument into ./config to see if it works there since my use case was a bit special, but I suspect it would. If I were not attempting to cross-compile with dockerized containers, I would prefer using ./config to ./Configure, as it does a decent enough job of examining the current environment to see what kind of binaries to create.
I hope this is useful.

using older version of a shared linux library while compiling C

I am trying to use libfann version 2.0.1 instead of the newest version 2.2.0, but could not figure out how to do so. Any thoughts on how to do that?
normally that works perfectly:
gcc fann_calculator.c -o run_fann_calculator -lfann -lm
where fann_calculator.c contains a program that calls a neural network.
Thanks
It depends upon where the two libraries sit. If they are installed in the same directory (e.g. both installed in /usr/lib/) you'll probably get the youngest one.
I suggest to carefully read the ld.so(8) and ldd(1) man pages. You certainly can trace what library is loaded (with e.g. the LD_DEBUG envirnonment variable). Don't forget to re-run ldconfig appropriately after library installation.
You could also play some LD_LIBRARY_PATH trick; for instance, set it to $HOME/lib:/usr/lib and install appropriate symlinks in your $HOME/lib/ to the precise library you want. For instance, you might do
ln -s /usr/lib/libfann.so.2.0.1 $HOME/lib/libfann.so.2
export LD_LIBRARY_PATH=$HOME/lib:/usr/lib:/lib
then check with ldd run_fann_calculator that you get the expected [version of the] libfann library.
Don't forget to read the Program Library Howto. You might want to pass appropriate flags to ld such as -rpath. You may need to pass them using gcc, perhaps with Gcc Link Options such as -Wl

why after setting LD-LIBRARY_PATH and ld.so.cache properly, there are still library-finding problems?

I have a certain shared object library in a special directory which I
make sure special directory is in $LD_LIBRARY_PATH
make sure this directory has read and execute permisions for all
make sure appropriate library directory is in ld.so.conf and that root has done a ldconfig
(verify by checking for library using ldconfig -p as normaluser.
make sure it is has no soname problems (i.e. create a few symlinks if necessary)
Now, say I compile a program that needs that special library, a program packaged in a typical Open Source manner which ./configure && make, etc) and it says -lspecialibrary cannot be found, an error which a lack of any of the above checks would also probably throw.
A workaround I have done is to symlink the library to /usr/local/lib64 and suddenly the library has ben found. Also when compiling a relatively simple package, I manually add -L/path/to/spec/lib and that also has worked. But I regard those two methods as hacks, so I was looking for any clues as to why my list of checks aren't good enough to find a library.
(I particularly find the $LD_LIBRARY_PATH of shallow use. In fact I can exclude certain libraries from it, and they will still be found in a compilation process).
$LD_LIBRARY_PATH and ldconfig are only used to locate libraries when running programs that need libraries, i.e. they are used by the loader not the compiler. Your program depends on libspeciallibrary.so. When running your program $LD_LIBRARY_PATH and ldconfig are consulted to find libspeciallibary.so.
These methods are not used by your compiler to find libraries. For your compiler, the -L option is the right way to go. Since your package uses the autotools, you should set the $LDFLAGS environment variable:
LDFLAGS=-L/path/to/lib ./configure && make
This is also documented in the configure help:
./configure --help

Resources