This is a question about centrally-located path specs, like PATH, LD_LIBRARY_PATH, and LIBRARY_PATH.
I know that there are two ways of specifying shared library paths for the loader: add them to LD_LIBRARY_PATH, or add files to /etc/ld.so.conf.d/. I also know that the latter is considered the more modern and preferred way to do it.
I also know that you can specify standard library paths for the linker by editing LIBRARY_PATH. Is this still the "modern" way to do it, or is there now a "ld.so.conf.d-style" alternative that I should be using?
EDIT: People are asking "why", so:
I'm using a Python package (Theano) that dynamically generates and compiles CUDA and C++ code when run. One of the libraries it links to is NVidia's cuDNN. I don't know why Theano's developer's have it link to the static lib and not the dynamic lib.
There isn't any equivalent to ld.so.conf.d/ for static libraries. You still just specify the standard linker search paths via the LIBRARY_PATH environment variable, and additional paths through command-line flags to the linker.
To be clear:
LIBRARY_PATH: Used by the linker at compile time. Is used by the linker to find both static and dynamic libraries.
LD_LIBRARY_PATH: Used by the loader at run time to find dynamic libraries.
static libraries are resolved at (static) link time and by definition don't have any runtime aspects.
My opinion is that you should avoid using static libraries and always prefer shared libraries.
Related
I'm compiling a C++ program on linux, and I can run make and it all compiles, but when I need to downgrade or change one of it's dependencies for another program, it breaks. I was wondering if it was possible to create a standalone executable, with dependencies bundled inside? There's not many dependencies, so size isn't an issue.
So, what you're asking is, can you link with static versions of libraries (which are included in the program directly) instead of dynamic versions of libraris (shared libraries) which are kept external to your program.
The answer is "yes", but it's not always straightforward. First you have to ensure you actually have the static versions of the libraries installed in your system: the static and dynamic libraries are different files and often the "standard" installation provides only the dynamic library.
If you're already compiling code against those libraries you probably already have the static libraries installed because, at least on GNU/Linux systems, the static libraries are often included in the "dev" packages along with the header files etc. needed to compile code.
To make this work you need to modify your linker command line. If you have a sufficiently new version of the binutils package (which provides the linker), you can change your link line to replace arguments like -lssl -lcrypto with arguments like -l:libssl.a -l:libcrypto.a (don't forget the colon after the -l) and that should do it.
The blog article "LD_LIBRARY_PATH – or: How to get yourself into trouble!" by the DTU Computing Center states:
3. Inconsistency: This is the most common problem. LD_LIBRARY_PATH forces an application to load a shared library it wasn’t linked against, and that is quite likely not compatible with the original version. This can either be very obvious, i.e. the application crashes, or it can lead to wrong results, if the picked up library not quite does what the original version would have done. Especially the latter is sometimes hard to debug.
Is this really true? LD_LIBRARY_PATH allows us to modify the search path for dynamic libraries, but does it really suppress the soname lookup that ensures binary compatibility?
(Because, by my interpretation, the Program Library HOWTO doesn't say any such thing.)
Or is the author unaware of the concept of maintaining a consistent library versioning scheme, and therefore assuming that one is not in use for the library in question?
I think the LD_LIBRARY should only be used for testing and not for a final installation, for it allows to use a specified library before the standard library location are used. But The linux documentation project says this about LD_LIBRARY_PATH and puts it more clear than I can.
3.3.1. LD_LIBRARY_PATH
You can temporarily substitute a different library for this particular
execution. In Linux, the environment variable LD_LIBRARY_PATH is a
colon-separated set of directories where libraries should be searched
for first, before the standard set of directories; this is useful when
debugging a new library or using a nonstandard library for special
purposes. The environment variable LD_PRELOAD lists shared libraries
with functions that override the standard set, just as
/etc/ld.so.preload does. These are implemented by the loader
/lib/ld-linux.so. I should note that, while LD_LIBRARY_PATH works on
many Unix-like systems, it doesn't work on all; for example, this
functionality is available on HP-UX but as the environment variable
SHLIB_PATH, and on AIX this functionality is through the variable
LIBPATH (with the same syntax, a colon-separated list).
LD_LIBRARY_PATH is handy for development and testing, but shouldn't be
modified by an installation process for normal use by normal users;
see ``Why LD_LIBRARY_PATH is Bad'' at
http://www.visi.com/~barr/ldpath.html for an explanation of why. But
it's still useful for development or testing, and for working around
problems that can't be worked around otherwise. If you don't want to
set the LD_LIBRARY_PATH environment variable, on Linux you can even
invoke the program loader directly and pass it arguments. For example,
the following will use the given PATH instead of the content of the
environment variable LD_LIBRARY_PATH, and run the given executable:
/lib/ld-linux.so.2 --library-path PATH EXECUTABLE
Just executing ld-linux.so without arguments will give you more help
on using this, but again, don't use this for normal use - these are
all intended for debugging.
taken at august 13th 2013 from: http://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html
The link inside the document is old, found a the intended article here: http://xahlee.info/UnixResource_dir/_/ldpath.html
edit
You can override a library to which a program is linked to during building/installing because of the order in which ld.so will lookup a library to load at runtime. A library found at in a location specified inside de environmental variable LD_LIBRARY_PATH will be loaded instead of a library specified the default path ( /lib and the /usr/lib)
from man 8 ld.so
ld.so loads the shared libraries needed by a program, prepares the pro‐
gram to run, and then runs it. Unless explicitly specified via the
-static option to ld during compilation, all Linux programs are incom‐
plete and require further linking at run time.
The necessary shared libraries needed by the program are searched for
in the following order
o Using the environment variable LD_LIBRARY_PATH
(LD_AOUT_LIBRARY_PATH for a.out programs). Except if the exe‐
cutable is a setuid/setgid binary, in which case it is ignored.
o From the cache file /etc/ld.so.cache which contains a compiled
list of candidate libraries previously found in the augmented
library path. Libraries installed in hardware capabilities
directories (see below) are prefered to other libraries.
o In the default path /lib, and then /usr/lib.
My questions is related to this: Creating both static and shared C++ libraries
I'm compiling a library in order to install it in ~/local on two different systems. It seems that every time I do this I end up with linker problems that take me hours to figure out. The specific library I'm looking at is primesieve. In that library, it's the default to build static libraries only. Unfortunately the example code count_primes.cpp wouldn't link with the static version of the library on one of my systems, for whatever reason. Eventually I figured out how to build the shared version and the code now compiles nicely, with no ugly hacks necessary.
Given the above, it seems to be that compiling both static and shared versions is a good idea if you're working with multiple systems and want the best chance of having your code compile. Is this true? Are there reasons not to build both versions? I realize that this is a bit of a subjective question but it's a serious programming issue that I think many people here have probably encountered.
PS.
This is what I ended up using to compile and install both shared and static versions of primesieve to ~/local:
make
make lib
make install PREFIX=~/local
make clean
make lib SHARED=yes
make install PREFIX=~/local
The make clean is because of this. I then added this to my .bash_profile:
export LIBRARY_PATH=$LIBRARY_PATH:~/local/lib
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:~/local/lib
export CPLUS_INCLUDE_PATH=$CPLUS_INCLUDE_PATH:~/local/include
Alternatively, without changing the environment variables I was able to compile the example program count_primes.cpp like this:
g++ -I ~/local/include/ -L ~/local/lib/ -lprimesieve count_primes.cpp
To use a static library you can just include it in the compilation as if it were a regular object file, e.g.
g++ -o foo foo.cpp /path/to/mylib.a
Of course, this means static linking too.
You can still statically link with a dynamic library, so there's not much use for static libraries really.
There is no reason not to build both. Neither library will "do" anything. The shared library will only be loaded if it is in a path viable to the dynamic linker (like you did by adding it to your LD library path). The static one won't be used unless you explicitly link against it - but that is only done at compile (link) time.
Lets say I have a massive project which consists of multiple dynamic libraries that will all get installed to /usr/lib or /usr/lib64. Now lets say that one of the libraries call into another of the compiled libraries. If I place both of the libraries that are dependent on eachother in the same location will the ld program be able to allow the two libraries to call eachother?
The answer is perhaps yes, but it is a very bad design to have circular references between two libraries (i.e. liba.so containing function fa, calling function fb from libb.so, calling function ga from liba.so).
You should merge the two libraries in one libbig.so. And don't worry, libraries can be quite big. (some corporations have Linux libraries of several hundred megabytes of code).
The gold linker from package binutils-gold on Debian should be useful to you. It works faster than the older linker from binutils.
Yes, as long as their location is present in set of directories ld searches for libraries in. You can override this set by using LD_LIBRARY_PATH enviroment variable.
See this manual, it will resolve your questions.
If you mean the runtime dynamic linker /lib/ld-linux* (as opposed to /usr/bin/ld), it will look for libraries in your LD_LIBRARY_PATH, which typically includes /usr/lib and /usr/lib64.
In general, /lib/ld-* are used for .so libraries at run-time; /usr/bin/ld is used for .a libraries at compile-time.
However, if your libraries are using dlopen() or similar to find one another (e.g. plug-ins), they may have other mechanisms for finding one another. For example, many plug-in systems will use dlopen to read every library in a certain (one or many) directory/ies.
I have an executable on Linux that loads libfoo.so.1 (that's a SONAME) as one of its dependencies (via another shared library). It also links to another system library, which, in turn, links to a system version, libfoo.so.2. As a result, both libfoo.so.1 and libfoo.so.2 are loaded during execution, and code that was supposed to call functions from library with version 1 ends up calling (binary-incompatible) functions from a newer system library with version 2, because some symbols stay the same. The result is usually stack smashing and a subsequent segfault.
Now, the library which links against the older version is a closed-source third-party library, and I can't control what version of libfoo it compiles against. Assuming that, the only other option left is rebuilding a bunch of system libraries that currently link with libfoo.so.2 to link with libfoo.so.1.
Is there any way to avoid replacing system libraries wiith local copies that link to older libfoo? Can I load both libraries and have the code calling correct version of symbols? So I need some special symbol-level versioning?
You may be able to do some version script tricks:
http://sunsite.ualberta.ca/Documentation/Gnu/binutils-2.9.1/html_node/ld_26.html
This may require that you write a wrapper around your lib that pulls in libfoo.so.1 that exports some symbols explicitly and masks all others as local. For example:
MYSYMS {
global:
foo1;
foo2;
local:
*;
};
and use this when you link that wrapper like:
gcc -shared -Wl,--version-script,mysyms.map -o mylib wrapper.o -lfoo -L/path/to/foo.so.1
This should make libfoo.so.1's symbols local to the wrapper and not available to the main exe.
I can only come up with a work-around. Which would be to statically link a version of the "system library" that you are using. For your static build, you could make it link against the same old version as the third-party library. Given that it does not rely on the newer version...
Perhaps it is also possible to avoid these problems with not linking to the third-party library the ordinary way. Instead, your program could load it at execution time. Perhaps then it could be shadowed against the rest. But I don't know much about that.