The blog article "LD_LIBRARY_PATH – or: How to get yourself into trouble!" by the DTU Computing Center states:
3. Inconsistency: This is the most common problem. LD_LIBRARY_PATH forces an application to load a shared library it wasn’t linked against, and that is quite likely not compatible with the original version. This can either be very obvious, i.e. the application crashes, or it can lead to wrong results, if the picked up library not quite does what the original version would have done. Especially the latter is sometimes hard to debug.
Is this really true? LD_LIBRARY_PATH allows us to modify the search path for dynamic libraries, but does it really suppress the soname lookup that ensures binary compatibility?
(Because, by my interpretation, the Program Library HOWTO doesn't say any such thing.)
Or is the author unaware of the concept of maintaining a consistent library versioning scheme, and therefore assuming that one is not in use for the library in question?
I think the LD_LIBRARY should only be used for testing and not for a final installation, for it allows to use a specified library before the standard library location are used. But The linux documentation project says this about LD_LIBRARY_PATH and puts it more clear than I can.
3.3.1. LD_LIBRARY_PATH
You can temporarily substitute a different library for this particular
execution. In Linux, the environment variable LD_LIBRARY_PATH is a
colon-separated set of directories where libraries should be searched
for first, before the standard set of directories; this is useful when
debugging a new library or using a nonstandard library for special
purposes. The environment variable LD_PRELOAD lists shared libraries
with functions that override the standard set, just as
/etc/ld.so.preload does. These are implemented by the loader
/lib/ld-linux.so. I should note that, while LD_LIBRARY_PATH works on
many Unix-like systems, it doesn't work on all; for example, this
functionality is available on HP-UX but as the environment variable
SHLIB_PATH, and on AIX this functionality is through the variable
LIBPATH (with the same syntax, a colon-separated list).
LD_LIBRARY_PATH is handy for development and testing, but shouldn't be
modified by an installation process for normal use by normal users;
see ``Why LD_LIBRARY_PATH is Bad'' at
http://www.visi.com/~barr/ldpath.html for an explanation of why. But
it's still useful for development or testing, and for working around
problems that can't be worked around otherwise. If you don't want to
set the LD_LIBRARY_PATH environment variable, on Linux you can even
invoke the program loader directly and pass it arguments. For example,
the following will use the given PATH instead of the content of the
environment variable LD_LIBRARY_PATH, and run the given executable:
/lib/ld-linux.so.2 --library-path PATH EXECUTABLE
Just executing ld-linux.so without arguments will give you more help
on using this, but again, don't use this for normal use - these are
all intended for debugging.
taken at august 13th 2013 from: http://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html
The link inside the document is old, found a the intended article here: http://xahlee.info/UnixResource_dir/_/ldpath.html
edit
You can override a library to which a program is linked to during building/installing because of the order in which ld.so will lookup a library to load at runtime. A library found at in a location specified inside de environmental variable LD_LIBRARY_PATH will be loaded instead of a library specified the default path ( /lib and the /usr/lib)
from man 8 ld.so
ld.so loads the shared libraries needed by a program, prepares the pro‐
gram to run, and then runs it. Unless explicitly specified via the
-static option to ld during compilation, all Linux programs are incom‐
plete and require further linking at run time.
The necessary shared libraries needed by the program are searched for
in the following order
o Using the environment variable LD_LIBRARY_PATH
(LD_AOUT_LIBRARY_PATH for a.out programs). Except if the exe‐
cutable is a setuid/setgid binary, in which case it is ignored.
o From the cache file /etc/ld.so.cache which contains a compiled
list of candidate libraries previously found in the augmented
library path. Libraries installed in hardware capabilities
directories (see below) are prefered to other libraries.
o In the default path /lib, and then /usr/lib.
Related
This is a question about centrally-located path specs, like PATH, LD_LIBRARY_PATH, and LIBRARY_PATH.
I know that there are two ways of specifying shared library paths for the loader: add them to LD_LIBRARY_PATH, or add files to /etc/ld.so.conf.d/. I also know that the latter is considered the more modern and preferred way to do it.
I also know that you can specify standard library paths for the linker by editing LIBRARY_PATH. Is this still the "modern" way to do it, or is there now a "ld.so.conf.d-style" alternative that I should be using?
EDIT: People are asking "why", so:
I'm using a Python package (Theano) that dynamically generates and compiles CUDA and C++ code when run. One of the libraries it links to is NVidia's cuDNN. I don't know why Theano's developer's have it link to the static lib and not the dynamic lib.
There isn't any equivalent to ld.so.conf.d/ for static libraries. You still just specify the standard linker search paths via the LIBRARY_PATH environment variable, and additional paths through command-line flags to the linker.
To be clear:
LIBRARY_PATH: Used by the linker at compile time. Is used by the linker to find both static and dynamic libraries.
LD_LIBRARY_PATH: Used by the loader at run time to find dynamic libraries.
static libraries are resolved at (static) link time and by definition don't have any runtime aspects.
My opinion is that you should avoid using static libraries and always prefer shared libraries.
I think a major design flaw in Linux is the shared object hell when it comes to distributing programs in binary instead of source code form.
Here is my specific problem: I want to publish a Linux program in ELF binary form that should run on as many distributions as possible so my mandatory dependencies are as low as it gets: The only libraries required under any circumstances are libpthread, libX11, librt and libm (and glibc of course). I'm linking dynamically against these libraries when I build my program using gcc.
Optionally, however, my program should also support ALSA (sound interface), the Xcursor, Xfixes, and Xxf86vm extensions as well as GTK. But these should only be used if they are available on the user's system, otherwise my program should still run but with limited functionality. For example, if GTK isn't there, my program will fall back to terminal mode. Because my program should still be able to run without ALSA, Xcursor, Xfixes, etc. I cannot link dynamically against these libraries because then the program won't start at all if one of the libraries isn't there.
So I need to manually check if the libraries are present and then open them one by one using dlopen() and import the necessary function symbols using dlsym(). This, however, leads to all kinds of problems:
1) Library naming conventions:
Shared objects often aren't simply called "libXcursor.so" but have some kind of version extension like "libXcursor.so.1" or even really funny things like "libXcursor.so.0.2000". These extensions seem to differ from system to system. So which one should I choose when calling dlopen()? Using a hardcoded name here seems like a very bad idea because the names differ from system to system. So the only workaround that comes to my mind is to scan the whole library path and look for filenames starting with a "libXcursor.so" prefix and then do some custom version matching. But how do I know that they are really compatible?
2) Library search paths: Where should I look for the *.so files after all? This is also different from system to system. There are some default paths like /usr/lib and /lib but *.so files could also be in lots of other paths. So I'd have to open /etc/ld.so.conf and parse this to find out all library search paths. That's not a trivial thing to do because /etc/ld.so.conf files can also use some kind of include directive which means that I have to parse even more .conf files, do some checks against possible infinite loops caused by circular include directives etc. Is there really no easier way to find out the search paths for *.so?
So, my actual question is this: Isn't there a more convenient, less hackish way of achieving what I want to do? Is it really so complicated to create a Linux program that has some optional dependencies like ALSA, GTK, libXcursor... but should also work without it! Is there some kind of standard for doing what I want to do? Or am I doomed to do it the hackish way?
Thanks for your comments/solutions!
I think a major design flaw in Linux is the shared object hell when it comes to distributing programs in binary instead of source code form.
This isn't a design flaw as far as creators of the system are concerned; it's an advantage -- it encourages you to distribute programs in source form. Oh, you wanted to sell your software? Sorry, that's not the use case Linux is optimized for.
Library naming conventions: Shared objects often aren't simply called "libXcursor.so" but have some kind of version extension like "libXcursor.so.1" or even really funny things like "libXcursor.so.0.2000".
Yes, this is called external library versioning. Read about it here. As should be clear from that description, if you compiled your binaries using headers on a system that would normally give you libXcursor.so.1 as a runtime reference, then the only shared library you are compatible with is libXcursor.so.1, and trying to dlopen libXcursor.so.0.2000 will lead to unpredictable crashes.
Any system that provides libXcursor.so but not libXcursor.so.1 is either a broken installation, or is also incompatible with your binaries.
Library search paths: Where should I look for the *.so files after all?
You shouldn't be trying to dlopen any of these libraries using their full path. Just call dlopen("libXcursor.so.1", RTLD_GLOBAL);, and the runtime loader will search for the library in system-appropriate locations.
What is the most portable and robust way to get the list of paths, configured by /etc/ld.so.conf and files included from it? Parsing the file manually seems to be not a good idea — the format is likely to change in the future revisions.
To allow better understanding of the question, I will give you specific details below. Note that, despite these details, this is a general programming question, applicable to other situations.
There is a program, called LuaRocks. It is a package manager for Lua programming language (somewhat like Ruby gems or Python eggs). LuaRocks packages are called "rocks".
As a convenience feature, LuaRocks allows a rock author to specify a list of external dependencies for a rock, formulated as a list of C header files and / or dynamic library files. (.so on Linux.) If the specified file does not exist, the rock can't be installed.
Currently, on Linux, LuaRocks by default checks .so file existance by searching for the file in two hardcoded paths, /usr/lib and /usr/local/lib.
I believe that this is incorrect behaviour, and it is broken by the recent changes in the Ubuntu and other Debian distributions.
Update: the paths are not hardcoded per se, but are user-configurable in the config file. Still, IMO, not a best solution.
Instead (as I understand it), LuaRocks should look up file in the paths, specified by /etc/ld.so.conf and files included from it.
(Now please re-read the question above ;-) )
You shouldn't need to parse /etc/ld.so.conf or any of the config files - if you run 'ldconfig', it will scan the configured directories and generate a cache file.
Then, subsequently when you attempt a dlopen it'll automatically find the files by iterating through the cached library directories. Same thing with compiling and giving -lSomeLib, you shouldn't need to specify -L/my/other/path if you've got it configured in ld.so.conf(.d)
autoconf accomplishes this by attempting to compile a test program that links to the shared library, but that's just a functional wrapper around the dlopen() call.
So, while other methods may not necessarily be 'wrong', at the root of it attempting to link to the library or doing a dlopen() are the 'most right' ways of doing it.
Consider this, if you attempt to link to a library in a directory that ISN'T cached in /etc/ld.so.cache, when you try to run the program it will fail because it won't be able to dlopen() the library!
Hence, any 'good' shared library will be in /etc/ld.so.cache and be linkable/dlopen()able, this means that gcc can use it to link and that the user-generated library or executable will be able to open it when it executes.
You can circumvent this by expressly setting the environment variable LD_LIBRARY_PATH, or LD_PRELOAD_PATH - but each of these has it's own caveats and should be avoided if possible for 'standard' use.
A good write-up on writing shared libraries covers some of these issues, and is a good read for anyone working on programmatic consuming of other-shared libraries. Ulrich Drepper's How to write shared libraries.
According to the FHS, the following are valid locations for dynamic libraries:
/lib*/
/opt/*/lib*/
/usr/lib*/
/usr/local/lib*/
(And most likely ~/lib*/ as well.)
All entries in my /etc/ld.so.conf.d/* conform to this. Some entries reference subdirectories below the FHS dirs, which probably means that you can use the libraries in there without path information.
Now I don't know enough about LuaRocks. If you're limited to Lua-path-style globs (only ?), you cannot match these and have to parse the configs. Otherwise, you could just try to find them anywhere in these directories.
This would break on non-FHS-conforming systems (only option: parse config) and if a directory is not included in the config, the installer might see libraries that the linker cannot find.
These two seem acceptable to me, therefore I'd simply ignore the config and look at these dirs.
(Another possibility could be trying to link the library, this should automagically use the right path. However, this is platform-specific and maybe dangerous.)
I know that LD_LIBRARY_PATH is a environment variable where the linker will look for the shared library (which contains shared objects) to link with the executable code.
But what does the LD Stands for, is it for Load? or List Directory?
Linker. The *nix linker is called ld. When a program with dynamic libraries is linked, the linker adds additional code to look for dynamic libraries to resolve symbols not statically linked. Usually this code looks in /lib and /usr/lib. LD_LIBRARY_PATH is a colon separated list of other directories to search.
"ldd" is a handy program to see where the libraries are: try "ldd /bin/ls", for example.
It could also mean "Loader", though. ;-)
Editorial:
As a (semi) interesting side-note: I think dynamic libraries will go away someday. They were needed when disk space and system memory was scarce. There is a performance hit to using them (i.e. the symbols need to be resolved and the object code edited). In these modern days of 3GB memory and 7 second bootup times, it might be appropriate to go back to static linking.
Except for the fact that every C++ program would magically grow to 3MB. ;-)
LD_LIBRARY_PATH - stands for LOAD LIBRARY PATH or some times called as LOADER LIBRARY PATH
I read some articles about problems in using the LD_LIBRARY_PATH, even as a part of a wrapper script:
http://linuxmafia.com/faq/Admin/ld-lib-path.html
http://blogs.oracle.com/ali/entry/avoiding_ld_library_path_the
In this case - what are the recommended alternatives?
Thanks.
You can try adding:
-Wl,-rpath,path/to/lib
to the linker options. This will save you the need to worry about the LD_LIBRARY_PATH environment variable, and you can decide at compile time to point to a specific library.
For a path relative to the binary, you can use $ORIGIN, eg
-Wl,-rpath,'$ORIGIN/../lib'
($ORIGIN may not work when statically linking to shared libraries with ld, use -Wl,--allow-shlib-undefined to fix this)
I've always set LD_LIBRARY_PATH, and I've never had a problem.
To quote you first link:
When should I set LD_LIBRARY_PATH? The short answer is never. Why? Some users seem to set this environment variable because of bad advice from other users or badly linked code that they do not know how to fix.
That is NOT what I call a definitive problem statement. In fact it brings to mind I don't like it. [YouTube, but SFW].
That second blog entry (http://blogs.oracle.com/ali/entry/avoiding_ld_library_path_the) is much more forthcoming on the nature of the problem... which appears to be, in a nutshell, library version clashes ThisProgram requires Foo1.2, but ThatProgram requires Foo1.3, hence you can't run both programs (easily). Note that most of these problems are negated by a simple wrapper script which sets the LD_LIBRARY_PATH for just the executing shell, which is (almost always) a separate child process of interactive shell.
Note also that the alternatives are pretty well explained in the post.
I'm just confused as to why you would post a question containing links to articles which apparently answer your question... Do you have a specific question which wasn't covered (clearly enough) in either of those articles?
the answer is in the first article you quoted.
In UNIX the location of a library can be specified with the -L dir option to the compiler.
....
As an alternative to using the -L and -R options, you can set the environment variable LD_RUN_PATH before compiling the code.
I find that the existing answers to not actually answer the question in a straightforward way:
LD_RUN_PATH is used by the linker (see ld) at the time you link your software. It is used only if you have no -rpath ... on the command line (-Wl,rpath ... on the gcc command line). The path(s) defined in that variable are added to the RPATH entry in your ELF binary file. (You can see that RPATH using objdump -x binary-filename—in most cases it is not there though! It appears in my development binaries, but once the final version gets installed RPATH gets removed.)
LD_LIBRARY_PATH is used at runtime, when you want to specify a directory that the dynamic linker (see ldd) needs to search for libraries. Specifying the wrong path could lead to loading the wrong libraries. This is used in addition to the RPATH value defined in your binary (as in 1.)
LD_RUN_PATH really causes no security threat unless you are a programmer and don't know how to use it. As I am using CMake to build my software, the -rpath is used all the time. That way I do not have to install everything to run my software. ldd can find all the .so files automatically. (the automake environment was supposed to do that too, but it was not very good at it, in comparison.)
LD_LIBRARY_PATH is a runtime variable and thus you have to be careful with it. That being said, many shared object would be really difficult to deal with if we did not have that special feature. Whether it is a security threat, probably not. If a hacker takes a hold of your computer, LD_LIBRARY_PATH is accessible to that hacker anyway. What could happen is that you use the wrong path(s) in that variable, your binary may not load, but if it loads you may end up with a crashing binary or at least a binary that does not work quite right. One concern is that over time you get new versions of the library and you are likely to forget to remove the LD_LIBRARY_PATH which means you may be using an unsecure version of the library.
The one other possibility for security is if the hacker installs a fake library of the same name as what the binary is searching, library that includes all the same functions, but that has some of those functions replaced with sneaky code. He can get that library loaded by changing the LD_LIBRARY_PATH variable. Then it will eventually get executed by the hacker. Again, if the hacker can add such a library to your system, he's already in and probably does not need to do anything like that in the first place (since he's in he has full control of your system anyway.) Because in reality, if the hacker can only place the library in his account he won't do anything much (unless your Unix box is not safe overall...) If the hacker can replace one of your /usr/lib/... libraries, he already has full access to your system. So LD_LIBRARY_PATH is not needed.