I have 3rd party library A, that requires some library B.
A is linked to binrary, which is linked with static version B.
Therefore there's no need in dynamic version of B any more.
A is not under my control and I cannot recompile it. Thus I want to remove NEEDED libA entry from DYNAMIC section of A.
Is there a way to do it with objcopy or other tool?
Is there a way to do it with objcopy or other tool?
I don't know of any existing tool that can do this, although elfsh might be able to.
It is quite trivial to write a C program to do what you want: the .dynamic section of libA.so is a table of fixed-size records (of type ElfW(Dyn)), terminated by an entry with .d_type == DT_NULL. To get rid of a particular DT_NEEDED entry, simply "slide" all following entries up (overwriting entry[n] with entry[n+1], etc.). This will leave your .dynamic with two DT_NULL entries at the end, but nothing should ever care.
One complication is that if libB.so contains versioned symbols that libA.so references, then there will be additional references to libB.so in DT_VERNEED table, and these are more difficult to get rid of. If you don't get rid of VERNEED references, the dynamic linker will fail assertions.
Related
From what I have gathered, symbol name lookup for a default-visibility symbol of a shared library iterates through the shared library dependency tree in breath-first order, with the executable program being the root of this search tree. All libraries that are linked by one DT_NEEDED list are at the same level of this tree.
Thus, when a symbol "foo" is looked up, it seems to me that its lookup is deterministically always bound to the same library or executable at runtime. Does the dynamic linker exploit this and has a "global symbol table" of sorts (perhaps associated with the link-map list), that knows what symbol belongs to what shared library once a symbol was first looked up, and takes the address of the symbol out of that libraries' GOT when the symbol is looked up by a different shared library a second time? Or will the symbol always be looked up as if this was its first lookup?
Thus, when a symbol "foo" is looked up, it seems to me that its lookup is deterministically always bound to the same library or executable at runtime.
This view is way oversimplified. There are many complications, such as presence of DT_SYMBOLIC on the referencing DSO, presence of RTLD_LOCAL when defining DSO is loaded (if it's not linked in directly), and I am sure some other complications I am not remembering at the moment.
Does the dynamic linker exploit this and has a "global symbol table" of sorts
GLIBC loader does not do that.
Or will the symbol always be looked up as if this was its first lookup?
Yes. You can observe this with LD_DEBUG=symbols,bindings ./a.out
Say I got (regular source) libraries A and B and executable E which depends on both.
Now, I want E to include the object files of A directly, whereas B should be added as a shared library (concrete use: B contains shared types of a plugin architecture). How would I do that with existing tools, preferably stack?
Is that possible or is it rather an all-or-nothing choice (use only shared libraries or link everything into the same binary)?
Optimally, I'd like to specify for each dependency if it should be linked statically or dynamically. Also, that should probably go into the .cabal file, but we have to work with what we got...
(Well, technically that's both statically linked, but in the second case the object code is split up in different files, you get the idea).
Analyzing this question I found out some things about behavior of weak symbol resolution in the context of dynamic loading (dlopen) on Linux. Now I'm looking for the specifications governing this.
Let's take an example. Suppose there is a program a which dynamically loads libraries b.so and c.so, in that order. If c.so depends on two other libraries foo.so (actually libgcc.so in that example) and bar.so (actually libpthread.so), then usually symbols exported by bar.so can be used to satisfy weak symbol linkages in foo.so. But if b.so also depends on foo.so but not on bar.so, then these weak symbols will apparently not be linked against bar.so. It seems as if foo.so inkages only look for symbols from a and b.so and all their dependencies.
This makes sense, to some degree, since otherwise loading c.so might change the behavior of foo.so at some point where b.so has already been using the library. On the other hand, in the question that got me started this caused quite a bit of trouble, so I wonder whether there is a way around this problem. And in order to find ways around, I first need a good understanding about the very exact details how symbol resolution in these cases is specified.
What is the specification or other technical document to define correct behavior in these scenarios?
Unfortunately, the authoritative documentation is the source code. Most distributions of Linux use glibc or its fork, eglibc. In the source code for both, the file that should document dlopen() reads as follows:
manual/libdl.texi
#c FIXME these are undocumented:
#c dladdr
#c dladdr1
#c dlclose
#c dlerror
#c dlinfo
#c dlmopen
#c dlopen
#c dlsym
#c dlvsym
What technical specification there is can be drawn from the ELF specification and the POSIX standard. The ELF specification is what makes a weak symbol meaningful. POSIX is the actual specification for dlopen() itself.
This is what I find to be the most relevant portion of the ELF specification.
When the link editor searches archive libraries, it extracts archive
members that contain definitions of undefined global symbols. The
member’s definition may be either a global or a weak symbol.
The ELF specification makes no reference to dynamic loading so the rest of this paragraph is my own interpretation. The reason I find the above relevant is that resolving symbols occurs at a single "when". In the example you give, when program a dynamically loads b.so, the dynamic loader attempts to resolve undefined symbols. It may end up doing so with either global or weak symbols. When the program then dynamically loads c.so, the dynamic loader again attempts to resolve undefined symbols. In the scenario you describe, symbols in b.so were resolved with weak symbols. Once resolved, those symbols are no longer undefined. It doesn't matter if global or weak symbols were used to defined them. They're already no longer undefined by the time c.so is loaded.
The ELF specification gives no precise definition of what a link editor is or when the link editor must combine object files. Presumably it's a non-issue because the document has dynamic-linking in mind.
POSIX describes some of the dlopen() functionality but leaves much up to the implementation, including the substance of your question. POSIX makes no reference to the ELF format or weak symbols in general. For systems implementing dlopen() there need not even be any notion of weak symbols.
http://pubs.opengroup.org/onlinepubs/9699919799/functions/dlopen.html
POSIX compliance is part of another standard, the Linux Standard Base. Linux distributions may or may not choose to follow these standards and may or may not go to the trouble of being certified. For example, I understand that a formal Unix certification by Open Group is quite expensive -- hence the abundance of "Unix-like" systems.
An interesting point about the standards compliance of dlopen() is made on the Wikipedia article for dynamic loading. dlopen(), as mandated by POSIX, returns a void*, but C, as mandated by ISO, says that a void* is a pointer to an object and such a pointer is not necessarily compatible with a function pointer.
The fact remains that any conversion between function and object
pointers has to be regarded as an (inherently non-portable)
implementation extension, and that no "correct" way for a direct
conversion exists, since in this regard the POSIX and ISO standards
contradict each other.
The standards that do exist contradict and what standards documents there are may not be especially meaningful anyway. Here's Ulrich Drepper writing about his disdain for Open Group and their "specifications".
http://udrepper.livejournal.com/8511.html
Similar sentiment is expressed in the post linked by rodrigo.
The reason I've made this change is not really to be more conformant
(it's nice but no reason since nobody complained about the old
behaviour).
After looking into it, I believe the proper answer to the question as you've asked it is that there is no right or wrong behavior for dlopen() in this regard. Arguably, once a search has resolved a symbol it is no longer undefined and in subsequent searches the dynamic loader will not attempt to resolve the already defined symbol.
Finally, as you state in the comments, what you describe in the original post is not correct. Dynamically loaded shared libraries can be used to resolve undefined symbols in previously dynamically loaded shared libraries. In fact, this isn't limited to undefined symbols in dynamically loaded code. Here is an example in which the executable itself has an undefined symbol that is resolved through dynamic loading.
main.c
#include <dlfcn.h>
void say_hi(void);
int main(void) {
void* symbols_b = dlopen("./dyload.so", RTLD_NOW | RTLD_GLOBAL);
/* uh-oh, forgot to define this function */
/* better remember to define it in dyload.so */
say_hi();
return 0;
}
dyload.c
#include <stdio.h>
void say_hi(void) {
puts("dyload.so: hi");
}
Compile and run.
gcc-4.8 main -fpic -ldl -Wl,--unresolved-symbols=ignore-all -o main
gcc-4.8 dyload.c -shared -fpic -o dyload.so
$ ./main
dyload.so: hi
Note that the main executable itself was compiled as PIC.
I have an executable with four shared libraries and the dependency tree look like this: Executable app does a dlopen of foo.so and bar.so. foo.so in turn links to fooHelper.so and bar.so links to barHelper.so.
Now, the issue is that fooHelper.so and barHelper.so have some of the same symbols. For instance, let us say we have a func with different implementations in fooHelper.so and barHelper.so. Is there a way to force foo.so to use fooHelper.so's implementation and bar.so to use barHelper.so's? What happens at present is that depending on the order of linking of the helpers, only one of the implementations of func is used by both foo.so and bar.so. This is because of the default Unix linkage model, if the definition of a symbol is already loaded, then any other definitions from shared libraries loaded subsequently are just discarded. Basically, func will be picked up from the helper library linked first. I need a way to explicitly specify the appropriate mapping without changing the source code of the shared libraries.
I'm working on Linux with g++ 4.4.
Is there a way to force foo.so to use fooHelper.so's implementation and bar.so to use barHelper.so's?
Yes: that's what RTLD_LOCAL is for (when dlopening foo.so and bar.so).
RTLD_LOCAL
This is the converse of RTLD_GLOBAL, and the default if neither flag
is specified. Symbols defined in this library are not made available
to resolve references in subsequently loaded libraries.
If both funcs happen to be in the same name-space, you're in a bit of trouble - if you are programming in C. The term to look for is "function overloading". There have been previous discussions on this topic, e.g. this one:
function overloading in C
EDIT: http://litdream.blogspot.de/2007/03/dynamic-loading-using-dlopen-api-in-c.html
With respect to the following link:
http://www.archlinux.org/news/libpnglibtiff-rebuilds-move-from-testing/
Could someone explain to me why a program should be rebuilt after one of its libraries has been updated?
How does that make any sense since the "main" file is not changed at all?
If the signatures of the functions involved haven't changed, then "rebuilding" the program means that the object files must be linked again. You shouldn't need to compile them again.
An API is contract that describes the interface to the public functions in a library. When the compiler generates code, it needs to know what type of variables to pass to each function, and in what order. It also needs to know the return type, so it knows the size and format of the data that will be returned from the function. When your code is compiled, the address of a library function may be represented as "start of the library, plus 140 bytes." The compiler doesn't know the absolute address, so it simply specifies an offset from the beginning of the library.
But within the library, the contents (that is, the implementations) of the functions may change. When that happens, the length of the code may change, so the addresses of the functions may shift. It's the job of the linker to understand where the entry points of each function reside, and to fill those addresses into the object code to create the executable.
On the other hand, if the data structures in the library have changed and the library requires the callers to manage memory (a bad practice, but unfortunately common), then you will need to recompile the code so it can account for the changes. For example, if your code uses malloc(sizeof(dataStructure)) to allocate memory for a library data structure that's doubled in size, you need to recompile your code because sizeof(dataStructure) will have a larger value.
There are two kinds of compatibility: API and ABI.
API compatibility is about functions and data structures which other programs may rely on. For instance if version 0.1 of libfoo defines an API function called "hello_world()", and version 0.2 removes it, any programs relying on "hello_world()" need updating to work with the new version of libfoo.
ABI compatibility is about the assumptions of how functions and, in particular, data structures are represented in the binaries. If for example libfoo 0.1 also defined a data structure recipe with two fields: "instructions" and "ingredients" and libfoo 0.2 introduces "measurements" before the "ingredients" field then programs based on libfoo 0.1 recipes must be recompiled because the "instructions" and "ingredients" fields will likely be at different positions in the 0.2 version of the libfoo.so binary.
What is a "library"?
If a "library" is only a binary (e.g. a dynamically linked library aka ".dll", ".dylib" or ".so"; or a statically linked library aka ".lib" or ".a") then there is no need to recompile, re-linking should be enough (and even that can be avoided in some special cases)
On the other hand, libraries often consist of more than just the binary object - e.g. the header-files might include some in-line (or macro) logic.
if so, re-linking is not enough, and you might to have to re-compile in order to make use of the newest version of the lib.