I'm currently linking with two third party shared libraries (A.so and B.so) on linux. the problem is that both of the two so's linked statically with another library, as a result there are about 400 functions from A.so and B.so having the same names.
When I compile and link with -lA -lB, or -lB -lA, depending on the order the functions are picked up from A or B separately as a result of function interposition which caused problem and the code cannot run. I'm wondering if there's a way to bind function names to their libraries so both libraries can be linked and used? because those overlapping function names are called internally within A and B, so I can't use things like objcopy, etc. will dlopen help?
I'm wondering if there's a way to bind function names to their libraries so both libraries can be linked and used?
When the two libraries were linked, they should have controlled the symbols they export, and should have hidden the "other" library, but they didn't ...
will dlopen help?
Yes: if you dlopen("A.so", RTLD_LOCAL); and dlopen("B.so", RTLD_LOCAL);, then neither library will be added to the global scope, and they will not "see" each other.
You'll have to explicitly lookup symbols you need from A.so and B.so, but that's the best you could do.
Update:
is there a quick way to link to a static library without exporting symbols from that "other" library while building A.so
This is best done by using -fvisibility=hidden flag and __attribute__((visibility("default"))) on symbols that should be exported. Example:
#define EXPORTED __attribute__((visibility("default")))
struct Foo {
void EXPORTED ExportedFunction();
void EXPORTED AnotherExportedFunction();
void InternalFunction();
};
void Foo::ExportedFunction() { }
void Foo::AnotherExportedFunction() { }
void Foo::InternalFunction() { }
gcc -shared -fPIC -o foo.so foo.cc
nm -CD foo.so | grep Foo::
00000000000005fc T Foo::ExportedFunction()
0000000000000610 T Foo::InternalFunction()
0000000000000606 T Foo::AnotherExportedFunction()
Without explicit export control, everything gets exported (including InternalFunction we don't want).
gcc -shared -fPIC -o foo.so foo.cc -fvisibility=hidden
nm -CD foo.so | grep Foo::
00000000000005bc T Foo::ExportedFunction()
00000000000005c6 T Foo::AnotherExportedFunction()
Voilà: only things we explicitly wanted to export are.
Related
Is there any way we can get gcc to detect a duplicate symbol in static libraries vs the main code (Or another static library ?)
Here's the situation:
main.c erroneously contained a function definition, e.g. with the signature uint foohash(const char*)
foo.c also contains a function definition with the signature uint foohash(const char*)
foo.c and other source files are compiled to a static util library, which the main program links in, i.e. something like:
gcc -o main main.o util.o -L ./libs -lfooutils
So, now main.o and libs/libfooutils.a both contain a foohash function. Presumably the linker found that symbol in main.o and doesn't bother looking for it elsewhere.
Is there any way we can get gcc to detect such a situation ?
Indeed as Simon Richter stated, --whole-archive option can be useful. Try to change your command-line to:
gcc -o main main.o util.o -L ./libs -Wl,--whole-archive -lfooutils -Wl,--no-whole-archive
and you'll see a multiple definition error.
gcc calls the ld program for linking. The relevant ld options are:
--no-define-common
--traditional-format
--warn-common
See the man page for ld. These should be what you need to experiment with to get the warnings sought.
Short answer: no.
GCC does not actually do anything with libraries. It is the task of ld, the linker (called automatically by GCC) to pull in symbols from libraries, and that's really a fairly dumb tool.
The linker has lots of complex jiggery pokery for combining different types of data from different sources, and supporting different file formats, and all the evil little details of binary executables, but in the end, all it really does is look for undefined symbols and find the definitions.
What you can do is a link trace (pass -t to gcc) to see what comes from where. Or else run nm on all the object files and libraries in your system, and write a script to detect duplicates.
Does anyone know how to use gcc to generate all possible binary files from object files ? I know you can use : "gcc -MM" to generate all the .o files for some given source files.
But how would you use gcc to generate all possible binary files from object files
in a project ?
Example: I use "gcc -MM" to generate: a.o, b.o, c.o, d.o
If one were trying to generate a list of binaries files built from each of the .o files like this:
a: b.o c.o d.o
b: a.o c.o d.o
c: a.o b.o d.o
d: a.o b.o b.o
I can do this with a Perl script, but I was just curious if there was some way to do it with gcc
Thanks
Short answer, "no, but..."
gcc -MM can give you foo.o: bar.h because foo.cc contains the directive:
#include "bar.h"
That's easy. But foo.cc can also contain the declaration:
int bar_f1(int);
How can gcc know which object file contains the binary code for this function? Or if there are two object files containing functions with this signature, which it should use? It can't.
...Unless...
Long answer, "yes, if..."
If you refrain from giving source files forward declarations of things in other source files, and also refrain from giving a header file declarations of anything not contained in the corresponding source file, and also give the source file containing int main(...) a fixed name like, say, main.cc, then you can take the output of gcc -MM:
bar.o: bar.h baz.h
foo.o: bar.h
main.o: foo.h zot.h
pan.o: pan.h
zot.o: zot.h
and transform it without too much trouble (using e.g Perl or sed) into:
main: bar.o foo.o zot.o
In theory you could get by without these restrictions(*) by scanning the object files and constructing the dependency tree; this might be equivalent to scanning for the presence of int main(...), segregating those files and linking each of them against all the others, or just listing them, if all you want is a list of possible executables without dependencies (I'm still not sure exactly what you want). These things still require some scripting on your part, I know of no way to do them with gcc alone.
(*)You must still refrain from having two definitions of the same thing.
I compiled an .cc file with the following command, which is in Makefile code:
bin/bash ../libtool --silent --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H -I.
-I.. -I../include/ -I.. -g -O2 -MT rtpsession_inet.lo -MD -MP -MF
.deps/rtpsession_inet.Tpo -c -o rtpsession_inet.lo rtpsession_inet.cc
There is a function named rtp_session_rtp_recv in the .cc file. However, it is said that the reference of this function cannot be found when I use the library generated by the Makefile.
So I checked the .o file generated by rtpsession_inet.cc and find that there is not a function named rtp_session_rtp_recv while the function name is changed to _Z20rtp_session_rtp_recvP11_RtpSessionj.
Meanwhile, there are several other functions changes their name, e.g. rtp_session_rtp_send -> _Z20rtp_session_rtp_sendP11_RtpSessionP4msgb.
But functions such as rtp_session_set_remote_addr_full are not changed at all.
What is the additional characters' meaning? How can I deal with this problem?
I compile the file in Linux and use command
nm rtpsession_inet.o
to read the .o file. (All the functions including the one with incorrect name are with T tag, which means the reference exists)
Thanks!
This is called name mangling.
It's for the benefit of the linker. A C++ compiler is able to resolve multiple functions of the same name, based on their argument types. For example, you might have a function called print that takes an int argument, and another that takes a char* argument. The compiler knows which one to generate a call to based on what type of argument you pass to it.
Calls across translation units, though, have to be resolved by the linker, which typically is not aware of C++ overloading rules and has to resolve calls based only on the name. So the C++ compiler decorates the name with enough information to resolve the overload.
The C++ standard doesn't specify how this is done, but if you look at the name you can probably work out how the mangled name is generated.
How can I deal with this problem?
The compiler and linker should resolve calls correctly. What problem are you referring to?
The additional characters are symbol "decorations" added by the compiler and are used to identify the function/method signature, i.e. the return type and the parameters. It helps the runtime determine which one of many functions of the same name (overloading) to invoke in any given invocation.
I have a program, myprogram, which is linked with a static convenience library, call it libconvenience.a, which contains a function, func(). The function func() isn't called anywhere in myprogram; it needs to be able to be called from a plugin library, plugin.so.
The symbol func() is not getting exported dynamically in myprogram. If I run
nm myprogram | grep func
I get nothing. However, it isn't missing from libconvenience.a:
nm libconvenience/libconvenience.a | grep func
00000000 T func
I am using automake, but if I do the last linking step by hand on the command line instead, it doesn't work either:
gcc -Wl,--export-dynamic -o myprogram *.o libconvenience/libconvenience.a `pkg-config --libs somelibraries`
However, if I link the program like this, skipping the use of a convenience library and linking the object files that would have gone into libconvenience.a directly, func() shows up in myprogram's symbols as it should:
gcc -Wl,--export-dynamic -o myprogram *.o libconvenience/*.o `pkg-config --libs somelibraries`
If I add a dummy call to func() somewhere in myprogram, then func() also shows up in myprogram's symbols. But I thought that --export-dynamic was supposed to export all symbols regardless of whether they were used in the program or not!
I am using automake 1.11.1 and gcc 4.5.1 on Fedora 14. I am also using Libtool 2.2.10 to build plugin.so (but not the convenience library.)
I didn't forget to put -Wl,--export-dynamic in myprogram_LDFLAGS, nor did I forget to put the source that contains func() in libconvenience_a_SOURCES (some Googling suggests that these are common causes of this problem.)
Can somebody help me understand what is going on here?
I managed to solve it. It was this note from John Calcote's excellent Autotools book that pointed me in the right direction:
Linkers add to the binary product every object file specified explicitly on the command line, but they only extract from archives those object files that are actually referenced in the code being linked.
To counteract this behavior, one can use the --whole-archive flag to libtool. However, this causes all the symbols from all the system libraries to be pulled in also, causing lots of double symbol definition errors. So --whole-archive needs to be right before libconvenience.a on the linker command line, and it needs to be followed by --no-whole-archive so that the other libraries aren't treated that way. This is a bit difficult since automake and libtool don't really guarantee keeping your flags in the same order on the command line, but this line in Makefile.am did the trick:
myprogram_LDFLAGS = -Wl,--export-dynamic \
-Wl,--whole-archive,libconvenience/libconvenience.a,--no-whole-archive
If you need func to be in plugin.so, you should try and locate it there if possible. Convenience libraries are meant to be just that -- a convenience to link to an executable or lib as an intermediate step.
I know that by default undefined symbols are ignored at compile time. However, I would also like them to be ignored at run-time. I need to distribute a .so that will run with and without MPI. I will know ahead of time if it is an MPI job and if it is not I won't make any MPI_* calls. If it's not an MPI run I need the application to not care that it cannot resolve the MPI_* symbols.
Is this possible? I could have sworn I've done this before but I can't get it working. Everytime I run I immediately get the following even though the logic in my code will never allow that symbol to be referenced:
undefined symbol: hpmp_comm_world
For what it's worth I am using the Intel Fortran Compiler to build the .so file.
EDIT
I found the linker flag: "-z lazy" which is supposed to resolve references to functions when the function is called which is what I want. That doesn't fix my problem, but hpmp_comm_world is a variable - not a function. Should that make a difference?
You can define a symbol to be a weak reference to its definition. Then, the symbol's value will be zero if the definition is not present.
For example, suppose the following is ref.c, which references a function and variable that may or may not be present; we'll use it to build libref.so (corresponding to your library, in your question):
#include <stdio.h>
void global_func(void);
void global_func(void) __attribute__ ((weak));
extern int global_variable __attribute__((weak));
void ref_func() {
printf("global_func = %p\n", global_func);
if (&global_variable)
global_variable++;
if (global_func)
global_func();
}
Here, global_func and global_variable are the weak references to the possibly-available function and variable. This code prints the function's address, increments the variable if it is present, and calls the function if it is present. (Note that the function's and variable's addresses are zero when they are not defined, so it is &global_variable that you must compare with zero.)
And suppose this is def.c, which defines global_func and global_variable; we'll use it to build libdef.so (corresponding to MPI, in your question):
#include <stdio.h>
int global_variable;
void global_func(void) {
printf("Hi, from global_func! global_variable = %d\n", global_variable);
}
And finally, suppose we have a main program, main.c, which calls ref_func from libref.so:
#include <stdio.h>
extern void ref_func(void);
int main(int argc, char **argv) {
printf("%s: ", argv[0]);
ref_func();
return 0;
}
Here's the Makefile that builds libref.so and libdef.so, and then builds two executables, both of which link against libref.so, but only one of which links against libdef.so:
all: ref-absent ref-present
ref-absent: main.o libref.so
$(CC) $(CFLAGS) $(LDFLAGS) $^ -o $#
ref-present: main.o libref.so libdef.so
$(CC) $(CFLAGS) $(LDFLAGS) $^ -o $#
lib%.so: %.o
$(CC) $(CFLAGS) $(LDFLAGS) -shared $^ -o $#
ref.o def.o: CFLAGS += -fpic
clean:
rm -f *.o *.so ref-absent ref-present
Do the build:
$ make
cc -c -o main.o main.c
cc -fpic -c -o ref.o ref.c
cc -shared ref.o -o libref.so
cc main.o libref.so -o ref-absent
cc -fpic -c -o def.o def.c
cc -shared def.o -o libdef.so
cc main.o libref.so libdef.so -o ref-present
$
Note that both ref-absent and ref-present linked without problems, even though there is no definition for global_name in ref-absent.
Now we can run the programs, and see that ref-absent skips the function call, while ref-present uses it. (We have to set LD_LIBRARY_PATH to allow the dynamic linker to find our shared libraries in the current directory.)
$ LD_LIBRARY_PATH=. ./ref-absent
./ref-absent: global_func = (nil)
$ LD_LIBRARY_PATH=. ./ref-present
./ref-present: global_func = 0x15d4ac
Hi, from global_func! global_variable = 1
$
The trick for you will be getting the ((weak)) attribute attached to every declaration of every MPI function your library references. However, as ref.c shows, there can be multiple declarations, and as long as one of them mentions the weak attribute, you're done. So you'll probably have to say something like this (I don't really know MPI):
#include <mpi.h>
mpi_fake_type_t mpi_function_foo(mpi_arg_type_t) __attribute__((weak));
mpi_fake_type_t mpi_function_bar(mpi_other_arg_type_t) __attribute__((weak));
Every reference to an MPI function needs to be in the scope of a ((weak)) declaration for that function; that's how the compiler decides what sort of symbol reference to put in the object file. You'll want to have automated tests to verify that you haven't accidentally generated any non-weak references.