Is it possible have a global variable defined inside an application, accessed by a shared library that has been loaded with dlopen()?
I have it declared as an extern but when the app loads and tries to use it I get an undefined symbol error.
I am loading the library with flags set to RTLD_LAZY | RTLD_GLOBAL.
You have to you build your application where you have a global variable with -rdynamic option for g++. This option instructs the linker (ld) to add all symbols, not only used ones, to the dynamic symbol table of your application.
This is an example how I build my test C++ application that loads a shared library. The shared library uses a global variable in main.cpp. So I have added -rdynamic when build my main application:
g++ -rdynamic -m64 -g main.cpp -o main -ldl
When g++ finds -rdynamic it passes the flag -export-dynamic to the ELF linker (ld).
This is from man ld (which actually creates the dynamic symbol table):
If you use "dlopen" to load a dynamic object which needs to refer back
to the symbols defined by the program, rather than some other dynamic
object, then you will probably need to use this option when linking
the program itself.
Useful links:
http://gcc.gnu.org/onlinedocs/gcc-4.9.0/gcc/Link-Options.html
http://linux.die.net/man/1/ld
Related
Let's say we have a library libutils which implements two functions: function1 and function2. And let's say function2 uses library libanother (so libutils depends on libanother).
Now I want to build my application that uses function1 only:
g++ myapplication.cpp -o my application -lutils
Is it required to include libanother in linkage:
g++ myapplication.cpp -o my application -lutils -lanother
Is there any linker requirements concerning that? Does the answer depend on whether the libraries libutils and libanother are shared or static?
Some first principles
Three kinds of files can be input to the linkage of a program:
object files
static libraries
shared libraries
But only two of those kinds of files can actually be linked into the program:
object files and shared libraries.1
If you input any object file p.o to the linkage of a program the linker
links it into the program unconditionally.
If you input a static library libx.a to the your linkage, the linker extracts from libx.a each archive member
libx.a(q.o) that it needs to carry on the linkage and links it into
the program. An archive member libx.a(q.o) is needed if it defines at least one
symbol that has been referenced, but not yet defined, in some file (object file or
shared library) that has already been linked into the program.
If you input a shared library liby.so, the story forks:-
If the linker (ld) option --as-needed is in effect when liby.so is reached,
then liby.so will be linked into the program if it is needed in the same sense as an archive member libx.a(q.o).
If --as-needed is not in effect, then liby.so is linked unconditionally, like
an object file.
If you invoke the linker via one of your distro's GCC frontends (gcc, g++, gfortran...) in the usual
way, then you will always get the --as-needed behaviour for shared libraries by default, or always not get it, as long as your
GCC toolchain remains the same, because your distro's build of GCC will configure --as-needed
as a default linkage option, or not, according to that distro's appraisal of the pros and cons of doing so.
Debian-derived distros (Ubuntu et al.) have defaulted to --as-needed in recent years. Last time I looked,
Redhat-derived distros (Fedora et al.) did not.2
When any file file.{o|so} is linked into the program it introduces 0 or more unresolved references into the program. Those unresolved references are
references to symbols in file.{o|so} that are not defined in file.{o|so} or in any other file already linked into the program.
The linker must resolve all references by the time it has finished the linkage of the
program, or the linkage fails with unresolved reference errors. Files must be input and linked that provide definitions for
all of the symbols that are referenced in the files that are linked.
Down to your case
When you say:
my application ... uses function1 only
you mean that some file will be linked that references function1. It's simple
to assume this file is some object file that will be linked unconditionally,
say the one that defines main, e.g.
main.c
#include <stdio.h>
extern void function1(void);
void call_function1(void) {
function1();
}
int main(void)
{
puts("Hello World!");
}
which we compile to an object file:
$ gcc -Wall -c main.c
Notice that main.o references function1 despite the fact that we cannot
run a program with that main function in any way that will call
function1. main.o defines a function,
call_function1, that calls function1; so function1 is referenced in main.o.
call_function1 is not referenced itself, but function1 is. If we look at
the symbol table of main.o:
$ nm main.o
0000000000000000 T call_function1
U function1
U _GLOBAL_OFFSET_TABLE_
000000000000000c T main
U puts
the symbols annotated T are defined. The ones annotated U are referenced
but not defined. So when we try to link main.o into a program:
$ gcc -o prog main.o
/usr/bin/ld: main.o: in function `call_function1':
main.c:(.text+0x5): undefined reference to `function1'
collect2: error: ld returned 1 exit status
we need to input a file that defines function1, even though a program linked
with this main.o cannot possibly call function1.
Your function1 is defined, somehow, within a library libutils, which also
defines function2. Your program does not reference function2, but
the definition of function2 references one or more symbols, say function3, that are defined,
somehow, within another library libanother.
Do you need to add libanother, as well as libutils to the linkage of your
program?
From first principles, we know you must add libanother to your linkage if you link a file that references function3.
function3 is referenced in the definition of function2. So you must link with libanother
if you link a file that contains the definition of function2.
Do you need to link such a file?
That depends on the implementation of libutils itself.
Case 1
Suppose you have implemented libutils as a static library in which you have
archived an object file that defines function1 and another object file that
defines function2. Like so:
function1.c
#include <stdio.h>
void function1(void)
{
puts(__func__);
}
function2.c
extern void function3();
void function2(void)
{
function3();
}
Compile the source files:
$ gcc -Wall -c function1.c function2.c
Archive the object files:
$ ar rcs libutils.a function1.o function2.o
Link prog like:
$ gcc -o prog main.o libutils.a
No problem. Run:
$ ./prog
Hello World!
Let's do the linkage again, this time with diagnostics to show what
object files and shared libraries were actually linked:
$ gcc -o prog main.o libutils.a -Wl,-trace
/usr/bin/ld: mode elf_x86_64
/usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/Scrt1.o
/usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/crti.o
/usr/lib/gcc/x86_64-linux-gnu/8/crtbeginS.o
main.o
(libutils.a)function1.o
libgcc_s.so.1 (/usr/lib/gcc/x86_64-linux-gnu/8/libgcc_s.so.1)
/lib/x86_64-linux-gnu/libc.so.6 (//lib/x86_64-linux-gnu/libc.so.6)
(//usr/lib/x86_64-linux-gnu/libc_nonshared.a)elf-init.oS
/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (//lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (//lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
libgcc_s.so.1 (/usr/lib/gcc/x86_64-linux-gnu/8/libgcc_s.so.1)
/usr/lib/gcc/x86_64-linux-gnu/8/crtendS.o
/usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/crtn.o
Most of those inputs are boilerplate object files and shared libraries that
gcc adds to program linkage by default. The files that we built are just:
main.o
(libutils.a)function1.o
The archive member libutils.a(function2.o) wasn't linked at all, because
no file was linked that references function2. So it doesn't matter that
libutils.a(function2.o) makes an undefined reference to function3. libutils.a(function2.o)
might as well not exist. The linkage:
$ gcc -o prog main.o libutils.a
is exactly the same linkage as:
gcc -o prog main.o function1.o
Case 2
Suppose you've implemented libutils as a static library that contains
one object file that defines both function1 and function2:
function12.c
#include <stdio.h>
void function1(void)
{
puts(__func__);
}
extern void function3(void);
void function2(void)
{
function3();
}
Compile that file:
$ gcc -Wall -c function12.c
Recreate libutils.a:
$ rm libutils.a
$ ar rcs libutils.a function12.o
Relink prog:
$ gcc -o prog main.o libutils.a -Wl,-trace
/usr/bin/ld: mode elf_x86_64
/usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/Scrt1.o
/usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/crti.o
/usr/lib/gcc/x86_64-linux-gnu/8/crtbeginS.o
main.o
(libutils.a)function12.o
libgcc_s.so.1 (/usr/lib/gcc/x86_64-linux-gnu/8/libgcc_s.so.1)
/lib/x86_64-linux-gnu/libc.so.6 (//lib/x86_64-linux-gnu/libc.so.6)
(//usr/lib/x86_64-linux-gnu/libc_nonshared.a)elf-init.oS
/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (//lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (//lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
libgcc_s.so.1 (/usr/lib/gcc/x86_64-linux-gnu/8/libgcc_s.so.1)
/usr/lib/gcc/x86_64-linux-gnu/8/crtendS.o
/usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/crtn.o
/usr/bin/ld: libutils.a(function12.o): in function `function2':
function12.c:(.text+0x32): undefined reference to `function3'
/usr/bin/ld: link errors found, deleting executable `prog'
collect2: error: ld returned 1 exit status
This time, as the -trace output shows, our object files:
main.o
(libutils.a)function12.o
were linked. main.o referenced function1, so the linker searched libutils.a
for an object file that defines function1. It found function12.o and linked
it, including the definition of function2 as well as the definition of function1.
The definition of function2 introduced an unresolved reference to function3,
into the program. No file was linked that resolved that reference, so the
linkage failed:
function12.c:(.text+0x32): undefined reference to `function3'
It doesn't matter that function2 cannot be called by the program -
just like it doesn't matter that function1 cannot be called.3
Case 3
You have implemented libutils as a shared library, by linking position-independent
object files function1.o and function2.o compiled from function1.c and function2.c
respectively:
$ gcc -Wall -fPIC -c function1.c function2.c
$ gcc -shared -o libutils.so function1.o function2.o
Notice here that although the shared library libutils.so is an ELF file produced
by linkage (not an ar archive of object files), we are able to link it successfully
despite the fact that function2.o contains an undefined reference to function3.
It is fine to link a shared library with undefined references. See its symbol table:
$ nm libutils.so
0000000000004030 b completed.7930
w __cxa_finalize##GLIBC_2.2.5
0000000000001060 t deregister_tm_clones
00000000000010d0 t __do_global_dtors_aux
0000000000003e18 t __do_global_dtors_aux_fini_array_entry
0000000000004028 d __dso_handle
0000000000003e20 d _DYNAMIC
0000000000001150 t _fini
0000000000001110 t frame_dummy
0000000000003e10 t __frame_dummy_init_array_entry
00000000000020f0 r __FRAME_END__
0000000000002020 r __func__.2361
0000000000001115 T function1
0000000000001142 T function2
U function3
0000000000004000 d _GLOBAL_OFFSET_TABLE_
w __gmon_start__
000000000000202c r __GNU_EH_FRAME_HDR
0000000000001000 t _init
w _ITM_deregisterTMCloneTable
w _ITM_registerTMCloneTable
U printf##GLIBC_2.2.5
0000000000001090 t register_tm_clones
There is U function3. But you can't link a program with undefined references:
$ gcc -o prog main.o libutils.so -Wl,-trace
/usr/bin/ld: mode elf_x86_64
/usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/Scrt1.o
/usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/crti.o
/usr/lib/gcc/x86_64-linux-gnu/8/crtbeginS.o
main.o
libutils.so
libgcc_s.so.1 (/usr/lib/gcc/x86_64-linux-gnu/8/libgcc_s.so.1)
/lib/x86_64-linux-gnu/libc.so.6 (//lib/x86_64-linux-gnu/libc.so.6)
(//usr/lib/x86_64-linux-gnu/libc_nonshared.a)elf-init.oS
/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (//lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (//lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
libgcc_s.so.1 (/usr/lib/gcc/x86_64-linux-gnu/8/libgcc_s.so.1)
/usr/lib/gcc/x86_64-linux-gnu/8/crtendS.o
/usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/crtn.o
/usr/bin/ld: libutils.so: undefined reference to `function3'
/usr/bin/ld: link errors found, deleting executable `prog'
collect2: error: ld returned 1 exit status
When we link:
main.o
libutils.so
the linkage fails just as with:
main.o
(libutils.a)function12.o
or indeed:
$ gcc -o prog main.o function1.o function2.o
/usr/bin/ld: function2.o: in function `function2':
function2.c:(.text+0x5): undefined reference to `function3'
collect2: error: ld returned 1 exit status
So:
Is it required to link in all dependencies?
Yes, when you're linking a program. But understand what the dependencies of
a program are:
The dependencies of the program are object files and/or shared libraries
that define the symbols that are referenced in the object files and shared libraries
that are actually linked.
A static library is never one those. It is a container of object files from which
the linker will extract and link ones that are dependencies of the program and ignore
the rest.
[1] What the linker does to link an object file into a program is fundamentally
different from what it does to link a shared library.
Roughly, to link an object file, it physically merges the parts of which the object
file is composed into the corresponding parts of the output program.
Roughly, to link a shared library liby.so, the linker just writes some concise information in
a standard format into the program that will be parsed by the runtime loader,
instructing it to find and load liby.so and merge the parts of which it is
composed into the process of the executed program.
[2] You can check whether your GCC toolchain defaults to --as-needed or not
by grepping for that option in the verbose output of a default GCC linkage, e.g.
$ gcc -v -o prog main.o function1.o 2>&1 | grep '\--as-needed'
[3] We can compile object files with a non-default compilation option -ffunction-sections
and link them in programs with a non-default linker option -gc-sections, with the
effect that the linker will be able to discard definitions of functions that
cannot called, so that they cannot give rise to any references that impose useless
dependencies on the program.
If no call made from second library, it is not required. Typically linker will give error if function in linking for which library or its definition is not present during linking as undefined.
This question already has answers here:
Use both static and dynamically linked libraries in gcc
(3 answers)
Closed 5 years ago.
First of all I would like to give some background information to avoid the XY Problem.
I am trying to compile a c++ program using makefiles and g++. I also have to build any dependencies statically into the program, but not the 'system libraries' (libz.so, libdl.so, libstdc++.so, libm.so, libpthread.so, libc.so and libgcc.so).
To achieve this I specify -static as linker flag, then all dependencies that have to be statically linked and then I use the -Wl, -Bdynamic option, that should tell the linker to link every library, after this option, to be linked in dynamically including the 'system libraries' because they get linked last.(please correct me if I am wrong.)
LDFLAGS += -Lpath/to/dependencies
# These libs should be linked statically
LDFLAGS += -static
LDFLAGS += -llib1
LDFLAGS += -llib2
LDFLAGS += -llib3
# Libs after this should be linked dynamically.
LDFLAGS += -Wl, -Bdynamic
LDFLAGS += -lz # If i dont specify these three libraries (z, pthread, dl)
LDFLAGS += -lpthread # I get undefined reference errors
LDFLAGS += -ldl
When I call make, the program compiles and links just fine, but when I try to execute it I get the error: bash: ./program: No such file or directory.
But the file DOES exist.
When I remove the -static flag from the linker, the program can be excecuted just fine, but the dependencies are linked dynamically, which is not what I want :(.
So when I call file on the program, that was made with the -static flag, I get this:
program: ELF 32-bit LSB executable, Intel 80386, version 1 (GNU/Linux), dynamically linked, interpreter /usr/lib/libc.so.1, for GNU/Linux 4.9.0, not stripped
The problem seems to be that the interpreter is set to /usr/lib/libc.so.1 and not to /lib/ld-linux.so.2 as it should usually be. At least it is, when I compile without the -static option.
What I found out is that the 'interpreter' is in reality the shared library loader and from what I read, I now assume that the bash tells me it can not find the program because the library loader is just wrong (Even though I don't quite understand the details of this).
So basically my question is: Why does the library loader get set to libc.so when I specify the -static option to the linker and how can I tell the linker to use the correct library loader when -static is specified?
Your error is mixing -static and -Bdynamic as both compiler and linker flags. Don't do that. If you use -Wl, gcc just blindly pass these flags to the linker, but if you don't, it rearranges the entire link line. (check what it does with gcc -v).
The mix creates an inconsistent and erroneous link command. I have no idea why gcc doesn't at least warn about that, but it doesn't, and silently sets the dynamic loader to a non-existent file.
You want to use -Wl,-Bstatic and -Wl,-Bdynamic consistently throughout. Not -Bstatic and -Bdynamic, as gcc logic is different from that of ld.
This will create a correct dynamically linked executable with some static libs linked in.
I am building a binary executable in Rust and it needs to link to a native library, say foo.a. foo.a contains a symbol void bar(void), which I would like to expose to the dynamic linker as a callback function that can be called by functions in a dlopen-style dynamically loaded library.
This can be done in ld by supplying -rdynamic if we use C source.
gcc -rdynamic -o a_dynamic main.c foo.c
What is the proper way of doing this in Rust? I have tried using cargo:rustc-flags=-rdynamic in build.rs, as well as
#![feature(link_args)]
#[link_args = "-rdynamic"]
Neither seems to work.
As of today, Rust toolchain discourages passing arbitrary flags to the linker. The closest we can do as a proper way is using cargo rustc and manually add the linking arguments.
cargo rustc -- -C link-args='-rdynamic'
I have a problem building a shared library with GCC/Linux. Currently this shared library is created with GCC/libtool option "-shared" and everything is fine.
Now there are two additional, static libraries (.a-files) that have to be added to this shared one since they provide some functionality that is required by the shared one. Adding these static libraries with option "-l" does not help, afterwards they are not part of the .so file.
So how can I force GCC/libtool to really add the code of these static libraries to the shared library?
Thanks!
You need --whole-archive linker option in this case to command the linker to include whole static libs' content into the shared lib.
g++ -shared sample.o -o libSample.so -Wl,-whole-archive -lmylib1.a -lmylib2.a -Wl,-no-whole-archive
From man ld:
For each archive mentioned on the command line after the --whole-archive option, include every object file in the archive in the link, rather than searching the archive for the required object files. This is normally used to turn an archive file into a shared library, forcing every object to be included in the resulting shared library. This option may be used more than once.
Two notes when using this option from gcc: First, gcc doesn't know about this option, so you have to use -Wl,-whole-archive. Second, don't forget to use -Wl,-no-whole-archive after your list of archives, because gcc will add its own list of archives to your link and you may not want this flag to affect those as well.
You only need the --whole-archive parameter to force the linker to include the library, but it should be able to infer its own needs from unmatched symbols.
Ensure that any static libraries on the command-line come after their dependent object-files e.g.:
g++ -Wl,-E -g -pipe -O2 -pipe -fPIC myobjectfile.o mystaticlibrary.a -shared -o mylib.so
I am working in Linux, Eclipse CDT, g++, with Boost library. Having existing program which uses Boost thread, I try to link it statically instead of dynamically. /usr/local/lib directory contains the following files:
libbost_thread.a
libbost_thread.so
libbost_thread.1.41.0
Dynamic linking works:
g++ -o"MyProgram" ./main.o -lboost_thread
Static linking:
g++ -static -o"MyProgram" ./main.o -lboost_thread
produces huge number of messages like:
undefined reference to `pthread_mutex_init'
How can I link statically to the Boost library?
For pthread_mutex_init, you want to compile/link with -pthread option:
g++ -static -pthread -o"MyProgram" ./main.o -lboost_thread
The problem is that functions like pthread_mutex_init are in a separate library. Dynamic libraries can include the metadata for the fact that it needs the separate library (so libboost_thread.so includes the fact that it needs libpthread).
But static libraries don't have that information. So you need to provide the reference to any necessary libraries when you link statically.
As for using -pthread instead of -lpthread, it's slightly preferable because it not only links the necessary library, but provides any other options that should be used (such a -D_REENTRANT to the compiler).
Try adding -lpthread to your invocation.
On Linux a dynamic library may automatically depend on other dynamic libraries so that when you link it, you get the other libraries for free. When linking statically, there is no such system and you have to specify the other libraries manually.