Let's say we have a library libutils which implements two functions: function1 and function2. And let's say function2 uses library libanother (so libutils depends on libanother).
Now I want to build my application that uses function1 only:
g++ myapplication.cpp -o my application -lutils
Is it required to include libanother in linkage:
g++ myapplication.cpp -o my application -lutils -lanother
Is there any linker requirements concerning that? Does the answer depend on whether the libraries libutils and libanother are shared or static?
Some first principles
Three kinds of files can be input to the linkage of a program:
object files
static libraries
shared libraries
But only two of those kinds of files can actually be linked into the program:
object files and shared libraries.1
If you input any object file p.o to the linkage of a program the linker
links it into the program unconditionally.
If you input a static library libx.a to the your linkage, the linker extracts from libx.a each archive member
libx.a(q.o) that it needs to carry on the linkage and links it into
the program. An archive member libx.a(q.o) is needed if it defines at least one
symbol that has been referenced, but not yet defined, in some file (object file or
shared library) that has already been linked into the program.
If you input a shared library liby.so, the story forks:-
If the linker (ld) option --as-needed is in effect when liby.so is reached,
then liby.so will be linked into the program if it is needed in the same sense as an archive member libx.a(q.o).
If --as-needed is not in effect, then liby.so is linked unconditionally, like
an object file.
If you invoke the linker via one of your distro's GCC frontends (gcc, g++, gfortran...) in the usual
way, then you will always get the --as-needed behaviour for shared libraries by default, or always not get it, as long as your
GCC toolchain remains the same, because your distro's build of GCC will configure --as-needed
as a default linkage option, or not, according to that distro's appraisal of the pros and cons of doing so.
Debian-derived distros (Ubuntu et al.) have defaulted to --as-needed in recent years. Last time I looked,
Redhat-derived distros (Fedora et al.) did not.2
When any file file.{o|so} is linked into the program it introduces 0 or more unresolved references into the program. Those unresolved references are
references to symbols in file.{o|so} that are not defined in file.{o|so} or in any other file already linked into the program.
The linker must resolve all references by the time it has finished the linkage of the
program, or the linkage fails with unresolved reference errors. Files must be input and linked that provide definitions for
all of the symbols that are referenced in the files that are linked.
Down to your case
When you say:
my application ... uses function1 only
you mean that some file will be linked that references function1. It's simple
to assume this file is some object file that will be linked unconditionally,
say the one that defines main, e.g.
main.c
#include <stdio.h>
extern void function1(void);
void call_function1(void) {
function1();
}
int main(void)
{
puts("Hello World!");
}
which we compile to an object file:
$ gcc -Wall -c main.c
Notice that main.o references function1 despite the fact that we cannot
run a program with that main function in any way that will call
function1. main.o defines a function,
call_function1, that calls function1; so function1 is referenced in main.o.
call_function1 is not referenced itself, but function1 is. If we look at
the symbol table of main.o:
$ nm main.o
0000000000000000 T call_function1
U function1
U _GLOBAL_OFFSET_TABLE_
000000000000000c T main
U puts
the symbols annotated T are defined. The ones annotated U are referenced
but not defined. So when we try to link main.o into a program:
$ gcc -o prog main.o
/usr/bin/ld: main.o: in function `call_function1':
main.c:(.text+0x5): undefined reference to `function1'
collect2: error: ld returned 1 exit status
we need to input a file that defines function1, even though a program linked
with this main.o cannot possibly call function1.
Your function1 is defined, somehow, within a library libutils, which also
defines function2. Your program does not reference function2, but
the definition of function2 references one or more symbols, say function3, that are defined,
somehow, within another library libanother.
Do you need to add libanother, as well as libutils to the linkage of your
program?
From first principles, we know you must add libanother to your linkage if you link a file that references function3.
function3 is referenced in the definition of function2. So you must link with libanother
if you link a file that contains the definition of function2.
Do you need to link such a file?
That depends on the implementation of libutils itself.
Case 1
Suppose you have implemented libutils as a static library in which you have
archived an object file that defines function1 and another object file that
defines function2. Like so:
function1.c
#include <stdio.h>
void function1(void)
{
puts(__func__);
}
function2.c
extern void function3();
void function2(void)
{
function3();
}
Compile the source files:
$ gcc -Wall -c function1.c function2.c
Archive the object files:
$ ar rcs libutils.a function1.o function2.o
Link prog like:
$ gcc -o prog main.o libutils.a
No problem. Run:
$ ./prog
Hello World!
Let's do the linkage again, this time with diagnostics to show what
object files and shared libraries were actually linked:
$ gcc -o prog main.o libutils.a -Wl,-trace
/usr/bin/ld: mode elf_x86_64
/usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/Scrt1.o
/usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/crti.o
/usr/lib/gcc/x86_64-linux-gnu/8/crtbeginS.o
main.o
(libutils.a)function1.o
libgcc_s.so.1 (/usr/lib/gcc/x86_64-linux-gnu/8/libgcc_s.so.1)
/lib/x86_64-linux-gnu/libc.so.6 (//lib/x86_64-linux-gnu/libc.so.6)
(//usr/lib/x86_64-linux-gnu/libc_nonshared.a)elf-init.oS
/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (//lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (//lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
libgcc_s.so.1 (/usr/lib/gcc/x86_64-linux-gnu/8/libgcc_s.so.1)
/usr/lib/gcc/x86_64-linux-gnu/8/crtendS.o
/usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/crtn.o
Most of those inputs are boilerplate object files and shared libraries that
gcc adds to program linkage by default. The files that we built are just:
main.o
(libutils.a)function1.o
The archive member libutils.a(function2.o) wasn't linked at all, because
no file was linked that references function2. So it doesn't matter that
libutils.a(function2.o) makes an undefined reference to function3. libutils.a(function2.o)
might as well not exist. The linkage:
$ gcc -o prog main.o libutils.a
is exactly the same linkage as:
gcc -o prog main.o function1.o
Case 2
Suppose you've implemented libutils as a static library that contains
one object file that defines both function1 and function2:
function12.c
#include <stdio.h>
void function1(void)
{
puts(__func__);
}
extern void function3(void);
void function2(void)
{
function3();
}
Compile that file:
$ gcc -Wall -c function12.c
Recreate libutils.a:
$ rm libutils.a
$ ar rcs libutils.a function12.o
Relink prog:
$ gcc -o prog main.o libutils.a -Wl,-trace
/usr/bin/ld: mode elf_x86_64
/usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/Scrt1.o
/usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/crti.o
/usr/lib/gcc/x86_64-linux-gnu/8/crtbeginS.o
main.o
(libutils.a)function12.o
libgcc_s.so.1 (/usr/lib/gcc/x86_64-linux-gnu/8/libgcc_s.so.1)
/lib/x86_64-linux-gnu/libc.so.6 (//lib/x86_64-linux-gnu/libc.so.6)
(//usr/lib/x86_64-linux-gnu/libc_nonshared.a)elf-init.oS
/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (//lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (//lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
libgcc_s.so.1 (/usr/lib/gcc/x86_64-linux-gnu/8/libgcc_s.so.1)
/usr/lib/gcc/x86_64-linux-gnu/8/crtendS.o
/usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/crtn.o
/usr/bin/ld: libutils.a(function12.o): in function `function2':
function12.c:(.text+0x32): undefined reference to `function3'
/usr/bin/ld: link errors found, deleting executable `prog'
collect2: error: ld returned 1 exit status
This time, as the -trace output shows, our object files:
main.o
(libutils.a)function12.o
were linked. main.o referenced function1, so the linker searched libutils.a
for an object file that defines function1. It found function12.o and linked
it, including the definition of function2 as well as the definition of function1.
The definition of function2 introduced an unresolved reference to function3,
into the program. No file was linked that resolved that reference, so the
linkage failed:
function12.c:(.text+0x32): undefined reference to `function3'
It doesn't matter that function2 cannot be called by the program -
just like it doesn't matter that function1 cannot be called.3
Case 3
You have implemented libutils as a shared library, by linking position-independent
object files function1.o and function2.o compiled from function1.c and function2.c
respectively:
$ gcc -Wall -fPIC -c function1.c function2.c
$ gcc -shared -o libutils.so function1.o function2.o
Notice here that although the shared library libutils.so is an ELF file produced
by linkage (not an ar archive of object files), we are able to link it successfully
despite the fact that function2.o contains an undefined reference to function3.
It is fine to link a shared library with undefined references. See its symbol table:
$ nm libutils.so
0000000000004030 b completed.7930
w __cxa_finalize##GLIBC_2.2.5
0000000000001060 t deregister_tm_clones
00000000000010d0 t __do_global_dtors_aux
0000000000003e18 t __do_global_dtors_aux_fini_array_entry
0000000000004028 d __dso_handle
0000000000003e20 d _DYNAMIC
0000000000001150 t _fini
0000000000001110 t frame_dummy
0000000000003e10 t __frame_dummy_init_array_entry
00000000000020f0 r __FRAME_END__
0000000000002020 r __func__.2361
0000000000001115 T function1
0000000000001142 T function2
U function3
0000000000004000 d _GLOBAL_OFFSET_TABLE_
w __gmon_start__
000000000000202c r __GNU_EH_FRAME_HDR
0000000000001000 t _init
w _ITM_deregisterTMCloneTable
w _ITM_registerTMCloneTable
U printf##GLIBC_2.2.5
0000000000001090 t register_tm_clones
There is U function3. But you can't link a program with undefined references:
$ gcc -o prog main.o libutils.so -Wl,-trace
/usr/bin/ld: mode elf_x86_64
/usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/Scrt1.o
/usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/crti.o
/usr/lib/gcc/x86_64-linux-gnu/8/crtbeginS.o
main.o
libutils.so
libgcc_s.so.1 (/usr/lib/gcc/x86_64-linux-gnu/8/libgcc_s.so.1)
/lib/x86_64-linux-gnu/libc.so.6 (//lib/x86_64-linux-gnu/libc.so.6)
(//usr/lib/x86_64-linux-gnu/libc_nonshared.a)elf-init.oS
/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (//lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 (//lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
libgcc_s.so.1 (/usr/lib/gcc/x86_64-linux-gnu/8/libgcc_s.so.1)
/usr/lib/gcc/x86_64-linux-gnu/8/crtendS.o
/usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/crtn.o
/usr/bin/ld: libutils.so: undefined reference to `function3'
/usr/bin/ld: link errors found, deleting executable `prog'
collect2: error: ld returned 1 exit status
When we link:
main.o
libutils.so
the linkage fails just as with:
main.o
(libutils.a)function12.o
or indeed:
$ gcc -o prog main.o function1.o function2.o
/usr/bin/ld: function2.o: in function `function2':
function2.c:(.text+0x5): undefined reference to `function3'
collect2: error: ld returned 1 exit status
So:
Is it required to link in all dependencies?
Yes, when you're linking a program. But understand what the dependencies of
a program are:
The dependencies of the program are object files and/or shared libraries
that define the symbols that are referenced in the object files and shared libraries
that are actually linked.
A static library is never one those. It is a container of object files from which
the linker will extract and link ones that are dependencies of the program and ignore
the rest.
[1] What the linker does to link an object file into a program is fundamentally
different from what it does to link a shared library.
Roughly, to link an object file, it physically merges the parts of which the object
file is composed into the corresponding parts of the output program.
Roughly, to link a shared library liby.so, the linker just writes some concise information in
a standard format into the program that will be parsed by the runtime loader,
instructing it to find and load liby.so and merge the parts of which it is
composed into the process of the executed program.
[2] You can check whether your GCC toolchain defaults to --as-needed or not
by grepping for that option in the verbose output of a default GCC linkage, e.g.
$ gcc -v -o prog main.o function1.o 2>&1 | grep '\--as-needed'
[3] We can compile object files with a non-default compilation option -ffunction-sections
and link them in programs with a non-default linker option -gc-sections, with the
effect that the linker will be able to discard definitions of functions that
cannot called, so that they cannot give rise to any references that impose useless
dependencies on the program.
If no call made from second library, it is not required. Typically linker will give error if function in linking for which library or its definition is not present during linking as undefined.
Related
I am trying to compile the mosquitto library with my custom c program. So WHat I have done is wrote a hello.c file, git cloned the latest mosquitto library from the below repository:
https://github.com/eclipse/mosquitto.git
and compiled it with the make command as below:
make
I had to remove the doc target as it was asking for some dependancy library. I don't have admin rights on this machine, hence don't want to be blocked by any dependancy lib. After the compilation what I have is the below:
src/mosquitto
./lib/libmosquitto.so.1
The I copied the libmosquitto.so.1 shared lib into a local folder called ~/hello/:
~/hello$ cp ~/mosquitto/lib/libmosquitto.so.1 .
then wrote a hello.c inside ~/hello/ which is as below:
#include <stdio.h>
int main()
{
printf("Hello World\n");
return 0;
}
I can compile the hello.c and run it as below:
gcc -o hello hello.c
./hello
Hello World
But if I try to link the binary with the mosquitto library I get an error like the below:
gcc -o hello hello.c -lmosquitto
/usr/bin/ld: cannot find -lmosquitto
collect2: error: ld returned 1 exit status
The libmosquitto.so.1 lives in the same folder as the hello.c. I don't want to install the mosquitto library, rather would like to keep in a local folder and be able to link it. I have also tried the below with the hope that the -L. would point the linker to the present directory for the shared lib file but still get the same error:
gcc -o hello hello.c -L. -lmosquitto
/usr/bin/ld: cannot find -lmosquitto
collect2: error: ld returned 1 exit status
My ultimate objective is to cross compile the library for an arm target. So really need to understand how the linking of the shared library is failing so that I can use the same experience while cross compiling and link for the target. At the moment I am doing this on a x86 platform.
Can anyone please help?
/usr/bin/ld: cannot find -lmosquitto
The linker doesn't look for libmosquitto.so.1 -- it only looks for libmosquitto.a or libmosquitto.so.
Solution: ln -s libmosquitto.so.1 libmosquitto.so
./pub: error while loading shared libraries: libmosquitto.so.1: cannot open shared object file: No such file or directory
The problem here is that the runtime loader doesn't look in the current directory for libmosquitto.so.1 -- it only looks in system-configured directories.
You could fix this by adding export LD_LIBRARY_PATH=$HOME/mosquitto/lib, but this is suboptimal -- your binary will work or not depending on the environment.
A better solution is to change your link command like so:
gcc -o hello hello.c -L. -lmosquitto -Wl,-rpath=$HOME/mosquitto/lib
I have a libSomelib.a that can be linked to an executable by the following command:
g++ -L. -lsomeLib -lpcrecpp -lpcre -lpthread main.cpp -o main
But how could I link a shared object from it, that contains all depentencies?
I want to achieve the following with my new someLib.so:
g++ -L. -lsomeLib main.cpp -o main
I have tried the following:
g++ -shared -L. -lsomeLib -lpcrecpp -lpcre -lpthread -o libSomelib_static.so
This gives me an .so file with no symbols.
PS: I'm completely beginer of compilers.
There are a few issues at play here:
Linkers only use object files from an archive that resolve unresolved symbols. This is why the order of archives in the command line is important. Correct order is object files, followed by static libraries, followed by shared libraries. E.g. g++ -o main -pthread main.cpp -L. -lsomeLib -lpcrecpp -lpcre.
When you build the shared library that archive does not resolve any symbols, hence the linker ignores it completely. See 1.
Object files for a shared library must be compiled as position independent code (-fPIC compiler flag). Archives are normally built without this flag.
Use -pthread flag both when compiling and linking multi-threaded applications, not -lpthread.
Is it possible have a global variable defined inside an application, accessed by a shared library that has been loaded with dlopen()?
I have it declared as an extern but when the app loads and tries to use it I get an undefined symbol error.
I am loading the library with flags set to RTLD_LAZY | RTLD_GLOBAL.
You have to you build your application where you have a global variable with -rdynamic option for g++. This option instructs the linker (ld) to add all symbols, not only used ones, to the dynamic symbol table of your application.
This is an example how I build my test C++ application that loads a shared library. The shared library uses a global variable in main.cpp. So I have added -rdynamic when build my main application:
g++ -rdynamic -m64 -g main.cpp -o main -ldl
When g++ finds -rdynamic it passes the flag -export-dynamic to the ELF linker (ld).
This is from man ld (which actually creates the dynamic symbol table):
If you use "dlopen" to load a dynamic object which needs to refer back
to the symbols defined by the program, rather than some other dynamic
object, then you will probably need to use this option when linking
the program itself.
Useful links:
http://gcc.gnu.org/onlinedocs/gcc-4.9.0/gcc/Link-Options.html
http://linux.die.net/man/1/ld
I try to build the example code provide by boost asio example:
http://www.boost.org/doc/libs/1_54_0/doc/html/boost_asio/example/cpp11/spawn/echo_server.cpp
I copy all the code and put it into a cpp file, compile it on linux with gcc4.7 and cmake, link with boost coroutine and boost context library, but the link is failed.
The output is list below:
Linking CXX executable ../../../output/bin/unit_test
cd /home/watson/ID_project/build/server_linux_makefile_gcc/abc/test/unit/abc_async && /usr/local/bin/cmake -E cmake_link_script CMakeFiles/unit_test.dir/link.txt --verbose=1
/usr/bin/c++ -std=c++11 -O3 -DNDEBUG -pthread -lrt -ltcmalloc -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free CMakeFiles/unit_test.dir/TestFileChannel.cpp.o CMakeFiles/unit_test.dir/TestStreamBuffer.cpp.o CMakeFiles/unit_test.dir/TestTimer.cpp.o CMakeFiles/unit_test.dir/TestThreadPool.cpp.o CMakeFiles/unit_test.dir/TestScheduler.cpp.o CMakeFiles/unit_test.dir/PCH.cpp.o CMakeFiles/unit_test.dir/main.cpp.o CMakeFiles/unit_test.dir/TestUDPNetwork.cpp.o CMakeFiles/unit_test.dir/TestTCPNetwork.cpp.o -o ../../../output/bin/unit_test -rdynamic ../../../../../../install/thirdparty_linux_makefile_gcc/lib/libboost_unit_test_framework-gcc47-mt-1_54.a ../../../../../../install/thirdparty_linux_makefile_gcc/lib/libboost_context-gcc47-mt-1_54.a ../../../../../../install/thirdparty_linux_makefile_gcc/lib/libboost_coroutine-gcc47-mt-1_54.a ../../../../../../install/thirdparty_linux_makefile_gcc/lib/libboost_thread-gcc47-mt-1_54.a ../../../../../../install/thirdparty_linux_makefile_gcc/lib/libboost_filesystem-gcc47-mt-1_54.a ../../../../../../install/thirdparty_linux_makefile_gcc/lib/libyaml-cpp.a ../../../../../../install/thirdparty_linux_makefile_gcc/lib/libmongoc.a ../../../../../../install/thirdparty_linux_makefile_gcc/lib/libboost_system-gcc47-mt-1_54.a ../../../../../../install/thirdparty_linux_makefile_gcc/lib/libprotobuf.a
../../../../../../install/thirdparty_linux_makefile_gcc/lib/libboost_coroutine-gcc47-mt-1_54.a(coroutine_context.o): In function `boost::coroutines::detail::coroutine_context::coroutine_context(void (*)(long), boost::coroutines::stack_context*)':
coroutine_context.cpp:(.text+0x103): undefined reference to `make_fcontext'
../../../../../../install/thirdparty_linux_makefile_gcc/lib/libboost_coroutine-gcc47-mt-1_54.a(coroutine_context.o): In function `boost::coroutines::detail::coroutine_context::jump(boost::coroutines::detail::coroutine_context&, long, bool)':
coroutine_context.cpp:(.text+0x1bc): undefined reference to `jump_fcontext'
collect2: error: ld returned 1 exit status
make[2]: *** [abc/output/bin/unit_test] Error 1
I print the symbol table from the .a file, and find the symbol jump_fcontext' andmake_fcontext' is existed:
nm libboost_context-gcc47-mt-1_54.a
make_x86_64_sysv_elf_gas.o:
U _GLOBAL_OFFSET_TABLE_
U _exit
000000000000002e t finish
0000000000000000 T make_fcontext
jump_x86_64_sysv_elf_gas.o:
0000000000000000 T jump_fcontext
Someone can give me any tips about it? I try google every place but without information.
Try swapping the linking order of boost_context and boost_coroutine.
The linker documentation states:
[...] the linker searches and processes libraries and object files in the order they are specified. Thus, ‘foo.o -lz bar.o’ searches library ‘z’ after file foo.o but before bar.o. If bar.o refers to functions in ‘z’, those functions may not be loaded.
In this case
Boost.Coroutine depends on Boost.Context. As such, boost_coroutine should appear before boost_context when linking. For more linker ordering details, consider reading this answer.
Just to add another possibility for this (specifically with the Boost Context library) - if building under MinGW on Windows, using an earlier version of MASM can produce static libraries that contain no linkable symbols due to the way the symbols are exported.
The solution is to rebuild Boost Context with MASM 8.
See the answer at https://stackoverflow.com/a/26874113/1678468 for more about this.
I have a llvm module that i've dumped as bitcode file with llvm::WriteBitcodeToFile. I want to turn this bitcode file into an native dynamically loadable library that contains the functions in the module.
How do i do this? i tried using llc for this, but this produces code that apparently is not relocatable, since after doing the following steps:
llc -enable-pie -cppgen=functions -filetype=asm executableModule -o em.s
then, assemblying with gnu as into an object file:
as -o mylib.o em.s
finally, trying to produce a shared library with:
gcc -shared -o libmyfile.so -fPIC mylib.o
fails with the error:
/usr/bin/ld: error: mylib.o: requires dynamic R_X86_64_PC32 reloc against 'X.foo' which may overflow at runtime; recompile with -fPIC
collect2: ld returned 1 exit status
You need to setup relocation model. Something like -llc -relocation-model=pic. Do not use PIE, because it's for executables, not for libraries. Also, -cppgen does not make any sense here, it's for cpp backend only.