make one static library from whole project with cmake - linux

c++-project, say, foo is maintained by the cmake.
One wants to create one library libfoo.a (with all classes/methods/functions created at the whole source-tree) to make possible creating programs that could linked to the library with -lfoo.
ok, let's consider now a toy example, and the prolbem will be clear. Directory foo (root of the project) contains directories a, and b. Two CmakeLists.txt are created:
# a/CMakeLists.txt
add_library(A <a_sources>)
# b/CMakeLists.txt
add_library(B <b_sources>)
And one CMakeLists.txt for root directory:
add_subdirectory(a)
add_subdirectory(b)
add_library(foo <foo_sources>
target_link_libraries(foo A B)
That was a surprise for me: after building libfoo.a contains only methods from foo_sources, and a_sources,b_sources are excluded.
That is ok in the case when executables are built with the same project: while creating executables cmake "guesses" that a and b must be linked if it is linked to foo.
But in the case executable is created "outside" project to use library foo one must link with -lfoo -la -lb, now imagine a project with lots of subdirectories - how to deal with it? so question is "how to create one library, aggregating methods from whole project with means of cmake?"
Googling led me to relatively recently embedded (appeared in 2.8.8) OBJECT library opportunity. Nice example of using it is shown here. Now the problem above can be solved with that:
# a/CMakeLists.txt
add_library(A OBJECT <a_sources>)
# b/CMakeLists.txt
add_library(B OBJECT <b_sources>)
# foo/CMakeLists.txt
add_subdirectory(a)
add_subdirectory(b)
add_library(foo <foo_sources> $<TARGET_OBJECTS:A> $<TARGET_OBJECTS:B>)
problem seems to be solved, unfortunately, not quite.
if dependency chain is longer than 2, for example, foo depends on A, which depends on B, problem still remains.
That is because,
Object libraries may contain only sources (and headers) that compile to object files.
and
Object libraries cannot be imported, exported, installed, or linked.
(quotes are taken from the same link)
I've tried several combinations of target_link_library(), add_library(), add_library(... OBJECT ..) trying to link A and B to foo without success (error during cmake-process.)
I must be loosing something simple, please help, thank you!
I am not sure is it important: project is maintained at the linux.

I think you're getting tangled up in the term "depends on". If you're building a library named foo and it has two parts, A and B, it doesn't matter whether A depends on B; the library should contain both. The CMake code you've shown will build foo properly.

Yep, I support answer #Pete Becker# . But it should be said as well that those libraries a $<TARGET_OBJECTS:A> and $<TARGET_OBJECTS:B> actually not a libraries at all, but rather cmake internal list of object modules. There is no dependencies between compilation of object modules (except auto-generated sources) so they can be done in any order and in parallel.
I guess more correct term for your intention is gathering together several TARGET_OBJECTS under single object library. That's really bad that you can't write add_library(B OBJECT b.cpp $<TARGET_OBJECTS:A>). But you always can implement this by yourself:
add_library(A OBJECT a.cpp)
set(A_OBJECTS $<TARGET_OBJECTS:A>)
add_library(B OBJECT b.cpp)
set(B_OBJECTS $<TARGET_OBJECTS:B> ${A_OBJECTS})
add_library(foo ${B_OBJECTS})
I.e just create special variables _OBJECTS to use them whenever you want to include those object libraries in library, executable or as part of other object library with that _OBJECTS flavor.

Related

cmake compiling but not linking a new source file in a library (libonion)

I am a cmake newbie (on Debian/Sid/Linux/x86-64)
I forked libonion on https://github.com/bstarynk/onion to enable customization of malloc with Boehm's garbage collector; see this mail thread.
I added two files there onion/src/low_util.c and onion_src/low_util.h (which is #include-d successfully in several other patched files.
It is compiled but not linked.
set(SOURCES onion.c codecs.c dict.c low_util.c request.c response.c handler.c
log.c sessions.c sessions_mem.c shortcuts.c block.c mime.c url.c ${POLLER_C}
listen_point.c request_parser.c http.c ${HTTPS_C} websocket.c ${RANDOM_C} ${SQLITE3_C})
later:
SET(INCLUDES_ONION block.h codecs.h dict.h handler.h http.h https.h listen_point.h low_util.h log.h mime.h onion.h poller.h request.h response.h server.h sessions.h shortcuts.h types.h types_internal.h url.h websocket.h ${SQLITE3_H})
MESSAGE(STATUS "Found include files ${INCLUDES_ONION}")
but when I build, my file low_util.c got compiled but not linked.
Linking C executable otemplate
CMakeFiles/opack.dir/__/__/src/onion/dict.c.o: In function `onion_dict_new':
dict.c:(.text+0x1bc): undefined reference to `onionlow_calloc'
CMakeFiles/opack.dir/__/__/src/onion/dict.c.o: In function `onion_dict_node_data_free':
dict.c:(.text+0x2ec): undefined reference to `onionlow_free'
CMakeFiles/opack.dir/__/__/src/onion/dict.c.o: In function `onion_dict_node_add':
Notice that libonion is a library (in C, providing HTTP service) and that I just want to add a low_util.c file (wrapping malloc & pthread_create etc... to make Boehm's GC happy: it is calling GC_malloc and GC_pthread_create ....) with its low_util.h header. Surprisingly, they get compiled, but do not seems to be linked. And I am not familiar with cmake and I am not familiar with how D.Moreno (the main author of libonion) has organized his cmake files.
Any clues?
Apply the following patch to make it link. The two executables which are being linked with the symbols generated from the .c file you added are missing and are added in the patch.
http://pastebin.com/mDMRiUQu
Based on what you posted, its hard to tell what could be wrong. The cake source code above says that a variable ${SOURCES} is equivalent to onion.c codecs.c dict.c low_util.c ... ${SQLITE3_C}, and a variable ${INCLUDE_ONION} is equivalent to block.h codecs.h dict.h ... ${SQLITE3_H}. You did not provide any targets or the files included in those targets.
A brief list of things that may help:
where do you define the top level library or executable? If your making a library, you will need the command add_library(). If you are making an executable, you will need the add_executable() command.
Use the command target_link_libraries() to resolve dependencies. Rather than placing all of the source files in a single library, group similar together in a single target (a target is defined by the add_* commands), and use this command to link the targets after compilation.
Use the find_package() to get any libraries which are defined on your system but not in you project. Then, link to that library using the target_link_libraries() command.
In this case, if the onion_dict_* functions are defined within the same library, your not including those files in library. When you use add_library or add_executable, ensure you add those files to the list. If the functions are within your project but not in the same library, use the target_link_libraries() command to link to the library which contains the correct files. If those commands are defined in an external library, then first find the library using find_package(), and then link to the library using target_link_libraries().

Explanation sought: libtool, automake, shared libraries (and Fortran)

The problem I had is solved. I'm posting this to solicit an explanation as to why the solution actually works. I've gotten great feedback here before.
I have a legacy code base that used a very simplistic build system, and my project is to migrate that to Autotools for customization and, particularly, building shared libraries. The main library is written in C, but must also be linkable from Fortran (for legacy purposes), and is distributed with some test codes in F77. The authors organized the source code into modules...
src_module1/
src_module2/
...
testc/
testf77/
Their built the library lib/libmain.a by compiling code in the src_*/ directories and archiving the objects with ranlib.
My first approach was to build a shared library from each src_*/ separately and "link" all of these into one shared library. Using Autotools, the src_module1/Makefile.am would contain
noinst_LTLIBRARIES = libmodule1.la
libmodule1_la_SOURCES = ...
and so on for the other modules, and finally the lib/Makefile.am would need only:
lib_LTLIBRARIES = libmain.la
libmain_la_SOURCES =
libmain_la_LIBADD = $(top_srcdir)/src_module1/libmodule1.la ...
This seemed to work perfectly. However, when the code in testc/ was compiled and linked against libmain.la, a "symbols not found" error was issued.
Thinking that this was an issue with libtool or shared libraries, I tried building static only, basically changing all .la to .a and all _LTLIBRARIES to _LIBRARIES. Same problem. This time, however, noticing the error "ranlib: warning for library: libmain.a the table of contents is empty (no object file members in the library define global symbols)" when trying to link libmain.a itself.
The solution that I found seems like a hack. I did not build Makefiles for any of the src_*/ directories, but instead used only for the lib/ directory and its Makefile.am had the lines:
lib_LTLIBRARIES = libmain.la
libmain_la_SOURCES = [all sources from all ../src_modules/ ]
This worked. The compiled programs in testc/ linked against libmain.la without issue. One of the "modules" is a set of Fortran bindings that wrap other C functions in the library. Even the Fortran codes in testf77/ linked against libmain.la properly.
Could someone carefully explain what happens when libtool builds a shared library? Or even when building a static library? Why is it that several static libraries can't be linked together to make one static library? Why are symbols only available when libtool/ranlib builds the library "from sources"? And what about installing a shared/static library, i.e. moving it to the /usr/local/lib --- what happens there? The Wikipedia article on static and shared libraries isn't really detailed enough for me.
I do appreciate all efforts to make sense of my longwinded question.
What you first tried ought to work. I am using this kind of setup all the time (in a C++ context). It's also documented, and part of the Automake test suite (although maybe not with Fortran).
A libtool library that is not installable, i.e., one declared with noinst_LTLIBRARIES, is called a libtool convenience library. That noinst_ makes a big difference in what is built. Even if Libtool is configured to build shared libraries, a libtool convenience library is not actually a shared library: it is just a set of object files (compiled as PIC so that they can be latter be used in a shared library) stored in an archive. You can use a libtool convenience library anywhere using this set of objects would make sense, e.g., to build a shared library.
When multiple libtool convenience libraries are LIBADDed to an installable libtool library (such as your libmain.la), Libtool has to unpack the archives containing the objects of each convenience library and link them into the final library.
There is a trap that is worth noting here: when building a shared library out of
convenience libraries, if the _SOURCES variable is empty Automake does not know which linker to use and default to the C linker. If you want to trick Automake into using the linking rule for some specific language, you can declare a nodist_EXTRA_..._SOURCES source file that do not have to exist. (See the Libtool Convenience Libraries section of the Automake manual for an example.)
Maybe that was your problem? If you have some Fortran files in the sources of some of your modules (your description suggests these are only C files), the Fortran linker will be used to build libmain.la only if a Fortran file appears in the source files declared for that libtool library. And the C linker will be used when libmain_la_SOURCES is empty.
Otherwise, I have no idea why it didn't work.
There is an small error in your example:
libmain_la_LIBADD = $(top_srcdir)/src_module1/libmodule1.la
should be
libmain_la_LIBADD = $(top_builddir)/src_module1/libmodule1.la
because the library is not created in the source directory. However I assume this is just a typo, and you won't see the difference unless you do a VPATH build or run make distcheck.
Your second try, using _LIBRARIES without Libtool is not expected to work.
_LIBRARIES can only be used to declare static archives, and in this case _LIBADD may only contain object files, not other static archives. Unpacking an archive to reuse its objects into another archive can be tricky to do portably. Automake's answer to this problem has always been: install Libtool and use _LTLIBRARIES (Libtool can be configured to build only static libraries).

What does the object code file ctr1.o do in the gcc compiler?

What does the obj file ctr1.o does in gcc compilier ?Why does the linker link this obj file whenever an executable is generated?
I think it contains very basic stuf (crt stands for C run time) like setting up argv and argc for your main function etc ... Here is a link with some explanation
If you don't want it, because you are writing a tiny bootloader for example, without any bit of the libc, you can use the --no-stdlib options to link your program. If you go this way, youwill also need to write your own linker script.
I'm not sure to understand your question but I guess you are referring to 'crt1.o' in the GCC package.
The crt is one of the base packages of the libc which provides basic functionality to access the computer. IIRC it contains methods like 'printf' and such.
That's why it is often even included in the most basic C applications.
Object files hold your compiled code, but are not in themselves executable. It is the job of the linker to take all the object files that make up a program, and join them into a whole. This involves resolving references between object files (extern symbols), checking that there is a main() entrypoint (for C programs), and so on.
Since each source file (.c or .cpp) compiles into a separate object file, which are then read by the linker, changes to a single C file mean only that can be re-compiled, generating a new object file, which is then linked with the existing object files into a new executable. This makes development faster.
UPDATE: As stated in another answer, the "crt.o" object files holds the C runtime code, which is assumed to be needed by most C programs. You can read the gcc linker options and find the --no-stdlib option, this will tell gcc that your particular program should not be linked with the standard C runtime files.

Link libraries with dependencies in Visual C++ without getting LNK4006

I have a set of statically-compiled libraries, with fairly deep-running dependencies between the libraries. For example, the executable X uses libraries A and B, A uses library C, and B uses libraries C and D:
X -> A
A -> C
X -> B
B -> C
B -> D
When I link X with A and B, I don't want to get errors if C and D were not also added to the list of libraries—the fact that A and B use these libraries internally is an implementation detail that X should not need to know about. Also, when new dependencies are added anywhere in the dependency tree, the project file of any program that uses A or B would have to be reconfigured. For a deep dependency tree, the list of required libraries can become really long and hard to maintain.
So, I am using the "Additional Dependencies" setting of the Librarian section in the A project, adding C.lib. And in the same section of B's project, I add C.lib and D.lib. The effect of this is that the librarian bundles C.lib into A.lib, and C.lib and D.lib into B.lib.
When I link X, however, both A.lib and B.lib contain their own copy of C.lib. This leads to tons of warnings along the lines of
A.lib(c.obj) : warning LNK4006 "symbol" (_symbol) already defined in B.lib(c.obj); second definition ignored.
How can I accomplish this without getting warnings? Is there a way to simply disable the warning, or is there a better way?
EDIT: I have seen more than one answer suggesting that, for the lack of a better alternative, I simply disable the warning. Well, this is part of the problem: I don't even know how to disable it!
As far as I know you can't disable linker warnings.
However, you can ignore some of them, using command line parameter of linker eg. /ignore:4006
Put it in your project properties under linker->command line setting (don't remember exact location).
Also read this:
Link /ignore
MSDN Forum - hiding LNK warnings
Wacek
Update If you can build all involved project in single solution, try this:
Put all project in one sln.
Remove all references to static libraries from projects' linker or librarian properties.
There is "Project Dependencies..." option in context menu for each project in Solution Explorer. Use it to define dependencies between project.
It should work. It doesn't invalidate anything I said before, the basic model of building C/C++ programs stays the same. VS (at least 2005 and newer) is simply smart enough to add all needed static libraries to linker command line. You can see it in project properties.
Of course this method won't help if you need to use already compiled static libraries. Then you need to add them all to exe or dll project that directly or indirectly uses them.
I don't think you can do anything about that. You should remove references to other static libs from static libs projects and add all needed static libs projects as dependences of exe or dll projects. You will just have to live with fact that any project that includes A.lib or B.lib also needs to include C.lib.
As an alternative you can turn your libraries into dlls which provide a richer model.
Statically compiled libraries simply aren't real libraries with dependency information, etc, like dlls. See how, when you build them, you don't really need to provide libraries they depend on? Headers are all that's needed. See? You can't even really say static libraries depend on something.
Static library is just an archive of compiled and not yet linked object code. It's not consistent whole. Each object file is compiled separately and remains separate entity inside the library. Linking happens when you build exe or dll. That's when you need to provide all object code. That's when all the symbol and dependency resolving happens.
If you add other static libraries to static library dependencies, librarian will simply copy all code together. Then, when building exe, linker will give you lots of warnings about duplicate symbols. You might be able to block those warnings (I don't know how) but be careful. It may conceal real problems like real duplicate symbols with differing definitions. And if you have static data defined in libraries, it probably won't work anyway.
Microsoft (R) Incremental Linker Version 9.00.x (link.exe) knows argument /ignore:4006
You could create one library which contains A, B, C & D and then link X against that.
Since it's a library, only object modules which are actually referenced will get linked into the final executable.
Note that one way of getting this warning is to define a member function in a header without the inline statement:
// Foo.h
class Foo
{
void someFunction();
};
void Foo:someFunction() // Warning! - should be "inline void Foo::someFunction()"
{
// do stuff
}
The problem is you are not localizing library C's symbols. So you have a ODR violation when you link in A and B. You need to have a way to make these private. By default all symbols are exported. One way to do this is to have a special linker definition file for both A and B that explicitly mention which files need to be exported.
[1] ODR = One Definition Rule.
I think the best course of action here will be to ignore/disable the linker warnings(LNK4006) since C.lib needs to be part of both A.Lib and B.lib and A.Lib does not need to know that B.lib itself uses C.Lib.
This may not fix your link error, but it might help with your dependency tree issue.
What I do, is just use a #pragma to include a lib in the .cpp file that needs it. For example:
#pragma comment(lib:"wsock32")
Like I said, I'm not sure it would keep the symbols in that object file, I'd have to whip up an example to try it out.
Poor flodin seems frustrated that nobody will explain how to disable the linker warnings. Well, I've had a similar problem, and for years I have simply lived with the fact that several hundred warnings were displayed. Now, however, thanks to the info from Link /ignore, I figured out how to disable the linker warnings.
I'm using Visual Studio 2008. In Project -> Settings -> Configuration Properties -> Librarian -> Command Line -> Additional Options, I added "/ignore:4006" (without the quotes). Now my warnings are gone!

The compilation process

Can anyone explain how compilation works?
I can't seem to figure out how compilation works..
To be more specific, here's an example.. I'm trying to write some code in MSVC++ 6 to load a Lua state..
I've already:
set the additional directories for the library and include files to the right directories
used extern "C" (because Lua is C only or so I hear)
include'd the right header files
But i'm still getting some errors in MSVC++6 about unresolved external symbols (for the Lua functions that I used).
As much as I'd like to know how to solve this problem and move on, I think it would be much better for me if I came to understand the underlying processes involved, so could anyone perhaps write a nice explanation for this? What I'm looking to know is the process.. It could look like this:
Step 1:
Input: Source code(s)
Process: Parsing (perhaps add more detail here)
Output: whatever is output here..
Step 2:
Input: Whatever was output from step 1, plus maybe whatever else is needed (libraries? DLLs? .so? .lib? )
Process: whatever is done with the input
Output: whatever is output
and so on..
Thanks..
Maybe this will explain what symbols are, what exactly "linking" is, what "object" code or whatever is..
Thanks.. Sorry for being such a noob..
P.S. This doesn't have to be language specific.. But feel free to express it in the language you're most comfortable in.. :)
EDIT: So anyway, I was able to get the errors resolved, it turns out that I have to manually add the .lib file to the project; simply specifying the library directory (where the .lib resides) in the IDE settings or project settings does not work..
However, the answers below have somewhat helped me understand the process better. Many thanks!.. If anyone still wants to write up a thorough guide, please do.. :)
EDIT: Just for additional reference, I found two articles by one author (Mike Diehl) to explain this quite well.. :)
Examining the Compilation Process: Part 1
Examining the Compilation Process: Part 2
From source to executable is generally a two stage process for C and associated languages, although the IDE probably presents this as a single process.
1/ You code up your source and run it through the compiler. The compiler at this stage needs your source and the header files of the other stuff that you're going to link with (see below).
Compilation consists of turning your source files into object files. Object files have your compiled code and enough information to know what other stuff they need, but not where to find that other stuff (e.g., the LUA libraries).
2/ Linking, the next stage, is combining all your object files with libraries to create an executable. I won't cover dynamic linking here since that will complicate the explanation with little benefit.
Not only do you need to specify the directories where the linker can find the other code, you need to specify the actual library containing that code. The fact that you're getting unresolved externals indicates that you haven't done this.
As an example, consider the following simplified C code (xx.c) and command.
#include <bob.h>
int x = bob_fn(7);
cc -c -o xx.obj xx.c
This compiles the xx.c file to xx.obj. The bob.h contains the prototype for bob_fn() so that compilation will succeed. The -c instructs the compiler to generate an object file rather than an executable and the -o xx.obj sets the output file name.
But the actual code for bob_fn() is not in the header file but in /bob/libs/libbob.so, so to link, you need something like:
cc -o xx.exe xx.obj -L/bob/libs;/usr/lib -lbob
This creates xx.exe from xx.obj, using libraries (searched for in the given paths) of the form libbob.so (the lib and .so are added by the linker usually). In this example, -L sets the search path for libraries. The -l specifies a library to find for inclusion in the executable if necessary. The linker usually takes the "bob" and finds the first relevant library file in the search path specified by -L.
A library file is really a collection of object files (sort of how a zip file contains multiple other files, but not necessarily compressed) - when the first relevant occurrence of an undefined external is found, the object file is copied from the library and added to the executable just like your xx.obj file. This generally continues until there are no more unresolved externals. The 'relevant' library is a modification of the "bob" text, it may look for libbob.a, libbob.dll, libbob.so, bob.a, bob.dll, bob.so and so on. The relevance is decided by the linker itself and should be documented.
How it works depends on the linker but this is basically it.
1/ All of your object files contain a list of unresolved externals that they need to have resolved. The linker puts together all these objects and fixes up the links between them (resolves as many externals as possible).
2/ Then, for every external still unresolved, the linker combs the library files looking for an object file that can satisfy the link. If it finds it, it pulls it in - this may result in further unresolved externals as the object pulled in may have its own list of externals that need to be satisfied.
3/ Repeat step 2 until there are no more unresolved externals or no possibility of resolving them from the library list (this is where your development was at, since you hadn't included the LUA library file).
The complication I mentioned earlier is dynamic linking. That's where you link with a stub of a routine (sort of a marker) rather than the actual routine, which is later resolved at load time (when you run the executable). Things such as the Windows common controls are in these DLLs so that they can change without having to relink the objects into a new executable.
Step 1 - Compiler:
Input: Source code file[s]
Process: Parsing source code and translating into machine code
Output: Object file[s], which consist[s] of:
The names of symbols which are defined in this object, and which this object file "exports"
The machine code associated with each symbol that's defined in this object file
The names of symbols which are not defined in this object file, but on which the software in this object file depends and to which it must subsequently be linked, i.e. names which this object file "imports"
Step 2 - Linking:
Input:
Object file[s] from step 1
Libraries of other objects (e.g. from the O/S and other software)
Process:
For each object that you want to link
Get the list of symbols which this object imports
Find these symbols in other libraries
Link the corresponding libraries to your object files
Output: a single, executable file, which includes the machine code from all all your objects, plus the objects from libraries which were imported (linked) to your objects.
The two main steps are compilation and linking.
Compilation takes single compilation units (those are simply source files, with all the headers they include), and create object files. Now, in those object files, there are a lot of functions (and other stuff, like static data) defined at specific locations (addresses). In the next step, linking, a bit of extra information about these functions is also needed: their names. So these are also stored. A single object file can reference functions (because it wants to call them when to code is run) that are actually in other object files, but since we are dealing with a single object file here, only symbolic references (their 'names') to those other functions are stored in the object file.
Next comes linking (let's restrict ourselves to static linking here). Linking is where the object files that were created in the first step (either directly, or after they have been thrown together into a .lib file) are taken together and an executable is created.
In the linking step, all those symbolic references from one object file or lib to another are resolved (if they can be), by looking up the names in the correct object, finding the address of the function, and putting the addresses in the right place.
Now, to explain something about the 'extern "C"' thing you need:
C does not have function overloading. A function is always recognizable by its name. Therefore, when you compile code as C code, only the real name of the function is stored in the object file.
C++, however, has something called 'function / method overloading'. This means that the name of a function is no longer enough to identify it. C++ compilers therefore create 'names' for functions that include the prototypes of the function (since the name plus the prototype will uniquely identify a function). This is known as 'name mangling'.
The 'extern "C"' specification is needed when you want to use a library that has been compiled as 'C' code (for example, the pre-compiled Lua binaries) from a C++ project.
For your exact problem: if it still does not work, these hints might help:
* have the Lua binaries been compiled with the same version of VC++?
* can you simply compile Lua yourself, either within your VC solution, or as a separate project as C++ code?
* are you sure you have all the 'extern "C"' things correct?
You have to go into project setting and add a directory where you have that LUA library *.lib files somewhere on the "linker" tab. Setting called "including libraries" or something, sorry I can't look it up.
The reason you get "unresolved external symbols" is because compilation in C++ works in two stages. First, the code gets compiled, each .cpp file in it's own .obj file, then "linker" starts and join all that .obj files into .exe file. .lib file is just a bunch of .obj files merged together to make distribution of libraries just a little bit simplier.
So by adding all the "#include" and extern declaration you told the compiler that somewhere it would be possible to find code with those signatures but linker can't find that code because it doesn't know where those .lib files with actual code is placed.
Make sure you have read REDME of the library, usually they have rather detailed explanation of what you had to do to include it in your code.
You might also want to check this out: COMPILER, ASSEMBLER, LINKER AND LOADER: A BRIEF STORY.

Resources