I compile twice the same .c and .h files and get object files with the same size but different md5sums.
Here is the only difference from objdump -d:
1) cpcidskephemerissegment.o: file format elf64-x86-64
Disassembly of section .text:
0000000000000000 <_ZN68_GLOBAL__N_sdk_segment_cpcidskephemerissegment.cpp_00000000_B8B9E66611MinFunctionEii>:
2) cpcidskephemerissegment.o: file format elf64-x86-64
Disassembly of section .text:
0000000000000000 <_ZN68_GLOBAL__N_sdk_segment_cpcidskephemerissegment.cpp_00000000_8B65537811MinFunctionEii>:
What can be the reason? Thanks!
I guess, the compiler didn't know how to name this namespace and used path to the source file plus some random number.
The compiler must guarantee that a symbol in unnamed namespace does not conflict with any other symbol in your program. By default this is achieved by taking full filename of the source, and appending a random hash value to it (it's legal to compile the same source twice (e.g. with different macros) and link the two objects into a single program, and the unnamed namespace symbols must still be distinct, so using just the source filename without the seed is not enough).
If you know that you are not linking the same source file more than once, and want to have a bit-identical object file on re-compile, the solution is to add -frandom-seed="abcd" to your compile line (replace "abcd" with anything you want; it's common to use the filename as the value of random seed). Documentation here.
The reasons can be many:
Using macros like __DATE__ and __TIME__
Embedding counters that are incremented for each build (the Linux kernel does this)
Timestamps (or similarly variable quantities) embedded in the .comments ELF section. One example of a compiler that does this is the xlC compiler on AIX.
Different names as a result of name mangling (e.g. C++)
Changes in environment variables which are affecting the build process.
Compiler bug(s) (however unlikely)
To produce bit identical builds, you can use GCC's -frandom-seed parameter. There were situations where it could break things before GCC 4.3, but GCC now turns functions defined in anonymous namespaces into static symbols. However, you will always be safe if you compile each file using a different value for -frandom-seed, the simplest way being to use
the filename itself as the seed.
Finally I've found the answer!
c++filt command gave the original name of the function:
{unnamed namespace}: MinFunction(int, int)
In the source was:
namespace
{
MinFunction(int a, int b) { ... }
}
I named the namespace and got stable checksum of object file!
As I guess, the compiler didn't know how to name this namespace and used path to the source file plus some random number.
Related
In our build system, we generate multiple .so files (foo.so, bar.so, ...) that are loaded during runtime by the main executable (biz). So the .so files are linked separately.
We also have our own util.a static library, that has some utility functions and global data.
The problem comes when some of the .so want to use util.a data/function, but we can't link each .so to util.a. It's because of the data section: global data must be unique in the program address space. If more than one .so is linked to util.a and has a copy of the data, the program behavior will be very surprising but hard to debug.
We can't link executable (biz) to util.a either. The linker will not put everything to the target, since biz doesn't reference the functions on behalf of .so.
Of course, unless linking util.a with -Wl,-whole-archive. But is there a better way to do this?
Solution 1: consider making util.a a dynamic library util.so.
Solution 2: don't let the linker export any symbols exported by util.a. When using gcc you can achieve this for example by using __attribute__((visibility("hidden"))):
int __attribute__((visibility("hidden"))) helperfunc(void *p);
You can use objdump to check which symbols are exported.
To answer myself's question, the eventual solution was like:
http://lists.gnu.org/archive/html/qemu-devel/2014-09/msg00099.html
TL;DR: Search for all the interesting symbols (that you want to pull from archives) inside the .so objects with nm (1), and inject into the compiling command line with -Wl,-u,$SYMBOL. Note that the -Wl,-u,$SYMBOL arguments need to come before archive names in the command line, so the linker knows that it needs to link them.
I need tool which is equivalent to:
$ echo 'char bar[] = {65, 66, 67};' >foo.c
$ gcc -c foo.c
I have a multi-megabyte binary file to be put to the bar array, and I need it without creating an .c file: I'd like the .o file be created directly from the binary file. Another option can be creating .s or .S files, but I'd like to avoid that as well. Is there a tool in binutils etc. which can do the job?
An update: gcc segfaults for a 9 MB binary file. as works, but it's slow and the the temporary .s file is too large.
You can use objcopy --add-section to create a section with contents found in a file. I think you'll need to use a linker script to add a symbol pointing at the start of the new section.
For a long time the easiest solution was creating an .s or an .S file.
binutils doesn't contain anything useful.
There is a 2-step trick which is fast and uses little memory:
Step 1. Create a and compile a .c file which contains the symbol of the right size, but it has a short signature instead of the real, long data. It should look like:
const char hi[1234567] = "SIGNATURE";
const char *hi_end = hi + sizeof(hi) / sizeof(char);
Step 2. Find the bytes SIGNATURE in the .o file generated by the compiler, and replace it (and the following '\0's) with the data from the real binary data file.
The Perl script cobjgen automates both steps. See this blog post for a more detailed analysis and usage instructions.
On Linux, is there a way to embed version information into an ELF binary? I would like to embed this info at compile time so it can then be extract it using a script later. A hackish way would be to plant something that can be extracted using the strings command. Is there a more conventional method, similar to how Visual Studio plant version info for Windows DLLs (note version tab in DLL properties)?
One way to do it if using cvs or subversion is to have a special id string formatted specially in your source file. Then add a pre-commit hook to cvs or svn that updates that special variable with the new version of the file when a change is committed. Then, when the binary is built, you can use ident to extract that indformation. For example:
Add something like this to your cpp file:
static char fileid[] = "$Id: fname.cc,v 1.124 2010/07/21 06:38:45 author Exp $";
And running ident (which you can find by installing rcs) on the program should show the info about the files that have an id string in them.
ident program
program:
$Id: fname.cc,v 1.124 2010/07/21 06:38:45 author Exp $
Note As people have mentioned in the comments this technique is archaic. Having the source control system automatically change your source code is ugly and the fact that source control has improved since the days when cvs was the only option means that you can find a better way to achieve the same goals.
To extend the #sashang answer, while avoiding the "$Id:$" issues mentioned by #cdunn2001, ...
You can add a file "version_info.h" to your project that has only:
#define VERSION_MAJOR "1"
#define VERSION_MINOR "0"
#define VERSION_PATCH "0"
#define VERSION_BUILD "0"
And in your main.c file have the line:
static char version[] = VERSION_MAJOR "." VERSION_MINOR "." VERSION_PATCH "." VERSION_BUILD;
static char timestamp[] = __DATE__ " " __TIME__;
(or however you want to use these values in your program)
Then set up a pre-build step which reads the version_info.h file, bumps the numbers appropriately, and writes it back out again. A daily build would just bump the VERSION_BUILD number, while a more serious release would bump other numbers.
If your makefile lists this on your object's prerequisite list, then the build will recompile what it needs to.
The Intel Fortran and C++ compilers can certainly do this, use the -sox option. So, yes there is a way. I don't know of any widespread convention for embedding such information in a binary and I generally use Emacs in hexl-mode for reading the embedded information, which is quite hackish.
'-sox' also embeds the compiler options used to build an executable, which is very useful.
If you declare a variable called program_version or similar you can find out at which address the variable is stored at and then proceed to extract its value. E.g.
objdump -t --demangle /tmp/libtest.so | grep program_version
0000000000600a24 g O .data 0000000000000004 program_version
tells me that program_version resides at address 0000000000600a24 and is of size 4. Then just read the value at that address in the file.
Or you could just write a simple program that links the library in questions and prints the version, defined either as an exported variable or a function.
What does the obj file ctr1.o does in gcc compilier ?Why does the linker link this obj file whenever an executable is generated?
I think it contains very basic stuf (crt stands for C run time) like setting up argv and argc for your main function etc ... Here is a link with some explanation
If you don't want it, because you are writing a tiny bootloader for example, without any bit of the libc, you can use the --no-stdlib options to link your program. If you go this way, youwill also need to write your own linker script.
I'm not sure to understand your question but I guess you are referring to 'crt1.o' in the GCC package.
The crt is one of the base packages of the libc which provides basic functionality to access the computer. IIRC it contains methods like 'printf' and such.
That's why it is often even included in the most basic C applications.
Object files hold your compiled code, but are not in themselves executable. It is the job of the linker to take all the object files that make up a program, and join them into a whole. This involves resolving references between object files (extern symbols), checking that there is a main() entrypoint (for C programs), and so on.
Since each source file (.c or .cpp) compiles into a separate object file, which are then read by the linker, changes to a single C file mean only that can be re-compiled, generating a new object file, which is then linked with the existing object files into a new executable. This makes development faster.
UPDATE: As stated in another answer, the "crt.o" object files holds the C runtime code, which is assumed to be needed by most C programs. You can read the gcc linker options and find the --no-stdlib option, this will tell gcc that your particular program should not be linked with the standard C runtime files.
Can anyone explain how compilation works?
I can't seem to figure out how compilation works..
To be more specific, here's an example.. I'm trying to write some code in MSVC++ 6 to load a Lua state..
I've already:
set the additional directories for the library and include files to the right directories
used extern "C" (because Lua is C only or so I hear)
include'd the right header files
But i'm still getting some errors in MSVC++6 about unresolved external symbols (for the Lua functions that I used).
As much as I'd like to know how to solve this problem and move on, I think it would be much better for me if I came to understand the underlying processes involved, so could anyone perhaps write a nice explanation for this? What I'm looking to know is the process.. It could look like this:
Step 1:
Input: Source code(s)
Process: Parsing (perhaps add more detail here)
Output: whatever is output here..
Step 2:
Input: Whatever was output from step 1, plus maybe whatever else is needed (libraries? DLLs? .so? .lib? )
Process: whatever is done with the input
Output: whatever is output
and so on..
Thanks..
Maybe this will explain what symbols are, what exactly "linking" is, what "object" code or whatever is..
Thanks.. Sorry for being such a noob..
P.S. This doesn't have to be language specific.. But feel free to express it in the language you're most comfortable in.. :)
EDIT: So anyway, I was able to get the errors resolved, it turns out that I have to manually add the .lib file to the project; simply specifying the library directory (where the .lib resides) in the IDE settings or project settings does not work..
However, the answers below have somewhat helped me understand the process better. Many thanks!.. If anyone still wants to write up a thorough guide, please do.. :)
EDIT: Just for additional reference, I found two articles by one author (Mike Diehl) to explain this quite well.. :)
Examining the Compilation Process: Part 1
Examining the Compilation Process: Part 2
From source to executable is generally a two stage process for C and associated languages, although the IDE probably presents this as a single process.
1/ You code up your source and run it through the compiler. The compiler at this stage needs your source and the header files of the other stuff that you're going to link with (see below).
Compilation consists of turning your source files into object files. Object files have your compiled code and enough information to know what other stuff they need, but not where to find that other stuff (e.g., the LUA libraries).
2/ Linking, the next stage, is combining all your object files with libraries to create an executable. I won't cover dynamic linking here since that will complicate the explanation with little benefit.
Not only do you need to specify the directories where the linker can find the other code, you need to specify the actual library containing that code. The fact that you're getting unresolved externals indicates that you haven't done this.
As an example, consider the following simplified C code (xx.c) and command.
#include <bob.h>
int x = bob_fn(7);
cc -c -o xx.obj xx.c
This compiles the xx.c file to xx.obj. The bob.h contains the prototype for bob_fn() so that compilation will succeed. The -c instructs the compiler to generate an object file rather than an executable and the -o xx.obj sets the output file name.
But the actual code for bob_fn() is not in the header file but in /bob/libs/libbob.so, so to link, you need something like:
cc -o xx.exe xx.obj -L/bob/libs;/usr/lib -lbob
This creates xx.exe from xx.obj, using libraries (searched for in the given paths) of the form libbob.so (the lib and .so are added by the linker usually). In this example, -L sets the search path for libraries. The -l specifies a library to find for inclusion in the executable if necessary. The linker usually takes the "bob" and finds the first relevant library file in the search path specified by -L.
A library file is really a collection of object files (sort of how a zip file contains multiple other files, but not necessarily compressed) - when the first relevant occurrence of an undefined external is found, the object file is copied from the library and added to the executable just like your xx.obj file. This generally continues until there are no more unresolved externals. The 'relevant' library is a modification of the "bob" text, it may look for libbob.a, libbob.dll, libbob.so, bob.a, bob.dll, bob.so and so on. The relevance is decided by the linker itself and should be documented.
How it works depends on the linker but this is basically it.
1/ All of your object files contain a list of unresolved externals that they need to have resolved. The linker puts together all these objects and fixes up the links between them (resolves as many externals as possible).
2/ Then, for every external still unresolved, the linker combs the library files looking for an object file that can satisfy the link. If it finds it, it pulls it in - this may result in further unresolved externals as the object pulled in may have its own list of externals that need to be satisfied.
3/ Repeat step 2 until there are no more unresolved externals or no possibility of resolving them from the library list (this is where your development was at, since you hadn't included the LUA library file).
The complication I mentioned earlier is dynamic linking. That's where you link with a stub of a routine (sort of a marker) rather than the actual routine, which is later resolved at load time (when you run the executable). Things such as the Windows common controls are in these DLLs so that they can change without having to relink the objects into a new executable.
Step 1 - Compiler:
Input: Source code file[s]
Process: Parsing source code and translating into machine code
Output: Object file[s], which consist[s] of:
The names of symbols which are defined in this object, and which this object file "exports"
The machine code associated with each symbol that's defined in this object file
The names of symbols which are not defined in this object file, but on which the software in this object file depends and to which it must subsequently be linked, i.e. names which this object file "imports"
Step 2 - Linking:
Input:
Object file[s] from step 1
Libraries of other objects (e.g. from the O/S and other software)
Process:
For each object that you want to link
Get the list of symbols which this object imports
Find these symbols in other libraries
Link the corresponding libraries to your object files
Output: a single, executable file, which includes the machine code from all all your objects, plus the objects from libraries which were imported (linked) to your objects.
The two main steps are compilation and linking.
Compilation takes single compilation units (those are simply source files, with all the headers they include), and create object files. Now, in those object files, there are a lot of functions (and other stuff, like static data) defined at specific locations (addresses). In the next step, linking, a bit of extra information about these functions is also needed: their names. So these are also stored. A single object file can reference functions (because it wants to call them when to code is run) that are actually in other object files, but since we are dealing with a single object file here, only symbolic references (their 'names') to those other functions are stored in the object file.
Next comes linking (let's restrict ourselves to static linking here). Linking is where the object files that were created in the first step (either directly, or after they have been thrown together into a .lib file) are taken together and an executable is created.
In the linking step, all those symbolic references from one object file or lib to another are resolved (if they can be), by looking up the names in the correct object, finding the address of the function, and putting the addresses in the right place.
Now, to explain something about the 'extern "C"' thing you need:
C does not have function overloading. A function is always recognizable by its name. Therefore, when you compile code as C code, only the real name of the function is stored in the object file.
C++, however, has something called 'function / method overloading'. This means that the name of a function is no longer enough to identify it. C++ compilers therefore create 'names' for functions that include the prototypes of the function (since the name plus the prototype will uniquely identify a function). This is known as 'name mangling'.
The 'extern "C"' specification is needed when you want to use a library that has been compiled as 'C' code (for example, the pre-compiled Lua binaries) from a C++ project.
For your exact problem: if it still does not work, these hints might help:
* have the Lua binaries been compiled with the same version of VC++?
* can you simply compile Lua yourself, either within your VC solution, or as a separate project as C++ code?
* are you sure you have all the 'extern "C"' things correct?
You have to go into project setting and add a directory where you have that LUA library *.lib files somewhere on the "linker" tab. Setting called "including libraries" or something, sorry I can't look it up.
The reason you get "unresolved external symbols" is because compilation in C++ works in two stages. First, the code gets compiled, each .cpp file in it's own .obj file, then "linker" starts and join all that .obj files into .exe file. .lib file is just a bunch of .obj files merged together to make distribution of libraries just a little bit simplier.
So by adding all the "#include" and extern declaration you told the compiler that somewhere it would be possible to find code with those signatures but linker can't find that code because it doesn't know where those .lib files with actual code is placed.
Make sure you have read REDME of the library, usually they have rather detailed explanation of what you had to do to include it in your code.
You might also want to check this out: COMPILER, ASSEMBLER, LINKER AND LOADER: A BRIEF STORY.