I have an executable on my Linux box which I know has been compiled either with OpenMPI or MPICH libraries.
Question: how to determine which one?
The following diagnostic procedure assumes that MPICH/MPICH2 and Open MPI are
the only possible MPI implementations that you may have linked with. Other
(especially commercial) MPI implementations do exist and may have different
library names and/or library symbols.
First determine if you linked dynamically:
% ldd my_executable
linux-vdso.so.1 => (0x00007ffff972c000)
libm.so.6 => /lib/libm.so.6 (0x00007f1f3c6cd000)
librt.so.1 => /lib/librt.so.1 (0x00007f1f3c4c5000)
libpthread.so.0 => /lib/libpthread.so.0 (0x00007f1f3c2a7000)
libc.so.6 => /lib/libc.so.6 (0x00007f1f3bf21000)
/lib64/ld-linux-x86-64.so.2 (0x00007f1f3c969000)
If you see libmpich.so in that list, then you have dynamically linked to
MPICH (or MPICH2). If you see libmpi.so then you have linked with Open MPI.
If neither is present, then you probably just linked statically. In that case
we need to examine the binary to look for distinguishing symbols:
% ( nm my_executable | grep MPIR_Free_contextid >/dev/null ) && echo "MPICH"
% ( nm my_executable | grep ompi_comm_set_name >/dev/null ) && echo "Open MPI"
Open MPI applications react to MCA parameters, that can be passed in environment variables. Just run the executable in singleton mode (i.e. without mpirun/mpiexec) with something like sysinfo_base_verbose set to 30:
$ OMPI_MCA_sysinfo_base_verbose=30 ./program
If you get an output like:
[hostname:pid] mca: base: components_open: Looking for sysinfo components
then this is more than robust indication that the executable uses Open MPI.
Related
I am trying to find the shared library which imported an external symbol. Currently I can get all imported symbols by using nm or many alternatives such as using radare2. I can also get the libraries which the binary is dependent on by using ldd. However, I got stuck at this point since I cannot find an efficient way to get which external symbol in my binary is dependent on which shared library. So, for example how can I find the shared library which exports the function named foo or printf or anything in an efficient way? I provide an example:
Output of nm -D myfile
w __cxa_finalize
U foo
w __gmon_start__
w _ITM_deregisterTMCloneTable
w _ITM_registerTMCloneTable
U __libc_start_main
U printf
U puts
Output of ldd
linux-vdso.so.1 (0x00007ffd30904000)
libfoo.so => /home/user/Desktop/dynamic_link_example/libfoo.so (0x00007f1b08aaf000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1b088a1000)
/lib64/ld-linux-x86-64.so.2 (0x00007f1b08abb000)
So, for example how can I find the shared library which exports the function named foo or printf or anything in an efficient way?
You can run your program with env LD_DEBUG=bindings ./a.out. This will produce a lot of output, which you can grep for foo and printf.
Note that the answer to "which external symbol in my binary is dependent on which shared library" is "whichever library defines this symbol first".
So if today your binary depends on lifoo.so for foo and on libc.so.6 for printf, nothing stops you from running with a different libfoo.so tomorrow, and that different version of libfoo.so may define different symbols. If the new version of libfoo.so defines printf, that would cause the answer to your question for symbol printf to change from libc.so.6 to libfoo.so.
[root#wdctc1281 bin]# ldd node
linux-vdso.so.1 => (0x00007fffd33f2000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f70f7855000)
librt.so.1 => /lib64/librt.so.1 (0x00007f70f764d000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f70f7345000)
libm.so.6 => /lib64/libm.so.6 (0x00007f70f7043000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f70f6e2d000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f70f6c10000)
libc.so.6 => /lib64/libc.so.6 (0x00007f70f684f000)
/lib64/ld-linux-x86-64.so.2 (0x00007f70f7a61000)
What does the first line and last line mean? They don't look like the normal
xxxx.so => /lib64/xxxxx.so (0xxxxxxxxxxxxxxxxxxxx)
format.
the first line is the VDSO. this is described in depth in the vdso(7) manpage. basically it's a shared library that's embedded in your kernel and automatically loaded whenever a new process is exec-ed. that's why there's no filesystem path on the right side -- there is none! the file only exists in the kernel memory (well, not 100% precise, but see the man page for more info).
the last line is the ELF interpreter. this is described in depth in the ld.so(8) manpage. the reason it has a full path is because your program has the full path hardcoded in it. the reason it doesn't have an entry on the right side is that it's not linked against (directly) and thus no search was performed. you can check this by running:
$ readelf -l node | grep interpreter
[Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
$ scanelf -i node
ET_EXEC /lib64/ld-linux-x86-64.so.2 node
all the other lines are libraries you've linked against. you can see those by looking at DT_NEEDED tags when you run readelf -d on the file. since those files lack full paths, the ld.so needs to perform a dynamic path search for it. that's actually what the lines are telling you: "libdl.so.2 is needed, so when i searched for it, i found it at /lib64/libdl.so.2 (and was loaded into memory at address 0x00007f70f7855000)"
ldd filename shows you the program shared libraries used by the file.
libc.so.6, for example, is libc shared object version 6, which sits in /lib64 and its memory location is 0x00007f70f684f000.
The last line talks about ld-linux-x86-64.so version 2 under /lib64. This fellow will find and load shared libraries node needs. It will prep those libraries and run them. So, speaking in very crude terms, ld-linux-x86-64 is the runner. libc.so.6 and others are loaded and ldd shows the location of those shared libraries and memory locations. That is my understanding.
I am trying to build a simple executable that uses boost_serialization and boost_iostreams.
#include <fstream>
#include <iostream>
#include <boost/archive/xml_iarchive.hpp>
#include <boost/archive/xml_oarchive.hpp>
#include <boost/iostreams/filtering_stream.hpp>
#include <boost/iostreams/filter/gzip.hpp>
#include <boost/iostreams/device/file.hpp>
int main()
{
using namespace boost::iostreams;
filtering_ostream os;
os.push(boost::iostreams::gzip_compressor());
os.push(boost::iostreams::file_sink("emptyGzipBug.txt.gz"));
}
Unfortunately the system I am working with has a very outdated version of boost_serialization in /usr/lib/, and I have no way to change that.
I am fairly certain when I build the example using
g++ -o main main.cpp -lboost_serialization -lboost_iostreams
that the linker errors result because gcc uses the system version of boost_serialization rather than my locally installed version. Setting LIBRARY_PATH and LD_LIBRARY_PATH to /home/andrew/install/lib doesnt work. When i build using
g++ -o main main.cpp -L/home/andrew/install/lib -lboost_serialization -lboost_iostreams
then everything works.
My questions are:
How can I get gcc to tell me the filenames of the libraries its using?
Is it possible to setup the environment so that I dont have to specify the absolute path to my local boost on the command line of gcc.
PS After typing the below info, I thought I'd be kind and add what you need for your specific case:
g++ -Wl,-rpath,/home/andrew/install/lib -o main main.cpp -I/home/andrew/install/include -L/home/andrew/install/lib -lboost_serialization -lboost_iostreams
gcc itself doesn't care about the libraries. The linker does ;).
Even though the linker needs to find the shared libraries so it can resolve
symbols, it doesn't store the path of those libraries in the executable normally.
So, for a start, lets find out what is actually in the binary after you linked it:
$ readelf -d main | grep 'libboost'
0x0000000000000001 (NEEDED) Shared library: [libboost_serialization.so.1.54.0]
0x0000000000000001 (NEEDED) Shared library: [libboost_iostreams.so.1.54.0]
Just the names thus.
The libraries that are actually used are detemined by /lib/ld-linux.so.*
at run time:
$ ldd main | grep libboost
libboost_serialization.so.1.54.0 => /usr/lib/x86_64-linux-gnu/libboost_serialization.so.1.54.0 (0x00007fd8fa920000)
libboost_iostreams.so.1.54.0 => /usr/lib/x86_64-linux-gnu/libboost_iostreams.so.1.54.0 (0x00007fd8fa700000)
The path is found by looking in /etc/ld.so.cache (which is normally
compiled by running ldconfig). You can print its contents with:
ldconfig -p | grep libboost_iostreams
libboost_iostreams.so.1.54.0 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libboost_iostreams.so.1.54.0
libboost_iostreams.so.1.49.0 (libc6,x86-64) => /usr/lib/libboost_iostreams.so.1.49.0
libboost_iostreams.so (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libboost_iostreams.so
but since that is only the cached result of a previous look up,
you are more interested in the output of:
$ ldconfig -v 2>/dev/null | egrep '^[^[:space:]]|libboost_iostreams'
/lib/i386-linux-gnu:
/usr/lib/i386-linux-gnu:
/usr/local/lib:
/lib/x86_64-linux-gnu:
/usr/lib/x86_64-linux-gnu:
libboost_iostreams.so.1.54.0 -> libboost_iostreams.so.1.54.0
/lib32:
/usr/lib32:
/lib:
/usr/lib:
libboost_iostreams.so.1.49.0 -> libboost_iostreams.so.1.49.0
which shows the paths that it looked in before finding a result.
Note if you are linking a 64bit program and it would find a 32bit
library first (or visa versa) then that would be skipped as being
incompatible. Otherwise, the first one found is used.
The paths used to search are specified in /etc/ld.so.conf which is
read (usually at boot time, or after installing something new)
when running ldconfig as root.
However, precedence take paths specified as a colon separated list
of paths in the environment variable LD_LIBRARY_PATH.
For example, if I'd do:
$ export LD_LIBRARY_PATH=/tmp
$ cp /usr/lib/libboost_iostreams.so.1.49.0 /tmp/libboost_iostreams.so.1.54.0
$ ldd main | grep libboost_iostreams
libboost_iostreams.so.1.54.0 => /tmp/libboost_iostreams.so.1.54.0 (0x00007f621add8000)
then it finds 'libboost_iostreams.so.1.54.0' in /tmp (even though it was a libboost_iostreams.so.1.49.0).
Note that you CAN hardcode a path in your executable by passing -rpath to
the linker:
$ unset LD_LIBRARY_PATH
$ g++ -Wl,-rpath,/tmp -o main main.cpp -lboost_serialization -lboost_iostreams
$ ldd main | grep libboost_iostreams
libboost_iostreams.so.1.54.0 => /tmp/libboost_iostreams.so.1.54.0 (0x00007fbd8bcd8000)
which can be made visible with
$ readelf -d main | grep RPATH
0x000000000000000f (RPATH) Library rpath: [/tmp]
Note that LD_LIBRARY_PATH even takes precedence over -rpath, unless
you also passed -Wl,--disable-new-dtags, along with the -rpath and provided that you are linking an executable and your linker supports
this flag.
You can show the search paths that gcc uses during compile(link) time with the -print-search-dirs command line option:
$ g++ -print-search-dirs | grep libraries
libraries: =/usr/lib/gcc/x86_64-linux-gnu/4.7/:/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../../x86_64-linux-gnu/lib/x86_64-linux-gnu/4.7/:/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../../x86_64-linux-gnu/lib/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../../x86_64-linux-gnu/lib/../lib/:/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../x86_64-linux-gnu/4.7/:/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../../lib/:/lib/x86_64-linux-gnu/4.7/:/lib/x86_64-linux-gnu/:/lib/../lib/:/usr/lib/x86_64-linux-gnu/4.7/:/usr/lib/x86_64-linux-gnu/:/usr/lib/../lib/:/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../../x86_64-linux-gnu/lib/:/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../:/lib/:/usr/lib/
This can be influenced by adding -L command line options. If a library can't be found in a path specified with the -L option then it looks in paths found through the environment variable GCC_EXEC_PREFIX (see the man page for that) and if that fails it uses the environment variable LIBRARY_PATH.
When you run g++ with the -v option, it will print the LIBRARY_PATH used.
LIBRARY_PATH=/tmp/lib g++ -v -o main main.cpp -lboost_serialization -lboost_iostreams 2>&1 | grep LIBRARY_PATH
LIBRARY_PATH=/tmp/lib/../lib/:/usr/lib/gcc/x86_64-linux-gnu/4.7/:/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../../lib/:/lib/x86_64-linux-gnu/:/lib/../lib/:/usr/lib/x86_64-linux-gnu/:/usr/lib/../lib/:/tmp/lib/:/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../:/lib/:/usr/lib/
Finally, note that especially for boost (but in general) you should
use header files that match the correct version! So, if the library that you
link with at run time is version xyz you should have used an -I command line option to get g++ to find the corresponding header files, or things might not link or worse, result in unexplainable crashes.
-nodefaultlibs
Do not use the standard system libraries when linking. Only the
libraries you specify are passed to the linker, and options
specifying linkage of the system libraries, such as
-static-libgcc or -shared-libgcc, are ignored. The standard
startup files are used normally, unless -nostartfiles is used.
The compiler may generate calls to "memcmp", "memset", "memcpy"
and "memmove". These entries are usually resolved by entries in
libc. These entry points should be supplied through some other
mechanism when this option is specified.
Haven't used it myself but it sounds exactly like what was asked for.
When executing ldd on a file, it returns a hex number in parentheses vor every library it found.
For example:
root#server> ldd wpa_supplicant
linux-gate.so.1 => (0xb779b000)
libnl.so.1 => /usr/lib/libnl.so.1 (0xb774d000)
libssl.so.1.0.0 => not found
libcrypto.so.1.0.0 => not found
libdl.so.2 => /lib/i686/cmov/libdl.so.2 (0xb7748000)
libc.so.6 => /lib/i686/cmov/libc.so.6 (0xb75ed000)
libm.so.6 => /lib/i686/cmov/libm.so.6 (0xb75c7000)
/lib/ld-linux.so.2 (0xb779c000)
If the hex number is not the one of the library the executable once got linked against, a version information error may occur.
I got two questions:
Where does this value originate?
How can I find out which hex value the executable is looking for? (i.e. the one it originally got linked against)
The hexadecimal numbers are the memory addresses the respective library gets loaded into. See https://stackoverflow.com/a/5130690/637284 for further explanation.
Linux binaries are usually dynamically linked to the core system library (libc). This keeps the memory footprint of the binary quite small but binaries which are dependent on the latest libraries will not run on older systems. Conversely, binaries linked to older libraries will run happily on the latest systems.
Therefore, in order to ensure our application has good coverage during distribution we need to figure out the oldest libc we can support and link our binary against that.
How should we determine the oldest version of libc we can link to?
Work out which symbols in your executable are creating the dependency on the undesired version of glibc.
$ objdump -p myprog
...
Version References:
required from libc.so.6:
0x09691972 0x00 05 GLIBC_2.3
0x09691a75 0x00 03 GLIBC_2.2.5
$ objdump -T myprog | fgrep GLIBC_2.3
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.3 realpath
Look within the depended-upon library to see if there are any symbols in older versions that you can link against:
$ objdump -T /lib/libc.so.6 | grep -w realpath
0000000000105d90 g DF .text 0000000000000021 (GLIBC_2.2.5) realpath
000000000003e7b0 g DF .text 00000000000004bf GLIBC_2.3 realpath
We're in luck!
Request the version from GLIBC_2.2.5 in your code:
#include <limits.h>
#include <stdlib.h>
__asm__(".symver realpath,realpath#GLIBC_2.2.5");
int main () {
realpath ("foo", "bar");
}
Observe that GLIBC_2.3 is no longer needed:
$ objdump -p myprog
...
Version References:
required from libc.so.6:
0x09691a75 0x00 02 GLIBC_2.2.5
$ objdump -T myprog | grep realpath
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 realpath
For further information, see http://web.archive.org/web/20160107032111/http://www.trevorpounds.com/blog/?p=103.
Unfortunately, #Sam's solution doesn't work well in my situation. But according to his way, I found my own way to solve that.
This is my situation:
I'm writing a C++ program using the Thrift framework(it's an RPC middleware). I prefer static link to dynamic link, so my program is linked to libthrift.a statically instead of libthrift.so. However, libthrift.a is dynamically linked to glibc, and since my libthrift.a is build on my system with glibc 2.15, my libthrift.a uses memcpy of version 2.14(memcpy#GLIBC_2.14) provided by glibc 2.15.
But the problem is that our server machines have only the glibc version 2.5 which has only memcpy#GLIBC_2.2.5. It is much lower than memcpy#GLIBC_2.14. So, of course, my server program can't run on those machines.
And I found this solusion:
Using .symver to obtain the ref to memcpy#GLIBC_2.2.5.
Write my own __wrap_memcpy function which just calls memcpy#GLIBC_2.2.5 directly.
When linking my program, add -Wl,--wrap=memcpy option to gcc/g++.
The code involved in steps 1 and 2 is here: https://gist.github.com/nicky-zs/7541169
To do this in a more automated fashion, you can use the following script to create a list of all the symbols that are newer in your GLIBC than in a given version (set on line 2). It creates a glibc.h file (filename set by the script argument) which contains all the necessary .symver declarations. You can then add -include glibc.h to your CFLAGS to make sure it gets picked up everywhere in your compilation.
This is sufficient if you don't use any static libraries that were compiled without the above include. If you do, and you don't want to recompile, you can use objcopy to create a copy of the library with the symbols renamed to the old versions. The second to bottom line of the script creates a version of your system libstdc++.a that will link against the old glibc symbols. Adding -L. (or -Lpath/to/libstdc++.a/) will make your program statically link libstdc++ without linking in a bunch of new symbols. If you don't need this, delete the last two lines and the printf ... redeff line.
#!/bin/bash
maxver=2.9
headerf=${1:-glibc.h}
set -e
for lib in libc.so.6 libm.so.6 libpthread.so.0 libdl.so.2 libresolv.so.2 librt.so.1; do
objdump -T /usr/lib/$lib
done | awk -v maxver=${maxver} -vheaderf=${headerf} -vredeff=${headerf}.redef -f <(cat <<'EOF'
BEGIN {
split(maxver, ver, /\./)
limit_ver = ver[1] * 10000 + ver[2]*100 + ver[3]
}
/GLIBC_/ {
gsub(/\(|\)/, "",$(NF-1))
split($(NF-1), ver, /GLIBC_|\./)
vers = ver[2] * 10000 + ver[3]*100 + ver[4]
if (vers > 0) {
if (symvertext[$(NF)] != $(NF-1))
count[$(NF)]++
if (vers <= limit_ver && vers > symvers[$(NF)]) {
symvers[$(NF)] = vers
symvertext[$(NF)] = $(NF-1)
}
}
}
END {
for (s in symvers) {
if (count[s] > 1) {
printf("__asm__(\".symver %s,%s#%s\");\n", s, s, symvertext[s]) > headerf
printf("%s %s#%s\n", s, s, symvertext[s]) > redeff
}
}
}
EOF
)
sort ${headerf} -o ${headerf}
objcopy --redefine-syms=${headerf}.redef /usr/lib/libstdc++.a libstdc++.a
rm ${headerf}.redef
glibc 2.2 is a pretty common minimum version. However finding a build platform for that version may be non-trivial.
Probably a better direction is to think about the oldest OS you want to support and build on that.