I have a stripped down spec file that looks like:
Name: some-binary
Version: 6.1.0
Release: 1
License: proprietary
Summary: a binary
%description
Whatever
%install
install -d %{buildroot}/fromrpm
cp %{_sourcedir}/thebinary %{buildroot}//fromrpm/thebinary
%files
/fromrpm/thebinary
Then I run:
$ cd ~/rpmbuild/
$ rpmbuild -bb SPECS/some-binary.spec 2>/dev/null
Executing(%install): /bin/sh -e /var/tmp/rpm-tmp.gxRpSj
Processing files: some-binary-6.1.0-1.x86_64
Provides: some-binary = 6.1.0-1 some-binary(x86-64) = 6.1.0-1
Requires(rpmlib): rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1
Checking for unpackaged file(s): /usr/lib/rpm/check-files /home/dspeyer/rpmbuild/BUILDROOT/some-binary-6.1.0-1.x86_64
Wrote: /home/dspeyer/rpmbuild/RPMS/x86_64/some-binary-6.1.0-1.x86_64.rpm
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.WR3Utx
$ rpm2cpio RPMS/x86_64/some-binary-6.1.0-1.x86_64.rpm | cpio -idmv
./fromrpm/thebinary
41456 blocks
$ readelf -a SOURCES/thebinary | grep rela\\.
[ 5] .rela.plt RELA 0000000000400338 00000338
00 .note.gnu.property .note.gnu.build-id .note.ABI-tag .note.go.buildid .rela.plt
Relocation section '.rela.plt' at offset 0x338 contains 29 entries:
$ readelf -a fromrpm/thebinary | grep rela\\.
[ 6] .rela.got.plt RELA 0000000000000000 0143d048
Relocation section '.rela.got.plt' at offset 0x143d048 contains 29 entries:
I'm not actually sure what the rela tags do, but I strongly suspect they're the reason the post-rpm binary when run on rhel 7.9 immediately crashes trying to crash with "Unexpected reloc type in static binary". Yes, the binary is statically linked. (And, yes, I did say "crashes trying to crash": sigsegv inside __libc_fatal)
Any ideas what rpmbuild is trying to do, or how to stop it?
Related
When I load Fedora 28's /usr/bin/ls file into GDB, I can access to the symbol abformat_init, even if it is not present as a string nor in the symbols table of the binary file.
$ file /usr/bin/ls
/usr/bin/ls: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=d6d0ea6be508665f5586e90a30819d090710842f, stripped, too many notes (256)
$ readelf -S /usr/bin/ls | grep abformat
$ nm /usr/bin/ls
nm: /usr/bin/ls: no symbols
$ strings /usr/bin/ls | grep abformat
$ gdb /usr/bin/ls
[...]
Reading symbols from /usr/bin/ls...Reading symbols from /usr/bin/ls...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Missing separate debuginfos, use: dnf debuginfo-install coreutils-8.29-7.fc28.x86_64
(gdb) info symbol abformat_init
abformat_init in section .text of /usr/bin/ls
Where does this symbol comes from? Is there a program that allows to extract them outside of GDB?
TL;DR:
There is a special .gnu_debugdata compressed section in Fedora binaries that GDB reads, and which contains mini-symbols.
Contents of that section can be conveniently printed with eu-readelf -Ws --elf-section /usr/bin/ls
readelf -S /usr/bin/ls | grep abformat
That command is dumping sections. You want symbols instead:
readelf -s /usr/bin/ls | grep abformat
readelf --all /usr/bin/ls | grep abformat
strings /usr/bin/ls | grep abformat
Strings tries to guess what you want, and doesn't output all strings found in the binary. See this blog post and try:
strings -a /usr/bin/ls | grep abformat
Update: I confirmed the results you've observed: abformat does not appear anywhere, yet GDB knows about it.
Turns out, there is a .gnu_debugdata compressed section (described here), which has mini-symbols.
To extract this data, normally you would do:
objcopy -O binary -j .gnu_debugdata /usr/bin/ls ls.mini.xz
However, that is broken on my system (produces empty output), so instead I used dd:
# You may need to adjust the numbers below from "readelf -WS /usr/bin/ls"
dd if=/usr/bin/ls of=ls.mini.xz bs=1 skip=151896 count=3764
xz -d ls.mini.xz
nm ls.mini | grep abformat
This produced:
00000000000005db0 t abformat_init
QED.
Additional info:
Confusing GDB no debugging symbols is addressed in this bug.
objcopy refusing to copy .gnu_debugdata is the subject of this bug.
There is a tool that can conveniently dump this info:
eu-readelf -Ws --elf-section /usr/bin/ls | grep abformat
37: 0000000000005db0 593 FUNC LOCAL DEFAULT 14 abformat_init
Is there a program that allows to extract them outside of GDB?
Yes, you can use nm to extract the symbol, but you should look for the symbol in a separate debug info file, because the binary itself is stripped.
You can use readelf or objdump to know separate debug info file name, see How to know the name and/or path of the debug symbol file which is linked to a binary executable?:
$ objdump -s -j .gnu_debuglink /usr/bin/ls
/usr/bin/ls: file format elf64-x86-64
Contents of section .gnu_debuglink:
0000 6c732d38 2e33302d 362e6663 32392e78 ls-8.30-6.fc29.x
0010 38365f36 342e6465 62756700 5cddcc98 86_64.debug.\...
On Fedora 29 the separate debug info file name for /usr/bin/ls is ls-8.30-6.fc29.x86_64.debug.
Normally, on Fedora, separate debug info is installed to /usr/lib/debug/ directory so the full path to debug info file is /usr/lib/debug/usr/bin/ls-8.30-6.fc29.x86_64.debug.
Now you can look for the symbol with nm:
$ nm /usr/lib/debug/usr/bin/ls-8.30-6.fc29.x86_64.debug | grep abformat_init
0000000000006d70 t abformat_init
Note that separate debug info should be installed with debuginfo-install, this is what gdb is telling you.
I have already seen Cannot enable kernel dynamic debugging on linux ; https://www.kernel.org/doc/html/v4.11/admin-guide/dynamic-debug-howto.html .
I have rebuilt the Raspbian 9 kernel with CONFIG_DYNAMIC_DEBUG, and booted into it; the file /sys/kernel/debug/dynamic_debug/control and is populated with 2k+ dynamic debug rule statements:
pi#raspberrypi:~ $ sudo ls -la /sys/kernel/debug/dynamic_debug/control
-rw-r--r-- 1 root root 0 Jan 1 1970 /sys/kernel/debug/dynamic_debug/control
pi#raspberrypi:~ $ sudo cat /sys/kernel/debug/dynamic_debug/control | wc -l
2358
pi#raspberrypi:~ $ sudo grep 'snd_device' /sys/kernel/debug/dynamic_debug/control
sound/core/device.c:132 [snd]snd_device_disconnect =_ "device disconnect %p (from %pS), not found\012"
sound/core/device.c:156 [snd]snd_device_free =_ "device free %p (from %pS), not found\012"
Ok, so I want to trace the is_connected_output_ep function, which is in sound/soc/soc-dapm.c. So I do this:
pi#raspberrypi:~ $ sudo bash -c "echo -n 'func is_connected_output_ep +p' > /sys/kernel/debug/dynamic_debug/control"
pi#raspberrypi:~ $ sudo cat /sys/kernel/debug/dynamic_debug/control | grep is_conn
pi#raspberrypi:~ $
pi#raspberrypi:~ $ sudo bash -c "echo 'file sound/soc/soc-dapm.c line 1175 +p' > /sys/kernel/debug/dynamic_debug/control"
pi#raspberrypi:~ $ sudo cat /sys/kernel/debug/dynamic_debug/control | grep dapm
pi#raspberrypi:~ $
... and I get no errors - but seemingly, nothing "sticks". (and yeah, I don't see this function being traced either).
The documentation says that +p does:
p enables the pr_debug() callsite.
I'm not sure what they mean by this - does it mean that if there are already existing pr_debug statements in the function, then they will be enabled (i.e. will print to syslog) with this? If so, what happens in the case when there are no such statements in the function - as is the case with is_connected_output_ep? Can I still setup dynamic debug to somehow trace this function - without having to manually insert printk or other statements and recompiling the kernel module?
Well, I did some more reading, and it seems the answer to:
does it mean that if there are already existing pr_debug statements in the function, then they will be enabled (i.e. will print to syslog) with this?
... is likely "yes" - so you cannot do dynamic debug of a function that does not have pr_debug statements in it already.
Also, it seems that the /sys/kernel/debug/dynamic_debug/control (upon read) is actually a list of all possible dynamic debug "probe points" if you will, along with their status (enabled or not), though I'm not sure about this.
Anyways, here is some more reading where this stuff is mentioned:
The dynamic debugging interface [LWN.net] 2011
Dynamic Debug, conference paper, 2009
So I cannot trace is_connected_output_ep with dynamic debug - so maybe I should look into ftrace or kprobes (dynamic probes) facilities of the Linux kernel...
EDIT: It turns out, dynamic_debug/control lists debuggable statements ONLY from currently loaded modules in the kernel! Example, there is a dev_dbg in the dpcm_path_get function in the soc-pcm.c source file, which ends up in the snd_soc_core kernel module (snd-soc-core.ko). This module by default is not loaded by Raspbian 9, so we get this:
pi#raspberrypi:~ $ lsmod | grep snd
snd_bcm2835 32768 1
snd_pcm 98304 1 snd_bcm2835
snd_timer 32768 1 snd_pcm
snd 69632 5 snd_timer,snd_bcm2835,snd_pcm
pi#raspberrypi:~ $ sudo grep 'soc-pcm' /sys/kernel/debug/dynamic_debug/control
pi#raspberrypi:~ $
Ok, now if the kernel module is loaded with modprobe, now suddenly the debuggable callsites appear in dynamic_debug/control:
pi#raspberrypi:~ $ sudo modprobe snd_soc_core
pi#raspberrypi:~ $ lsmod | grep snd
snd_soc_core 200704 0
snd_compress 20480 1 snd_soc_core
snd_pcm_dmaengine 16384 1 snd_soc_core
snd_bcm2835 32768 1
snd_pcm 98304 3 snd_pcm_dmaengine,snd_bcm2835,snd_soc_core
snd_timer 32768 1 snd_pcm
snd 69632 7 snd_compress,snd_timer,snd_bcm2835,snd_soc_core,snd_pcm
pi#raspberrypi:~ $ sudo grep 'soc-pcm' /sys/kernel/debug/dynamic_debug/control
sound/soc/soc-pcm.c:1367 [snd_soc_core]dpcm_prune_paths =_ "ASoC: pruning %s BE %s for %s\012"
sound/soc/soc-pcm.c:1373 [snd_soc_core]dpcm_prune_paths =_ "ASoC: found %d old BE paths for pruning\012"
...
pi#raspberrypi:~ $ sudo grep 'dpcm_path_get' /sys/kernel/debug/dynamic_debug/control
sound/soc/soc-pcm.c:1331 [snd_soc_core]dpcm_path_get =_ "ASoC: found %d audio %s paths\012"
And finally, we can now enable this print statement:
pi#raspberrypi:~ $ sudo bash -c "echo 'func dpcm_path_get +p' > /sys/kernel/debug/dynamic_debug/control"
pi#raspberrypi:~ $ sudo grep 'dpcm_path_get' /sys/kernel/debug/dynamic_debug/control
sound/soc/soc-pcm.c:1331 [snd_soc_core]dpcm_path_get =p "ASoC: found %d audio %s paths\012"
Apparently, the disabled lines have a =_ symbol in the line, and the enabled lines have =p ...
Now all that I'd want, is to enable some statements before the driver is loaded, so I could monitor printouts in _probe functions of kernel module drivers...
You can add following argument to insmod
dyndbg="+p"
When I run qmake, it ends up defining CXX as /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++. which clang++ though sends me to /usr/bin/clang++. No big deal right?
Consider hello.cpp
#include <string>
int main() {
return 0;
}
when I run clang++ hello.cpp it compiles with no issues. But if I use the qmake clang++,
Tims-Air:hello tim$ /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++ hello.cpp
In file included from hello.cpp:1:
In file included from /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/string:470:
In file included from /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/string_view:171:
In file included from /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/__string:56:
In file included from /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/algorithm:638:
In file included from /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/cstring:61:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/string.h:61:15: fatal error: 'string.h' file not found
#include_next <string.h>
^~~~~~~~~~
1 error generated.
How can I overcome this issue in my qmake Makefile besides hardcoding the clang++ from /usr/bin?
Update. Details reveal that the search paths are different!
Tims-Air:Projects tim$ /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++ -v -E -x c++ - < /dev/null
Apple LLVM version 9.0.0 (clang-900.0.39.2)
Target: x86_64-apple-darwin17.0.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
"/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang" -cc1 -triple x86_64-apple-macosx10.13.0 -Wdeprecated-objc-isa-usage -Werror=deprecated-objc-isa-usage -E -disable-free -disable-llvm-verifier -discard-value-names -main-file-name - -mrelocation-model pic -pic-level 2 -mthread-model posix -mdisable-fp-elim -fno-strict-return -masm-verbose -munwind-tables -target-cpu penryn -target-linker-version 305 -v -dwarf-column-info -debugger-tuning=lldb -resource-dir /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/9.0.0 -stdlib=libc++ -fdeprecated-macro -fdebug-compilation-dir /Users/tim/Projects -ferror-limit 19 -fmessage-length 165 -stack-protector 1 -fblocks -fobjc-runtime=macosx-10.13.0 -fencode-extended-block-signature -fcxx-exceptions -fexceptions -fmax-type-align=16 -fdiagnostics-show-option -fcolor-diagnostics -o - -x c++ -
clang -cc1 version 9.0.0 (clang-900.0.39.2) default target x86_64-apple-darwin17.0.0
ignoring nonexistent directory "/usr/include/c++/v1"
ignoring nonexistent directory "/usr/include"
#include "..." search starts here:
#include <...> search starts here:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1
/usr/local/include
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/9.0.0/include
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include
/System/Library/Frameworks (framework directory)
/Library/Frameworks (framework directory)
End of search list.
# 1 "<stdin>"
# 1 "<built-in>" 1
# 1 "<built-in>" 3
# 342 "<built-in>" 3
# 1 "<command line>" 1
# 1 "<built-in>" 2
# 1 "<stdin>" 2
and
Tims-Air:Projects tim$ clang++ -v -E -x c++ - < /dev/null
Apple LLVM version 9.0.0 (clang-900.0.39.2)
Target: x86_64-apple-darwin17.0.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
"/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang" -cc1 -triple x86_64-apple-macosx10.13.0 -Wdeprecated-objc-isa-usage -Werror=deprecated-objc-isa-usage -E -disable-free -disable-llvm-verifier -discard-value-names -main-file-name - -mrelocation-model pic -pic-level 2 -mthread-model posix -mdisable-fp-elim -fno-strict-return -masm-verbose -munwind-tables -target-cpu penryn -target-linker-version 305 -v -dwarf-column-info -debugger-tuning=lldb -resource-dir /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/9.0.0 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk -I/usr/local/include -stdlib=libc++ -fdeprecated-macro -fdebug-compilation-dir /Users/tim/Projects -ferror-limit 19 -fmessage-length 165 -stack-protector 1 -fblocks -fobjc-runtime=macosx-10.13.0 -fencode-extended-block-signature -fcxx-exceptions -fexceptions -fmax-type-align=16 -fdiagnostics-show-option -fcolor-diagnostics -o - -x c++ -
clang -cc1 version 9.0.0 (clang-900.0.39.2) default target x86_64-apple-darwin17.0.0
ignoring nonexistent directory "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/usr/include/c++/v1"
ignoring nonexistent directory "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/usr/local/include"
ignoring nonexistent directory "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/Library/Frameworks"
#include "..." search starts here:
#include <...> search starts here:
/usr/local/include
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/9.0.0/include
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/usr/include
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/System/Library/Frameworks (framework directory)
End of search list.
# 1 "<stdin>"
# 1 "<built-in>" 1
# 1 "<built-in>" 3
# 342 "<built-in>" 3
# 1 "<command line>" 1
# 1 "<built-in>" 2
# 1 "<stdin>" 2
Workaround. It can be fixed by adding the 10.13 include as it is missing in the search path. But it sure feels ghetto.
Tims-Air:hello tim$ /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++ -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/usr/include hello.cpp
-isysroot does the trick as well
I fixed it by removing all the references to .qmake.stash in the output folder
qmake will also search up the directory hierarchy. If you run qmake directly, you can be sure that you're not using some cached information if it says "Info: creating file .... .qmake.stash"
This question already has an answer here:
Error during Cross-compiling C code with Dynamic libraries
(1 answer)
Closed 6 years ago.
I am using VAR-SOM-AM33 board and want to run sample code like hello world run on device and it gives -sh:./test:not found error
Toolchain used for compile code is gcc-linaro-arm-linux-gnueabihf-4.7-2013.03-20130313_linux.
Code is as below
#include <stdio.h>
int main(){
printf("hello world");
return 0;
}
for crosscompile file
arm-linux-gnueabihf-gcc test.c -march=armv7-a -marm -mthumb-interwork -mfloat-abi=hard -mfpu=neon -mtune=cortex-a8 -o test
output binary is shows as follows
test: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.31, BuildID[sha1]=2ce1c5b3d97dac2093fe2cd2d340cdaa9989923f, not stripped
After copy that file into hardware and run it shows following error
root#am335x-evm:~# ./test
-sh: ./test: not found
File permission also change by
root#am335x-evm:~# chmod +x test
but result shows same not found error.
Demo file which is running on hardware,its architecture as follows
root#am335x-evm:~# readelf -A /usr/bin/hello
Attribute Section: aeabi
File Attributes
Tag_CPU_name: "7-A"
Tag_CPU_arch: v7
Tag_CPU_arch_profile: Application
Tag_ARM_ISA_use: Yes
Tag_THUMB_ISA_use: Thumb-2
Tag_VFP_arch: VFPv3
Tag_Advanced_SIMD_arch: NEONv1
Tag_ABI_PCS_wchar_t: 4
Tag_ABI_FP_denormal: Needed
Tag_ABI_FP_exceptions: Needed
Tag_ABI_FP_number_model: IEEE 754
Tag_ABI_align8_needed: Yes
Tag_ABI_align8_preserved: Yes, except leaf SP
Tag_ABI_enum_size: int
Tag_ABI_HardFP_use: SP and DP
and file which is cross compiled,its architecture as follows
root#am335x-evm:~# readelf -A test
Attribute Section: aeabi
File Attributes
Tag_CPU_name: "7-A"
Tag_CPU_arch: v7
Tag_CPU_arch_profile: Application
Tag_ARM_ISA_use: Yes
Tag_THUMB_ISA_use: Thumb-2
Tag_VFP_arch: VFPv3
Tag_Advanced_SIMD_arch: NEONv1
Tag_ABI_PCS_wchar_t: 4
Tag_ABI_FP_denormal: Needed
Tag_ABI_FP_exceptions: Needed
Tag_ABI_FP_number_model: IEEE 754
Tag_ABI_align8_needed: Yes
Tag_ABI_align8_preserved: Yes, except leaf SP
Tag_ABI_enum_size: int
Tag_ABI_HardFP_use: SP and DP
Tag_ABI_VFP_args: VFP registers
Tag_CPU_unaligned_access: v6
Tag_unknown_44: 1 (0x1)
Also hardware cpuinfo is as follows
root#am335x-evm:~# cat /proc/cpuinfo
Processor : ARMv7 Processor rev 2 (v7l)
BogoMIPS : 718.02
Features : swp half thumb fastmult vfp edsp thumbee neon vfpv3 tls
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x3
CPU part : 0xc08
CPU revision : 2
Hardware : Variscite VAR-SOM-AM33
Revision : 0000
Serial : 0000000000000000
I have tried running ldd command on target device.
root#am335x-evm:~# ldd
-sh: ldd: not found
So I suspect that the issue is related to the linker.
If I simply compile the file, without linking it.
arm-linux-gnueabihf-gcc -mtune=cortex-a8 -march=armv7 -O -c test.c -o test
Now if I run this file I get this error.
root#am335x-evm:~# chmod +x test
root#am335x-evm:~# ./test
./test: line 1: syntax error: word unexpected (expecting ")")
please suggest how to resolve this.
Thanks for solution, I found error with linker.
existing file has linker ld-linux.so.3 and crosscompiled file has linker ld-linux-armhf.so.3
root#am335x-evm:/usr/bin# readelf -l hello
Elf file type is EXEC (Executable file)
Entry point 0x82fc
There are 8 program headers, starting at offset 52
Program Headers:
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
EXIDX 0x00044c 0x0000844c 0x0000844c 0x00008 0x00008 R 0x4
PHDR 0x000034 0x00008034 0x00008034 0x00100 0x00100 R E 0x4
INTERP 0x000134 0x00008134 0x00008134 0x00013 0x00013 R 0x1
[Requesting program interpreter: /lib/ld-linux.so.3]
crosscompiled file program header
root#am335x-evm:~# readelf -l test
Elf file type is EXEC (Executable file)
Entry point 0x82f9
There are 8 program headers, starting at offset 52
Program Headers:
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
EXIDX 0x000450 0x00008450 0x00008450 0x00008 0x00008 R 0x4
PHDR 0x000034 0x00008034 0x00008034 0x00100 0x00100 R E 0x4
INTERP 0x000134 0x00008134 0x00008134 0x00019 0x00019 R 0x1
[Requesting program interpreter: /lib/ld-linux-armhf.so.3]
after chang that linker program runs on target device.
root#am335x-evm:~# cd /lib/
root#am335x-evm:/lib# ls -l ld-linux.so.3
lrwxrwxrwx 1 1000 1000 12 Aug 7 2012 ld-linux.so.3 -> ld-2.12.2.so
root#am335x-evm:/lib# ln -s /li ld-2.12.2.so ld-linux-armhf.so.3
/lib/ /linuxrc
root#am335x-evm:/lib# ln -s /lib/ld-2.12.2.so ld-linux-armhf.so.3
root#am335x-evm:/lib# ldconfig
root#am335x-evm:/lib# cd
root#am335x-evm:~# ./test
hello worldroot#am335x-evm:~#
Edit: added important note that it is about debugging MPI application
System installed shared library doesn't have debugging symbols:
$ readelf -S /usr/lib64/libfftw3.so | grep debug
$
I have therefore compiled and instaled in my home directory my owne version, with debugging enabled (--with-debug CFLAGS=-g):
$ $ readelf -S ~/lib64/libfftw3.so | grep debug
[26] .debug_aranges PROGBITS 0000000000000000 001d3902
[27] .debug_pubnames PROGBITS 0000000000000000 001d8552
[28] .debug_info PROGBITS 0000000000000000 001ddebd
[29] .debug_abbrev PROGBITS 0000000000000000 003e221c
[30] .debug_line PROGBITS 0000000000000000 00414306
[31] .debug_str PROGBITS 0000000000000000 0044aa23
[32] .debug_loc PROGBITS 0000000000000000 004514de
[33] .debug_ranges PROGBITS 0000000000000000 0046bc82
I have set both LD_LIBRARY_PATH and LD_RUN_PATH to include ~/lib64 first, and ldd program confirms that local version of library should be used:
$ ldd a.out | grep fftw
libfftw3.so.3 => /home/narebski/lib64/libfftw3.so.3 (0x00007f2ed9a98000)
The program in question is parallel numerical application using MPI (Message Passing Interface). Therefore to run this application one must use mpirun wrapper (e.g. mpirun -np 1 valgrind --tool=callgrind ./a.out). I use OpenMPI implementation.
Nevertheless, various profilers: callgrind tool in Valgrind, CPU profiling google-perfutils and perf doesn't find those debugging symbols, resulting in more or less useless output:
calgrind:
$ callgrind_annotate --include=~/prog/src --inclusive=no --tree=none
[...]
--------------------------------------------------------------------------------
Ir file:function
--------------------------------------------------------------------------------
32,765,904,336 ???:0x000000000014e500 [/usr/lib64/libfftw3.so.3.2.4]
31,342,886,912 /home/narebski/prog/src/nonlinearity.F90:__nonlinearity_MOD_calc_nonlinearity_kxky [/home/narebski/prog/bin/a.out]
30,288,261,120 /home/narebski/gene11/src/axpy.F90:__axpy_MOD_axpy_ij [/home/narebski/prog/bin/a.out]
23,429,390,736 ???:0x00000000000fc5e0 [/usr/lib64/libfftw3.so.3.2.4]
17,851,018,186 ???:0x00000000000fdb80 [/usr/lib64/libmpi.so.1.0.1]
google-perftools:
$ pprof --text a.out prog.prof
Total: 8401 samples
842 10.0% 10.0% 842 10.0% 00007f200522d5f0
619 7.4% 17.4% 5025 59.8% calc_nonlinearity_kxky
517 6.2% 23.5% 517 6.2% axpy_ij
427 5.1% 28.6% 3156 37.6% nl_to_direct_xy
307 3.7% 32.3% 1234 14.7% nl_to_fourier_xy_1d
perf events:
$ perf report --sort comm,dso,symbol
# Events: 80K cycles
#
# Overhead Command Shared Object Symbol
# ........ ....... .................... ............................................
#
32.42% a.out libfftw3.so.3.2.4 [.] fdc4c
16.25% a.out 7fddcd97bb22 [.] 7fddcd97bb22
7.51% a.out libatlas.so.0.0.0 [.] ATL_dcopy_xp1yp1aXbX
6.98% a.out a.out [.] __nonlinearity_MOD_calc_nonlinearity_kxky
5.82% a.out a.out [.] __axpy_MOD_axpy_ij
Edit Added 11-07-2011:
I don't know if it is important, but:
$ file /usr/lib64/libfftw3.so.3.2.4
/usr/lib64/libfftw3.so.3.2.4: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, stripped
and
$ file ~/lib64/libfftw3.so.3.2.4
/home/narebski/lib64/libfftw3.so.3.2.4: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, not stripped
If /usr/lib64/libfftw3.so.3.2.4 is listed in callgrind output, then your LD_LIBRARY_PATH=~/lib64 had no effect.
Try again with export LD_LIBRARY_PATH=$HOME/lib64. Also watch out for any shell scripts you invoke, which might reset your environment.
You and Employed Russian are almost certainly right; the mpirun script is messing things up here. Two options:
Most x86 MPI implementations, as a practical matter, treat just running the executable
./a.out
the same as
mpirun -np 1 ./a.out.
They don't have to do this, but OpenMPI certainly does, as does MPICH2 and IntelMPI. So if you can do the debug serially, you should just be able to
valgrind --tool=callgrind ./a.out.
However, if you do want to run with mpirun, the issue is probably that your ~/.bashrc
(or whatever) is being sourced, undoing your changes to LD_LIBRARY_PATH etc. Easiest is just to temporarily put your changed environment variables in your ~/.bashrc for the duration of the run.
The way recent profiling tools typically handle this situation is to consult an external, matching non-stripped version of the library.
On debian-based Linux distros this is typically done by installing the -dbg suffixed version of a package; on Redhat-based they are named -debuginfo.
In the case of the tools you mentioned above; they will typically Just Work (tm) and find the debug symbols for a library if the debug info package has been installed in the standard location.