Problem with cross compiling libc.so.6 for armhf - inconsistent architectures - linux

I am cross-compiling a go (golang) program to armhf. The program compiles successfully, but when I try to run it on an ARM board, it crashes with 'Illegal instruction' error.
Chasing down the problem, I am almost sure this is due to fact that glib.so.6 has inconsistent architecture.
Facts:
Host: ubuntu 18 (bionic), amd64
GCC: gcc version 7.5.0 (Ubuntu/Linaro 7.5.0-3ubuntu1~18.04)
I am using multiarch (armhf)
My Cortex-a5 ARM processor:
$ cat /proc/cpuinfo
Processor : ARMv7 Processor rev 1 (v7l)
BogoMIPS : 262.96
Features : swp half thumb fastmult vfp edsp vfpv3 vfpv3d16 tls vfpv4 <br>
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part : 0xc05
CPU revision : 1
<br>
Hardware : SAMA5D3X-EK
Revision : 0000
Serial : 0000000000000000
All my armhf shared libraries have the following architecture:
$ readelf -A /lib/arm-linux-gnueabihf/ld-2.27.so
Attribute Section: aeabi
File Attributes
Tag_CPU_name: "7-A"
Tag_CPU_arch: v7
Tag_CPU_arch_profile: Application
Tag_ARM_ISA_use: Yes
Tag_THUMB_ISA_use: Thumb-2
Tag_FP_arch: VFPv3-D16
Tag_ABI_PCS_wchar_t: 4
Tag_ABI_FP_rounding: Needed
Tag_ABI_FP_denormal: Needed
Tag_ABI_FP_exceptions: Needed
Tag_ABI_FP_number_model: IEEE 754
Tag_ABI_align_needed: 8-byte
Tag_ABI_align_preserved: 8-byte, except leaf SP
Tag_ABI_enum_size: int
Tag_ABI_VFP_args: VFP registers
Tag_ABI_optimization_goals: Aggressive Speed
Tag_CPU_unaligned_access: v6
However, the armhf libc.so.6 is:
$ readelf -A /lib/arm-linux-gnueabihf/libc-2.27.so
Attribute Section: aeabi
File Attributes
Tag_CPU_name: "7-A"
Tag_CPU_arch: v7
Tag_CPU_arch_profile: Application
Tag_ARM_ISA_use: Yes
Tag_THUMB_ISA_use: Thumb-2
Tag_FP_arch: VFPv3
Tag_Advanced_SIMD_arch: NEONv1
Tag_ABI_PCS_wchar_t: 4
Tag_ABI_FP_rounding: Needed
Tag_ABI_FP_denormal: Needed
Tag_ABI_FP_exceptions: Needed
Tag_ABI_FP_user_exceptions: Needed
Tag_ABI_FP_number_model: IEEE 754
Tag_ABI_align_needed: 8-byte
Tag_ABI_enum_size: int
Tag_ABI_VFP_args: VFP registers
Tag_CPU_unaligned_access: v6
If I understand things correctly, this is wrong, as my processor does not support neon.
An old version of libc.so.6 on the ARM board looks as follows:
$ readelf -A /media/marek/sd/board/libc.so.6
Attribute Section: aeabi
File Attributes
Tag_CPU_name: "7-A"
Tag_CPU_arch: v7
Tag_CPU_arch_profile: Application
Tag_ARM_ISA_use: Yes
Tag_THUMB_ISA_use: Thumb-2
Tag_FP_arch: VFPv3-D16
Tag_ABI_PCS_wchar_t: 4
Tag_ABI_FP_denormal: Needed
Tag_ABI_FP_exceptions: Needed
Tag_ABI_FP_number_model: IEEE 754
Tag_ABI_align_needed: 8-byte
Tag_ABI_align_preserved: 8-byte, except leaf SP
Tag_ABI_enum_size: int
Tag_ABI_HardFP_use: Deprecated
Tag_ABI_VFP_args: VFP registers
Tag_CPU_unaligned_access: v6
Tag_DIV_use: Not allowed
Which looks consistent with the rest.
I wrote a trivial hello world program, and it crashes, too. But when I replace the armhf libc.so.6 with the old version on the board, it runs OK. However, I need a new version of libc (>= 2.25). Thus, it looks like it is sufficient to cross compile it from sources.
I am using configure with the following options:
../configure arm-linux-gnueabihf --prefix=pwd CFLAGS='-march=armv7-a -mfpu=vfpv3-d16 -g -O2 -w' LDFLAGS='-march=armv7-a -mfpu=vfpv3-d16 -g -w'
(also tried without LDFLAGS). All static libraries have the desired architecture but libc.so still has the wrong one (as above) - VFPv3 and NEONv1 are set). What am I doing wrong?

Related

Incorrect cross compiling to ARMv6 in Go project

I'm trying to cross compile a Go program for the armv6l architecture, specifically to get it running on a Raspberry Pi Zero W.
$ cat /proc/cpuinfo
processor : 0
model name : ARMv6-compatible processor rev 7 (v6l)
BogoMIPS : 697.95
Features : half thumb fastmult vfp edsp java tls
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part : 0xb76
CPU revision : 7
Hardware : BCM2835
Revision : 9000c1
Serial : 0000000059e0c5a9
Model : Raspberry Pi Zero W Rev 1.1
I was doing what I thought was right by running the command:
env CC=arm-linux-gnueabihf-gcc CGO_ENABLED=1 GOOS=linux GOARM=6 GOARCH=arm go build -o build/dht_sensor_exporter-$(VERSION)-linux-armv6 ./cmd/main.go
At first glance, after running file dht_sensor_exporter-$(VERSION)-linux-armv6 it looks good, as it outputs:
dht_sensor_exporter-$(VERSION)-linux-armv6: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, BuildID[sha1]=d84b6c4b0dc4b77b9c65db7bc30823e5dbe1078c, for GNU/Linux 3.2.0, with debug_info, not stripped
When I try to run it on the Pi Zero W, it then outputs Illegal instruction. After a bit of digging, it seems that the cross compile didn't actually generate the file for the right architecture.
I found this using the following command:
$ readelf -A dht_sensor_exporter
Attribute Section: aeabi
File Attributes
Tag_CPU_name: "7-A"
Tag_CPU_arch: v7
Tag_CPU_arch_profile: Application
Tag_ARM_ISA_use: Yes
Tag_THUMB_ISA_use: Thumb-2
Tag_FP_arch: VFPv3-D16
Tag_ABI_PCS_wchar_t: 4
Tag_ABI_FP_rounding: Needed
Tag_ABI_FP_denormal: Needed
Tag_ABI_FP_exceptions: Needed
Tag_ABI_FP_number_model: IEEE 754
Tag_ABI_align_needed: 8-byte
Tag_ABI_enum_size: int
Tag_ABI_VFP_args: VFP registers
Tag_CPU_unaligned_access: v6
Looking at the Tag_CPU_arch it seems that it's been compiled for ARMv7 actually.
Any help would be much appreciated. The repo in question is here and is built using github actions (workflow, makfile) on the latest ubuntu version.
I'm still new at this and had to do some "interesting," potentially incorrect things with the C/C++ compiler (arm-linux-gnueabihf-gcc)

Running 32-bit ARM binary on aarch64 not working despite CONFIG_COMPAT

I've got a 64-bit ARM machine that I'd like to run 32-bit ARM binaries on. As a test case I've built a small hello world for 32-bit ARM using the arm-linux-gnueabihf-gcc toolchain. file shows it as:
root#ubuntu:/home/ubuntu# file hello
hello: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=61ffe5e22117a6d4c2ae37a1f4c76617d3e5facc, not stripped
But trying to run it produces:
root#ubuntu:/home/ubuntu# ./hello
bash: ./hello: cannot execute binary file: Exec format error
Based on a prior question, I checked whether the kernel was built with the CONFIG_COMPAT option, and it was:
root#ubuntu:/home/ubuntu# grep CONFIG_COMPAT= /boot/config-$(uname -r)
CONFIG_COMPAT=y
I also verified that the armhf architecture has been added and that the armhf version of the loader is present. Note that the loader itself doesn't run either:
root#ubuntu:/home/ubuntu# dpkg --print-foreign-architectures
armhf
root#ubuntu:/home/ubuntu# dpkg -l libc6:armhf
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-=============-============-=================================
ii libc6:armhf 2.30-0ubuntu2 armhf GNU C Library: Shared libraries
root#ubuntu:/home/ubuntu# file /lib/arm-linux-gnueabihf/ld-2.30.so
/lib/arm-linux-gnueabihf/ld-2.30.so: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, BuildID[sha1]=dff2b536287d61ddca68f3e001e14b7c235bbf68, stripped
root#ubuntu:/home/ubuntu# /lib/arm-linux-gnueabihf/ld-2.30.so
bash: /lib/arm-linux-gnueabihf/ld-2.30.so: cannot execute binary file: Exec format error
Other relevant system info:
root#ubuntu:/home/ubuntu# cat /proc/cpuinfo
processor : 0
BogoMIPS : 400.00
Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics cpuid asimdrdm
CPU implementer : 0x43
CPU architecture: 8
CPU variant : 0x1
CPU part : 0x0af
CPU revision : 2
root#ubuntu:/home/ubuntu# uname -a
Linux ubuntu 5.3.0-24-generic #26-Ubuntu SMP Thu Nov 14 01:14:25 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux
I'm running out of things to try at this point. Any idea how to get the kernel to recognize these binaries and run them?
Ok, it looks like the underlying problem here is not in software, unfortunately. The CPU we're using, the Cavium ThunderX2, is one of the few 64-bit ARM chips that does not have aarch32 support. Quoting from WikiChip:
Only the 64-bit AArch64 execution state is support. No 32-bit AArch32 support.
So, this explains why it's not able to run 32-bit ARM binaries. I'm still fairly sure that other 64-bit ARM chips, like the Cortex-A57, are able to do this.
Update: older 32-bit ARM binaries do indeed work on aarch64 with a CPU that supports it, as shown below on an AWS ARM a1.metal instance:
ubuntu#ip-172-31-12-156:~$ cat /proc/cpuinfo | tail
processor : 15
BogoMIPS : 166.66
Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
CPU implementer : 0x41
CPU architecture: 8
CPU variant : 0x0
CPU part : 0xd08
CPU revision : 3
ubuntu#ip-172-31-12-156:~$ uname -a
Linux ip-172-31-12-156 4.15.0-1054-aws #56-Ubuntu SMP Thu Nov 7 16:18:50 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux
ubuntu#ip-172-31-12-156:~$ dpkg --print-foreign-architectures
armhf
ubuntu#ip-172-31-12-156:~$ file hello_hf
hello_hf: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-, for GNU/Linux 3.2.0, BuildID[sha1]=c95f0c46dfab925db53506751d7677156e334e5c, not stripped
ubuntu#ip-172-31-12-156:~$ ./hello_hf
hello, world!
This question has become more relevant as 32bit ARM chips are being phased out.
Me, I like to see old 32 bit binaries supported on new 64 bit hardware. Not so important, but convenient.
I took out my old Raspberry Pi Zero (32 bit) and Raspberry Pi 3B+ (64 bit) and for the first time installed a 64 bit OS on the 3B+ despite warnings of bugs and such and did some compiling.
No special make parameters. Just plain ordinary 32 bit compiling.
Runs fine on 32 bit and 64 bit.
Controllers and VMs will be running 32 bit architecture for many more years.
4GB memory limit is a pain yes. But for small apps its not a problem.

Validate Android NDK compiled library is compiled for correct architecture

I have a number of libraries built using either CMake or Automake and I'd like to make sure they are compiled correctly for the various architectures. Specifically I'd like to make sure the libraries I targeted for armeabi and armeabi-v7a are compiled with ARM v5 or ARM v7-a instructions.
I have found that readelf on linux provides a degree of certainty that the library was compiled with the desired options and you don't need to be on Linux, because the tool is in the NDK. The following examples were compiled with Clang using NDK r13b.
# -h option instead of -A is helpful if you're inspecting x86 binaries
arm-linux-androideabi-readelf -A library.so
readelf is in the NDK toolchain so even if you're on macOS you can still use it.
armeabi
Attribute Section: aeabi
File Attributes
Tag_CPU_name: "5TE"
Tag_CPU_arch: v5TE
Tag_ARM_ISA_use: Yes
Tag_THUMB_ISA_use: Thumb-1
Tag_FP_arch: VFPv2
Tag_ABI_PCS_wchar_t: 4
Tag_ABI_FP_denormal: Needed
Tag_ABI_FP_exceptions: Needed
Tag_ABI_FP_number_model: IEEE 754
Tag_ABI_align_needed: 8-byte
Tag_ABI_enum_size: int
Tag_ABI_optimization_goals: Aggressive Speed
armeabi-v7a
Attribute Section: aeabi
File Attributes
Tag_CPU_name: "ARM v7"
Tag_CPU_arch: v7
Tag_CPU_arch_profile: Application
Tag_ARM_ISA_use: Yes
Tag_THUMB_ISA_use: Thumb-2
Tag_FP_arch: VFPv3
Tag_Advanced_SIMD_arch: NEONv1
Tag_ABI_PCS_wchar_t: 4
Tag_ABI_FP_denormal: Needed
Tag_ABI_FP_exceptions: Needed
Tag_ABI_FP_number_model: IEEE 754
Tag_ABI_align_needed: 8-byte
Tag_ABI_enum_size: int
Tag_ABI_HardFP_use: Deprecated
Tag_ABI_optimization_goals: Aggressive Speed
Tag_CPU_unaligned_access: v6

Compile Automake (configure) based project using Android NDK

I want to compile a number of libraries for Android using the NDK toolchain. I'm having trouble compiling libiconv, libtiff, libxml2 for example and validate they are built correctly for each target architecture. How can I for example compile a trivial project like libiconv and validate that it was built correctly?
Example by cross-compiling libiconv
You can find these files on GitHub
Build a standalone toolchain
When compiling using Automake you'll need a standalone toolchain that embodies one target architecture and a specific platform revision. The procedure is outlined in the following script.
generate-standalone.sh
#!/bin/bash
ANDROID_NDK_DIR=/opt/android-ndk-r13b
ANDROID_API=19
ANDROID_STL=gnustl
INSTALL_PREFIX=/opt/standalone-r13b
declare -a COMPILE_ARCHITECTURES=("arm" "x86")
pushd ${ANDROID_NDK_DIR}
for ARCH in "${COMPILE_ARCHITECTURES[#]}"
do
INSTALL_DIR=${INSTALL_PREFIX}-${ARCH}
build/tools/make_standalone_toolchain.py \
--arch ${ARCH} \
--api ${ANDROID_API} \
--stl ${ANDROID_STL} \
--install-dir ${INSTALL_DIR}
done
popd
Replace config.sub and config.guess
On some older projects you'll find that the config.guess and config.sub files won't recognize the Android NDK toolchain so you'll need to grab a newer one from automake. In this project I copied the files from Automake 1.15 and just overwrote the ones in the libiconv project.
Compiling for 3 architectures
Flags chosen for the architectures come from Google's ABI Management and Standalone Toolchain pages.
Note: I haven't compiled 64-bit targets in this example. I would need to raise the Android API to 21 to add ARM64 and x86_64.
ANDROID_NDK_DIR=/opt/standalone-r13b
LIBICONV_INSTALL_DIR=${HOME}/Development/libiconv
declare -a COMPILE_ARCHITECTURES=("arm" "armv7a" "x86")
SAVED_PATH="${PATH}"
for ARCH in "${COMPILE_ARCHITECTURES[#]}"
do
COMPILER_GROUP=""
COMPILER_PREFIX=""
case ${ARCH} in
"arm" )
COMPILER_GROUP=arm
;;
"armv7a" )
COMPILER_GROUP=arm
;;
"x86" )
COMPILER_GROUP=x86
;;
esac
export ANDROID_NDK_ROOT="${ANDROID_NDK_DIR}-${COMPILER_GROUP}"
ANDROID_NDK_BIN="${ANDROID_NDK_ROOT}/bin"
ANDROID_SYSROOT_DIR="${ANDROID_NDK_ROOT}/sysroot"
export PATH="${ANDROID_NDK_BIN}:${SAVED_PATH}"
export CFLAGS="--sysroot=${ANDROID_SYSROOT_DIR}"
export LDFLAGS=""
case ${ARCH} in
"arm" )
ABI_NAME=armeabi
COMPILER_PREFIX=arm-linux-androideabi
;;
"armv7a" )
ABI_NAME=armeabi-v7a
COMPILER_PREFIX=arm-linux-androideabi
CFLAGS="${CFLAGS} -march=armv7-a -mfpu=neon -mfloat-abi=softfp -mthumb"
LDFLAGS="${LDFLAGS} -march=armv7-a -Wl,--fix-cortex-a8"
;;
"x86" )
ABI_NAME=x86
COMPILER_PREFIX=i686-linux-android
CFLAGS="${CFLAGS} -march=i686 -mtune=intel -mssse3 -mfpmath=sse -m32"
;;
esac
export CC=${ANDROID_NDK_BIN}/${COMPILER_PREFIX}-gcc
export CPP=${ANDROID_NDK_BIN}/${COMPILER_PREFIX}-cpp
export CXX=${ANDROID_NDK_BIN}/${COMPILER_PREFIX}-g++
export LD=${ANDROID_NDK_BIN}/${COMPILER_PREFIX}-ld
export AR=${ANDROID_NDK_BIN}/${COMPILER_PREFIX}-ar
export RANLIB=${ANDROID_NDK_BIN}/${COMPILER_PREFIX}-ranlib
export STRIP=${ANDROID_NDK_BIN}/${COMPILER_PREFIX}-strip
echo "---- Compiling for ${ARCH}"
./configure --enable-static --host="${COMPILER_PREFIX}" --prefix="${LIBICONV_INSTALL_DIR}/${ABI_NAME}"
make clean
make -j4
make install
done
export PATH="${SAVED_PATH}"
Validating the libraries target the desired architecture
From the NDK you can find arm-linux-androideabi-readelf to determine if for example I'm using Thumb-1 or Thumb-2 instructions in armeabi and armeabi-v7a respectively.
arm-linux-androideabi-readelf -A libiconv.so.2.5.1.so
armeabi
Attribute Section: aeabi
File Attributes
Tag_CPU_name: "5TE"
Tag_CPU_arch: v5TE
Tag_ARM_ISA_use: Yes
Tag_THUMB_ISA_use: Thumb-1
Tag_FP_arch: VFPv2
Tag_ABI_PCS_wchar_t: 4
Tag_ABI_FP_denormal: Needed
Tag_ABI_FP_exceptions: Needed
Tag_ABI_FP_number_model: IEEE 754
Tag_ABI_align_needed: 8-byte
Tag_ABI_enum_size: int
Tag_ABI_optimization_goals: Aggressive Speed
armeabi-v7a
If your project has support for advanced features like NEON you'll definitely want to validate that you've compiled the library with the correct options.
Attribute Section: aeabi
File Attributes
Tag_CPU_name: "ARM v7"
Tag_CPU_arch: v7
Tag_CPU_arch_profile: Application
Tag_ARM_ISA_use: Yes
Tag_THUMB_ISA_use: Thumb-2
Tag_FP_arch: VFPv3
Tag_Advanced_SIMD_arch: NEONv1
Tag_ABI_PCS_wchar_t: 4
Tag_ABI_FP_denormal: Needed
Tag_ABI_FP_exceptions: Needed
Tag_ABI_FP_number_model: IEEE 754
Tag_ABI_align_needed: 8-byte
Tag_ABI_enum_size: int
Tag_ABI_HardFP_use: Deprecated
Tag_ABI_optimization_goals: Aggressive Speed
Tag_CPU_unaligned_access: v6

Cross compile from linux to ARM-ELF (ARM926EJ-S/MT7108)

I have installed all cross compile packages on my ubuntu system so far but am having a
problem and need some help.
Processor : ARM926EJ-S rev 5 (v5l)
BogoMIPS : 184.72
Features : swp half thumb fastmult edsp java
CPU implementer : 0x41
CPU architecture: 5TEJ
CPU variant : 0x0
CPU part : 0x926
CPU revision : 5
Cache type : write-back
Cache clean : cp15 c7 ops
Cache lockdown : format C
Cache format : Harvard
I size : 32768
I assoc : 4
I line length : 32
I sets : 256
D size : 32768
D assoc : 4
D line length : 32
D sets : 256
Hardware : MT7108
Revision : 0000
Serial : 0000000000000000
This is the target machine I need to cross compile for. What flags should I
use when compiling?
You have an ARMv5 with no floating-point processor. It should have been enough with -march=armv5 and -mfloat-abi=soft flags.
However if those flags doesn't work for you, I would suggest writing the smallest c application for testing the toolchain.
/* no includes */
int main(void) {
return 42;
}
and compiling it with most complete/strict flags
$arm-linux-gnueabi-gcc -Wall --static -O2 -marm -march=armv5 simple.c -o simple
after this, push simple to target, run it then issue an echo $? to verify if you would get 42. If it works, try to see if you can get printf working. If that one also works, you are pretty much set for everything. If printf fails, easiest solution would be to find right toolchain for your target.
apt-cache search arm | grep ^gcc- gives the following list,
gcc-4.7-aarch64-linux-gnu - GNU C compiler
gcc-4.7-arm-linux-gnueabi - GNU C compiler
gcc-4.7-arm-linux-gnueabi-base - GCC, the GNU Compiler Collection (base package)
gcc-4.7-arm-linux-gnueabihf - GNU C compiler
gcc-4.7-arm-linux-gnueabihf-base - GCC, the GNU Compiler Collection (base package)
gcc-4.7-multilib-arm-linux-gnueabi - GNU C compiler (multilib files)
gcc-4.7-multilib-arm-linux-gnueabihf - GNU C compiler (multilib files)
gcc-aarch64-linux-gnu - The GNU C compiler for arm64 architecture
gcc-arm-linux-gnueabi - The GNU C compiler for armel architecture
gcc-arm-linux-gnueabihf - The GNU C compiler for armhf architecture
You should install gcc-arm-linux-gnueabi which is an alias for gcc-4.7-arm-linux-gnueabi. gcc-4.7-multilib-arm-linux-gnueabi is also possible, but more complicated. Use the flags, -march=armv5te -mtune=arm926ej-s -msoft-float -mfloat-abi=soft. You can do more tuning by specifying the --param NAME=VALUE option to gcc with parameters tuned to your systems memory sub-system timing.
You may not be able to use these gcc versions as your Linux maybe compiled with OABI and/or be quite ancient compared to the one the compiler was built for. In some cases, the libc will call a newer Linux API, which may not be present. If the compiler/libc was not configured to be backwards compatible, then it may not work with your system. You can use crosstool-ng to create a custom compiler that is built to suit your system, but this is much more complex.

Resources