I have setup a chroot for an aarch64 rootfs. I am using qemu-aarch64-static as an emulator. This works. I can login to the chroot and execute aarch64 binaries.
Now I would like to run a native (x86_64) cross compiler from within this environment. (I have a large application which does not build using a cross compiler. Using a qemu emulated gcc is too slow). I cannot find a way to run x86 executables from the chroot.
First I mount the native filesystem into the chroot
mount -o bind / /mnt/rpi_rootfs/mnt/native
prepare chroot
cd /mnt/rpi_rootfs
sudo mount -t proc /proc proc/
sudo mount --rbind /sys sys/
sudo mount --rbind /dev dev/
login to the chroot
sudo chroot /mnt/rpi_rootfs/
Create a link to the x86 dynamic linker/loader
ln -s /mnt/native/lib/ld-linux.so.2 /lib/ld-linux.so.2
Try to run any x86 native binary.
LD_LIBRARY_PATH=/mnt/native/lib:/mnt/native/usr/lib /mnt/native/bin/pwd
Error:
>/mnt/native/bin/pwd: No such file or directory
I was inspired by this approach: https://gitlab.com/postmarketOS/pmbootstrap/-/issues/1731
Notes:
On the native system: ls /proc/sys/fs/binfmt_misc/ shows the various registered emulators, such as qemu-aarch64.
In the chroot ls /proc/sys/fs/binfmt_misc/ is empty.
I use the 'pwd' app as an example.
Execute
file /bin/pwd
/bin/pwd: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2
This shows that actually /lib64/ld-linux-x86-64.so.2 is required to run the application. Thus step 4 above needs to be changed.
Note: /lib64/ld-linux-x86-64.so.2 is a symlink.
Activate the chroot and next, inside the chroot environment create a symlink from the expected location of the dynamic linker to the actual file on the host:
>ln -s /mnt/native/lib/x86_64-linux-gnu/ld-2.31.so /lib64/ld-linux-x86-64.so.2
When this is done it is finally possible to run native x86 applications in the aarch64 chroot. This allows one to run high performance cross compilers from within the chroot.
>LD_LIBRARY_PATH=/mnt/native/lib:/mnt/native/usr/lib:/mnt/native/lib/x86_64-linux-gnu /mnt/native/bin/pwd
>/
Related
I have a container built from base image alpine:3.11
Now I have a binary my_bin that I copied into the running container. From within the running container I moved to /usr/local/bin and I confirmed that the binary is there with the right permissions. Eg
/ # ls -l /usr/local/bin/my_bin
-rwxr-xr-x 1 root root 55662376 Jun 12 18:52 /usr/local/bin/my_bin
But when I attempt to execute/run this binary I get the following:
/ # my_bin init
/bin/sh: my_bin: not found
This is also the case if I switch into /usr/local/bin/ and run via ./my_bin
also if I try using the full path
/# /usr/local/bin/my_bin init
/bin/sh: /usr/local/bin/my_bin: not found
Why am I seeing this behavior? and how do I get to be able to execute the binary?
EDIT 1
I installed file and I can also confirm that the binary is copied and is an executable
file /usr/local/bin/my_bin
/usr/local/bin/my_bin: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=b36f0aad307c3229850d8db8c52e00033eae900c, for GNU/Linux 3.2.0, not stripped
Maybe this gives some extra clues?
Edit 2
As suggested by #BMitch in the answer I also ran ldd and here is the output
# ldd /usr/local/bin/my_bin
/lib64/ld-linux-x86-64.so.2 (0x7f91a79f3000)
libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x7f91a79f3000)
libdl.so.2 => /lib64/ld-linux-x86-64.so.2 (0x7f91a79f3000)
libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f91a79f3000)
** Edit 3 **
Based on the output of ldd and more googling, I find that running apk add libc6-compat installed the missing libraries and I could then run the binary.
For a binary, this most likely indicates a missing dynamic library. You can run ldd /usr/local/bin/my_bin to see all the libraries that binary uses. With alpine, the most common library to be missing from an externally compiled program is libc. Alpine is built with musl instead of libc, and therefore you'll want to compile programs specifically for Alpine.
For others that may encounter this error in docker containers, I cover various issues in my faq presentation and other questions on the site.
/ # my_bin init
/bin/sh: my_bin: not found
When you execute above line it says file which you are trying to execute can't be found, my_bin is the file in your case.
Check if file is copied properly and with the same name or you might be trying to execute file from different location.
e.g. Try /usr/local/bin/my_bin init if you are not doing cd /usr/local/bin after ls -l /usr/local/bin/my_bin command.
I am trying to build a basic root filesystem using Buildroot, for an embedded system (the Banana PI D1).
I am using a kernel from an SDK supplied by the SoC vendor. From this repo I am using only the kernel, found in src/kernel.
There's nothing remarkable about the Buildroot configuration. It builds with no errors and the resulting root filesystem looks like it contains everything I would expect.
I have configured it to build the filesystem as an initramfs embedded within the zImage.
The kernel appears to start up correctly, but cannot load init and then panics:
Booting Linux on physical CPU 0
Linux version 3.4.35 (harmic#penski.harmic.moo.org) (gcc version 4.8.4 (Buildroot 2015.02) ) #7 Sat Mar 21 22:59:18 AEDT 2015
CPU: ARM926EJ-S [41069265] revision 5 (ARMv5TEJ), cr=00053177
...
Kernel command line: root=/dev/mtdblock1 ro init=/sbin/init mem=64M console=ttySAK0,115200
...
Freeing init memory: 4632K
Failed to execute /init
Failed to execute /sbin/init. Attempting defaults...
mmc0: host does not support reading read-only switch. assuming write-enable.
mmc0: new SDHC card at address 0007
mmcblk0: mmc0:0007 SD08G 7.42 GiB
mmcblk0: p1
Kernel panic - not syncing: No init found. Try passing init= option to kernel. See Linux Documentation/init.txt for guidance.
I have tried a number of troubleshooting steps:
I've built a root filesystem using this miniroot project (took some doing, as it is quite out of date). It booted OK, using the same kernel as I am trying to use with the buildroot root fs.
I've tried using both uClibc and eglibc
I've tried using Buildroot's own cross-tools as well as the cross tools supplied by the SoC vendor
I've confirmed that the built rootfs does include an /init (it does!)
There is a gist here containing the buildroot configuration, a copy of the kernel boot log, and a listing of the contents of the generated filesystem.
What steps can I take to troubleshoot this further?
Update:
The generated rootfs.cpio.gz weighs in at 2139200 bytes. I have read that there is a maximum size of initramfs you can use, but I have not been able to find where the hard limit is documented.
I have attached a listing of the generated root filesystem to the gist linked above.
I have unpacked the rootfs on the host and inspected it. /init contains this:
#!/bin/sh
# devtmpfs does not get automounted for initramfs
/bin/mount -t devtmpfs devtmpfs /dev
exec 0</dev/console
exec 1>/dev/console
exec 2>/dev/console
exec /sbin/init $*
/sbin/init is a symlink to /bin/busybox.
/bin/busybox is dynamically linked:
$ file busybox
busybox: setuid ELF 32-bit MSB executable, ARM, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.16, stripped
$ ../../../host/usr/bin/armeb-buildroot-linux-gnueabi-readelf -a busybox | grep "Shared library:"
0x00000001 (NEEDED) Shared library: [libc.so.6]
libc.so.6 is present in /lib. /lib32 is a symlink to /lib for good measure.
The device has 64M RAM.
Both the vendor's cross tools and the buildroot cross tools are set up for eabi
As suggested by #sawdust, the problem was that the CPU was not supposed to be run in big endian mode.
After changing the target to 'ARM (Little Endian)', cleaning and re-building, it now boots correctly.
In retrospect this should have been obvious - inspecting the vendor's kernel image:
$ file zImage
zImage: Linux kernel ARM boot executable zImage (little-endian)
My 32-bit application is running under Wine, and to help it better integrate with the environment, it runs some shell scripts. I was just testing under Ubuntu 14.04 64-bit, and my program crashed with this error:
err:process:create_process starting 64-bit process L"Z:\\bin\\sh" not supported in 32-bit wineprefix
I've tried searching for a 32-bit build of "sh" on my system, but couldn't find one. Any creative ideas on how I can get past this issue?
I am a user of the program in question and I did some experimenting with it.
Not 32-bit vs. 64-bit but "shared object" vs. "executable" ?
Running file /bin/dash it prints:
/bin/dash: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), ...
Running file /bin/bash however prints:
/bin/bash: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), ...
dash is a "shared object" while bash is an "executable". Clearly /bin/dash seems to work like an executable in some way (I don't know the technical details here), but it seems that this difference matters to Wine.
I got the same error as you reported (can't start 64 bit process) for Wine 1.4, but the error I got on newer versions of Wine was wine: Bad EXE format for Z:\bin\sh..
If you actually just replace /bin/sh with /bin/bash (even though that's a 64-bit binary) it will work. Wine also didn't seem to like running a symlink, but copying over /bin/bash worked.
So first back up the existing (symlinked) /bin/sh with:
sudo cp /bin/sh /bin/sh_orig
Then copy bash to sh:
sudo cp /bin/bash /bin/sh
Then when I ran Wine with the program and its calls to /bin/sh work fine.
Alternatively, you download a 32-bit shell directly
Pull down the .deb file for the 32-bit bash shell:
wget http://us.archive.ubuntu.com/ubuntu/pool/main/b/bash/bash_4.3-6ubuntu1_i386.deb
I your home directory, extract it into a folder:
mkdir ~/bash_4.3-6ubuntu1_i386
dpkg -x bash_4.3-6ubuntu1_i386.deb ~/bash_4.3-6ubuntu1_i386
Copy the bash script into /bin/sh:
sudo mv /bin/sh /bin/sh64original
sudo cp ~/bash_4.3-6ubuntu1_i386/bin/bash /bin/sh
sudo chown root:root /bin/sh
Or run schroot, but still must copy /bin/bash to /bin/sh
Basile Starynkevitch mentioned above about setting up a 32-bit shell in an schroot environment. I did that with an Ubuntu 14.04 32-bit environment and ran into the same issue with the dash vs. bash "shared object" vs. "executable" (but when I copied /bin/bash to /bin/sh it worked), so that helped me realize that the distinction wasn't the 32-bit vs. 64-bit difference but the format of the shell executables that mattered to Wine.
If you would like I can post details for setting up the schroot evnironment but basically I followed the instructions at https://help.ubuntu.com/community/DebootstrapChroot but needed to configure the /etc/apt/sources.list to have the full list of packages (as are installed in my default host system) for apt-get install wine to work.
Following the Linux from Scratch book I have managed to build a toolchain for an ARM on
an ARM. This is till chapter 6 of the book, and on the ARM board itself I could go on further with no problems.
My question is if I can use the prepared environment to continue building the soft from chapter 6 on my x86_64 Fedora 16 laptop?
I thought that while I have all the binaries set up I could just copy them to laptop, chroot inside and feel myself as on the ARM board, but using the command from the book gives no result:
`# chroot "$LFS" /tools/bin/env -i HOME=/root TERM="$TERM" PS1='\u:\w\$
PATH=/bin:/usr/bin:/sbin:/usr/sbin:/tools/bin /tools/bin/bash --login +h
chroot: failed to run command `/tools/bin/env': No such file or directory`
The binary is there, but it doesn't belong to this system:
`# ldd /tools/bin/env
not a dynamic executable`
The binary is compiled as per the book:
# readelf -l /tools/bin/env | grep interpreter
[Requesting program interpreter: /tools/lib/ld-linux.so.3]
So I wonder if there is a way, like using proper environment variables for CC LD READELF, to continue building for ARM using these tools on x86_64 host.
Thank you.
Yes, you certainly can chroot into an ARM rootfs on an x86 box.
Basically, like this:
$ sudo chroot /path/to/arm/rootfs /bin/sh
sh-4.3# ls --version 2>&1 | head
/bin/ls: unrecognized option '--version'
BusyBox v1.22.1 (2017-03-02 15:41:43 CST) multi-call binary.
Usage: ls [-1AaCxdLHRFplinsehrSXvctu] [-w WIDTH] [FILE]...
List directory contents
-1 One column output
-a Include entries which start with .
-A Like -a, but exclude . and ..
sh-4.3# ls
bin css dev home media proc sbin usr wav
boot data etc lib mnt qemu-arm sys var
My rootfs is for a small embedded device, so everything is BusyBox-based.
How is this working? Firstly, I have the binfmt-misc support running in the kernel. I didn't have to do anything; it came with Ubuntu 18. When the kernel sees an ARM binary, it hands it off to the registered interpreter /usr/bin/qemu-arm-static.
A static executable by that name is found inside my rootfs:
sh-4.3# ls /usr/bin/q*
/usr/bin/qemu-arm-static
I got it from a Ubuntu package. I installed:
$ apt-get install qemu-user-static
and then copied /usr/bin/qemu-arm-static into the usr/bin subdirectory of the rootfs tree.
That's it; now I can chroot into that rootfs without even mentioning QEMU on the chroot command line.
Nope. You can't run ARM binaries on x86, so you can't enter its chroot. No amount of environment variables will change that.
You might be able to continue the process by creating a filesystem image for the target and running it under an emulator (e.g, qemu-system-arm), but that's quite a different thing.
No you cannot, at least not using chroot. What you have in your hands is a toolchain with an ARM target for an ARM host. Binaries are directly executable only on architectures compatible with their host architecture - and x86_64 is not ARM-compatible.
That said, you might be able to use an emulated environment. qemu, for example, offers two emulation modes for ARM: qemu-system-arm that emulates a whole ARM-based system and qemu-arm that uses ARM-native libraries to provide a thinner emulation layer for running ARM Linux executables on non-ARM hosts.
I 've cross compiled a Linux Kernel (for ARM on i686 - using Cross-LFS).
Now I'm trying to boot this Kernel using QEMU.
$ qemu-system-arm -m 128 -kernel /mnt/clfs-dec4/boot/clfskernel-2.6.38.2 --nographic -M versatilepb
Then, it shows this line and waits for infinite time !!
Uncompressing Linux... done, booting the kernel.
So, I want to debug the kernel, so that I can study what exactly is happening.
I'm new to these kernel builds, Can someone please help me to debug my custom built kernel as it is not even showing anything after that statement. Is there any possibility of the kernel being broken? ( I dont think so, b'se it didnot give any error while compiling )
And my aim is to generate a custom build very minimal Linux OS. Any suggestions regarding any tool-chains etc which would be easy & flexible depending on my requirements like drivers etc.,
ThankYou
You can use GDB to debug your kernel with QEMU you can use -s -S options. If you want a simple and reliable toolchain, you can use ELDK from DENX (http://www.denx.de/wiki/DULG/ELDK).
You can install it like this (It's not the last version, but you got the idea):
wget http://ftp.denx.de/pub/eldk/4.2/arm-linux-x86/iso/arm-2008-11-24.iso
sudo mkdir -p /mnt/cdrom (if necessary)
sudo mount -o loop arm-2008-11-24.iso /mnt/cdrom
/mnt/cdrom/install -d $HOME/EMBEDDED_TOOLS/ELDK/
The command above should install the toolchain under $HOLE/EMBEDDED_TOOLS/ELDK (modify it if you need)
echo "export PATH=$PATH:$HOME/EMBEDDED_TOOLS/ELDK/ELDK42/usr/bin" >> $HOME/.bashrc
You can then see the version of your ARM toolchain like this:
arm-linux-gcc -v
You can test a hello_world.c program like this:
arm-linux-gcc hello_world.c -o hello_world
And you type: file hello_wrold to see the target architecture of the binary, it should be something like this:
hello_wrold: ELF 32-bit LSB executable, ARM, version 1 (SYSV)
Now if you want to compile a production kernel, you need to optimize it (i suggest using busybox) and if you want just one for testing now, try this steps:
Create a script to set your chain tool set_toolchain.sh:
#! /usr/bin/sh
PATH=$PATH:$HOME/EMBEDDED_TOOLS/ELDK/ELDK42/usr/bin
ARCH=arm
CROSS_COMPILE=arm-linux-gnueabi-
export PATH ARCH CROSS_COMPILE
And run your script (source ./set_toolchain.sh)
Download a linux kernel and unzip it (Let's assume 2.6.x, it's an old kernel, but there are a lot of chances that it work without compilation errors).
Inside your unzipped kernel:
cd ~/linux-2.6.29/arch/arm/configs
make versatile_defconfig
Here we use versatile chip, you may need to use make menuconfig to modify the option OABI and set it to ARM EABI, this option is under Kernel features menu
After all this steps, you can compile you kernel:
make
if you want verbose compilation make v=1
After this you got your kernel under arch/arm/boot/zImage.
Hope this help.
Regards.
I would suggest to build your kernel by activating the option in the section Kernel hacking of your configuration file.
Then you may use kdb or kgdb which is easier to use but requires another machine running gdb.
`
You can also connect Qemu and GDB. Qemu has the -s and -S options that run a GDB server and allow you to connect to it via TCP to localhost:1234. Then you can load your kernel image (the unzipped one) in GDB and see how far your kernel boots.