I want to build a custom image for my arm board so i would like to know what are the information's in machine tune files aswellas the arch-arm files
The Custom image for your ARM board is a collection of packages (softwares) cross-compiled using Yocto.
Yocto uses gcc to cross-compile these packages. And gcc has options to optimize (tune) the code for a particular Architecture (armv8) and CPU (cortex-a53).
GCC guys have explained the Arch and CPU tuning options here -
https://gcc.gnu.org/onlinedocs/gcc/AArch64-Options.html#index-march
Now, Yocto needs to know which of these tuning options need to be passed to gcc while compiling a package for your ARM board. This is done using Machine tune files in Yocto.
Go through the machine tune files to understand more - https://github.com/openembedded/openembedded-core/blob/master/meta/conf/machine/include/arm/arch-armv8a.inc
The default CPU and Application Binary Interface (ABI) tunings are set in BSP layer (machine.conf) of your board.
For example for Raspberry Pi 4 64 it is set here - https://github.com/agherzan/meta-raspberrypi/blob/ffc402ef5e9660de4db27379215fc88681d06dc7/conf/machine/raspberrypi4-64.conf#L13
I am very newbie in this kind of business. I have just cross compiled Linux kernel. But I have few question to ask which I have to know.
When we compile a Linux kernel I am using this piece of command, because my target platform is ARM.
make ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabi-
Could I cross compile any Open Source software like that or Is it depends on the software release that the software supports cross compilation or not?
The Linux kernel source contains a arch folder for separate architectures but gcc, gLibc, binutils doesn't have, why?
But those can be cross-compiled. Can any one tell me why this kind of behavior happens?
Is there any standard way to cross-compile different kind of software as per requirement?? Please lead me if any one proficient in this kind of business.
Thank you.
There is a general way for cross compilation of software in linux if that is having configure
script.
Extract the source code of the package that you want to install .
See whether that has any configure script in it.
If that is , then run
./configure --help
to find the options supported for compilation .
I usually use the following command to cross compile.
./configure --host=arm-none-linux-eabi --prefix=/path/to/where/you/want/to/install
Based on the package may be required to give additional options.
Examples like --with-out= libtiff etc.
If that is not having any configure script then tweak into the make file.
The linux kernel has its own, very particular, build-system that is set up - not only for cross-compilation - but multiple architecture cross-compilation. This is why a series of arch folders exists.
A large amount (but by no means all) of open-source user-space software uses GNU autoconf to manage the configuration and build process. The purpose of autoconf is somewhat different from the kernel build script - it allows software to be built on a wide variety of subtly different UNIX-like build hosts for a equally wide variety of build targets.
autoconf can be used used for cross-compilation with a bit of work. There are some hints here. In principle, the build process needs to know:
Which set of tools to use (e.g. gcc, binutils)
Where the target's headers and libraries are staged
Where to install the resulting product.
gcc and binutils are slightly special case in that cross-tools are installed on a development host alongside the host's own tools. Since build processes might well use both, it's untenable that selection of tools is done entirely by the executable search path. Instead, cross-tools are named with a target-specific name format - e.g.
arm-linux-gnueabi-gcc
and
i686-apple-darwin11-llvm-gcc
I have a Gdium Liberty 1000 laptop (Loongson 2F processor) on it's way to me. I will use it mainly to explore the MIPS assembly language. The inner architecture of the Loongson is supposed to be different from the MIPS CPUs. Does it in fact run MIPS assembly (MIPS64?) with no modification or are there any particular quirks to the Loongson 2F architecture? I realize that I can run SPIM (or MARS) as MIPS simulators on my Intel/AMD x86 processors, it just seemed like fun to try to run it natively.
Does it run MIPS assembly (MIPS64?) with no modification?
What tools are available on the Gdium (it runs a version of Mandriva Linux) for learning MIPS assembly?
Loongson 2F is MIPS III-compatible, so this means that you will be able to write MIPS III compatible assembly, compile it and run it. If available, native toolchain should be a better choice than any cross toolchain.
I would suggest reading a UserGuide which will tell you what instructions you have available. Note that the instruction set for this chip has been extended with number of different integer/fp/media instructions.
From wikipedia on Loongson:
The current Loongson instruction set is a MIPS64,
but the internal microarchitecture is independently developed by ICT.
Early implementations of the family lacked four instructions
patented by MIPS Technologies to avoid legal issues.
Yes, it is certainly fun to run MIPS assembly on the real hardware (actually, for similar reasons I've bought myself an old PowerPC Mac as this was a great option to get a decent PowerPC processor).
Regarding the problem with development, I would recommend:
Setup the remote access (run 'sshd' there) to this technical wonder
Download some precompiled gcc toolchain for MIPS
Compile binaries locally, deploy them using 'ssh' to the netbook.
Setup 'gdbserver' there (on the netbook) if you want remote debugging or stick to extensive logging
I'm building an embedded system using buildroot and i want to replace nginx+php with node.js.
My system is using eglibc but i want to reduce the size of the system so i want to switch to uclibc. node.js can be compiled using uclibc-0.9.32-nptl?
More, while my current test hardware is x86 based, alix, i'll switch, in a couple of months, to a plug computer, that is arm based.
Will node.js works on arm based hardware?
Just for your interest: I compile Node.js 0.4.7 on sheevaplug just taking into account
http://code.google.com/p/v8/issues/detail?id=836
https://github.com/joyent/node/issues/883
Enjoy!
As it is built in an OpenEmbedded recipe, you should be able to build it in Buildroot (with some adaptation to transform bitbake script to makefile one), yet nothing limit node.js to be compiled on an ARM platform or with a uclibc (at least till version 0.4.2).
How do the Linux kernel developers test their code locally and after they have it committed? Do they use some kind of unit testing and build automation? Test plans?
The Linux kernel has a heavy emphasis on community testing.
Typically, any developer will test their own code before submitting, and quite often they will be using a development version of the kernel from Linus, or one of the other unstable/development trees for a project relevant to their work. This means they are often testing both their changes and other people's changes.
There tends not to be much in the way of formal test plans, but extra testing may be asked for before features are merged into upstream trees.
As Dean pointed out, there's also some automated testing: The Linux Test Project and the kernel Autotest (good overview).
Developers will often also write automated tests targeted to test their change, but I'm not sure there's a (often used) mechanism to centrally collect these ad hoc tests.
It depends a lot on which area of the kernel is being changed of course - the testing you'd do for a new network driver is quite different to the testing you'd do when replacing the core scheduling algorithm.
Naturally, the kernel itself and its parts are tested prior to the release, but these tests cover only the basic functionality. There are some testing systems which perform testing of Linux Kernel:
The Linux Test Project (LTP) delivers test suites to the open source community that validate the reliability and stability of Linux. The LTP test suite contains a collection of tools for testing the Linux kernel and related features.
Autotest—a framework for fully automated testing. It is designed primarily to test the Linux kernel, though it is useful for many other purposes such as qualifying new hardware, virtualization testing, and other general user space program testing under Linux platforms. It's an open-source project under the GPL and is used and developed by a number of organizations, including Google, IBM, Red Hat, and many others.
Also there are certification systems developed by some major GNU/Linux distribution companies. These systems usually check complete GNU/Linux distributions for compatibility with hardware. There are certification systems developed by Novell, Red Hat, Oracle, Canonical, and Google.
There are also systems for dynamic analysis of the Linux kernel:
Kmemleak is a memory leak detector included in the Linux kernel. It provides a way of detecting possible kernel memory leaks in a way similar to a tracing garbage collector with the difference that the orphan objects are not freed, but only reported via /sys/kernel/debug/kmemleak.
Kmemcheck traps every read and write to memory that was allocated dynamically (i.e., with kmalloc()). If a memory address is read that has not previously been written to, a message is printed to the kernel log. It is also is a part of the Linux kernel.
Fault Injection Framework (included in the Linux kernel) allows for infusing errors and exceptions into an application's logic to achieve a higher coverage and fault tolerance of the system.
How do the Linux kernel developers test their code locally and after they have it committed?
Do they use some kind of unit testing and build automation?
In the classic sense of words, no.
For example, Ingo Molnar is running the following workload:
build a new kernel with a random set of configuration options
boot into it
go to 1
Every build fail, boot fail, bug or runtime warning is dealt with. 24/7. Multiply by several boxes, and one can uncover quite a lot of problems.
Test plans?
No.
There may be a misunderstanding that there is a central testing facility, but there is none. Everyone does what he/she wants.
In-tree tools
A good way to find test tools in the kernel is to:
make help and read all targets
look under tools/testing
In v4.0, this leads me to:
kselftest under tools/testing/selftests. Run with make kselftest. Must be running built kernel already. See also: Documentation/kselftest.txt , https://kselftest.wiki.kernel.org/
ktest under tools/testing/ktest. See also: http://elinux.org/Ktest , http://www.slideshare.net/satorutakeuchi18/kernel-auto-testbyktest
Static analysers section of make help, which contains targets like:
checkstack: Perl: what does checkstack.pl in linux source do?
coccicheck for Coccinelle (mentioned by askb)
Kernel CI
https://kernelci.org/ is a project that aims to make kernel testing more automated and visible.
It appears to do only build and boot tests (TODO how to test automatically that boot worked Source should be at https://github.com/kernelci/).
Linaro seems to be the main maintainer of the project, with contributions from many big companies: https://kernelci.org/sponsors/
Linaro Lava
http://www.linaro.org/initiatives/lava/ looks like a CI system with focus on development board bringup and the Linux kernel.
ARM LISA
https://github.com/ARM-software/lisa
Not sure what it does in detail, but it is by ARM and Apache Licensed, so likely worth a look.
Demo: https://www.youtube.com/watch?v=yXZzzUEngiU
Step debuggers
Not really unit testing, but may help once your tests start failing:
QEMU + GDB: https://stackoverflow.com/a/42316607/895245
KGDB: https://stackoverflow.com/a/44226360/895245
My own QEMU + Buildroot + Python setup
I also started a setup focused on ease of development, but I ended up adding some simple testing capabilities to it as well: https://github.com/cirosantilli/linux-kernel-module-cheat/tree/8217e5508782827320209644dcbaf9a6b3141724#test-this-repo
I haven't analyzed all the other setups in great detail, and they likely do much more than mine, however I believe that my setup is very easy to get started with quickly because it has a lot of documentation and automation.
It’s not very easy to automate kernel testing. Most Linux developers do the testing on their own, much like adobriyan mentioned.
However, there are a few things that help with debugging the Linux Kernel:
kexec: A system call that allows you to put another kernel into memory and reboot without going back to the BIOS, and if it fails, reboot back.
dmesg: Definitely the place to look for information about what happened during the kernel boot and whether it works/doesn't work.
Kernel Instrumentation: In addition to printk's (and an option called 'CONFIG_PRINTK_TIME' which allows you to see (to microsecond accuracy) when the kernel output what), the kernel configuration allows you to turn on a lot of tracers that enable them to debug what is happening.
Then, developers usually have others review their patches. Once the patches are reviewed locally and seen not to interfere with anything else, and the patches are tested to work with the latest kernel from Linus without breaking anything, the patches are pushed upstream.
Here's a nice video detailing the process a patch goes through before it is integrated into the kernel.
In addition to the other answers, this emphasise more on the functionality testing, hardware certification testing and performance testing the Linux kernel.
A lot of testing actually happen through scripts, static code analysis tools, code reviews, etc. which is very efficient in catching bugs, which would otherwise break something in the application.
Sparse – An open-source tool designed to find faults in the Linux kernel.
Coccinelle is another program does matching and transformation engine which provides the language SmPL (Semantic Patch Language) for specifying desired matches and transformations in C code.
checkpatch.pl and other scripts - coding style issues can be found in the file Documentation/CodingStyle in the kernel source tree. The important thing to remember when reading it is not that this style is somehow better than any other style, just that it is consistent. This helps developers easily find and fix coding style issues. The script scripts/checkpatch.pl in the kernel source tree has been developed for it. This script can point out problems easily, and should always be run by a developer on their changes, instead of having a reviewer waste their time by pointing out problems later on.
There are also:
MMTests which is collection of benchmarks and scripts to analyze the results.
Trinity which is Linux system call fuzz tester.
Also the LTP pages at SourceForge are quite outdated and the project has moved to GitHub.
I would imagine they use virtualization to do quick tests. It could be something like QEMU, VirtualBox or Xen, and some scripts to perform configurations and automated tests.
Automated testing is probably done by trying either many random configurations or a few specific ones (if they are working with a specific issue). Linux has a lot of low-level tools (such as dmesg) to monitor and log debug data from the kernel, so I imagine that is used as well.
As far as I know, there is an automatically performance regression check tool (named lkp/0 day) running/funding by the Intel. It will test each valid patch sent to the mailing list and check the scores changed from different microbenchmarks such as hackbench, fio, unixbench, netperf, etc.
Once there is a performance regression/improvement, a corresponding report will be sent directly to the patch author and a Cc related maintainers.
LTP and Memtests are generally preferred tools.
adobriyan mentioned Ingo's loop of random configuration build testing. That is pretty much now covered by the 0-day test bot (aka kbuild test bot). A nice article about the infrastructure is presented here: Kernel Build/boot testing
The idea behind this set-up is to notify the developers ASAP so that they can rectify the errors soon enough (before the patches make it into Linus' tree in some cases as the kbuild infrastructure also tests against maintainer's subsystem trees).
Once after contributors submit their patch files and after making a merge request, Linux gatekeepers are checking the patch by integrating and reviewing it. Once it succeeds, they will merge the patch into the relevant branch and a make new version release.
The Linux Test Project is the main source which provides test scenarios (test cases) to run against the kernel after applying patches. This may take around 2 ~ 4 hours, and it depends.
Please note regarding the file system of the selected kernel is going to test against.
Example: ext4 generates different results against ext3 and so on.
Kernel Testing procedure.
Get latest kernel source from the repository (The Linux Kernel Archives or GitHub)
Apply the patch file (using a diff tool)
Build the new kernel.
Test against test procedures in LTP (Linux Test Project)
I had done Linux kernel compilation and done some modifications for Android (Android 6.0 (Marshmallow) and Android 7.0 (Nougat)) in which I use Linux version 3. I cross-compiled it on a Linux system, debugged the errors manually and then ran its boot image file in Android and checked if it was going in a loop-hole or not. If it runs perfect then it means it is compiled perfectly according to system requirements.
For MotoG kernel Compilation
Note: The Linux kernel will change according to requirements which depend on system hardware