I wanted to validate/test/benchmark set of features I have added to the ext4 kernel_tree/fs.
I came across Spruce Linux file system driver verification. Especially for filesystem.
The project is hosted #https://code.google.com/p/spruce/wiki/GettingStarted.
and this is for x86.
I work on arm target, and I have few questions before starting off.
Has anybody worked on Spruce earlier.
how to use Spruce project for ARM, Do we need to port for ARM?
Is cross compilation straight forward or any changes need to be done.
I have gone through this paper: http://syrcose.ispras.ru/2012/files/submissions/25_syrcose2012_submission_21.pdf
there is no information on ARM and its support.
Please someone explain/help who has any work experience/knowledge on Spruce project.
Spruce was intended to work as follows. It provides a set of tests that make the kernel module for a given file system execute as many paths in the code as possible. It allows to use some external analyzers (such as the tools from KEDR framework) to detect different kinds of errors: memory leaks, etc.
All that was primarily intended for x86.
While it might be possible to port the tests themselves to ARM, one will need to choose the analyzers that work on that platform too. KEDR tools are currently for x86 only but one may try Kmemleak, Fault injection facilities and other tools on ARM instead.
Spruce seems to be a work in progress still. I see, you opened a ticket concerning ARM support in their issue tracker, I think, it is the right thing to do.
I would also suggest to take a look at Phoronix Test Suite. It is currently widely used for testing and benchmarking, including the analysis of file system kernel modules. See this article for example. It seems to work on ARM although I haven't tried it there myself.
The best tool for testing/validating a file system is xfstests. I have written tools to make it easy to validate xfstests for ext4. See: http://thunk.org/gce-xfstests for more details.
There is also an alpha-test level support for using this on ARM directly: http://thread.gmane.org/gmane.comp.file-systems.ext4/53649/focus=53659
This has been used successfully to test ext4 on an Android device, although to be honest, most of the time what I do is to bludgeon an Android kernel until it will build on x86, and then use kvm-xfstests gce-xfstests, since it's much more convenient. In particular with gce-xfstests, I can just do a "fire and forget", and then when the test completes I get a test report in my e-mail. Where as with the Android arm xfstests tarball, the automation isn't done yet, so you have to manually set up an external USB-attached USB device, hook it up via some kind of USB C hub, or if you are going to use an OTG usb adapter, you need to make sure the Android device can receive power while it is also driving the OTG usb port --- and you have to manually set up the chroot. Unless the BSP kernel has been badly abused so you can't figure out how to make it build on x86 (getting the MSM kernel to work on x86 wasn't easy) testing on gce-xfstests may be much simpler at the end of the day.
Related
I am trying to build minimal kernel under 1 Mb with Buildroot. It is intended for small board with qspi memory and basic functionality, ethernet, usb, spi, and some GPIO's. Basic terminal access via ssh and UART.
My first thoughts are if it is even possible to modify kernel .config via linux-menuconfig to reach this size.
Also if it is possible to identify the redundant parts without deep knowledge about kernel architecture and exclude them from compilation.
If someone can direct me to good direction how to solve this problem or even specify some tools and ways how to do it would be very helpful.
Thank you!
If you have working build root for your board, than, it's better to continue to work with it. Technic for disabling kernel options should be the same. In the article he reached ~0,7MB uImage with lost a lot of functionality (p40). He started with minimal (bare) config (p27) and add blocks of configs. So instead of identify the redundant parts you can build smallest possible uImage you can boot. Than add to it more options: ext2, serial and so on. Actually this work require a lot of testing. And you probably brake dependencies.
Kernel configs (working and new one) could be compared using diff -Naur, so you can see what changed.
Offtopic:
Looks like yocto officially supported by altera. here are instructions how to build altera-image-minimal. If you fine with it size, than use it and don't spend time on minimizing uImage. If you need extra packages installed into it, than you can ease extend it.
And here are instructions about building Angstrom (yocto based distribution). You can create you custom image based on console-image-minimal.
I use Angstrom in production. Must say, it was really hard to use it first time.
Whether or not you build the kernel with buildroot is not really relevant. The important thing is to configure it so it fits in 1MB. When you build the kernel from buildroot, you can do that with make linux-menuconfig, as you mention.
That said, getting a kernel under 1MB will be quite hard. Biff once did this for an x86-based platform, bifferboard. But that was without networking or USB.
You can refer to the kernel size tuning guide, which has links to some patches to reduce the size. But it's not been updated in a couple of years.
I'd like to collect some ideas about the solution of following problem.
I've got a TOF camera, and its driver for linux x86/64. It works fine. But in fact the camera would be used on an ARM based embedded device if that's possible.
Questions:
should I have to decompile the driver binaries and recompile with ARM compiler? is there any available decompiler tool?
is there any ARM>>x86 emulator which is available?
any other ideas?
ps: the pure source is very expensive, so I don't like to purchase it anyway :)
AFAIK, as of today, there's no such decompiler that will generate compilable code from machine code. You will have to manually fix (usually a LOT of fixes) the generated code. You can check this question for Linux decompilers.
Check QEmu. Underlying architecture is not relevant as long as you can execute it ;)
There's not much besides what you've stated.
What you're attempting is (in part) reverse engineering. RE is a complex process and requires great knowledge in the thing you're attempting to reverse; in your case camera drivers. If you have knowledge in this area, go ahead. If you don't, I wouldn't waste my time on it (or get the knowledge first ;))
See the following answer for details. It lists some of the problems you could encounter attempting to automate such a translation:
Convert object file to another architecture
Recently created Eltechs ExaGear Server, available on ARM as A Service, runs x86 applications on ARM devices.
It is new and proprietary, but it does exist.
If your driver is a kernel object, there is no emulator (to my knowledge) capable of running x86 kernel code inside ARM kernel. If your "driver" is in fact a userspace library hacked on top of a generic driver (v4l2, etc.), you may have luck with QEMU or Exagear mentioned here. As a side note, you will probably end up with an x86 application software, too, as calling an x86 library from ARM code is not supported by QEMU or Exagear out of the box.
So, I know Linux kernel is quite "heavy" when considering lower scale embedded systems, but currently but we're a 2 man team trying to see how to create our own embedded system.
I'm the one in charge of all software (the other guy is a HW guy), and thus I would like to re-use existing libraries and frameworks as much as possible, and I would like to bounce off some ideas with gurus around here.
I am fairly comfortable in Linux, but the booting and initialization process is new to me, and I need to dive in to that soon enough. Any book recommendations are welcome as well!
I haven't designed any embedded systems before.. Only own some ARM dev boards (beagleboard and raspberry pi).
Current I have prototype of the software running on beagleboard already, and now we're thinking how to minimize the cost, and to create something our own..
It's a system connected to the internet, and I need to run a tiny web server with some scripting support. Performance wise I don't think it needs to be too powerful.
I would like to minimize all bootloader etc work, since I'm a one man SW team, and just concentrate on the application itself.
Of course I understand that I need to configure our kernel for this, but this is indeed why I thought selecting some SoC would be good, since they usually have some linux and bootloaders ready..
First I thought that Cirrus EP9301 would be perfect, since it seems to be a good package, and not very expensive.. But it seems that it's already in end-of-life, and also support for this is very bad (people on the cirrus forums constantly complain about it).
Are there some good choices for this kind of project, which would enable us "easily" to get linux kernel up and running, with still maintaining some kind of decent BOM (hopefully 20USD or so) ?
Your hardware guy should already know this, but go with an existing reference design. Take the raspberry pi, the beagleboard/bone, open-rd, or any number of other existing systems and clone the part you need. As a result the linux porting will be a matter of removing what you are not using from the reference design instead of adding new stuff and hoping it works. If you go with flat pack parts you can do the work in your garage, if you go with bgas you need the equipment for that or pay someone to do it. (can you tell yet that I hate bgas?).
Is linux a requirement, if not that opens the door to a lot more devices using freertos or chibios or a number of other solutions. the stm32f4 discovery board for example is $20, uses what can barely be called a microcontroller for all the features it has (cortex-m4). Supposedly possible to run uclinux on a cortex-m, but definitely possible to run any number of rtoses and have an ip stack, etc. stellaris (ti.com) has a number of eval boards, one/some with ethernet already (use as a reference design). You can also take the wiznet approach (or a spi ethernet) and use any microcontroller (puts you into the avr/msp430 level and price range). Bang for buck the cortex-m's are good, arm based so comfortable to work with, etc.
Using linux if you are already not an experienced at porting to an embedded platform, and dont want to learn that on this go around, I would definitely go with a clone of an existing design, leverage as much as you can from a project with folks that are experienced at porting linux to a platform. If need be take an existing board (beagle/raspi/openrd) and go through the motions of porting to the platform with the cheat sheet of having access to an existing port, see if you cant get uboot ported and linux booting, etc. (dont really need uboot at all, that is possibly an unnecessary complication, just get dram up and pass the atags, etc to linux and just branch to it, pretty easy to launch linux from bare metal).
You could probably do worse than taking the Broadcom BCM2835 - used on the Raspberry Pi - as your starting point - especially if you want to avoid kernel and boot-loader work and a source of reference schematics. If this proves too expensive, check out other devices in the Broadcom range.
A few bits of advice
You probably want some flash rather than a MMC card interface for production use. eMMC is an option. NAND flash is a nightmare due to rapid component obsolescence and the need to get own and dirty with the MTD drivers.
USB Ethernet will be easier to integrate than a controller hanging off a general purpose bus, but won't perform as well. SmSC seems to be popular source for either
You could also have a look at the work that Olimex is doing with their linux boards. Perhaps even order a som and then combine it with other external components.
I have a Linux network driver that was originally written for 2.4 kernel. It works perfect.
I want to port it to kernel 2.6.31 and then to ARM Linux with same kernel i.e. 2.6.31. I have actually done some minor changes to the driver so that it is able to compile under kernel 2.6.31 and it also loads and unloads without crashing. It also cross compiles for the ARM Linux. But I am unable to test it on ARM so far.
How do I check that the driver is fully compatible with the target kernel, and what considerations shall be made to make it compatible with ARM.
The driver is a virtual network device driver.
Thanks in advance.
Maybe you could use Qemu ( http://wiki.qemu.org/Main_Page) to emulate an ARM platform to be able to test your driver.
You cannot check the driver like that - you have to consider the API changes within 2.6.x series kernel. The changes are quite significant and the overall of the API's from the 2.4 series which is not currently in use.
I would suggest you to go here to the Amazon book store for this book in particular. The book is called 'Essential Linux Device Drivers', by Sreekrishnan Venkateswaran. A very well detailed explanation that will be your guidance in ensuring it works properly.
Since you mentioned the device driver is a network, presumably char device (You're not accessing it in blocks), well, the good news is that the 2.6.x series kernel APIs for the character devices are significantly easier and more centralized to focus on - in fact a lot of the framework is already in place in which the author of said book explains very clearly.
By the way, the book focusses on the latter 2.6.x series after 2.6.19, so this will help you clue in on what needs to be done to ensure your driver works.
You did not specify the ARM chipset you're targetting?
As for testing... well.. perhaps the best way to do this, this is dependant on how you answer the above question to you regarding ARM chipset - if its ARMv6, then perhaps, a cheap android handset that you can easily unlock and root, and pop the kernel in there and see what happens - sorry for sounding contrived but that's the best thing I can think of and that's what pops into my head, to enable you to test it out for ease of testing :)
PS: A lot of cheap ARMv6 handsets would have kernel 2.6.32 running Froyo if that's of any help!
How do the Linux kernel developers test their code locally and after they have it committed? Do they use some kind of unit testing and build automation? Test plans?
The Linux kernel has a heavy emphasis on community testing.
Typically, any developer will test their own code before submitting, and quite often they will be using a development version of the kernel from Linus, or one of the other unstable/development trees for a project relevant to their work. This means they are often testing both their changes and other people's changes.
There tends not to be much in the way of formal test plans, but extra testing may be asked for before features are merged into upstream trees.
As Dean pointed out, there's also some automated testing: The Linux Test Project and the kernel Autotest (good overview).
Developers will often also write automated tests targeted to test their change, but I'm not sure there's a (often used) mechanism to centrally collect these ad hoc tests.
It depends a lot on which area of the kernel is being changed of course - the testing you'd do for a new network driver is quite different to the testing you'd do when replacing the core scheduling algorithm.
Naturally, the kernel itself and its parts are tested prior to the release, but these tests cover only the basic functionality. There are some testing systems which perform testing of Linux Kernel:
The Linux Test Project (LTP) delivers test suites to the open source community that validate the reliability and stability of Linux. The LTP test suite contains a collection of tools for testing the Linux kernel and related features.
Autotest—a framework for fully automated testing. It is designed primarily to test the Linux kernel, though it is useful for many other purposes such as qualifying new hardware, virtualization testing, and other general user space program testing under Linux platforms. It's an open-source project under the GPL and is used and developed by a number of organizations, including Google, IBM, Red Hat, and many others.
Also there are certification systems developed by some major GNU/Linux distribution companies. These systems usually check complete GNU/Linux distributions for compatibility with hardware. There are certification systems developed by Novell, Red Hat, Oracle, Canonical, and Google.
There are also systems for dynamic analysis of the Linux kernel:
Kmemleak is a memory leak detector included in the Linux kernel. It provides a way of detecting possible kernel memory leaks in a way similar to a tracing garbage collector with the difference that the orphan objects are not freed, but only reported via /sys/kernel/debug/kmemleak.
Kmemcheck traps every read and write to memory that was allocated dynamically (i.e., with kmalloc()). If a memory address is read that has not previously been written to, a message is printed to the kernel log. It is also is a part of the Linux kernel.
Fault Injection Framework (included in the Linux kernel) allows for infusing errors and exceptions into an application's logic to achieve a higher coverage and fault tolerance of the system.
How do the Linux kernel developers test their code locally and after they have it committed?
Do they use some kind of unit testing and build automation?
In the classic sense of words, no.
For example, Ingo Molnar is running the following workload:
build a new kernel with a random set of configuration options
boot into it
go to 1
Every build fail, boot fail, bug or runtime warning is dealt with. 24/7. Multiply by several boxes, and one can uncover quite a lot of problems.
Test plans?
No.
There may be a misunderstanding that there is a central testing facility, but there is none. Everyone does what he/she wants.
In-tree tools
A good way to find test tools in the kernel is to:
make help and read all targets
look under tools/testing
In v4.0, this leads me to:
kselftest under tools/testing/selftests. Run with make kselftest. Must be running built kernel already. See also: Documentation/kselftest.txt , https://kselftest.wiki.kernel.org/
ktest under tools/testing/ktest. See also: http://elinux.org/Ktest , http://www.slideshare.net/satorutakeuchi18/kernel-auto-testbyktest
Static analysers section of make help, which contains targets like:
checkstack: Perl: what does checkstack.pl in linux source do?
coccicheck for Coccinelle (mentioned by askb)
Kernel CI
https://kernelci.org/ is a project that aims to make kernel testing more automated and visible.
It appears to do only build and boot tests (TODO how to test automatically that boot worked Source should be at https://github.com/kernelci/).
Linaro seems to be the main maintainer of the project, with contributions from many big companies: https://kernelci.org/sponsors/
Linaro Lava
http://www.linaro.org/initiatives/lava/ looks like a CI system with focus on development board bringup and the Linux kernel.
ARM LISA
https://github.com/ARM-software/lisa
Not sure what it does in detail, but it is by ARM and Apache Licensed, so likely worth a look.
Demo: https://www.youtube.com/watch?v=yXZzzUEngiU
Step debuggers
Not really unit testing, but may help once your tests start failing:
QEMU + GDB: https://stackoverflow.com/a/42316607/895245
KGDB: https://stackoverflow.com/a/44226360/895245
My own QEMU + Buildroot + Python setup
I also started a setup focused on ease of development, but I ended up adding some simple testing capabilities to it as well: https://github.com/cirosantilli/linux-kernel-module-cheat/tree/8217e5508782827320209644dcbaf9a6b3141724#test-this-repo
I haven't analyzed all the other setups in great detail, and they likely do much more than mine, however I believe that my setup is very easy to get started with quickly because it has a lot of documentation and automation.
It’s not very easy to automate kernel testing. Most Linux developers do the testing on their own, much like adobriyan mentioned.
However, there are a few things that help with debugging the Linux Kernel:
kexec: A system call that allows you to put another kernel into memory and reboot without going back to the BIOS, and if it fails, reboot back.
dmesg: Definitely the place to look for information about what happened during the kernel boot and whether it works/doesn't work.
Kernel Instrumentation: In addition to printk's (and an option called 'CONFIG_PRINTK_TIME' which allows you to see (to microsecond accuracy) when the kernel output what), the kernel configuration allows you to turn on a lot of tracers that enable them to debug what is happening.
Then, developers usually have others review their patches. Once the patches are reviewed locally and seen not to interfere with anything else, and the patches are tested to work with the latest kernel from Linus without breaking anything, the patches are pushed upstream.
Here's a nice video detailing the process a patch goes through before it is integrated into the kernel.
In addition to the other answers, this emphasise more on the functionality testing, hardware certification testing and performance testing the Linux kernel.
A lot of testing actually happen through scripts, static code analysis tools, code reviews, etc. which is very efficient in catching bugs, which would otherwise break something in the application.
Sparse – An open-source tool designed to find faults in the Linux kernel.
Coccinelle is another program does matching and transformation engine which provides the language SmPL (Semantic Patch Language) for specifying desired matches and transformations in C code.
checkpatch.pl and other scripts - coding style issues can be found in the file Documentation/CodingStyle in the kernel source tree. The important thing to remember when reading it is not that this style is somehow better than any other style, just that it is consistent. This helps developers easily find and fix coding style issues. The script scripts/checkpatch.pl in the kernel source tree has been developed for it. This script can point out problems easily, and should always be run by a developer on their changes, instead of having a reviewer waste their time by pointing out problems later on.
There are also:
MMTests which is collection of benchmarks and scripts to analyze the results.
Trinity which is Linux system call fuzz tester.
Also the LTP pages at SourceForge are quite outdated and the project has moved to GitHub.
I would imagine they use virtualization to do quick tests. It could be something like QEMU, VirtualBox or Xen, and some scripts to perform configurations and automated tests.
Automated testing is probably done by trying either many random configurations or a few specific ones (if they are working with a specific issue). Linux has a lot of low-level tools (such as dmesg) to monitor and log debug data from the kernel, so I imagine that is used as well.
As far as I know, there is an automatically performance regression check tool (named lkp/0 day) running/funding by the Intel. It will test each valid patch sent to the mailing list and check the scores changed from different microbenchmarks such as hackbench, fio, unixbench, netperf, etc.
Once there is a performance regression/improvement, a corresponding report will be sent directly to the patch author and a Cc related maintainers.
LTP and Memtests are generally preferred tools.
adobriyan mentioned Ingo's loop of random configuration build testing. That is pretty much now covered by the 0-day test bot (aka kbuild test bot). A nice article about the infrastructure is presented here: Kernel Build/boot testing
The idea behind this set-up is to notify the developers ASAP so that they can rectify the errors soon enough (before the patches make it into Linus' tree in some cases as the kbuild infrastructure also tests against maintainer's subsystem trees).
Once after contributors submit their patch files and after making a merge request, Linux gatekeepers are checking the patch by integrating and reviewing it. Once it succeeds, they will merge the patch into the relevant branch and a make new version release.
The Linux Test Project is the main source which provides test scenarios (test cases) to run against the kernel after applying patches. This may take around 2 ~ 4 hours, and it depends.
Please note regarding the file system of the selected kernel is going to test against.
Example: ext4 generates different results against ext3 and so on.
Kernel Testing procedure.
Get latest kernel source from the repository (The Linux Kernel Archives or GitHub)
Apply the patch file (using a diff tool)
Build the new kernel.
Test against test procedures in LTP (Linux Test Project)
I had done Linux kernel compilation and done some modifications for Android (Android 6.0 (Marshmallow) and Android 7.0 (Nougat)) in which I use Linux version 3. I cross-compiled it on a Linux system, debugged the errors manually and then ran its boot image file in Android and checked if it was going in a loop-hole or not. If it runs perfect then it means it is compiled perfectly according to system requirements.
For MotoG kernel Compilation
Note: The Linux kernel will change according to requirements which depend on system hardware