Using NDK r8c, Eclipse 4.2, Windows 7 64
I've used remote debuggers before (on other platforms, via gigabit ethernet) for large C++ codebases that felt no different than local debugging. The Java debugger that comes with the SDK runs fast too. Therefore I'm quite baffled why gdb is so slow to connect and step over lines of code.
In my current application, which is around 20 static libraries and 1500 source files, it takes about 15 seconds to connect, and about 2 seconds to step. I'm more concerned about stepping.
Has anyone ever profiled gdb to see what the problem is? If so, any suggestions?
I have. My cohorts and I at NVIDIA have contributed several commits to AOSP to address this problem, although our focus has been on shared libraries (symbol load performance, and pending symbol resolution.) We have sped up solib load processing by a factor of 6x. (Although, after doing our own work we discovered that 3x of that 6x had already been solved upstream by GNU, in 7.5... so we abandoned our reinvention, and submitted the relevant 7.5 patches up to Google's NDK repository, which was based on the older 7.3 GDB.) I believe all of our speedups are present in r8d... but I haven't checked.
I cannot think of any reason why static libraries would slow things down, but I must admit I haven't given any thought to them. Do you have a specific reason for believing so, or was that just comment to give perspective about the size and scope of your debugging needs?
We have begun to work on the stepping problem, but don't have anything to share yet. Basically, the bottleneck is ADB (especially on Windows.) Additionally, there is a lot of chatty communication between GDB and gdbserver, when stepping, especially if you are using an IDE with local window, register window, expression window, stack window, etc., all updating with each step. That's a lot of chatter that could likely be optimized for the IDE use-case.
Just some of the fixes that we are considering for speeding up stepping will be IDE-specific:
Using python scripting to pre-process watch expressions in GDB, rather than in the IDE.
Implementing "super-packets" communicating between GDB and gdbserver... packets that encapsulate IDE-specific communications in a way that minimizes chatter between GDB and gdbserver.
We intend to share all of this with the Android community.
Related
My app is having tearing problems when I scroll fast horizontally. Normally I'd just enable vsync, but I don't know how to do that in Qt. The tearing is occurring on my dev PC which runs Linux with a X-server, but it will also need to work well on the target system which is either Android or Linux+EGLFS.
One google result was from the Qt4 era and said that the tearing will be gone in Qt 5. But apparently it isn't.
Many google results are about env vars for embedded linux systems, such as QT_QPA_EGLFS_FORCEVSYNC, but I'd prefer the solution to work for my dev PC as well.
One google result said that vsync being enabled/disabled depends on the renderloop in use. I don't override the renderloop, so, since I'm on linux, I think I'm using the threaded renderloop.
Here's a bugreport about nvidia binary drivers causing this.
https://bugreports.qt.io/browse/QTBUG-45480
I too am using those drivers, so this may be it - just a driver bug. If so, at least it's unlikely to happen on my target hardware (I still haven't tested there), so that's good.
I am new to linux programming & interested to tweak linux kernel(though I am not sure, what to tweak, I am planning to write drivers for particular device). To learn internal of kernel, I have started from historic kernel release (first release).
My problem is, how to test whatever changes I am doing for development, without disturbing my current os environment.(ubuntu 12, 64 bit). Is there any way like virtual box, sandbox?
Along with these, if anybody send some good approaches to learn these things, I would be really greatful.
Thank You.
If you're new to linux programming then you really don't want to be tweaking the kernel. You really want to be an advanced programmer capable of programming drivers and complex software first.
But yes there is, you can can create a virtual machine using openbox or vmware. If you're really keen on tweaking the kernel you probably want to first just try compiling and configuring the kernel and seeing if that works.
Also make sure you're well acquainted with how the kernel works and advanced OS designs in general.
Search in google fr "Kernel configuration" you u will get many links how to configure your own kernel.
And one more thing do not use a outdated version of kernel ,always use latest stable release , because a lot of code and API is changed in new versions and no book in market is updated so ,, u have to read from kernel documentation. Thats the best way to learn the most updated information about linux kernel
Yes, you can test your changes on any of the commonly available virtual machines (VMs); that way, whatever changes you make to the VM kernel won't affect native OS.
Personally, I prefer using CentOS 64 bit on VMWare Player. With this setup, I got away with minimal system maintenance while was able to focus on the actual job at hand. Once the VM is up & running, you can download and compile one of the latest stable releases from kernel.org. Instructions on compiling your downloaded version of kernel could be found here and here; however, this may require little tweaking based on your actual setup. Once the VM is running on your desired version of kernel, using a combination of cscope and ctags will help you immensely in kernel code browsing.
Finally, if you want to become a serious kernel programmer and write your own device drivers, you need to get familiar with it in the first place. Below are a few excellent references -
Linux Device Driver by Corbet, Rubini, Kroah-Hartman, 3rd edition
Linux Kernel Development by Robert Love, 3rd edition
Understanding the Linux Kernel by Bovet, Cesati
Linux kernel source (ideally placed into your /usr/src/$(DESIRED_KERNEL) path, symlinked to /usr/src/linux)
Going through these books is a tedious job and chances are that you may hit the roadblock from time to time. kernelnewbies mailing list and StackOverflow are some of the few reliable places where people would be happy to answer to your queries.
Good luck!
As an Android developer I've been moving away from Eclipse to Intellij IDEA for production code in anticipation of Google's Android studios which shares a code base with IDEA.
My experience has been a good one up to this point. I've only been using IDEA at the office, where I have a 4x core Intel i7 machine running Ubuntu 12.04 LTS (Sun JDK/JRE), up to this point and I've never noticed what the performance of IDEA really is.
Now however after setting IDEA up on my personal computer at home the performance is abysmal. Memory usage is normal, but the constant CPU usage bounces between 80%-100% (over the whole application lifecycle). And that is when nothing else is running on the machine and no work is being done, by me or visually by the IDE.
This makes IDEA unusable when working on it, and I can forget about having anything else running along side it.
My home specs and software are:
Intel Core 2 duo 3GHz
8 GB RAM
Ubuntu 12.04 x64 LTS (3.8.0-35-generic) running of SSD SATA
Intellij IDEA 13.0-0ubuntu1 build: IC-133.193
Tried both OpenJDK and Sun
And the strange thing is that this happens as well with Android Studios.
All help in trying to debug this behaviour would be appreciated.
#Edit 1:
Noticed that the CPU load falls down to 20% when bringing up dialogs (Project structure, Settings, etc) and then goes right back up when dismissing them.
#Edit 2:
I tested simply getting the tarball straight from JetBrains, instead of using the one in Canonical's ppa. The performance was significantly better for at least an hour (20-30% CPU usage while idle). Seems that the native file watcher in C-PPA wasn't working properly and was indexing the whole filesystem.
However the performance became worse after the first hour or so, going back to 90-100% CPU.
The issue turned out to be the native-file watcher being out-of-date. IntelliJ was re indexing my whole drive it seems. Was fixed by uninstalling the version gotten from Canonical's ppa and installing directly from JetBrain's own webpage.
Are you using any plugins outside of the included ones which might cause issues.
I don't run Ubuntu anymore but can't recall any issues with high CPU-usage when i did. (I use Fedora with KDE a colleague uses Fedora with GNOME though. )
Does this always happen or only when you have a project open?
I'm thinking if this might have something to do with the background-compile that IDEA does.
Might be worth trying to turn this off.
Found under Project Settings -> Compiler -> Make Project Automatically
worst case it is a Unity-integration issue or something. Haven't used unity so can't say.
Usualy I manage to fix it by deleting IDE's index files rm -rf ~/.RubyMine60/system,
don't forget to change .RubyMine60 to IDEA's config folder
If you're willing to do some sleuthing, you could run the Oracle JVM and use the VisualVM profiler to see where the IDE is spending all its time, presuming it's a Java-based process that's actually eating your CPU cycles.
How do the Linux kernel developers test their code locally and after they have it committed? Do they use some kind of unit testing and build automation? Test plans?
The Linux kernel has a heavy emphasis on community testing.
Typically, any developer will test their own code before submitting, and quite often they will be using a development version of the kernel from Linus, or one of the other unstable/development trees for a project relevant to their work. This means they are often testing both their changes and other people's changes.
There tends not to be much in the way of formal test plans, but extra testing may be asked for before features are merged into upstream trees.
As Dean pointed out, there's also some automated testing: The Linux Test Project and the kernel Autotest (good overview).
Developers will often also write automated tests targeted to test their change, but I'm not sure there's a (often used) mechanism to centrally collect these ad hoc tests.
It depends a lot on which area of the kernel is being changed of course - the testing you'd do for a new network driver is quite different to the testing you'd do when replacing the core scheduling algorithm.
Naturally, the kernel itself and its parts are tested prior to the release, but these tests cover only the basic functionality. There are some testing systems which perform testing of Linux Kernel:
The Linux Test Project (LTP) delivers test suites to the open source community that validate the reliability and stability of Linux. The LTP test suite contains a collection of tools for testing the Linux kernel and related features.
Autotest—a framework for fully automated testing. It is designed primarily to test the Linux kernel, though it is useful for many other purposes such as qualifying new hardware, virtualization testing, and other general user space program testing under Linux platforms. It's an open-source project under the GPL and is used and developed by a number of organizations, including Google, IBM, Red Hat, and many others.
Also there are certification systems developed by some major GNU/Linux distribution companies. These systems usually check complete GNU/Linux distributions for compatibility with hardware. There are certification systems developed by Novell, Red Hat, Oracle, Canonical, and Google.
There are also systems for dynamic analysis of the Linux kernel:
Kmemleak is a memory leak detector included in the Linux kernel. It provides a way of detecting possible kernel memory leaks in a way similar to a tracing garbage collector with the difference that the orphan objects are not freed, but only reported via /sys/kernel/debug/kmemleak.
Kmemcheck traps every read and write to memory that was allocated dynamically (i.e., with kmalloc()). If a memory address is read that has not previously been written to, a message is printed to the kernel log. It is also is a part of the Linux kernel.
Fault Injection Framework (included in the Linux kernel) allows for infusing errors and exceptions into an application's logic to achieve a higher coverage and fault tolerance of the system.
How do the Linux kernel developers test their code locally and after they have it committed?
Do they use some kind of unit testing and build automation?
In the classic sense of words, no.
For example, Ingo Molnar is running the following workload:
build a new kernel with a random set of configuration options
boot into it
go to 1
Every build fail, boot fail, bug or runtime warning is dealt with. 24/7. Multiply by several boxes, and one can uncover quite a lot of problems.
Test plans?
No.
There may be a misunderstanding that there is a central testing facility, but there is none. Everyone does what he/she wants.
In-tree tools
A good way to find test tools in the kernel is to:
make help and read all targets
look under tools/testing
In v4.0, this leads me to:
kselftest under tools/testing/selftests. Run with make kselftest. Must be running built kernel already. See also: Documentation/kselftest.txt , https://kselftest.wiki.kernel.org/
ktest under tools/testing/ktest. See also: http://elinux.org/Ktest , http://www.slideshare.net/satorutakeuchi18/kernel-auto-testbyktest
Static analysers section of make help, which contains targets like:
checkstack: Perl: what does checkstack.pl in linux source do?
coccicheck for Coccinelle (mentioned by askb)
Kernel CI
https://kernelci.org/ is a project that aims to make kernel testing more automated and visible.
It appears to do only build and boot tests (TODO how to test automatically that boot worked Source should be at https://github.com/kernelci/).
Linaro seems to be the main maintainer of the project, with contributions from many big companies: https://kernelci.org/sponsors/
Linaro Lava
http://www.linaro.org/initiatives/lava/ looks like a CI system with focus on development board bringup and the Linux kernel.
ARM LISA
https://github.com/ARM-software/lisa
Not sure what it does in detail, but it is by ARM and Apache Licensed, so likely worth a look.
Demo: https://www.youtube.com/watch?v=yXZzzUEngiU
Step debuggers
Not really unit testing, but may help once your tests start failing:
QEMU + GDB: https://stackoverflow.com/a/42316607/895245
KGDB: https://stackoverflow.com/a/44226360/895245
My own QEMU + Buildroot + Python setup
I also started a setup focused on ease of development, but I ended up adding some simple testing capabilities to it as well: https://github.com/cirosantilli/linux-kernel-module-cheat/tree/8217e5508782827320209644dcbaf9a6b3141724#test-this-repo
I haven't analyzed all the other setups in great detail, and they likely do much more than mine, however I believe that my setup is very easy to get started with quickly because it has a lot of documentation and automation.
It’s not very easy to automate kernel testing. Most Linux developers do the testing on their own, much like adobriyan mentioned.
However, there are a few things that help with debugging the Linux Kernel:
kexec: A system call that allows you to put another kernel into memory and reboot without going back to the BIOS, and if it fails, reboot back.
dmesg: Definitely the place to look for information about what happened during the kernel boot and whether it works/doesn't work.
Kernel Instrumentation: In addition to printk's (and an option called 'CONFIG_PRINTK_TIME' which allows you to see (to microsecond accuracy) when the kernel output what), the kernel configuration allows you to turn on a lot of tracers that enable them to debug what is happening.
Then, developers usually have others review their patches. Once the patches are reviewed locally and seen not to interfere with anything else, and the patches are tested to work with the latest kernel from Linus without breaking anything, the patches are pushed upstream.
Here's a nice video detailing the process a patch goes through before it is integrated into the kernel.
In addition to the other answers, this emphasise more on the functionality testing, hardware certification testing and performance testing the Linux kernel.
A lot of testing actually happen through scripts, static code analysis tools, code reviews, etc. which is very efficient in catching bugs, which would otherwise break something in the application.
Sparse – An open-source tool designed to find faults in the Linux kernel.
Coccinelle is another program does matching and transformation engine which provides the language SmPL (Semantic Patch Language) for specifying desired matches and transformations in C code.
checkpatch.pl and other scripts - coding style issues can be found in the file Documentation/CodingStyle in the kernel source tree. The important thing to remember when reading it is not that this style is somehow better than any other style, just that it is consistent. This helps developers easily find and fix coding style issues. The script scripts/checkpatch.pl in the kernel source tree has been developed for it. This script can point out problems easily, and should always be run by a developer on their changes, instead of having a reviewer waste their time by pointing out problems later on.
There are also:
MMTests which is collection of benchmarks and scripts to analyze the results.
Trinity which is Linux system call fuzz tester.
Also the LTP pages at SourceForge are quite outdated and the project has moved to GitHub.
I would imagine they use virtualization to do quick tests. It could be something like QEMU, VirtualBox or Xen, and some scripts to perform configurations and automated tests.
Automated testing is probably done by trying either many random configurations or a few specific ones (if they are working with a specific issue). Linux has a lot of low-level tools (such as dmesg) to monitor and log debug data from the kernel, so I imagine that is used as well.
As far as I know, there is an automatically performance regression check tool (named lkp/0 day) running/funding by the Intel. It will test each valid patch sent to the mailing list and check the scores changed from different microbenchmarks such as hackbench, fio, unixbench, netperf, etc.
Once there is a performance regression/improvement, a corresponding report will be sent directly to the patch author and a Cc related maintainers.
LTP and Memtests are generally preferred tools.
adobriyan mentioned Ingo's loop of random configuration build testing. That is pretty much now covered by the 0-day test bot (aka kbuild test bot). A nice article about the infrastructure is presented here: Kernel Build/boot testing
The idea behind this set-up is to notify the developers ASAP so that they can rectify the errors soon enough (before the patches make it into Linus' tree in some cases as the kbuild infrastructure also tests against maintainer's subsystem trees).
Once after contributors submit their patch files and after making a merge request, Linux gatekeepers are checking the patch by integrating and reviewing it. Once it succeeds, they will merge the patch into the relevant branch and a make new version release.
The Linux Test Project is the main source which provides test scenarios (test cases) to run against the kernel after applying patches. This may take around 2 ~ 4 hours, and it depends.
Please note regarding the file system of the selected kernel is going to test against.
Example: ext4 generates different results against ext3 and so on.
Kernel Testing procedure.
Get latest kernel source from the repository (The Linux Kernel Archives or GitHub)
Apply the patch file (using a diff tool)
Build the new kernel.
Test against test procedures in LTP (Linux Test Project)
I had done Linux kernel compilation and done some modifications for Android (Android 6.0 (Marshmallow) and Android 7.0 (Nougat)) in which I use Linux version 3. I cross-compiled it on a Linux system, debugged the errors manually and then ran its boot image file in Android and checked if it was going in a loop-hole or not. If it runs perfect then it means it is compiled perfectly according to system requirements.
For MotoG kernel Compilation
Note: The Linux kernel will change according to requirements which depend on system hardware
A linux machine freezes few hours after booting and running software (including custom drivers). I'm looking a method to debug such problem. Recently, there has been significant progress in Linux Kernel debugging techniques, hasn't it?
I kindly ask to share some experience on the topic.
If you can reproduce the problem inside a VM, there is indeed a fairly new (AFAIK) technique which might be useful: debugging the virtual machine from the host machine it runs on.
See for example this:
Debugging Linux Kernel in VMWare with Windows host
VMware Workstation 7 also enables a powerful technique that lets you record system execution deterministically and then replay it as desired, even backwards. So as soon as the system crashes you can go backwards and see what was happening then (and even try changing something and see if it still crashes). IIRC I read somewhere you can't do this and debug the kernel using VMware/gdb at the same time.
Obviously, you need a VMM for this. I don't know what VMM's other than VMware's VMM family support this, and I don't know if any free VMware versions support this. Likely not; one can't really expect a commercial company to give away everything for free. The trial version is 30 days.
If your custom drivers are for hardware inside the machine, then I suppose this probably won't work.
SystemTap seems to be to Linux what Dtrace is to Solaris .. however I find it rather hostile to use. Still, you may want to give it a try. NB: compile the kernel with debug info and spend some time with the kernel instrumentation hooks.
This is why so many are still using printk() after empirically narrowing a bug down to a specific module.
I'm not recommending it, just pointing out that it exists. I may not be smart enough to appreciate some underlying beauty .. I just write drivers for odd devices.
There are many and varied techniques depending on the sort of problems you want to debug. In your case the first question is "is the system really frozen?". You can enable the magic sysrq key and examine the system state at freeze and go from there.
Probably the most directly powerful method is to enable the kernel debugger and connect to it via a serial cable.
One option is to use Kprobes. A quick search on google will show you all the information you need. It isn't particularly hard to use. Kprobes was created by IBM I believe as a solution for kernel debugging. It is essentially a elaborate form of printk() however it allows you to handle any "breakpoints" you insert using handlers. It may be what you are looking for. All you need to do is write and 'insmod' a module into the kernel which will handle any "breakpoints" hit that you specify in the module.
Hope that can be a useful option...
How I debug this kind of bug, was to run my OS inside the VirtualBox, and compile the kernel with kgdb builtin. Then I setup a serial console on the VirtualBox so that I can gdb to the kernel inside the VirtualBox's OS via the serial console. Anytime the OS hang, just like magic sysrq key, I can enter ctrl-c on the gdb to stop and understand the kernel at that point in time.
Normally kernel stack tracing is just too difficult to pinpoint the culprit process, so the best way I think is still generic "top" command, just looking at the application logs to see what are the cause of hanging - this will need a reboot to see the log of course.