The question is in the subject. Let me explain "why".
I am running my application on Red Hat Enterprise Linux Server 7.7. When I was checking performance using htop I found that few threads take too much of the CPU.
I added some debug logging and found that the threads with high CPU are not created in my code. So I assume that these CPU greedy threads are created in 3rd party shared libs which I am using.
So there is a question:
Say I have a thread id (17405). Is there any way to find which shared lib started this thread?
I apologize if the question is too trivial - I started work with Linux OS not long time ago.
Thank you
Actually I found solution which looks satisfactory to me.
I start gdb, attach to my process, then i can list all threads in the process "info threads", then i select thread I am interested in, and - voila - i can see the stack trace by issuing bt command.
Works I think
Related
There is a lot about the "procedure" LINUX undertakes when a reboot or 'imminent shutdown' is underway. I wish to understand, programmatically, and react to, that a REBOOT IS IMMINIENT.
My application gets sent TERM and HUP signals which is all good and understood.
How can I tell the difference between a signal for immediate REBOOT reasons or just from another user?
CONTEXT:
If the system is 100% rebooting then my application will SAVE IT's STATE for LOAD on BOOT.
If however, a user, even "root" arbitrately kills my application then I DO NOT WISH to save the STATE.
I am using RedHat if it makes a difference.
I have attempted to research this quite extensively but so far I have not got the solution.
So you know... calling STACK OVERFLOW!
Thank you in-advance.
For: QNX Software Development Platform 6.5.0
I have run into a problem on a QNX 6.5.0 system where my program silently exists, and has been found to be due to a race condition similar to this post here:
Thread stops randomly in the middle of a while loop
I have done some research and found that QNX has some built in tools to monitor memory and detect any leaks that are present in the program, however the instructions I have come across are for the QNX 6.5.0 IDE GUI, and I am running QNX on a server in command line.
example: http://www.qnx.com/developers/docs/6.5.0/index.jsp?topic=%2Fcom.qnx.doc.ide.userguide%2Ftopic%2Fmemory_DetecMemLeaks_.html
I'm kind of stuck with this as there isn't really a simple way to do this as the software designed is for logging purposes and is taking thousands of entries per second, and silently exists after a few hours. So I can't sit here waiting 2 hours each round.
Has anyone had experience with debugging mem leaks in QNX?
Edit: I am also using boost::lockfree::spsc_queue which may be causing the crash.
I was able to solve this by utilising Valgrind. I compiled my program and valgrind for linux and was able to debug my issue this way.
Are there any watchdog tools or libraries on Linux for the following purpose? I would like to build a watchdog executable which starts 2 processes and restarts them if:
processes crash
processes become unresponsive (e.g. hang for some reason)
Internet search found watchdog.c but I am not sure if that can be used for my purposes, it looks pretty low level.
I could run my processes as init programs (daemons) as suggested here, but I am not sure if Linux would then recognize that the process is hanging (e.g. due to a deadlock)
We use monit here: http://mmonit.com/monit/ it will let you do the restart thing it is also highly customizable regarding how to check and how to react via scripts
At college I'm studying Operative Systems, and as a first part of the project we have to modify the Timer Interrupt to execute my own code, may be with threads, and I think that Linux present less restrictions to access the Interrupt Vector that Windows does, is not?
Can you give me more details if it's better use Windows or Linux (like Ubuntu) to do this.
Thanks.
I would use Linux, because I think you might fail your assignment if you use Windows. The reason being that the commonly accessible timers (i.e. non-driver stuff) under Windows are not really interrupts, they're messages posted to your thread's message queue.
Whereas under Linux signal/sigaction in combination with timer_create will send a signal, which really counts as "interrupt".
A linux machine freezes few hours after booting and running software (including custom drivers). I'm looking a method to debug such problem. Recently, there has been significant progress in Linux Kernel debugging techniques, hasn't it?
I kindly ask to share some experience on the topic.
If you can reproduce the problem inside a VM, there is indeed a fairly new (AFAIK) technique which might be useful: debugging the virtual machine from the host machine it runs on.
See for example this:
Debugging Linux Kernel in VMWare with Windows host
VMware Workstation 7 also enables a powerful technique that lets you record system execution deterministically and then replay it as desired, even backwards. So as soon as the system crashes you can go backwards and see what was happening then (and even try changing something and see if it still crashes). IIRC I read somewhere you can't do this and debug the kernel using VMware/gdb at the same time.
Obviously, you need a VMM for this. I don't know what VMM's other than VMware's VMM family support this, and I don't know if any free VMware versions support this. Likely not; one can't really expect a commercial company to give away everything for free. The trial version is 30 days.
If your custom drivers are for hardware inside the machine, then I suppose this probably won't work.
SystemTap seems to be to Linux what Dtrace is to Solaris .. however I find it rather hostile to use. Still, you may want to give it a try. NB: compile the kernel with debug info and spend some time with the kernel instrumentation hooks.
This is why so many are still using printk() after empirically narrowing a bug down to a specific module.
I'm not recommending it, just pointing out that it exists. I may not be smart enough to appreciate some underlying beauty .. I just write drivers for odd devices.
There are many and varied techniques depending on the sort of problems you want to debug. In your case the first question is "is the system really frozen?". You can enable the magic sysrq key and examine the system state at freeze and go from there.
Probably the most directly powerful method is to enable the kernel debugger and connect to it via a serial cable.
One option is to use Kprobes. A quick search on google will show you all the information you need. It isn't particularly hard to use. Kprobes was created by IBM I believe as a solution for kernel debugging. It is essentially a elaborate form of printk() however it allows you to handle any "breakpoints" you insert using handlers. It may be what you are looking for. All you need to do is write and 'insmod' a module into the kernel which will handle any "breakpoints" hit that you specify in the module.
Hope that can be a useful option...
How I debug this kind of bug, was to run my OS inside the VirtualBox, and compile the kernel with kgdb builtin. Then I setup a serial console on the VirtualBox so that I can gdb to the kernel inside the VirtualBox's OS via the serial console. Anytime the OS hang, just like magic sysrq key, I can enter ctrl-c on the gdb to stop and understand the kernel at that point in time.
Normally kernel stack tracing is just too difficult to pinpoint the culprit process, so the best way I think is still generic "top" command, just looking at the application logs to see what are the cause of hanging - this will need a reboot to see the log of course.