I'm trying to debug an issue in a Qt5 application where, for some reason, even after the application reaches its exit point (log message placed right before return 0 of int main shows), the process persists and when running "ps -e" and grepping for the process, it will show a process in the background.
Is there a way I can diagnose where this thread is in the background? All my log messages indicate that all Qt windows have been closed, and the "setQuitOnLastWindowClosed" flag is set to true. So the only thing I can think of is that a thread spawned by the application is still running in the background.
I should note that this does not ALWAYS happen. When the user exits the application normally, this does not happen. But when the machine detects a power cycle, it forces a close, but it seems something is missing in the code it runs in this case, so figuring out what's still running will help me find that.
The application was built in Qt5 and it's running on Scientific Linux 6.4 if that matters.
Related
I have a routine that crashes linux and force a reboot using a system function.
Now I have the problem that I need to crash linux when a certain process dies. Using a script starting the process and if the script ends restart the server is not appropriate since it takes some ms.
Another idea is spawning the shooting processes alongside and use polling of a counter and if the counter is not incremented reboot the server would be another idea.
This would result in an almost instant reaction.
Now the question is what would be a good timeframe. I have no idea how the scheduler of linux would guarantee a certain update of any such counter and what a good timeout would be.
Also I would like to hear some alternatives to this second process spawning. Is there a possibility to advice linux to run a certain routine in case of a crash of the given process or a listener meachanism for the even of problems with a given process?
The timeout idea is already implemented in the kernel. You can register any application as a software watchdog, but you'll have to lower the default timeout. Have a look at http://linux.die.net/man/8/watchdog for some ideas. That application can also handle user-defined tests. Realistically unless you're trying to run kernel like linux-rt, having timeouts lower than 100ms can be dangerous on a system with heavy load - especially if the check needs to poll another application.
In cases of application crashes, you can handle them if your init supports notifications. For example both upstart and systemd can do that by monitoring files (make sure coredumps are created in the right place).
But in any case, I'd suggest rethinking the idea of millisecond-resolution restarts. Do you really need to kill the system in that time, or do you just need to isolate it? Just syncing the disks will take a few extra milliseconds and you probably don't want to miss that step. Instead of just killing the host, you could make sure the affected app isn't working (SIGABRT?) and kill all networking (flush iptables, change default to DROP).
I am very much new to linux and to this forum. I am working on one issue for a customer where they have 10+ Red Hat Linux 5.5 64 bits servers. They want to stop the tomcat process using the stop script (the script uses 'kill -15')
On some servers, the script works fine and stops the tomcat process within seconds.
On some servers, sometimes it stops quickly, sometimes it keeps running for minutes and finally customer has to use 'kill -9' command to stop tomcat. Logs are not indicating anything.
Do you have any idea why there is an intermittent behaviour of this script? How can we catch it in logs etc?
Processes may ignore the SIGTERM(15) signal.
From Wikipedia: http://en.wikipedia.org/wiki/Unix_signal
The SIGTERM signal is sent to a process to request its termination. Unlike the SIGKILL signal, it can be caught and interpreted or ignored by the process. This allows the process to perform nice termination releasing resources and saving state if appropriate.
If it's ignored the only option left is to kill it with SIGKILL(9) which can't be ignored.
I am trying to track down the reason why my WPF application is not ending cleanly while debugging. By 'cleanly' I mean that all the windows are closed, I can see various messages in the Output window showing that the app has ended but the process is still active and the 'Stop' button in the debugger is still active.
I call the Shutdown() method but something is stopping the application from ending. I am pretty sure it has something to do with the ethernet connection to an IO device but cannot see what I am doing wrong. (When I comment out the call to connect the device the app can exit cleanly)
I was wondering if VSE 2010 can list all active threads as this might give a clue as to what is still 'alive' after the main program ends. Or is there an external tool that might help here?
You should be able to use the Visual Studio Threads window to see which threads are still active. I'm not entirely sure this window is available in the Express edition (the documentation doesn't mention such a limitation), but should you not have it, then you can also use WinDbg to list all threads. WinDbg is part of the debugging tools for Windows. You might need to install the latest version of the Windows SDK to get it.
Use the debugger first. Debug + Break All, Debug + Windows + Threads to see what threads are still running. You can double-click one and use Debug + Windows + Call Stack to see what it is doing. The typical case is a thread you started but forgot to tell to terminate. The Thread.IsBackground property is a way to let the CLR abort a thread automatically for you.
Technically it is possible to have a problem with a device that prevents a process from shutting down. The Threads window would then typically show only one thread with an otherwise impenetrable stack trace. If you use Task Manager, Processes tab, View + Select Columns, tick Handles, then you may see only one handle still in use. The diagnostic then is that you have a lousy device driver on your machine that doesn't properly support I/O cancellation. Which could leave a kernel thread running that doesn't quit, preventing the process from terminating. Very unusual, look for the reasons given in the first paragraph first.
What are the advantages of "daemonizing" a server application over running the program in console mode?
Having it run as a daemon means you can
log out without loosing the service (which saves some resources)
do not risk loosing the service from an accidental ctrl-c
does not offer a minor security risk from someone accessing the terminal, hitting ctrl-c and taking your session
Essentially all 'real' services that are running 'in production' (as opposed to debug mode) run that way.
I think it is preventing from accidentally closing an app and you have one more terminal free.
But I personally don't see big difference between "screen" program and "daemonizing"
The main point would be to detach the process from the terminal so that the process does not terminate when the user logs out from the terminal. If you run a program in console mode, it will terminate when you log out, because this is the default behavior for a process when it receives a SIGHUP signal.
Note that there is more to writing a daemon than just calling daemon(3). See How to write a unix daemon for more information.
I have a daemon process which does the configuration management. all the other processes should interact with this daemon for their functioning. But when I execute a large action, after few hours the daemon process is unresponsive for 2 to 3 hours. And After 2- 3 hours it is working normally.
Debugging utilities for Linux process hang issues?
How to get at what point the linux process hangs?
strace can show the last system calls and their result
lsof can show open files
the system log can be very effective when log messages are written to track progress. Allows to box the problem in smaller areas. Also correlate log messages to other messages from other systems, this often turns up interesting results
wireshark if the apps use sockets to make the wire chatter visible.
ps ax + top can show if your app is in a busy loop, i.e. running all the time, sleeping or blocked in IO, consuming CPU, using memory.
Each of these may give a little bit of information which together build up a picture of the issue.
When using gdb, it might be useful to trigger a core dump when the app is blocked. Then you have a static snapshot which you can analyze using post mortem debugging at your leisure. You can have these triggered by a script. The you quickly build up a set of snapshots which can be used to test your theories.
One option is to use gdb and use the attach command in order to attach to a running process. You will need to load a file containing the symbols of the executable in question (using the file command)
There are a number of different ways to do:
Listening on a UNIX domain socket, to handle status requests. An external application can then inquire as to whether the application is still ok. If it gets no response within some timeout period, then it can be assumed that the application being queried has deadlocked or is dead.
Periodically touching a file with a preselected path. An external application can look a the timestamp for the file, and if it is stale, then it can assume that the appliation is dead or deadlocked.
You can use the alarm syscall repeatedly, having the signal terminate the process (use sigaction accordingly). As long as you keep calling alarm (i.e. as long as your program is running) it will keep running. Once you don't, the signal will fire.
You can seamlessly restart your process as it dies with fork and waitpid as described in this answer. It does not cost any significant resources, since the OS will share the memory pages.