We have about 40 computers running identical hardware and software. They all run Ubuntu 11.10. They all have just one user account to log in. The .profile file is set up to launch a daemon process. The code for the daemon is written in C.
Once in a few weeks, we get a report that the daemon is no longer running. This does not happen on all computers but just one or two. We cannot reproduce the problem consistently.
Looking at the code, the application quits when it receives either SIGHUP or SIGTERM.
As I understand, SIGHUP is generated when a user logs off. In our case, the user never logs off. I am wondering if it is possible that SIGHUP could have been generated for some other reason. Any other thought would be appreciated.
Well, there are a couple of things to note about SIGHUP. Firstly, its origin is from the concept of a hang-up, i.e. loss of connection to a console over something like a modem. In modern parlance this generally means it has lost its controlling tty. Unless you've taken care to detach from your tty, any program started in a given terminal will receive a SIGHUP when the terminal is closed. See here for details on how to do this in your program. Other options include:
running your program inside screen or tmux
run your program with nohup or some other daemonising framework
The other possibility is something is deliberately sending your process a SIGHUP which by "tradition" is often used to signal a process that it should re-read its configuration.
Signals can be sent using kill utility or kill syscall.
Of course, you can try and find out who is sending that signal or disconnecting your terminals or network connections, but there is simpler practical way to fix your problem.
When code is supposed to run as a daemon, but really isn't (just like yours), there is a wrapper that can turn any program into daemon. Surprise - this wrapper is called daemon! It has lots of options, probably most importantly for you, option to automatically restart your utility should it ever die for any reason.
If this command is not installed on your Ubuntu, just sudo apt-get install daemon, and man daemon to get started.
Related
I was running gdb over an SSH shell. After a while the shell disconnected due to being idle. On reconnecting I see the gdb instance still running.
How do I take control of the running gdb instance?
Can I start a new gdb instance and take over the session from the running gdb instance?
Note: This is not about keeping the SSH session alive. This is more about taking control of gdb from another shell instance. Regardless of whether it is running from SSH or locally.
You probably can't do anything with the gdb that is already running. It probably no longer has a controlling tty at all. What you probably want to do is kill it and then start a new gdb process, but do it inside a program like tmux or screen. If you do that, then if/when you get disconnected you can easily reattach to the tmux/screen session any time. Just check out the manual for those programs. They do pretty much the same thing. I think tmux is a little more powerful.
This question sounds very stupid to me, but if this is somehow possible it would be really helpful.
My application is crashing and I need to debug it. And I run this application in another computer, via SSH (it's an HTTP server). If I could leave a terminal running the application over GDB and SSH all the time, I'd be able to find the bugs. But I don't have a free computer to do that. What can I do? Is there a way to start GDB with nohup(1) plus &> and stuff like that, so I can see GDB output (where command, for example) later?
A classical Unix program called screen is your friend (or its competitor tmux). It allows to keep a virtual console open across multiple logins:
screen
starts such a session; using you can detach from that; using
screen -r
you can reconnect to it later.
However, you don't even need that; just make your program leave a core dump when it crashes; ulimit -c unlimited says "every program that crashes leaves a core dump"; you can then just open the core dump using gdb later on, and everything will be as if you ran the program inside gdb when it crashed.
gdb core.123456
I'm currently working on an automation script that connects users via telnet to different ports (for different command interpreters) on an embedded system. Since the amount of available memory is pretty low on the system, the number of telnet sessions is limited. If a user doesn't close the telnet session normally, the server-side session will just hang and use up an available telnet session. Is there a way in tcl/expect to send commands/execute a procedure before an xterm window closes?
I did some googling, at it turns out expect supports the trap command which allows to run a script when a certain Unix signal is sent to the process hosting the interpreter.
It appears you have to trap SIGHUP but may be you'll also need to trap SIGTERM and/or SIGQUIT. man 7 signal — if on a Linux-based OS — for more info (on a different OS flavor the manual page section may be different).
Are there any watchdog tools or libraries on Linux for the following purpose? I would like to build a watchdog executable which starts 2 processes and restarts them if:
processes crash
processes become unresponsive (e.g. hang for some reason)
Internet search found watchdog.c but I am not sure if that can be used for my purposes, it looks pretty low level.
I could run my processes as init programs (daemons) as suggested here, but I am not sure if Linux would then recognize that the process is hanging (e.g. due to a deadlock)
We use monit here: http://mmonit.com/monit/ it will let you do the restart thing it is also highly customizable regarding how to check and how to react via scripts
I'm pretty new to linux...
I was running an install script for something on my ubuntu vps and I got disconnected from the server. I'm able to log back in again but I'm worried this install script might be waiting for prompts in the background.
What's the best action to take in this situation? Should I just run the script again, or should I do some kind of cleanup first?
Thanks
When you disconnect (from the terminal), the install script receives a signal (SIGHUP) which usually leads to program termination. If it is stuck, you probably should kill and re-run it, but there is no 100% confidence it will work.
Next time use screen. If you disconnect, you can then reconnect to the running screen instance using screen -dr. It has many other features you can read about in the manual or numerous tutorials.