In the Supervisord conf files you can specify to autorestart a certain program with:
autorestart=true
But is there an equivalent for [Supervisord] itself?
What is the recommended method of making sure Supervisord continues running unconditionally, especially if the Supervisord process gets killed.
Thanks!
Actually your question is a particular application of the famous "Quis custodiet ipsos custodes?" that is "Who will guard the guards?".
In a modern Linux system the central guarding point is init process (the process number 1). If init dies, the Linux kernel immediately panics, and thus you have to go to your data center (I mean go afoot) and press reset button. There're a lot of alternative init implementations, here is one of those "comparison tables" :)
The precise answer how to configure a particular init implementation depends on what init version you use in that system. For example systemd has its own machinery for configure service restart upon their deaths (directives Restart=, RestartSec=, WatchdogSec= etc in a corresponding unit-file. Other init implementations like Ubuntu Upstart also has its analogues (respawn directive in a service configuration file). Even old good SysV init has respawn option for a service line in /etc/inittab, but usually user-level services aren't started directly inittab, only virtual console managers (getty, mgetty etc)
Related
Is there any way to detect in a program that it is run by systemd as a daemon?
systemd API
sd_booted()
is used to detected if the whole system is booted by systemd, but says nothing about the program itself.
Thanks
Get the parent process id and see whether that process is systemd.
Starting from systemd v232, an environment variable INVOCATION_ID is given to all processes started as (part of) a service unit. This is a nice trait from systemd and not any other service manager, so it can be used as a convenient way to detect systemd, but not necessarily reliable.
Personally I use this to disable timestamp in logging as systemd journal already does that.
You could set a magic environment variable in the daemon's service file and look for this variable.
Context:
On a linux (RedHat family) system, I have an init-script-based service that is started/stopped manually most of the time, and in general is only run in response to specific, uncommon situations. The init scripts are thin wrappers around code that I do not have control over.
If the service is killed without running the stop action on its init script, it is aborted uncleanly and leaves the system in a broken state that requires manual intervention to fix.
When the systems running the service shut down, they kill it uncleanly. I can register the service with chkconfig such that it gets shut down via the init script when the host shuts down; this solves the problem on a per-host basis.
Question:
I would like to automate the configuration of this service to stop-at-shutdown via Puppet.
How can I tell Puppet to register a service with chkconfig such that the service will be stopped via the init script when the system shuts down, but will not otherwise be managed by Puppet?
What I've Tried:
I made a hokey exec statement in Puppet that calls chkconfig directly, but that feels inelegant (and will probably break in some way I haven't thought of).
I played around with the noop flag to the service type in Puppet, but it didn't seem to have the desired effect.
Puppet does not have any built-in support for configuring which runlevels a service runs in, nor any built-in, generalized support for chkconfig. Ordinarily it is a service-installation responsibility to register the service with chkconfig; services that are installed from the system RPMs are registered that way.
Furthermore, chkconfig recognizes structured comments at the top of initscripts to determine which runlevels the service will run in by default, according to LSB convention. A proper initscript need only be registered with chkconfig to have the default runlevels set -- in particular, for it to be set to be stopped in runlevels 0 and 6, which is what you're after.
If you're rolling your own initscripts and deploying them manually or directly via Puppet (as opposed to packaging them up and installing them via Yum) then your best bet is probably to build a defined type that manages the initscript and its registration. You do not need and probably do not want a Service resource for it, but a File resource to put the proper file in place and an Exec resource to handle registration sounds about right.
I have a script to start and stop my services. My server is based on Linux. How do I automate the process such that when OS is shutdown the stop script runs and when it is starting up, the start script runs?
You should install init script for your program. The standard way is to follow Linux Standards Base section 20 subsections 2-8
The idea is to create a script that will start your application when called with argument start, stop it when called with argument stop, restart it when called with argument restart and make it reload configuration when called with argument reload. This script should be installed in /etc/init.d and linked in various /etc/rd.* directories. The standard describes a comment to put at the beginning of the script and a uitlity to handle the installation.
Please, refer to the documentation; it is to complicated to explain everything in sufficient detail here.
Now that way should be supported by all Linux distribution. But Linux community is currently searching for better init system and there are two new, improved, systems being used:
systemd is what most of the world seems to be going to
upstart is a solution Ubuntu created and sticks to so far
They provide some better options like ability to restart your application when it fails, but your script will then be specific to the chosen system.
I'm interested in how rebooting is implemented in Linux. When I press ctrl-alt-del or click "restart" in the menu bar, what happens next?
Thanks!
brings the system down in a secure way. All logged-in users are notified that the system is going down, and login(1) is blocked. It is possible to shut the system down immediately or after a specified delay. All processes are first notified that the system is going down by the signal SIGTERM.
It does its job by signalling the init process, asking it to change the runlevel. Runlevel 0 is used to halt the system, runlevel 6 is used to reboot the system, and runlevel 1 is used to put to system into a state where administrative tasks can be performed;
So basically reboot calls the "shutdown".
Quick answer is that all the scripts that are in /etc/rc6.d are executed.
scripts that start with "K" are executed with the "stop" parameter.
scripts that start with "S" are executed with the "start"parameter.
For more you can start reading about runlevels here: http://en.wikipedia.org/wiki/Runlevel
There are different init systems on Linux, and they also control what happens on restart/shutdown. See https://unix.stackexchange.com/questions/18209/detect-init-system-using-the-shell to tell which you're using.
If you're using SysVinit, then there is a runlevel associated with the overall system status. The init system will first run all the kill scripts associated with your current runlevel and then the start scripts associated with runlevel 6. If your current runlevel was 5, it would run /etc/rc5.d/K* and then /etc/rc6.d/S*. They might be in another directory, such as /etc/init.d/rc5.d/k*, depending on your Linux distribution.
If you're using systemd, then instead of having an overall "runlevel", there would be a list of defined targets and services. A list of targets is essentially a runlevel. These are defined in .service and .target files under /etc/systemd. There will likely be a "reboot.target" defined under there, and other services with a dependency on that will be run on reboot. See the systemd homepage or this stackexchange question for an example.
Some Ubuntu versions also use upstart, but I think it's been replaced by systemd in more recent versions. If you are using upstart, see this guide or this askubuntu question.
One thing to be careful of is that regardless of which init system you're using you may be using init scripts generally associated with another one. So you may be using sysVinit, but some of the rc*.d scripts may be links to things that invoke systemd scripts. Or vice-versa.
I am a node.js developer. I use Amazon ec2 to deploy my node.js apps.
I want to have my node.js service running permanently - restarted if it fails for any reason.
I came across 2 tools . Forever and Upstart
Is there any advantages of using one over the other ?
Is there any other tool which is better ?
Upstart is a system service controller, similar to SysV Init and will start/stop/restart essentially any service registered for it, Node.js-based or not, and it will also automatically start services on system start for you. But Upstart is essentially specific to Ubuntu, and Upstart-specific services won't run on other Linux distros.
Upstart has a SysV Init compatibility layer that you could target,instead, to maintain as broad of a compatibility layer as possible.
Forever is a Node.js application that monitors and restarts other Node.js applications as needed, and as defined by its configuration JSON. Lots of options and fine-grained control over your service without the effort that would be needed to duplicate it in a custom SysV Init script. However, Forever isn't a system service, so if the server is restarted, you'll have to manually start your forever scripts again.
Beyond that, if all you need is something that will restart your script if/when it crashes, and you don't care about it starting automatically on system start, all you need is a bash script as simple as:
#!/bin/bash
while true
do
node ./myScript.js
done
Just to correct a misleading statement in the accepted answer...it is not true that upstart is an Ubuntu-only technology. See:
https://serverfault.com/questions/291546/centos-6-and-upstart
http://searchenterpriselinux.techtarget.com/tip/RHEL-6-ditches-System-V-init-for-Upstart-What-Linux-admins-need-to-know
http://en.wikipedia.org/wiki/Upstart#Adoption
With that, I think it is a much more compelling solution.