inittab not restarting service after service crash in Red Hat 6.7 - linux

NOTE: I am running Red Hat 6.7
I have a service that is configured with the Linux init system to start a process as a service when the machine boots. This was done by doing this one-time configuration from the command line:
ln -snf /home/me/bin/my_service /etc/init.d/my_service
chkconfig --add my_service
chkconfig --level 235 my_service on
When the OS reboots, the service starts as expected.
I ALSO need the service to be restarted if the service (my_service) crashes. From what I've read, all I need to do is add an entry to /etc/inittab that looks like this:
mysvc:235:respawn:/home/me/bin/my_service_starter
Where my_service_starter looks like:
#!/bin/bash
/home/me/bin/my_service start
My understanding is that when the init system detects that my_service is not running, it will attempt to restart it by running "my_service_starter".
However this does not seem to be working.
I need to understand how to tell the Linux init system to restart my service when the service crashes.

Given an entry like:
mysvc:235:respawn:/home/me/bin/my_service_starter
Then inittab will:
call /home/me/bin/my_service_starter
which will call /home/me/bin/my_service start
...and then exit, so init will thing your service has failed
so init will call /home/me/bin/my_service_starter again
...and so forth, which will result in init deciding that your script is respawning too fast, after which it will ignore it completely.
A process started by inittab is not expected to exit. If you really want to use inittab to maintain your service, you could remove /etc/init.d/my_service, and then in /etc/inittab you would have something like:
mysvc:235:respawn:/home/me/bin/my_service
And you would need to ensure that my_service runs in the foreground (some programs automatically daemonize by default, although these will often have some sort of --run-in-foreground flag).
If you upgrade to CentOS 7 or something else with systemd, this all becomes easier.
You can also investigate "third-party" process supervisors like "supervisord" or "runit" that you could use for process monitoring/restarting on CentOS 6.
Update
As mangotang points out, and I forgot, RHEL 6 actually shipped with upstart, even though it used almost exclusively SysV-style init scripts. So a better solution would be to create an upstart service instead. There are some reasonable getting-started docs here.

On RHEL 6.X, at top of the /etc/inittab file it says:
# inittab is only used by upstart for the default runlevel.
#
# ADDING OTHER CONFIGURATION HERE WILL HAVE NO EFFECT ON YOUR SYSTEM
RHEL 6.X uses Upstart instead of the System V init system. See the man pages for initctl(8) and initctl(5), or ask Google about Upstart.

Related

Make chosen version of Elasticsearch run as a service in Linux

I have an issue with later versions of ES, so have to use 7.10.2 currently.
This means that the previous method I used to install ES as a service, i.e. apt-get, doesn't work You can't choose an older version this way: it currently installs 7.16.3.
So I followed the procedure on this page for 7.10: everything worked: I was able to run ES as an app and also as a "daemon". Clearly I could simply put the "daemon" startup line in a script which runs on boot.
But what's the optimum way of turning this "daemon arrangement" into a service which you can control with systemctl, and which starts automatically when the machine boots?
PS I don't want to get involved with Docker. I'm sure that's a useful thing but I'm convinced there is a simpler way of doing it, using available Linux sys tools.
I found a workaround... this doesn't in fact create a service of the "systemd" type which can be controlled by systemctl. There seem to be one or two problems which make this non-trivial.
1) You can't start ES as root! I assume (not sure) that most services are being run by root. Anyway this was something I couldn't find a solution to.
2) I am not sure whether a shell script file called by a service is allowed to end... or should continue endlessly: initially I thought this would be sufficient. This is a shell script (run_es_daemon.sh) which does indeed start up ES (as a daemon process) when run by manually in a terminal. There is no issue to do with the fact that the script ends and you then close the terminal: the daemon process continues to run:
#!/bin/bash
# start ES as a daemon...
cd /home/mike/Elasticsearch/elasticsearch-7.10.2
./bin/elasticsearch -d -p pid
... but it never worked using a xxx.service file in /etc/systemd/system/ (maybe because of 1) above). So I also tried adding these lines under the above ones:
while true
do
echo "bubbles"
sleep 60
done
... didn't work either.
In the end I found a simple workaround solution was to start up the daemon process by using crontab:
#reboot /home/mike/sysadmin/run_es_daemon.sh
... but I'd still like to know how to set it up as a true service, which starts at boot...

How to execute automatically command at rapsbian startup

I want my raspberry to execute 2 commands when he starts but i don't know how to setup it
the commands are
cd /var/www/restaurant && php -S 10.0.0.1:8000 -t public
i have tried to edit /etc/rc.local and add my command but it doesn't work
The "official" way to run a program at boot time on systemd-based Raspbian systems (and, in fact, most modern Linux distributions) is to create a systemd unit file. There are specific instructions for Raspbian here:
https://www.raspberrypi.org/documentation/linux/usage/systemd.md
The entry After=network.target is particular relevant in this case, because I imagine your program will need network interfaces to be up.
rc.local is a hold-over from the SysV init days, and I've heard reports of it not working reliably in Raspbian. Creating a systemd unit file provides a simple way to test the service using systemctl without actually having to reboot. If it fails on boot, you'll probably need to use journalctl to see the error messages.

Running a terminal command permanently

I am currently hosting my database for free on Openshift and have my program running on a linux box on my local server. I need to pass the data from the program to my openshift database. I want to run the linux box headless.
To do this I run the command:
rhc port-forward -a webapp
My question is how can I run this command permanently without it timing out (some checking to see if process is running?) and without a terminal running (background process)?
You could add that command in the startup settings of your Linux computer. So a systemd configuation, or an init one (details could depend upon your particular distribution and system). See systemd(1) and/or sysvinit
You could also use crontab(5). It can be used for periodic tasks, but also for started once tasks, thru some #reboot entry.
At last, you might use batch facilities, look into at (& batch)
Perhaps you may just want nohup(1) (or screen(1)...)

Why is my script to start UWSGI not functioning at bootup?

I wonder if you can help.
I am running the following versions:
OS: SMP Debian 3.2.81-1 x86_64
uWSGI: uWSGI 2.0.11.2
I installed uWSGI manually, as I want to use a specific version. Using the following commands: -
apt-get install build-essential psmisc python-dev libxml2 libxml2-dev python-setuptools
cd /opt/
wget http://projects.unbit.it/downloads/uwsgi-2.0.11.2.tar.gz
tar -zxvf uwsgi-2.0.11.2.tar.gz
mv uwsgi-2.0.11.2/ uwsgi/
cd uwsgi/
python setup.py install
I am trying to replicate the setup on another server that the project is already working on in a live environment (I am essentially setting up a test server environment).
The original server has uWSGI running on boot. To figure out how this is happening, I used
htops
I've been able to identify that uWSGI is running on the existing server with a set of command line switches. I've managed to track down the script that initialises uWSGI with these switches in the init.d folder.
I copied this script to my test server, and ran it using
service script.sh start
After various troubleshooting, mainly involving permissions on socket folders etc, now when I run this script it starts, and if I run htop I can see uWSGI is running and it has the exact same command switches I need.
I thought simply putting the script in init.d and giving it execute permission
chmod +x script.sh
Would be enough so that it starts when the server is switched on... but this appears to not be the case. Because when I issue
reboot
At the terminal, the terminal reboots but when I go into htops, and check for the uWSGI process it is not running.
If however directly after reboot I issue the following command
service script.sh start
The service starts just fine, and I can once again see it in htops.
Research online lead me to the suggestion that I should try to set the script to run automatically using chkconfig. I installed chkconfig using
apt-get chkconfig
and then ran the following command
chkconfig --list
I noticed that all the runtime levels where set to off for the script I am trying to get to execute on boot.
I ran the following command
chkconfig /etc/init.d/script.sh on
And now when I check the script runtime switches with chkconfig, it shows me the following output for my script:
script.sh 0:off 1:off 2:on 3:on 4:on 5:on 6:off
However when I reboot the uWSGI process is still not starting.
Yet if I simply type
service script.sh start
At the terminal the service runs ok, and uWSGI runs fine.
How can I set the script to run when the server restarts?
Edit:
Further research on the live server that is working fine, has determined that it does not appear to be using systemd to launch uWSGI on startup. I logged into the live server and while there is a
/etc/systemd
folder, it has just one folder in it system and no files. The system folder has the following files in it:
multi-user.target.wants sockets.target.wants syslog.service
So there does not appear to be anything uWSGI related in here.
Also what is making me think this is likely something to do with the
/etc/init.d
folder, is that when I run htop and examine the running services (or daemons) not quite sure of the correct terminology in linux. uWSGI is showing in here as running with a signature of command line switches, and the script I have found in /etc/init.d has this exact uWSGI command and same signature of switches, so I'm fairly convinced this is the part of the system that is starting the uWSGI daemon , I just can't figure out what I need to do , to get it to run apart from copying the same file to /etc/init.d on the new server and giving it execute permission.
The OS of the live server is :
SMP Debian 3.2.73-2+deb7u1 x86_64
and the OS I am running on the new server is
SMP Debian 3.2.81-1 x86_64
So they seem fairly similiar? Although I'm not sure how significant the 8 incremements in the least significant digits in the version number is.
On the new server there is no /etc/systemd folder , and on the live server there is a /etc/systemd as explained above. So it does appear to have been installed seperately to the main OS install (as I have a later version of Debian and it wasn't installed on my system by default) - so perhaps there is something related to systemd that is causing the script to start on the live server, but I'm not too sure.
Jessie
In the recent Debian (Jessie) the initv scripts do not work the way they did. And given your kernel version you are not running a Debian that uses initv scripts. The current Debian uses systemd and scripts in /etc/rc.d are run by compatibility features of systemd (the service command is now a systemd command that tries to behave like the old initv command).
You have two options:
Add a line calling the script from /etc/rc.local:
/etc/rc.d/script.sh
This is a rather dirty fix, since it depends on another compatibility feature of systemd. Also, the location of the script does not matter anymore.
Write a full systemd service for uwsgi (this is what I do, and what is recommended by the uwsgi documentation). You would need to create a file called /etc/systemd/system/uwsgi.service with a content similar to:
[Unit]
Description=uwsgi emperor
After=rsyslog.service
[Service]
PIDFile=/run/uwsgi-emperor.pid
ExecStart=/bin/uwsgi --ini /etc/uwsgi/emperor.ini
ExecReload=/bin/uwsgi --reload /run/uwsgi-emperor.pid
ExecStop=/bin/uwsgi --stop /run/uwsgi-emperor.pid
Restart=always
KillSignal=SIGQUIT
Type=notify
StandardError=syslog
NotifyAccess=all
[Install]
WantedBy=multi-user.target
I use the emperor mode (which is also the mode recommended by uwsgi for use with systemd), although it is possible to hack it to run a single process uwsgi (see further reading below).
You will also need to enable the service to be used by the multi-user.target, which will run at boot. You need to perform this as root:
systemctl enable uwsgi.service
And uwsgi will start with the next boot (it will not start straight away, to make it start you need systemctl start uwsgi.service).
Further reading:
The Arch linux wiki about systemd is very thorough
The Debian wiki on systemd is good, but outdated in some places (notably, it tells you that you need to install it which is not the case in Jessie)
Weezy
You're mixing things up a little there: chkconfig is a script of the RedHat family of OSes. Making it work for Debian was not easy in the past, and I do not believe it is easy to do so now.
Weezy still uses the initv rc.d folders alright, for each runlevel one rc.d folder:
/etc/rc.d/rc0.d/
/etc/rc.d/rc1.d/
/etc/rc.d/rc2.d/
/etc/rc.d/rc3.d/
/etc/rc.d/rc4.d/
/etc/rc.d/rc5.d/
/etc/rc.d/rc6.d/
You can check the runlevel you are in with the (appropriately named) runlevel command. Then you need to check whether there is a softlink to the script in the correct /etc/rc.d/rc*.d folder. If there is no softlink to the script you need to add it with something of the lines:
ln -s /etc/rc.d/init.d/script.dh /etc/rc.d/rc$(runlevel | cut -d ' ' -f 2).d/script.sh
And that is almost all about how initv scripts work. If you are going into runlevel 2 when the machine boots (I believe that's the default on Debian), what init performs is simply service <script> start for every file in /etc/rc.d/rc2.d.

Linux Invoke custom script during OS shutdown

I wrote a simple JAVA application which runs as a service. When my application is up and running, I maintain the PID in a file which will be used while stopping the application.
Issue:
When I restart the OS the stop script is not called (not sure how to make this happen) and the old PID is left as it is in the PID file. Now, after reboot (which start my app) when I stop the app using stop script now my stop script will try to clean up all the PID listed in the file. Most of the time, I will get "No such process". But there are chance the same PID might have been used for some other process
Question:
How I can make sure my stop script will be invoked when I shutdown ore reboot the OS? I am looking a solution for RHEL environment.
I think your are looking for a init script. (startup/shutdown services at different run levels)
This is a good reference
http://blog.rimuhosting.com/2009/09/30/one-java-init-script/
this has a good refernce to Linux init.d scripts
http://coreymaynard.com/blog/creating-a-custom-initd-script-on-fedora/
Hope it helps
If you are looking for scripts that run after reboot, I guess you can write the script in /etc/rc.local and then you can start your service.
This script will run after all your init scripts have run while your machine starts. Using this you can delete the old PID file.

Resources