I want to auto restart my application "Fiware IoT Agent" if it stopped, the problem is that it depends of Mongo Db Data Base and the Mosquitto broker. My OS is centOS 7
Here is the commands that I use to launch my three application in the following order:
*Mongo:
/usr/local/iot/mongodb-linux-x86_64-3.0.5/bin/mongod --dbpath /usr/local/iot/mongodb-linux-x86_64-3.0.5/data/db$
*Mosquitto broker
/usr/sbin/mosquitto -c /etc/iot/mosquitto.conf &
pid=$!
echo $pid > /var/run/iot/mosquitto.pid
Iot Agent:
than I start my application using this command
export LD_LIBRARY_PATH=/usr/local/iot/lib
/usr/local/iot/bin/iotagent -i 192.168.1.11 -p 80 -v DEBUG -d /usr/local/iot/lib -c /etc/iot/config.json
how can I start my application if it stopped known that it depends of the other two application? If for example Mongo DB stopped, I must be able to restart it and then to restart my application.
CentOS 7 uses systemd. You can create systemd service for each of your applications and specify dependencies between them. And specify "Restart=always" for service which need to be auto restarted.
You can create your own watch dog code. When you start your application get the pid of the process and the pid of mongo DB.
Every couple of second like 10 seconds check that the pid of both process still exist, or you can also make the programs touch a file every couple of seconds as well then check the file modification time to see if the programs are still alive.
If the program hasn't touched the file or if you go jus the pid route and the pid doesn't exist. Then the program has died.
Restart the program and get the new pid and go about again in a forever while loop.
Related
I have an app created in .NET Core and I want to run it under Ubuntu, so to run it this is what I do:
sudo dotnet App.dll &
and to stop it I have to remember the process id and then run:
sudo -kill kill <procId>
The questions:
How should the .sh file look like to run/stop the service?
I mentioned "remember the process Id" because when I run the service and then run ps -a then I can see the process Id, but if I log off and run this command later then it does not show me the procId. Why is that? I also tried some other commands to show the running procs but with those commands/utils I was unable to distinguish my process.
If you would like to auto-start your dotnet application after reboot in Ubuntu, you can reference UpstartHowto - Community Help Wiki to write your own initscript.
When you run your application in the background, you can keep the proccess Id in the $! variable immediately. Please check the screenshot below:
After you log off and login again. If you want to kill your process, you can try this: kill `cat mymvc.pid`
Actually I'm on my way to switch to debian 9 for the new production servers of the company and want to provision them with ansible. So far, everything works fine, but now I'm stuck with redis-server.
By default, Debian 9 comes with redis version 3.2. I'm installing the package via apt-get install redis-server. After that, redis starts up as a daemon in the background. Now I want to apply some custom configuration, like binding to 2 different IPs (127.0.0.1 and the server IP).
After changing this as well as the daemonize option (to yes), redis is no longer willing to start in the background. Whenever doing either service redis-server start or /etc/init.d/redis-server start, the command just stucks.
journalctl -xe tells me, that the pid file is not readable (redis-server.service: PID file /var/run/redis/redis-server.pid not readable (yet?) after start-post: No such file or directory) even though it should be created according to init.d script:
start)
echo -n "Starting $DESC: "
mkdir -p $RUNDIR
touch $PIDFILE
chown redis:redis $RUNDIR $PIDFILE
chmod 755 $RUNDIR
After all, I can see, that both service redis-server start and /etc/init.d/redis-server are starting the server and I'm also able to connect to the server via redis-cli. But the damn process stucks.
Can anyone help? If you need further information, just let me know. I'll provide what ever possible if this solves the problem!
best
Chris
I had a similar situation on a Centos 7 server.
The resolution was to change supervised from no to auto
# By default Redis does not run as a daemon. Use 'yes' if you need it.
# Note that Redis will write a pid file in /var/run/redis.pid when daemonized.
daemonize yes
# If you run Redis from upstart or systemd, Redis can interact with your
# supervision tree. Options:
# supervised no - no supervision interaction
# supervised upstart - signal upstart by putting Redis into SIGSTOP mode
# supervised systemd - signal systemd by writing READY=1 to $NOTIFY_SOCKET
# supervised auto - detect upstart or systemd method based on
# UPSTART_JOB or NOTIFY_SOCKET environment variables
# Note: these supervision methods only signal "process is ready."
# They do not enable continuous liveness pings back to your supervisor.
supervised auto
When you run the process as daemon it need to interact with systemd for process management (if I read well some documentation).
Thanks
I have a systemd service for my spring boot application connected to consul server, behind haproxy. consul provides consul-template to automatically update the service location in haproxy configuration file via consul-template command.
consul-template takes a template file and writes to the final haproxy configuration file and then reload the haproxy.
Now, consul-template process needs to run in background always along with my application, so that as the application comes up, it can detect new application startup and update its location in the configuration file.
Here is my systemd service file for this.
[Unit]
Description=myservice
Requires=network-online.target
After=network-online.target
[Service]
Type=forking
PIDFile=/home/dragon/myservice/run/myservice.pid
ExecStart=/home/dragon/myservice/bin/myservice-script start
ExecReload=/home/dragon/myservice/bin/myservice-script reload
ExecStop=/home/dragon/myservice/bin/myservice-script stop
ExecStartPost=consul-template -template '/etc/haproxy/haproxy.cfg.template:/etc/haproxy/haproxy.cfg:sudo systemctl reload haproxy'
User=dragon
[Install]
WantedBy=multi-user.target
Now, when I start systemctl start myservice, my application starts and the call to consul-template also works, but consul-template process doesn't go in background. I have to press Ctl+C and then systemctl comes back and I have both my application and consul-template process running.
Is there way to run the consul-template process in background specified in ExecStartPost?
I was trying to add & at the end of the ExecStartPost command, but then consul-template complains that it is an additional invalid argument and it fails.
I was also trying to make the command as /bin/sh -c "consul-template command here...", but then this also doesn't work. Even nohup in this command wasn't working.
Any help is really appreciated.
A workaround would be to have a bash file as your entrypoint, add all you need in there, then it will all magically work
I was trying to accomplish the same task. I wanted to fire off some HTTP requests to Tomcat once the service had started, so that I could warmup our servers ahead of the first user request.
I went through a lot of trial and error with using trying to use ExecStartPost to fire off an async process, but actually worked. By calling a shell script, I could trigger off background processes, but from my testing Systemd appears to kill the process thread when ExecStartPost finishes, so any child processes end up getting killed too. I tried various combinations of using &, setsid, nohup, etc, even some Perl to try and trigger off the an executable in it's own thread, but as soon as the shell script exite from ExecStartPost any processes running where killed. It's possible there's some solution that would work using ExecStartPost, but I couldn't find it.
However, what did work is creating a new service (like #divinedragon mentions) which piggy backs off the service I wanted to monitor (in this case Tomcat).
Since it took me a little research to get something working the way I wanted, I wanted to share my solution in case it helps someone.
The first step is to create a new service (e.g. /usr/lib/systemd/system/tomcat-service-listener.service):
[Unit]
Description=Tomcat start/stop event listener
# make sure to stop the service when Tomcat stops
BindsTo=tomcat.service
# waits for both Nginx & Tomcat to be started before this service is started
After=nginx.service tomcat.service
[Service]
Type=oneshot
ExecStart=/path/to/your/script.sh start
ExecStop=/path/to/your/script.sh stop
RemainAfterExit=yes
TimeoutStartSec=300
[Install]
# When the service is enabled, forces this service to start when Tomcat is started
WantedBy=tomcat.service
Some notes on what is happening here:
The BindsTo make sure the service gets stopped when Tomcat is stopped. This triggers the ExecStop command.
The After make sure that on server reboot, this service does not start until both Nginx & Tomcat have started.
The WantedBy will create the wants symlink for Tomcat (when the service is enabled), which will force Tomcat to start this service any time it's restarted.
The RemainAfterExit=yes is necessary for the ExecStop to work. If you only care about triggering something when you're service is started and don't care about when the service is stopped, you can set this to no and remove the ExecStop line.
Make the TimeoutStartSec long enough for whatever task you plan on running.
To get this service working, you then need to do the following:
# make the service executable
chmod 664 /usr/lib/systemd/system/tomcat-service-listener.service
# make Systemd aware of the new service
systemctl daemon-reload
# register the service so it's started/stopped with Tomcat
systemctl enable tomcat-service-listener.service
Now all you need script to trigger off the logic you want. In my case, I wanted to warmup some servers once Tomcat started so my /path/to/your/script.sh looks something like:
#!/bin/sh
SCRIPT_MODE="$1"
LOGFILE=/var/logs/myscript.log
log_message() {
local MESSAGE="$1"
echo "$(date '+%Y-%m-%d %H:%M:%S') $MESSAGE" >> "$LOGFILE"
return 0
}
warmup_server() {
local SERVER_ADDRESS="$1"
local SERVER_DESCRIPTION="$2"
log_message "Warming up $SERVER_DESCRIPTION..."
# we want to track the time it took to warm up the server
local START_TIME=$(date +%s)
# server restarts can take a while for all services to start, so we must retry long enough for all relevant services to start
HTTP_STATUS=$(curl --insecure --location --silent --show-error --fail --retry 60 --retry-delay 2 --retry-max-time 240 --output /dev/null --write-out "%{http_code}" '$SERVER_ADDRESS')
# we want to track the time it took to warm up the server
local TOTAL_STARTUP_TIME=$(($(date +%s)-$START_TIME))
log_message "$SERVER_DESCRIPTION started in $TOTAL_STARTUP_TIME seconds... (Status: $HTTP_STATUS)"
return 0
}
# monitor when Tomcat has stopped
if [ "$SCRIPT_MODE" == "stop" ]; then
log_message "Tomcat listener shutting down..."
exit 0
elif [ "$SCRIPT_MODE" == "start" ]; then
log_message "Tomcat listener started..."
fi
# servers to warm up
warmup_server 'https://127.0.0.1' 'Localhost #1'
warmup_server 'https://127.0.0.2' 'Localhost #2'
This seems to be working exactly as I want. The service starts up when the server is reboot and starting/stopping/restarting Tomcat fires off the expected events. Since it's independent of the Tomcat service, I can restart this warmup script if needed. It also doesn't delay the Tomcat startup time, since it is its own service, therefore running asynchronously like I wanted.
When I run dsc cassandra on CoreOS(tarball) using telnet everything comes up fine. But when i close the telnet session, it kills the process. How do i keep the cassandra server running?
I tried sudo bin/cassandra and sudo bin/cassandra -f
both didnt help.
I have no issues in other OS.
Option Description
-f Start the cassandra process in foreground. The default is to start as background process.
-h Help.
-p filename Log the process ID in the named file. Useful for stopping Cassandra by killing its PID.
-v Print the version and exit.
When you are starting cassandra using -f it runs in foreground, hence it will stop as soon as terminal is closed. Same is true for background process.
This will happen with any application you run in telnet session.
You can try
sudo service cassandra start OR nohup bin/cassandra this will keep your application running even when terminal is closed
You need to run Cassandra as a systemd service, as described here: https://coreos.com/os/docs/latest/getting-started-with-systemd.html
Running in the foreground with cassandra -f as your ExecStart= command will allow systemd to manage the state of the process (ideally inside a container).
While this is a bit different than what you're used to, it will lead to an overall more stable mechanism since you'll be using an init system that understands dependency chains, restart and reboot behavior, logging, etc.
Run the process in a screen or tmux session. Detaching from the screen session should allow the process to keep running.
We would like to monitor our CouchDB installation using the default pid file method with MONIT, however although couchdb is working fine there is no pid file generated under /var/run/couchdb, there is only a couch.uri file.
Permissions on /var/run/couchdb are good (couch:couch) and service couchdb stop and start work fine, although for MONIT to stop/start we would need the /etc/init.d/couchdb start/stop option (which again isn't present).
For info we just installed using apt-get install couchdb on Ubuntu 14.04.
Any advice appreciated.
Best regards
RichBos
I have done this with an older version (1.3) of CouchDB installed from source. Please check if this is working for you:
check process couchdb with pidfile
/usr/local/var/run/couchdb/couchdb.pid
group database
start program = "/etc/init.d/couchdb start -u couchdb"
stop program = "/etc/init.d/couchdb stop -u couchdb"
if failed host 127.0.0.1 port 5984 then restart
if cpu is greater than 40% for 2 cycles then alert
if cpu > 60% for 5 cycles then restart
if 10 restarts within 10 cycles then timeout
If you have installed it via a package manager, you will most likely find the pid in /var/run/couchdb/couchdb.pid
The place of the pid file did not change since 1.3. So chances are good, that it's working for you.