I'm trying to use a cron job to call a healthcheck script I've written to check the status of a web app (api) I've written (a url call doesn't suffice to test full functionality, hence the custom healthcheck). The healthcheck app has several endpoints which are called from a shell script (see below), and this script restarts the bigger web app we are checking. Naturally, I'm having trouble.
How it works:
1) cron job runs every 60s
2) healthcheck script is run by cron job
3) healthcheck script checks url, if url returns non-200 response, it stops and start a service
What works:
1) I can run the script (healthcheck.sh) as the ec2-user
2) I can run the script as root
3) The cron job calls the script and it runs, but it doesn't stop/start the service (I can see this by watching /tmp/crontest.txt and ps aux).
It totally seems like a permissions issue or some very basic linux thing that I'm not aware of.
The log when I run as root or ec2-user (/tmp/crontest.txt):
Fri Nov 23 00:28:54 UTC 2012
healthcheck.sh: api not running, restarting service!
api start/running, process 1939 <--- it restarts the service properly!
The log when the cron job runs:
Fri Nov 23 00:27:01 UTC 2012
healthcheck.sh: api not running, restarting service! <--- no restart
Cron file (in /etc/cron.d):
# Call the healthcheck every 60s
* * * * * root /srv/checkout/healthcheck/healthcheck.sh >> /tmp/crontest.txt
Upstart script (/etc/init/healthcheck.conf)- this is for the healthcheck app, which provides endpoints which we call from the shell script healthcheck.sh:
#/etc/init/healthcheck.conf
description "healthcheck"
author "me"
env USER=ec2-user
start on started network
stop on stopping network
script
# We run our process as a non-root user
# Upstart user guide, 11.43.2 (http://upstart.ubuntu.com/cookbook/#run-a-job-as-a-different-user)
exec su -s /bin/sh -c "NODE_ENV=production /usr/local/bin/node /srv/checkout/healthcheck/app.js" $USER
end script
Shell script permissions:
-rwxr-xr-x 1 ec2-user ec2-user 529 Nov 23 00:16 /srv/checkout/healthcheck/healthcheck.sh
Shell script (healthcheck.sh):
#!/bin/bash
API_URL="http://localhost:4567/api"
echo `date`
status_code=`curl -s -o /dev/null -I -w "%{http_code}" $API_URL`
if [ 200 -ne $status_code ]; then
echo "healthcheck.sh: api not running, restarting service!"
stop api
start api
fi
Add path to start/stop command to your script:
#!/bin/bash
PATH=$PATH:/sbin/
or your full path to start and stop commands:
/sbin/stop api
you can check path to them using whereis:
$ whereis start
/sbin/start
Answer found in another question!
Basically the cron jobs operate in a limited environment, so in 'start [service]', the start command is not found!
Modifying the script to look like so makes it work:
#!/bin/bash
PATH="/bin:/sbin:/usr/bin:/usr/sbin:/opt/usr/bin:/opt/usr/sbin:/usr/local/bin:/usr/local/sbin"
...
Related
I need a script to run a bash script that stops nginx dows a bunch of other stuff and then starts nginx
the crontab looks like this
5 10 * * 0 sh ~/runnow/script.sh
and I edited the crontab well logged in as root.
the script runs fine however the lines for stop and start of nginx give the error service not found.
echo "Stopping Nginx"
service nginx-sp stop
gives
Stopping Nginx
/root/run/script.sh: 2: /root/run/script.sh: service: not found
if I manually run the script logged on as root it runs fine.
my thought is that crontab is not being run as root but searching for that it seems if you exited your crontab when logged in as root with crontab -e it should be run as root. and it must be since I am using ~/run/script.sh to run the script and the real path is /root/run/script.sh
so... im lost....
As cyrus pointed ...
use which to find service path in this way:
which service
Then add full path in your script.
I have a Docker container in which I have my Python tools installed, including my Luigi pipeline interface. I would like to run a shell script which kicks off my Luigi pipeline on a weekly basis using cron.
I have tried high and low to get cron to work within a Docker container. I cannot, for the life of me, get my crontab -e file to run.
In my file I have:
0 0 * * Sun /data/myscript.sh
followed by a new line. Cron is running in the background - ps aux | grep cron shows /usr/sbin/cron is running. Furthermore, in my /var/log/syslog file, I have:
/USR/SBIN/CRON[2037]: (root) CMD (/data/myscript.sh)
I've also tried using 0 0 * * Sun . /root/.bashrc ; sh /data/myscript.sh
However, my script does not run (when I run my script manually using bash myscript.sh, I get the expected results).
Suggestions?
Scheduled tasks won't run inside of a normal container since there is no scheduler running. The only active task will be that which you have elected to run via the CMD keyword or the Entrypoint.
In order to execute schedule tasks, it's more prudent to utilize the host scheduler and docker exec commands:
docker exec <container> <command>
docker exec <container> /data/myscript.sh
So you would end up with a cron on your host something like :
(Crontab Style)
0 * * * * root docker exec mycontainer /data/myscript.sh
If you have a cluster, you would have to query the cluster first to locate the container, or even have a script do it for you.
A container is meant to only one run main process. You either need to run crond as the main process for a container, or ensure that crond is running alongside your main process. This kind of breaks the contracts / point of containers, but sometimes it's easier to set it up this way. Instructions below:
My Dockerfile has the following ENTYPOINT:
ENTRYPOINT ["/startup.sh"]
And then within startup.sh I do a couple of things to spin up the container, but most importantly before executing the last command, I have this:
crond
exec start_my_service
crond starts the daemon that executes the crons, and start_my_service then becomes the primary process for my container.
I have a LAMP container with supervisor.
I add a simple cron
* * * * * root /bin/date >> /var/log/cron.log
from my Dockerfile
ADD ./crons/test /etc/cron.d/test
RUN chmod 0777 /etc/cron.d/test
I start cron via supervisor with a supervisor-cron.conf like this:
[program:cron]
command=/bin/bash -c "cron -f"
numprocs=1
autostart=true
autorestart=true
startretries=2
Cron starts fine and stays up and running. The strange thing is that no cronjob is running automatically [as it should] but when I execute docker exec lamp crontab /etc/cron.d/test the cron job starts and works as expected.
Am I missing something? Everywhere I have read that cron jobs are executed automatically by cron.
I solved it.
I tried both setting them up in /etc/crontab and /etc/cron.d/ .
Cron didn’t auto-start the cron jobs .
However, when I run docker exec lamp crontab /etc/cron.d/my_cronjob_file all played nice. This made me suspicious , and then I read this . So, after adding my_cronjob_file in the container [in the dockerfile] I added RUN crontab /etc/cron.d/my_cronjob_file . This essentially ‘installs’ the cronjob to the crontab table. [I don’t know the internals of cron/tab but that’s the gist I understood.] .
After that , the cron service comes up by supervisor and the cronjob runs like a charm.
This can be solved with the bash file, due to the layered architecture of the Docker, cron service doesn't get initiated with RUN/CMD/ENTRYPOINT commands.
Simply add a bash file which will initiate the cron and other services (if required)
DockerFile
FROM gradle:6.5.1-jdk11 AS build
# apt
RUN apt-get update
RUN apt-get -y install cron
# Setup cron to run every minute to print (you can add/update your cron here)
RUN touch /var/log/cron-1.log
RUN (crontab -l ; echo "* * * * * echo testing cron.... >> /var/log/cron-1.log 2>&1") | crontab
# entrypoint.sh
RUN chmod +x entrypoint.sh
CMD ["bash","entrypoint.sh"]
entrypoint.sh
#!/bin/sh
service cron start & tail -f /var/log/cron-2.log
If any other service is also required to run along with cron then add that service with & in the same command, for example: /opt/wildfly/bin/standalone.sh & service cron start & tail -f /var/log/cron-2.log
Once you will get into the docker container there you can see that testing cron.... will be getting printed every minute in file: /var/log/cron-1.log
I am getting a Django 1.6 set up started on a Linux (Debian Whiskey) server on Google Compute Engine. I've got Celery 3.1 running in the background to help with some processes. When I start a new instance (using a snapshot I've created), I always need to start Celery. I am looking for a way to start Celery automatically on server-load. This is particularly helpful if the server decides to restart, as they seem to do now and then. To achieve this, I've edited the rc.local file:
$ sudo nano /etc/rc.local
It used to contain the following:
exit 0
[ -x /sbin/initctl ] && initctl emit --no-wait google-rc-local-has-run || true
I've edited the file such that it now reads:
cd /home/user/gce_app celery -A myapp.tasks --concurrency=1 --loglevel=info worker > output.log 2> errors.log &
exit 0
[ -x /sbin/initctl ] && initctl emit --no-wait google-rc-local-has-run || true
The directory:
/home/user/gce_app
is where my Django project resides and the directory I need to be in to start Celery. However, after restarting the instance, when I type in:
$ celery status
Error: No nodes replied within time constraint.
Opening the errors.log file, I see:
/etc/rc.local: 14: /etc/rc.local: celery: not found
Surely the cd at the start of that code string should address this? Is there a way (within the Django project itself) to start the Celery instance when the project is started to make the code more platform-independent and immune to inevitable OS updates?
I think you're missing a semicolon between your 'cd' and celery invocations. Also, I suspect rc.local may not be searching your path, so you may need to give an absolute path to celery. e.g.
cd /home/user/gce_app; /usr/bin/celery ...
Alternatively, you might look at using a startup script from the GCE metadata to avoid needing to modify rc.local.
Since you seem to be using upstart this might help you:
description "runs celery"
start on runlevel [2345]
stop on runlevel [!2345]
console log
env VENV='/srv/myvirtualenv'
env PROJECT='/srv/run/mydjangoproject'
exec su -s /bin/sh -c 'exec "$0" "$#"' www-data -- /usr/bin/env PATH=$VENV:$PATH $VENV/python $PROJECT/manage.py celeryd
respawn
respawn limit 10 5
I have the following in a .conf file in the /etc/init/ directory of my CentOS server:
#!upstart
description "shortnr server for fmc.io"
author "Felix Milea-Ciobanu"
start on startup
stop on shutdown
respawn
respawn limit 10 30
script
export HOME="/root"
exec /usr/local/bin/node /var/www/fmc.io/nodejs/app.js >> /var/www/fmc.io/logs/shortnr.upstart.log 2>&1
end script
pre-start script
echo "[`date -u +%Y-%m-%dT%T.%3NZ`] (sys) Starting" >> /var/www/fmc.io/logs/shortnr.upstart.log
end script
pre-stop script
echo "[`date -u +%Y-%m-%dT%T.%3NZ`] (sys) Stopping" >> /var/www/fmc.io/logs/shortnr.upstart.log
end script
It's a pretty simple and straight forward upstart script. I named this service shortnr, after the nodejs software that the script starts up.
At the command line if I type in start shortnr I get something along the lines of shortnr start/running, process 28350.
However, I can't seem to access the nodejs server; If I do ps aux | grep shortnr at the command shell, nothing comes up.
If I do stop shortnr after running start, I get stop: Unknown instance:, meaning that the original service never started up.
The log file that I setup in the Upstart script looks something like this:
[2012-10-05T17:00:17.174Z] (sys) Starting
[2012-10-05T17:00:17.181Z] (sys) Starting
[2012-10-05T17:00:17.190Z] (sys) Starting
[2012-10-05T17:00:17.197Z] (sys) Starting
[2012-10-05T17:00:17.204Z] (sys) Starting
Basically the script is trying to start multiple times a second when I issued the start command, meaning that the service must be crashing on start or something and trying to respawn?
However, if I copy the command after exec and paste it in the shell prompt, the nodejs script starts up and runs properly.
So that means something must be wrong with my Upstart script.
If I try start/stop the service with the initctl command, I get the same results.
I'm running CentOS 6.3 and Upstart 0.6.5
Anyone have any idea what could be causing this or how to fix my script?
While I couldn't figure out the answer to my problem, I just ended up using forever instead: https://github.com/nodejitsu/forever-monitor
I am also running into similar problems on CentOS 6.3, Upstart 0.6.5 and Node.Js 0.10.5. I specifically upgraded Node so I could use the daemon module and be able to put the daemonized Node app under Upstart control.
Here's my /etc/init/job-worker.conf:
description "job-worker under Upstart/init control"
start on job-worker-start-event
stop on job-worker-stop-event
expect daemon
script
#setuid myuser
exec /root/BasicJobWorker/bin/basic-job-worker
#sleep 5
end script
respawn
respawn limit 10 5
And here's my basic-job-worker script:
\#!/usr/bin/env node
// this code is run twice
// see implementation notes below
console.log(process.pid);
// after this point, we are a daemon
require('daemon')();
// different pid because we are now forked
// original parent has exited
console.log(process.pid);
var BasicJobWorker = require('../lib/basic-job-worker.js');
new BasicJobWorker().boot();
I have tried using "expect fork", "expect daemon" as well as no expect at all. In all cases the job is respawned too fast and it is eventually stopped.