I have a shell script that does a backup. I set this script in a cron but the problem is that the backup is heavy so it is possible to execute a second rsync before the first ends up.
I thought to launch rsync in a script and then get PID and write a file that script checks if the process exist or not (if this file exist or not).
If I put rsync in background I get the PID but I don't know how to know when rsync ends up but, if I set rsync (no background) I can't get PID before the process finish so I can't write a file whit PID.
I don't know what is the best way to "have rsync control" and know when it finish.
My script
#!/bin/bash
pidfile="/home/${USER}/.rsync_repository"
if [ -f $pidfile ];
then
echo "PID file exists " $(date +"%Y-%m-%d %H:%M:%S")
else
rsync -zrt --delete-before /repository/ /mnt/backup/repositorio/ < /dev/null &
echo $$ > $pidfile
# If I uncomment this 'rm' and rsync is running in background, the file is deleted so I can't "control" when rsync finish
# rm $pidfile
fi
Can anybody help me?!
Thanks in advance !! :)
# check to make sure script isn't still running
# if it's still running then exit this script
sScriptName="$(basename $0)"
if [ $(pidof -x ${sScriptName}| wc -w) -gt 2 ]; then
exit
fi
pidof finds the pid of a process
-x tells it to look for scripts too
${sScriptName} is just the name of the script...you can hardcode this
wc -w returns the word count by words
-gt 2 no more than one instance running (instance plus 1 for the pidof check)
if more than one instance running then exit script
Let me know if this works for you.
Test both for presence of pid file and status of the running process like this:
#!/bin/bash
pidfile="/home/${USER}/.rsync_repository"
is_running =0
if [ -f $pidfile ];
then
echo "PID file exists " $(date +"%Y-%m-%d %H:%M:%S")
previous_pid=`cat $pidfile`
is_running=`ps -ef | grep $previous_pid | wc -l`
fi
if [ $is_running -gt 0 ];
then
echo "Previous process didn't quit yet"
else
rsync -zrt --delete-before /repository/ /mnt/backup/repositorio/ < /dev/null &
echo $$ > $pidfile
fi
Hope this helps!!!
Related
I am having this script which looks for the process filebeat and restarts it if is not running. Cron runs this script every 5 minutes. Most of the time this works fine except sometime it creates multiple filebeat process. Can someone please point out what is the issue in my script.
#!/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
service=filebeat
servicex=/usr/share/filebeat/bin/filebeat
pid=`pgrep -x "filebeat"`
if [ $pid > /dev/null ]
then
echo "$(date) $service is running!!!"
else
echo "$(date) starting $service"
cd /home/hpov/beats/filebeat
./filebeat -c filebeat.yml &
fi
#!/bin/bash
pidof script.x86 >/dev/null
if [[ $? -ne 0 ]] ; then
echo "Restarting script: $(date)" >> /var/log/script.txt
/etc/script/script.x86 &
fi
Super easy :D
I have BBB based custom Embedded Linux based board with busybox shell(ash)
I have a situation where my script must run in background with following condition
There must only one instance of the script.
wrapper script need to know if script started successfully in background or not.
There is another wrapper script which starts and stops my script, wrapper script is as mentioned below.
#!/bin/sh
export PATH=/bin:/sbin:/usr/bin:/usr/sbin
readonly TEST_SCRIPT_PATH="/home/testscript.sh"
readonly TEST_SCRIPT_LOCK_PATH="/var/run/${TEST_SCRIPT_PATH##*/}.lock"
start_test_script()
{
local pid_of_testscript=0
local status=0
#Run test script in background
"${TEST_SCRIPT_PATH}" &
#---------Now When this point is hit, lock file must be created.-----
if [ -f "${TEST_SCRIPT_LOCK_PATH}" ];then
pid_of_testscript=$(head -n1 ${TEST_SCRIPT_LOCK_PATH})
if [ -n "${pid_of_testscript}" ];then
kill -0 ${pid_of_testscript} &> /dev/null || status="${?}"
if [ ${status} -ne 0 ];then
echo "Error starting testscript"
else
echo "testscript start successfully"
fi
else
echo "Error starting testscript.sh"
fi
fi
}
stop_test_script()
{
local pid_of_testscript=0
local status=0
if [ -f "${TEST_SCRIPT_LOCK_PATH}" ];then
pid_of_testscript=$(head -n1 ${TEST_SCRIPT_LOCK_PATH})
if [ -n "${pid_of_testscript}" ];then
kill -0 ${pid_of_testscript} &> /dev/null || status="${?}"
if [ ${status} -ne 0 ];then
echo "testscript not running"
rm "${TEST_SCRIPT_LOCK_PATH}"
else
#send SIGTERM signal
kill -SIGTERM "${pid_of_testscript}"
fi
fi
fi
}
#Script starts from here.
case ${1} in
'start')
start_test_script
;;
'stop')
stop_test_script
;;
*)
echo "Usage: ${0} [start|stop]"
exit 1
;;
esac
Now actual script "testscript.sh" looks something like this,
#!/bin/sh
#Filename : testscript.sh
export PATH=/bin:/sbin:/usr/bin:/usr/sbin
set -eu
LOCK_FILE="/var/run/${0##*/}.lock"
FLOCK_CMD="/bin/flock"
FLOCK_ID=200
eval "exec ${FLOCK_ID}>>${LOCK_FILE}"
"${FLOCK_CMD}" -n "${FLOCK_ID}" || exit 0
echo "${$}" > "${LOCK_FILE}"
# >>>>>>>>>>-----Now run the code in background---<<<<<<
handle_sigterm()
{
# cleanup
"${FLOCK_CMD}" -u "${FLOCK_ID}"
if [ -f "${LOCK_FILE}" ];then
rm "${LOCK_FILE}"
fi
}
trap handle_sigterm SIGTERM
while true
do
echo "do something"
sleep 10
done
Now in above script you can see "---Now run the code in background--" at that point I am sure that either lock file is successfully created or instance of this script is already running. So Then I can safely run other code in background and wrapper script can check for lockfile and find out if the process mentioned in the lock file is running or not.
can shellscript itself make it to run in background ?
if not is there a better way to meet all the conditions ?
I think you can look into job control built-in, specifically bg.
Job Control Commands
When processes say they background themselves, what they actually do is fork and exit the parent. You can do the same by running whichever commands, functions or statements you want with & and then exiting.
#!/bin/sh
echo "This runs in the foreground"
sleep 3
while true
do
sleep 10
echo "doing background things"
done &
I have Linux centralize server – Linux 5.X.
In some cases on my Linux server the get_hosts.ksh script could be run from some other different hosts.
For example get_hosts.ksh could run on my Linux machine three or more times at the same time.
My question:
How to avoid running multiple instances of process/script?
A common solution for your problem on *nix systems is to check for a lock file existence.
Usually lock file contains current process PID.
This is an example ksh script:
#!/bin/ksh
pid="/var/run/get_hosts.pid"
trap "rm -f $pid" SIGSEGV
trap "rm -f $pid" SIGINT
if [ -e $pid ]; then
exit # pid file exists, another instance is running, so now we politely exit
else
echo $$ > $pid # pid file doesn't exit, create one and go on
fi
# your normal workflow here...
rm -f $pid # remove pid file just before exiting
exit
UPDATE: Answering to OP comment, I add handling program interruptions and segfaults with trap command.
The normal way of doing this is to write the process id into a file. The first thing the script does is check for the existence of the file, read the pid, check if a process with that pid exists, and for extra paranoia points, if that process actually runs the script. If yes, the script exits.
Here's a simple example. The process in question is a binary, and this script makes sure the binary runs only once. This is not exactly what you need, but you should be able to adapt this:
RUNNING=0
PIDFILE=$PATH_TO/var/run/example.pid
if [ -f $PIDFILE ]
then
PID=`cat $PIDFILE`
ps -eo pid | grep $PID >/dev/null 2>&1
if [ $? -eq 0 ]
then
RUNNING=1
fi
fi
if [ $RUNNING -ne 1 ]
then
run_binary
PID=$!
echo $PID > $PIDFILE
fi
This is not very elaborate but should get you on the right track.
You can use a pid file to keep track of when the process is running. At the top of the script, check for the existence of the pid file and if it doesn't exist, create it and run the script, otherwise return.
Some sample code can be seen in this answer to a similar question.
You might consider using the (optional) lockfile(1) command (provided by procmail package on Debian).
I have a lot of scripts, and using this below code for prevent multiple/simulate run:
PID="/var/scripts/PID.txt" # Temp file
if [ ! -f "$PID" ]; then
echo $$ > "$PID" # Print actual PID into a file
else
ps -p $(cat "$PID") > /dev/null && exit || echo $$ > "$PID"
fi
Building on wallenborn's answer I also added a "staleness" check just in case the PID lock file is beyond a certain expected age in seconds.
# prevent simultaneous executions within an hourish
pid_file="$HOME/.harness.pid"
max_stale_seconds=3600
if [ -f $pid_file ]; then
pid="$(cat "$pid_file")"
let age_in_seconds="$(date +%s) - $(date -r "$pid_file" +%s)"
if ps $pid >/dev/null && [ $age_in_seconds -lt $max_stale_seconds ]; then
exit 1
fi
fi
echo $$>"$pid_file"
trap "rm -f \"$pid_file\"" SIGSEGV
trap "rm -f \"$pid_file\"" SIGINT
This could be made "smarter" to kill off the other executions should the PID be valid but this would be dangerous. Consider a sudden power failure and reset situation where the PID file contains a number that may now reference a completely different process.
I've got a redirect dilemma that I can't get past in a bash backup script I'm developing in CentOS 6.4. I want to redirect all output to two separate files: one tmp and one permanent. The script loops through an external source list and I'd like for the tmp log files to be specific to the source, so that I can send an email if that specific source had errors containing that log (and conversely remove the tmp if the backup completes without error).
I'm using exec to tee my output:
exec > >(tee -a ${templog} /var/log/rob/rob.log) 2>&1
This works if I place at the top of the script, but here the variable isn't defined yet, so I can't do source-specific logs.
If I place this within the while loop, it grabs the variable, but writes a copy of each line determined by the total iterations of the loop; for the example below, I have four sources it iterates through, so I get output for each source in quadruplicate:
-S-07/11/14 09:15:35 ROB-Source Process for cc2-gamma has started-S-
-S-07/11/14 09:15:35 ROB-Source Process for cc2-gamma has started-S-
-S-07/11/14 09:15:35 ROB-Source Process for cc2-gamma has started-S-
-S-07/11/14 09:15:35 ROB-Source Process for cc2-gamma has started-S-
Share cc2-gamma is not Mounted. Try 1 of 5 to mount...
Share cc2-gamma is not Mounted. Try 1 of 5 to mount...
Share cc2-gamma is not Mounted. Try 1 of 5 to mount...
Share cc2-gamma is not Mounted. Try 1 of 5 to mount...
Is there a different way to tee the output within the loop to prevent this (without touching each line of course)? Or is there something rotten in my loops that I'm not seeing? Here's the whole script. Please excuse the mess and style.. I'm clearly not finished. I didn't include the config.conf and backup source file as they don't affect the output. Let me know if needed. Thanks.
#!/bin/bash
#V.2014.0723 - Radation Oncology Backup script
#declarations
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/rob
source /rob/conf/config.conf
while read smbdir 'smbpath' exclfile drive foldername; do
#loop declarations
mountedfile=/rob/${smbdir}.MOUNTED
runningfile=/rob/${smbdir}.RUNNING
lastrunfile=/rob/${smbdir}_${foldername}.LASTRUN
templog=/rob/${smbdir}_${foldername}.TMPLOG
errorfile=/rob/${smbdir}_${foldername}.HAD_ERRORS
backupfile=/rob/${baname:0:3}_rtbackup.sql.bz2 # for the -l seccton below -- sql backup of backup.sql
#exec > >(tee -a ${templog} /var/log/rob/rob.log) 2>&1
### SOURCE BACKUP ##############################################################################################
if [ "$1" == "-s" ]
then
exec > >(tee -a /var/log/rob/rob.log ${templog}) 2>&1
#Write Source STDOUT and STDERR to both permanent and temporary log file. Must be in loop to use variables.
#exec > >(tee -a ${templog} /var/log/rob/rob.log) 2>&1
#exec > >(tee -a /var/log/rob.log ${templog}) 2>
if [ "${sources_active}" == "1" ]
then
echo "-S-$(date "+%m/%d/%y %T") ROB-Source Process for $smbdir has started-S-"
# unmount all cifs shares, due to duplicate mounts, write file to prevent concurrentcy
umount -a -t cifs > /dev/null
# The following will test to see if the souce is mounted, and if not, mount it.
for i in {1..5}
do
if mountpoint -q /mnt/${smbdir}/${drive}/${foldername}
then
echo "Share ${smbdir} is Mounted."
touch $mountedfile
break
else
sleep 2
echo "Share ${smbdir} is not Mounted. Try $i of 5 to mount..."
mkdir -p /mnt/${smbdir}/${drive}/${foldername} > /dev/null
mount -t cifs ${smbpath} -o ro,username=<USER>,password=<PW>,workgroup=<DOMAIN> /mnt/${smbdir}/${drive}/${foldername}
fi
done
# Test to see if above was successful, and if rob is not already running, run the backup.
if [[ -f ${mountedfile}&& ! -f ${runningfile} ]]
then
src="/mnt/${smbdir}/$drive"
dst="/backup/rob/"
touch ${runningfile}
/root/bin/rtbackup -m /mnt -p ${src}/${foldername} -b ${dst} -x #${exclfile}
if [ "$?" -ne "0" ]; then
#Errors Running RTBackup
rm -f ${runningfile} > /dev/null 2>&1
rm -f ${mountedfile}> /dev/null 2>&1
echo "$(date "+%m/%d/%y %T") Source Process for ${smbdir} had errors running:-SSS"
echo "$errors" >&2
touch ${errorfile}
exit 1
else
echo "What the hell is this doing?"
fi
#NO Errors Running RTBACKUP
rm -f ${templog}
rm -f ${runningfile} > /dev/null 2>&1
rm -f ${mountedfile} > /dev/null 2>&1
echo "$(date "+%m/%d/%y %T") Source Process for ${smbdir} did not have any errors"
else
#backup will *NOT* run, cleaning up and logging
rm -f ${mountedfile} > /dev/null 2>&1
echo "$(date "+%m/%d/%y %T") ${smbdir} could not be mounted, or is already in progress. Backup could not complete."
touch ${errorfile}
tail /var/log/rob/robso.log | mail -s "ROBSO Failed to run for ${smbdir} on ${baname}" ${email}
fi
echo "-F-$(date "+%m/%d/%y %T") ROB-Source Process for ${smbdir} has finished-F-"
#break
elif [[ "${sources_active}" == "0" ]]
then
echo "***$(date "+%m/%d/%y %T") ROB-Source Process for ${smbdir} did not run because the job is not set as active***"
#break
fi
done < /rob/conf/${baname}.conf
if [ $? -eq 10 ]; then exit 0; fi
You can use curly braces to redirect a set of commands; as it says in the bash manual about command grouping, "When commands are grouped, redirections may be applied to the entire command list". It behaves more-or-less like an anonymous function.
{
command1
command2
} > >(tee -a ${templog} /var/log/rob/rob.log) 2>&1
You can do the same with a named function, too, if you're so inclined, but I don't know offhand what environment would be used to expand the redirections. (If you do, please edit this answer!)
# Untested. This MIGHT work.
your_log_command() {
command1
command2
} > >(tee -a $1 /var/log/rob/rob.log) 2>&1
your_log_command $templog
your_log_command $something_else
I have a process that fails regularly & sometimes starts duplicate instances..
When I run:
ps x |grep -v grep |grep -c "processname"
I will get:
2
This is normal as the process runs with a recovery process..
If I get
0
I will want to start the process
if I get:
4
I will want to stop & restart the process
What I need is a way of taking the result of ps x |grep -v grep |grep -c "processname"
Then setup a simple 3 option function
ps x |grep -v grep |grep -c "processname"
if answer = 0 (start process & write NOK & Time to log /var/processlog/check)
if answer = 2 (Do nothing & write OK & time to log /var/processlog/check)
if answer = 4 (stot & restart the process & write NOK & Time to log /var/processlog/check)
The process is stopped with
killall -9 process
The process is started with
process -b -c /usr/local/etc
My main problem is finding a way to act on the result of ps x |grep -v grep |grep -c "processname".
Ideally, I would like to make the result of that grep a variable within the script with something like this:
process=$(ps x |grep -v grep |grep -c "processname")
If possible.
Programs to monitor if a process on a system is running.
Script is stored in crontab and runs once every minute.
This works with if process is not running or process is running multiple times:
#! /bin/bash
case "$(pidof amadeus.x86 | wc -w)" in
0) echo "Restarting Amadeus: $(date)" >> /var/log/amadeus.txt
/etc/amadeus/amadeus.x86 &
;;
1) # all ok
;;
*) echo "Removed double Amadeus: $(date)" >> /var/log/amadeus.txt
kill $(pidof amadeus.x86 | awk '{print $1}')
;;
esac
0 If process is not found, restart it.
1 If process is found, all ok.
* If process running 2 or more, kill the last.
A simpler version. This just test if process is running, and if not restart it.
It just tests the exit flag $? from the pidof program. It will be 0 of process is running and 1 if not.
#!/bin/bash
pidof amadeus.x86 >/dev/null
if [[ $? -ne 0 ]] ; then
echo "Restarting Amadeus: $(date)" >> /var/log/amadeus.txt
/etc/amadeus/amadeus.x86 &
fi
And at last, a one liner
pidof amadeus.x86 >/dev/null ; [[ $? -ne 0 ]] && echo "Restarting Amadeus: $(date)" >> /var/log/amadeus.txt && /etc/amadeus/amadeus.x86 &
This can then be used in crontab to run every minute like this:
* * * * * pidof amadeus.x86 >/dev/null ; [[ $? -ne 0 ]] && echo "Restarting Amadeus: $(date)" >> /var/log/amadeus.txt && /etc/amadeus/amadeus.x86 &
cccam oscam
I adopted the #Jotne solution and works perfectly! For example for mongodb server in my NAS
#! /bin/bash
case "$(pidof mongod | wc -w)" in
0) echo "Restarting mongod:"
mongod --config mongodb.conf
;;
1) echo "mongod already running"
;;
esac
I have adopted your script for my situation Jotne.
#! /bin/bash
logfile="/var/oscamlog/oscam1check.log"
case "$(pidof oscam1 | wc -w)" in
0) echo "oscam1 not running, restarting oscam1: $(date)" >> $logfile
/usr/local/bin/oscam1 -b -c /usr/local/etc/oscam1 -t /usr/local/tmp.oscam1 &
;;
2) echo "oscam1 running, all OK: $(date)" >> $logfile
;;
*) echo "multiple instances of oscam1 running. Stopping & restarting oscam1: $(date)" >> $logfile
kill $(pidof oscam1 | awk '{print $1}')
;;
esac
While I was testing, I ran into a problem..
I started 3 extra process's of oscam1 with this line:
/usr/local/bin/oscam1 -b -c /usr/local/etc/oscam1 -t /usr/local/tmp.oscam1
which left me with 8 process for oscam1. the problem is this..
When I run the script, It only kills 2 process's at a time, so I would have to run it 3 times to get it down to 2 process..
Other than killall -9 oscam1 followed by /usr/local/bin/oscam1 -b -c /usr/local/etc/oscam1 -t /usr/local/tmp.oscam1, in *)is there any better way to killall apart from the original process? So there would be zero downtime?
If you changed awk '{print $1}' to '{ $1=""; print $0}' you will get all processes except for the first as a result. It will start with the field separator (a space generally) but I don't recall killall caring. So:
#! /bin/bash
logfile="/var/oscamlog/oscam1check.log"
case "$(pidof oscam1 | wc -w)" in
0) echo "oscam1 not running, restarting oscam1: $(date)" >> $logfile
/usr/local/bin/oscam1 -b -c /usr/local/etc/oscam1 -t /usr/local/tmp.oscam1 &
;;
2) echo "oscam1 running, all OK: $(date)" >> $logfile
;;
*) echo "multiple instances of oscam1 running. Stopping & restarting oscam1: $(date)" >> $logfile
kill $(pidof oscam1 | awk '{ $1=""; print $0}')
;;
esac
It is worth noting that the pidof route seems to work fine for commands that have no spaces, but you would probably want to go back to a ps-based string if you were looking for, say, a python script named myscript that showed up under ps like
root 22415 54.0 0.4 89116 79076 pts/1 S 16:40 0:00 /usr/bin/python /usr/bin/myscript
Just an FYI
The 'pidof' command will not display pids of shell/perl/python scripts. So to find the process id’s of my Perl script I had to use the -x option i.e. 'pidof -x perlscriptname'
I cannot get case to work at all.
Heres what I have:
#! /bin/bash
logfile="/home/name/public_html/cgi-bin/check.log"
case "$(pidof -x script.pl | wc -w)" in
0) echo "script not running, Restarting script: $(date)" >> $logfile
# ./restart-script.sh
;;
1) echo "script Running: $(date)" >> $logfile
;;
*) echo "Removed duplicate instances of script: $(date)" >> $logfile
# kill $(pidof -x ./script.pl | awk '{ $1=""; print $0}')
;;
esac
rem the case action commands for now just to test the script. the above pidof -x command is returning '1', the case statement is returning the results for '0'.
Anyone have any idea where I'm going wrong?
Solved it by adding the following to my BIN/BASH Script:
PATH=$PATH:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
In case you're looking for a more modern way to check to see if a service is running (this will not work for just any old process), then systemctl might be what you're looking for.
Here's the basic command:
systemctl show --property=ActiveState your_service_here
Which will yield very simple output (one of the following two lines will appear depending on whether the service is running or not running):
ActiveState=active
ActiveState=inactive
And if you'd like to know all of the properties you can get:
systemctl show --all your_service_here
If you prefer that alphabetized:
systemctl show --all your_service_here | sort
And the full code to act on it:
service=$1
result=`systemctl show --property=ActiveState $service`
if [[ "$result" == 'ActiveState=active' ]]; then
echo "$service is running" # Do something here
else
echo "$service is not running" # Do something else here
fi
If you are using CentOS, no need to write a script and set cron job. Here is one of the smartest ways to ensure systemd services restart on failure.
Make following changes to /usr/lib/systemd/system/mariadb.service
Then under the [Service] section in the file, add the following 2 lines:
Restart=always
RestartSec=3
After saving the file we need to reload the daemon configurations to ensure systemd is aware of the new file
systemctl daemon-reload
Read the following link for the complete steps -
https://jonarcher.info/2015/08/ensure-systemd-services-restart-on-failure/