How to write a script could restart the job when exceeding a limit time in Linux? - linux

I'm writing a script could do the same job 100 times and each in a different directory (named run1 to run100). However, the job will stuck for a long time sometimes and I have to delete the directory containing that run and restart it.
Is it possible to write a script that could 1. stop and delete the directory (e.g., run13) if that run exceeds 6 hours and 2. restart that run again?
Here is my original shell script
PREFIX=earlymigration
for i in {1..100}
do
mkdir run$i
cp ${PREFIX}.tpl ${PREFIX}.est ${PREFIX}_jointMAFpop1_0.obs run$i"/"
cd run$i
fsc26 -t ${PREFIX}.tpl -e ${PREFIX}.est -m -0 -C 10 -n 200000 -L 40 -s0 -M -c 10
cd ..
done

So do exactly that. Timeout the command, and if it times out, restart it.
while true; do
timeout $((6 * 60 * 60)) fsc26 ....
ret=$?
if ((ret == 124)); then
rm the_directory_containing_that_run
continue
fi
break
done
See man timeou.

Related

CRON on Rpi simply will not run

I am using a Raspberry pi.
I need to turn on a LED whenever I'm connected to the net and turn off the LED if the connection ever fails. I want to use a cron job running once per minute to do this.
I wrote and compiled two programs in 'C' (ledon, ledoff) that handles the GPIO pin. Those programs work.
I am logged in as 'pi'.
I used crontab -e to write the following:
*/1 * * * * /home/pi/cron_scripts/nettest
I was informed by someone that the first asterisk must have '/1' in order to run properly at the once-per-minute rate that I want. There is no space to the left of the first '/1' and one space after the '1' and each '*' thereafter.
FOR TESTING ONLY, The contents of /home/pi/cron_scripts/nettest is -
#!/bin/bash
ping -c 1 -q 8.8.8.8
if [ "$?" -eq 0 ]; then
printf "%s\n\n" "SUCCESS\n"
else
printf "%s\n\n" "FAIL\n"
fi
exit 0
I used sudo chmod +x /home/pi/cron_scripts/nettest
to make the script executable.
I will replace the printf lines with "ledon" and "ledoff" for the final version.
BUT IT WILL NOT RUN!
echo $(ping -c 1 -q 8.8.8.8)
etc.

service restart doesn't happen when using cron

I have a script that I call from a cron job. The script is
#!/bin/bash
python /home/ubuntu/gateway-haproxy-config.py | tee /etc/haproxy/haproxy.cfg.new
DIFF=$(diff /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.new)
if [ "$DIFF" != "" ]
then
mv /etc/haproxy/haproxy.cfg.new /etc/haproxy/haproxy.cfg
service haproxy restart
else
echo "unmodified"
fi
The script works exactly as expected when I run it from a shell prompt.
I installed it as a cron job as follows (for root using sudo crontab -e):
* * * * * cd /home/ubuntu && ./gateway-config-cron
When the cron runs, the script successfully writes a new configuration file, does the diff and even replaces the old one with the new one when the diff is not empty.
The service haproxy restart never happens when running as a cron job. I am forced to manually restart the service.
This might have been a path related problem I was able to make it work as expected by providing the full path to service.
#!/bin/bash
python /home/ubuntu/gateway-haproxy-config.py | tee /etc/haproxy/haproxy.cfg.new
DIFF=$(diff /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.new)
if [ "$DIFF" != "" ]
then
mv /etc/haproxy/haproxy.cfg.new /etc/haproxy/haproxy.cfg
/usr/sbin/service haproxy restart
else
echo "unmodified"
fi

Linux: run a command every 50 minutes randomly

I need to run a curl request to locahost at least once in every 30 mins. So the command will be curl http://localhost:8080.
The catch here is, I want to select a time randomly between 5min - 30 min and then execute the curl command. The pseudo-code might look like this
while(true)
n = random number between 5-30
run curl http://localhost:8080 after 'n' minutes
A detailed answer would be nice since I don't have much knowledge about linux.
while true; do
sleep $(((RANDOM%25+5)*60))
curl http://localhost:8080
done
If you run above script, you must run as background process and make sure it will not be killed by something (OS, other users,...)
Another way is use cronjob to trigger automatically but the script is more complex.
Cronjob setting:
* * * * * bash test_curl.sh >> log_file.log
Shell script test_curl.sh:
#!/bin/bash
# Declare some variable
EXECUTE_TIME_FILE_PATH="./execute_time"
# load expected execute time
EXPECTED_EXECUTE_TIME=$(cat $EXECUTE_TIME_FILE_PATH)
echo "Start at $(date)"
# calculate current time and compare with expected execute time
CURRENT_MINUTE_OF_TIME=$(date +'%M')
if [[ "$EXPECTED_EXECUTE_TIME" == "$CURRENT_MINUTE_OF_TIME" ]];
then
curl http://localhost:8080
# Random new time from 5 -> 30
NEXT_RANDOM=$((RANDOM%25+5))
# Get current time
CURRENT_TIME=$(date +'%H:%M')
# Calculate next expected execute time = Current Time + Next Random
NEXT_EXPECTED_EXECUTE_TIME=$(date -d "$CURRENT_TIME $NEXT_RANDOM minutes" +'%M')
# Save to file
echo -n $NEXT_EXPECTED_EXECUTE_TIME > $EXECUTE_TIME_FILE_PATH
echo "Next Executed time is $(date -d "$CURRENT_TIME $NEXT_RANDOM minutes" +'%H:%M')"
else
echo "This $(date +'%H:%M') is not expected time to run test"
fi
echo "End at $(date)"
I commented out in line so you can read it easily.
**
Update: Importance: file execute_time must has initial value. For
example, the current minute of first time you execute.
**

Process continues to hold file after file deletion

I created a log file by running the iostat command to a text file, and ran the command in the background using nohup.
#nohup iostat -xm 5 > /z/logfile.txt &
Later on, I created a cronjob that runs every ten minutes doing the same as above, after I realized my process was being killed by a reboot.
I've also setup log-rotation as below:
/z/logfile.txt {
size 20M
rotate 0
create 0644 root root
missingok
notifempty
}
Now I have realized that the logfile.txt gets deleted but the iostat command keeps pointing at deleted files as shown by the lsof -n | grep deleted command. There the disk space is not freed.
How can I make sure the files are rotated and thereafter iostat points to the newly created file, freeing up disk space?
Any ideas how to set it up correctly?
One solution would be to write a program that will read from iostat, write to the output file, and accept a signal to reopen the file. For example, if you did: iostat -xm 5 | log-daemon /z/logfile.txt where log-daemon is a simple script like:
#!/bin/bash
echo $$ > /var/run/log-daemon
exec > $1
trap 'exec > $1' SIGHUP
read line
while test $? -le 0; do
echo $line
read line
done
Then add a postrotate clause in the logrotate config to send a HUP to the log-daemon:
postrotate
/usr/bin/kill -HUP $(cat /var/run/log-daemon)
Would pointing your cronjob iostat command at a softlink not work?
ln -s /z/logfile.txt iostat_link.txt
nohup iostat -xm 5 > /z/iostat_link.txt &
I haven't used logrotate before, but I tested this by manually changing the file in the background while I had this running:
#Make the files
touch afile1.txt
ln -s afile1.txt file.txt
#Kick off loop
for i in {1..1000};do echo "running still $i" >> file.txt;sleep 3;done &
[localhost (2017-05-15 20:30:55) IP: 26.176 ~]# cat afile1.txt
running still 7
running still 8
running still 9
#Change the file out from under the loop
mv afile1.txt afile1.txt.backup;touch afile1.txt
[localhost (2017-05-15 20:31:21) IP: 26.176 ~]# cat afile1.txt
running still 15
running still 16
running still 17
Check if file system where log is recording is full. If you have such a case find and kill process or reboot server in worst case.

Test a weekly cron job [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 4 years ago.
Improve this question
I have a #!/bin/bash file in cron.week directory.
Is there a way to test if it works? Can't wait 1 week
I am on Debian 6 with root
Just do what cron does, run the following as root:
run-parts -v /etc/cron.weekly
... or the next one if you receive the "Not a directory: -v" error:
run-parts /etc/cron.weekly -v
Option -v prints the script names before they are run.
A wee bit beyond the scope of your question... but here's what I do.
The "how do I test a cron job?" question is closely connected to "how do I test scripts that run in non-interactive contexts launched by other programs?" In cron, the trigger is some time condition, but lots of other *nix facilities launch scripts or script fragments in non-interactive ways, and often the conditions in which those scripts run contain something unexpected and cause breakage until the bugs are sorted out. (See also: https://stackoverflow.com/a/17805088/237059 )
A general approach to this problem is helpful to have.
One of my favorite techniques is to use a script I wrote called 'crontest'. It launches the target command inside a GNU screen session from within cron, so that you can attach with a separate terminal to see what's going on, interact with the script, even use a debugger.
To set this up, you would use "all stars" in your crontab entry, and specify crontest as the first command on the command line, e.g.:
* * * * * crontest /command/to/be/tested --param1 --param2
So now cron will run your command every minute, but crontest will ensure that only one instance runs at a time. If the command takes time to run, you can do a "screen -x" to attach and watch it run. If the command is a script, you can put a "read" command at the top to make it stop and wait for the screen attachment to complete (hit enter after attaching)
If your command is a bash script, you can do this instead:
* * * * * crontest --bashdb /command/to/be/tested --param1 --param2
Now, if you attach with "screen -x", you'll be facing an interactive bashdb session, and you can step through the code, examine variables, etc.
#!/bin/bash
# crontest
# See https://github.com/Stabledog/crontest for canonical source.
# Test wrapper for cron tasks. The suggested use is:
#
# 1. When adding your cron job, use all 5 stars to make it run every minute
# 2. Wrap the command in crontest
#
#
# Example:
#
# $ crontab -e
# * * * * * /usr/local/bin/crontest $HOME/bin/my-new-script --myparams
#
# Now, cron will run your job every minute, but crontest will only allow one
# instance to run at a time.
#
# crontest always wraps the command in "screen -d -m" if possible, so you can
# use "screen -x" to attach and interact with the job.
#
# If --bashdb is used, the command line will be passed to bashdb. Thus you
# can attach with "screen -x" and debug the remaining command in context.
#
# NOTES:
# - crontest can be used in other contexts, it doesn't have to be a cron job.
# Any place where commands are invoked without an interactive terminal and
# may need to be debugged.
#
# - crontest writes its own stuff to /tmp/crontest.log
#
# - If GNU screen isn't available, neither is --bashdb
#
crontestLog=/tmp/crontest.log
lockfile=$(if [[ -d /var/lock ]]; then echo /var/lock/crontest.lock; else echo /tmp/crontest.lock; fi )
useBashdb=false
useScreen=$( if which screen &>/dev/null; then echo true; else echo false; fi )
innerArgs="$#"
screenBin=$(which screen 2>/dev/null)
function errExit {
echo "[-err-] $#" | tee -a $crontestLog >&2
}
function log {
echo "[-stat-] $#" >> $crontestLog
}
function parseArgs {
while [[ ! -z $1 ]]; do
case $1 in
--bashdb)
if ! $useScreen; then
errExit "--bashdb invalid in crontest because GNU screen not installed"
fi
if ! which bashdb &>/dev/null; then
errExit "--bashdb invalid in crontest: no bashdb on the PATH"
fi
useBashdb=true
;;
--)
shift
innerArgs="$#"
return 0
;;
*)
innerArgs="$#"
return 0
;;
esac
shift
done
}
if [[ -z $sourceMe ]]; then
# Lock the lockfile (no, we do not wish to follow the standard
# advice of wrapping this in a subshell!)
exec 9>$lockfile
flock -n 9 || exit 1
# Zap any old log data:
[[ -f $crontestLog ]] && rm -f $crontestLog
parseArgs "$#"
log "crontest starting at $(date)"
log "Raw command line: $#"
log "Inner args: $#"
log "screenBin: $screenBin"
log "useBashdb: $( if $useBashdb; then echo YES; else echo no; fi )"
log "useScreen: $( if $useScreen; then echo YES; else echo no; fi )"
# Were building a command line.
cmdline=""
# If screen is available, put the task inside a pseudo-terminal
# owned by screen. That allows the developer to do a "screen -x" to
# interact with the running command:
if $useScreen; then
cmdline="$screenBin -D -m "
fi
# If bashdb is installed and --bashdb is specified on the command line,
# pass the command to bashdb. This allows the developer to do a "screen -x" to
# interactively debug a bash shell script:
if $useBashdb; then
cmdline="$cmdline $(which bashdb) "
fi
# Finally, append the target command and params:
cmdline="$cmdline $innerArgs"
log "cmdline: $cmdline"
# And run the whole schlock:
$cmdline
res=$?
log "Command result: $res"
echo "[-result-] $(if [[ $res -eq 0 ]]; then echo ok; else echo fail; fi)" >> $crontestLog
# Release the lock:
9<&-
fi
After messing about with some stuff in cron which wasn't instantly compatible I found that the following approach was nice for debugging:
crontab -e
* * * * * /path/to/prog var1 var2 &>>/tmp/cron_debug_log.log
This will run the task once a minute and you can simply look in the /tmp/cron_debug_log.log file to figure out what is going on.
It is not exactly the "fire job" you might be looking for, but this helped me a lot when debugging a script that didn't work in cron at first.
I'd use a lock file and then set the cron job to run every minute. (use crontab -e and * * * * * /path/to/job) That way you can just keep editing the files and each minute they'll be tested out. Additionally, you can stop the cronjob by just touching the lock file.
#!/bin/sh
if [ -e /tmp/cronlock ]
then
echo "cronjob locked"
exit 1
fi
touch /tmp/cronlock
<...do your regular cron here ....>
rm -f /tmp/cronlock
What about putting it into cron.hourly, waiting until the next run of hourly cron jobs, then removing it? That would run it once within an hour, and in the cron environment. You can also run ./your_script, but that won't have the same environment as under cron.
Aside from that you can also use:
http://pypi.python.org/pypi/cronwrap
to wrap up your cron to send you an email upon success or failure.
None of these answers fit my specific situation, which was that I wanted to run one specific cron job, just once, and run it immediately.
I'm on a Ubuntu server, and I use cPanel to setup my cron jobs.
I simply wrote down my current settings, and then edited them to be one minute from now. When I fixed another bug, I just edited it again to one minute from now. And when I was all done, I just reset the settings back to how they were before.
Example: It's 4:34pm right now, so I put 35 16 * * *, for it to run at 16:35.
It worked like a charm, and the most I ever had to wait was a little less than one minute.
I thought this was a better option than some of the other answers because I didn't want to run all of my weekly crons, and I didn't want the job to run every minute. It takes me a few minutes to fix whatever the issues were before I'm ready to test it again. Hopefully this helps someone.
The solution I am using is as follows:
Edit crontab(use command :crontab -e) to run the job as frequently
as needed (every 1 minute or 5 minutes)
Modify the shell script which should be executed using cron to prints the output into some file (e.g: echo "Working fine" >>
output.txt)
Check the output.txt file using the command : tail -f output.txt, which will print the latest additions into this file, and thus you can track the execution of the script
I normally test by running the job i created like this:
It is easier to use two terminals to do this.
run job:
#./jobname.sh
go to:
#/var/log and run
run the following:
#tailf /var/log/cron
This allows me to see the cron logs update in real time. You can also review the log after you run it, I prefer watching in real time.
Here is an example of a simple cron job. Running a yum update...
#!/bin/bash
YUM=/usr/bin/yum
$YUM -y -R 120 -d 0 -e 0 update yum
$YUM -y -R 10 -e 0 -d 0 update
Here is the breakdown:
First command will update yum itself and next will apply system updates.
-R 120 : Sets the maximum amount of time yum will wait before performing a command
-e 0 : Sets the error level to 0 (range 0 - 10). 0 means print only critical errors about which you must be told.
-d 0 : Sets the debugging level to 0 - turns up or down the amount of things that are printed. (range: 0 - 10).
-y : Assume yes; assume that the answer to any question which would be asked is yes
After I built the cron job I ran the below command to make my job executable.
#chmod +x /etc/cron.daily/jobname.sh
Hope this helps,
Dorlack
sudo run-parts --test /var/spool/cron/crontabs/
files in that crontabs/ directory needs to be executable by owner - octal 700
source: man cron and NNRooth's
I'm using Webmin because its a productivity gem for someone who finds command line administration a bit daunting and impenetrable.
There is a "Save and Run Now" button in the "System > Scheduled Cron Jobs > Edit Cron Job" web interface.
It displays the output of the command and is exactly what I needed.

Resources