timer to check the end of the file bash shell - linux

I am using a code basically runs to restart a calculation after one finishes. The code was working fine. But i wanna improvise the code. Right now it runs on a timer function sleep. It never checks for the end of the file. So it waits 22 min to submit the next job irrespective of the previous job. So i want to change the code in such a way it checks the end of the file every one minute for the keyword "End of program" from the log file and submits the next job. Please drop me a msg if you have any ideas ...
while ( 1 )
#sleep for X seconds/minutes
$SLEEP 22m
#cut JobID into peeces:
#e.g.: 31380.chic1i20.informatik.tu-chemnitz.de Info: Your job definition is checked against CHiC policies. --> 31380.chic1i20
#set JOBnr = `echo $JOBID | $CUT -c-15`
set JOBnr = `echo $JOBID | $CUT -d " " -f1 | $CUT -d "." -f-2`
$QSTAT -nu "$USER" > .qstat_outfile
if ( "$?" ) then
echo "autorestart: qstat error"
$CAT .qstat_outfile
continue
endif

Try this:
UNWRITTEN=0
while [ $UNWRITTEN -eq 0 ]; do
token=$(tail -n 2 .qstat_outfile)
echo $token
if [[ $token == "End of program" ]]
then
echo "autorestart..."
UNWRITTEN=1
fi
sleep 60
done
Explanation:
Program continually loops until it find the correct line at the end of the logfile. Then it restarts your calculations and exits.
tail -n 2 reads the last two lines (which is actually the last line if there is a newline after "End of program". You may need to adjust this to -n 1 on your system.)

If I got you right, a while-loop like this may help you:
stop=0
while [ $stop == 0 ]; do
tail -1 $LOGFILE | grep -q "End of program" && stop=1 || sleep 60
done
(It checks the last line (tail -1) of LOGFILE for the presence of your search text, stops
if text is found or sleeps for 60 seconds if not).

Related

While loop within sleep and retry the proccess

I have a control file file.txt that always changes in value anytime depends on the other proccess.
The value of the file is like 1, 2, or 3.
I want to execute my new job when the value of control file is 3. So i need to check first the control file before do the execution job.
I was try while loop below using sleep for a moment and retry the proccess automatically every 5 seconds for 3 times, but the retry proccess is not running. How i resolved this?
#!/bin/bash
myname="kyy"
while read file; do
if [ $file == 2 ]; then
echo "`date +%Y%m%d:%H:%M:%S:%N` [ERROR]:-Value of Control File : $file "
echo "`date +%Y%m%d:%H:%M:%S:%N` [NOTICE]:-Nothing to do.. Script will exiting.."
else
echo "`date +%Y%m%d:%H:%M:%S:%N` [SUCCESS]:-Go to next step "
echo "`date +%Y%m%d:%H:%M:%S:%N` [SUCCESS]:-My name is $myname "
fi
sleep 3
((c++)) && ((c==3)) && break
done < /home/hcuseros/file.txt
This would be a simple way, where you only need to fill in your commands depending on the outcome of the test:
for i in {1..3}; do
file=$(cat test.txt)
if [ $file -eq "3" ]; then
...
else
...
fi
sleep 5
done

linux printf to specified terminal line

In yocto, when I use bibake to build one recipe, stages related with this recipe will be printed in multiple lines, very beautiful.
So I want implement a tiny example like it.
get current cursor's row
run 2 threads, one outputs something at row + 1, another outputs
something at row + 2
As a result, I failed. Then I run 2 process which do same things, failed again.
Can some one give me some suggestions?
What I want is like:
ts:/home/test$ ./program1 &; ./program1 &
program1's output.....................
program2's output.....................
In Bash:
#!/usr/bin/env bash
printf '\n\n'
echo -ne "\033[6n"
read -rs -d\[ _
read -rs -dR foo
cursor_pos=$(cut -d";" -f1 <<< "$foo")
upper_row=$((cursor_pos - 2))
lower_row=$((cursor_pos - 1))
echo -ne "\033[${upper_row};0H"
echo upper row
sleep 2
echo -ne "\033[${lower_row};0H"
echo lower low
sleep 2
echo -ne "\033[${upper_row};0H"
printf "\r\e[0K%s\n" "upper again"
sleep 2
echo -ne "\033[${lower_row};0H"
printf "\r\e[0K%s\n" "lower again"

Run subshells in parallel and collect exit codes

by using shell scripting, I am dividing one long data file into 8 files and run them in parallel in 8 instance.
function_child()
{
while read -r record
do
###process to get the data by arsdoc get##
exit 12 ## if get fails##
### fp2pdf ###
EXIT 12 ## IF AFP2PDF FAILS ###
### logic ###
exit 12 ## if logic fails####
done < $1
}
## main ##
for file in /$MY_WORK/CCN_split_files/*; do
proceed_task "$file" &
PID="$!"
echo "$PID:$file" | tee $tmp_file
PID_LIST+="$PID "
done
how can take\ monitor the exit code and pid's of the child process when there is an failure.
I tryed this below, Once all the process are sent to background, I am using ‘wait’ function to wait for the PID from our PID_LIST to exit and then capture and print the respective exit status.
for process in "${PID_LIST[#]}";do
wait "$process"
exit_status=$?
file_name=`egrep $process $tmp_file | awk -F ":" '{print $2}' | rev | awk -F "/" '{print $2}' | rev`
echo "$file_name exit status: $exit_status"
done
but it gives an error
line 49: wait: `23043 23049 ': not a pid or valid job spec
grep: 23049: No such file or directory
could someone help me on this, Thank you.
See: help jobs and help wait
Collect return status at end of your code
for pid in $(jobs -rp); do
printf "Job %d handling file %q is still running\n" "$pid" "${pids[pid]}"
done
for pid in ${jobs -sp); do
printf "Job %s handling file %q has returned with status %d\n" "$pid" "${pids[pid]}" "$(wait "$pid")"
done
The double quotes around the argument to wait creates a single string. Remove the quotes to have the shell break up the string into individual PIDs.
Use wait on proper pid numbers.
function_child() {
while read -r record; do
# let's return a random number!
exit ${RANDOM}
done <<<'a'
}
mkdir -p my-home/dir
touch my-home/dir/{1..9}
for file in my-home/dir/*; do
function_child "$file" &
pid=$!
echo "Backgrounded: $file (pid=$pid)"
pids[$pid]=$file
done
for i in "${!pids[#]}"; do
wait "$i"
ret=$?
echo ${pids[$i]} returned with $ret
done
outputs on repl:
Backgrounded: my-home/dir/1 (pid=84)
Backgrounded: my-home/dir/2 (pid=85)
Backgrounded: my-home/dir/3 (pid=86)
Backgrounded: my-home/dir/4 (pid=87)
Backgrounded: my-home/dir/5 (pid=88)
Backgrounded: my-home/dir/6 (pid=89)
Backgrounded: my-home/dir/7 (pid=90)
Backgrounded: my-home/dir/8 (pid=91)
Backgrounded: my-home/dir/9 (pid=92)
my-home/dir/1 returned with 241
my-home/dir/2 returned with 59
my-home/dir/3 returned with 235
my-home/dir/4 returned with 11
my-home/dir/5 returned with 6
my-home/dir/6 returned with 222
my-home/dir/7 returned with 230
my-home/dir/8 returned with 189
my-home/dir/9 returned with 195
But I think just use xargs or other tool designed to run such jobs in parallel.
printf "%s\n" my-home/dir/* | xargs -n$'\n' -P8 sh -c 'echo "$1"; ###process to get the data by arsdoc get' --
#KamilCuk, appologies , updated the code.
The PID_LIST+="$PID " creates a one long variable with spaces. The "${PID_LIST[#]}" is an expansion used for arrays. It works that way, that ${PID_LIST[#]} just expands to the value of the variable PID_LIST, as if "$PID_LIST", so it expands to "23043 23049 ". Because it is in quotes it iterates over one element, hence it runs wait "23043 23049 ", hence you see the error message.
Not recommended: You could depend on shell space splitting
for process in $PID_LIST; do
wait "$process"
But just use an array:
PID_LIST+=("$PID")
done
for process in "${PID_LIST[#]}"; do
wait "$process"
If you feel not safe with your pids[$pid]=$file associative array, use two arrays instead:
onlypids+=("$pid")
files+=("$files")
done
for i in "${!onlypids[#]}"; do
pid="${onlypids[$i]}"
file="${files[$i]}"
wait "$pid"
Note that by convention, upper case variable names are for exported variables.
You mention in the comments that you do not want to use GNU Parallel, so this answer is for people who do not have that restriction.
doit() {
record="$1"
###process to get the data by arsdoc get##
exit 12 ## if get fails##
### fp2pdf ###
EXIT 12 ## IF AFP2PDF FAILS ###
### logic ###
exit 12 ## if logic fails####
}
export -f doit
cat /$MY_WORK/CCN_split_files/* |
parallel --joblog my.log doit
# Field 7 of my.log is the exit value
# If you have an unsplit version of the input you can have GNU Parallel process it:
# cat /$MY_WORK/CNN_big_file |
# parallel --joblog my.log doit

Bash concurrent jobs gets stuck

I've implemented a way to have concurrent jobs in bash, as seen here.
I'm looping through a file with around 13000 lines. I'm just testing and printing each line, as such:
#!/bin/bash
max_bg_procs(){
if [[ $# -eq 0 ]] ; then
echo "Usage: max_bg_procs NUM_PROCS. Will wait until the number of background (&)"
echo " bash processes (as determined by 'jobs -pr') falls below NUM_PROCS"
return
fi
local max_number=$((0 + ${1:-0}))
while true; do
local current_number=$(jobs -pr | wc -l)
if [[ $current_number -lt $max_number ]]; then
echo "success in if"
break
fi
echo "has to wait"
sleep 4
done
}
download_data(){
echo "link #" $2 "["$1"]"
}
mapfile -t myArray < $1
i=1
for url in "${myArray[#]}"
do
max_bg_procs 6
download_data $url $i &
((i++))
done
echo "finito!"
I've also tried other solutions such as this and this, but my issue is persistent:
At a "random" given step, usually between the 2000th and the 5000th iteration, it simply gets stuck. I've put those various echo in the middle of the code to see where it would get stuck but it the last thing it prints is the $url $i.
I've done the simple test to remove any parallelism and just loop the file contents: all went fine and it looped till the end.
So it makes me think I'm missing some limitation on the parallelism, and I wonder if anyone could help me out figuring it out.
Many thanks!
Here, we have up to 6 parallel bash processes calling download_data, each of which is passed up to 16 URLs per invocation. Adjust per your own tuning.
Note that this expects both bash (for exported function support) and GNU xargs.
#!/usr/bin/env bash
# ^^^^- not /bin/sh
download_data() {
echo "link #$2 [$1]" # TODO: replace this with a job that actually takes some time
}
export -f download_data
<input.txt xargs -d $'\n' -P 6 -n 16 -- bash -c 'for arg; do download_data "$arg"; done' _
Using GNU Parallel it looks like this
cat input.txt | parallel echo link '\#{#} [{}]'
{#} = the job number
{} = the argument
It will spawn one process per CPU. If you instead want 6 in parallel use -j:
cat input.txt | parallel -j6 echo link '\#{#} [{}]'
If you prefer running a function:
download_data(){
echo "link #" $2 "["$1"]"
}
export -f download_data
cat input.txt | parallel -j6 download_data {} {#}

Shell function to tail a log file for a specific string for a specific time

I need to the following things to make sure my application server is
Tail a log file for a specific string
Remain blocked until that string is printed
However if the string is not printed for about 20 mins quit and throw and exception message like "Server took more that 20 mins to be up"
If string is printed in the log file quit the loop and proceed.
Is there a way to include time outs in a while loop ?
#!/bin/bash
tail -f logfile | grep 'certain_word' | read -t 1200 dummy_var
[ $? -eq 0 ] && echo 'ok' || echo 'server not up'
This reads anything written to logfile, searches for certain_word, echos ok if all is good, otherwise after waiting 1200 seconds (20 minutes) it complains.
You can do it like this:
start_time=$(date +"%s")
while true
do
elapsed_time=$(($(date +"%s") - $start_time))
if [[ "$elapsed_time" -gt 1200 ]]; then
break
fi
sleep 1
if [[ $(grep -c "specific string" /path/to/log/file.log) -ge 1 ]]; then
break
fi
done
You can use signal handlers from shell scripts (see http://www.ibm.com/developerworks/aix/library/au-usingtraps/index.html).
Basically, you'd define a function to be called on, say, signal 17, then put a sub-script in the background that will send that signal at some later time:
timeout(pid) {
sleep 1200
kill -SIGUSR1 $pid
}
watch_for_input() {
tail -f file | grep item
}
trap 'echo "Not found"; exit' SIGUSR1
timeout($$) &
watch_for_input
Then if you reach 1200 seconds, your function is called and you can choose what to do (like signal your tail/grep combo that is watching for your pattern in order to kill it)
time=0
found=0
while [ $time -lt 1200 ]; do
out=$(tail logfile)
if [[ $out =~ specificString ]]; then
found=1
break;
fi
let time++
sleep 1
done
echo $found
The accepted answer doesn't work and will never exit (because althouth read -t exits, the prior pipe commands (tail -f | grep) will only be notified of read -t exit when they try to write to output, which never happens until the string matches).
A one-liner is probably feasible, but here are scripted (working) approaches.
Logic is the same for each one, they use kill to terminate the current script after the timeout.
Perl is probably more widely available than gawk/read -t
#!/bin/bash
FILE="$1"
MATCH="$2"
# Uses read -t, kill after timeout
#tail -f "$FILE" | grep "$MATCH" | (read -t 1 a ; kill $$)
# Uses gawk read timeout ability (not available in awk)
#tail -f "$FILE" | grep "$MATCH" | gawk "BEGIN {PROCINFO[\"/dev/stdin\", \"READ_TIMEOUT\"] = 1000;getline < \"/dev/stdin\"; system(\"kill $$\")}"
# Uses perl & alarm signal
#tail -f "$FILE" | grep "$MATCH" | perl -e "\$SIG{ALRM} = sub { `kill $$`;exit; };alarm(1);<>;"

Resources