bash script run to send process to background - linux

Hi Im making a script to do some rsync process, for the rsync process, Sys admin has created the script, when it run it is asking select options, so i want to create a script to pass that argument from script and run it from cron.
list of directories to rsync take from file.
filelist=$(cat filelist.txt)
for i in filelist;do
echo -e "3\nY" | ./rsync.sh $i
#This will create a rsync log file
so i check the some value of log file and if it is empty i moving to the second file. if the file is not empty, i have to start rsync process as below that will take more that 2 hours.
if [ a != 0 ];then
echo -e "3\nN" | ./rsync.sh $i
above rsync process need to send to the background and take next file to loop. i check with the screen command, but screen is not working with server. also i need to get the duration that take to run process and passing to the log, when i use the time command i am unable to pass the echo variable. Also need to send this to background and take next file. appreciate any suggestions to success this task.
Question
1. How to send argument with Time command
echo -e "3\nY" | time ./rsync.sh $i
above one not working
how to send this to background and take next file to rsync while running previous rsync process.
Full Code
#!/bin/bash
filelist=$(cat filelist.txt)
Lpath=/opt/sas/sas_control/scripts/Logs/rsync_logs
date=$(date +"%m-%d-%Y")
timelog="time_result/rsync_time.log-$date"
for i in $filelist;do
#echo $i
b_i=$(basename $i)
echo $b_i
echo -e "3\nY" | ./rsync.sh $i
f=$(cat $Lpath/$(ls -tr $Lpath| grep rsync-dry-run-$b_i | tail -1) | grep 'transferred:' | cut -d':' -f2)
echo $f
if [ $f != 0 ]; then
#date=$(date +"%D : %r")
start_time=`date +%s`
echo "$b_i-start:$start_time" >> $timelog
#time ./rsync.sh $i < echo -e "3\nY" 2> "./time_result/$b_i-$date" &
time { echo -e "3\nY" | ./rsync.sh $i; } 2> "./time_result/$b_i-$date"
end_time=`date +%s`
s_time=$(cat $timelog|grep "$b_i-start" |cut -d ':' -f2)
duration=$(($end_time-$s_time))
echo "$b_i duration:$duration" >> $timelog
fi
done

Your question is not very clear, but I'll try:
(1) If I understand you correctly, you want to time the rsync.
My first attempt would be to use echo xxxx | time rsycnc. On my bash, this was however broken (or not supposed to work?). I'm normally using Zsh instead of bash, and on zsht, this indeed runs fine.
If it is important for you to use bash, an alternative (since the time for the echo can likely be neglected) would be to time the whole pipe, i.e. time (echo xxxx | time rsync), or even simpler time rsync <(echo xxxx)
(2) To send a process to the background, add an & to the line. However, the time command produces of course output (that's it purpose), and you don't want to receive output from a program in background. The solution is to redirect the output:
(time rsync <(echo xxxx) >output.txt 2>error.txt) &

If you want to time something, you can use:
time sleep 3
If you want to time two things, you can do a compound statement like this (note semicolon after second sleep):
time { sleep 3; sleep 4; }
So, you can do this to time your echo (which will take no time at all) and your rsync:
time { echo "something" | rsync something ; }
If you want to do that in the background:
time { echo "something" | rsync something ; } &
Full Code
#!/bin/bash
filelist=$(cat filelist.txt)
Lpath=/opt/sas/sas_control/scripts/Logs/rsync_logs
date=$(date +"%m-%d-%Y")
timelog="time_result/rsync_time.log-$date"
for i in $filelist;do
#echo $i
b_i=$(basename $i)
echo $b_i
echo -e "3\nY" | ./rsync.sh $i
f=$(cat $Lpath/$(ls -tr $Lpath| grep rsync-dry-run-$b_i | tail -1) | grep 'transferred:' | cut -d':' -f2)
echo $f
if [ $f != 0 ]; then
#date=$(date +"%D : %r")
start_time=`date +%s`
echo "$b_i-start:$start_time" >> $timelog
#time ./rsync.sh $i < echo -e "3\nY" 2> "./time_result/$b_i-$date" &
time { echo -e "3\nY" | ./rsync.sh $i; } 2> "./time_result/$b_i-$date"
end_time=`date +%s`
s_time=$(cat $timelog|grep "$b_i-start" |cut -d ':' -f2)
duration=$(($end_time-$s_time))
echo "$b_i duration:$duration" >> $timelog
fi
done

Related

How to limit concurrent SSH or Dropbear Tunnel connections

I need to limit concurrent SSH/Dropbear Tunnel connections to 1 login per user.
I have a script that takes care of that.
But it doesn't work for me because when there are many users it becomes saturated and it takes a long time to kick the users.
Another problem with this script is that if the user logs out and logs back in it is detected as multilogin.
Maxlogins and MaxSessions does not work on Dropbear.
Below is the script I am using:
#!/bin/bash
# This script locates all users who have multiple active dropbear
# processes and kills processes in excess of one for each user.
if [ "$EUID" -ne 0 ]; then
printf "Please run as root.\n"
exit
fi
IFS=+
while true; do
PIDFILE=$(mktemp)
AUTHFILE=$(mktemp)
USERS=$(mktemp)
ps aux | grep dropbear | grep -v grep | awk 'BEGIN{} {print $2}' > $PIDFILE
journalctl -r | grep dropbear | grep auth > $AUTHFILE
while read LINE; do
USER=$(printf "%s" $LINE | sed "s/^.* '//" | sed "s/'.*$//" -)
PID=$(printf "%s" $LINE | sed "s/^.*\[//" | sed "s/].*$//" -)
if grep -Fxq $(printf "%s" $USER) $USERS; then
:
else
printf "%s\n" $USER >> $USERS
fi
USERFILE=$(printf "/tmp/%s" $USER)
if [ ! -f $USERFILE ]; then
touch $USERFILE
fi
if grep -Fxq $(printf "%s" $PID) $PIDFILE; then
printf "%s\n" $PID >> $USERFILE
else
:
fi
done < $AUTHFILE
while read USER; do
i=1
while read PID; do
if [ $i -gt 1 ]; then
printf "Kill PID %s of user %s\n" $PID $USER
kill -9 $(printf "%s" $PID)
curl -k "https://redesprivadasvirtuales.com/modules/servers/openvpn/vega.php?secret=DD8sPD&user=$USER"
else
:
fi
((i++))
done < $(printf "/tmp/%s" $USER)
rm $(printf "/tmp/%s" $USER)
done < $USERS
rm $PIDFILE
rm $AUTHFILE
rm $USERS
done
Suggestions:
journalctl -r is very expensive. Limit journalctl to time since last search.
Line with USER=$(...) and PID=$(...). Replace printf and sed commands, with single awk command.
Research pgrep and pkill commaonds.
Replace file $PIDFILE $AUTHFILE $USERS with array variables (research readarray command).
While loop over $AUTHFILE could be implemented as loop over bash array.
While loop over $USERS (including internal loop) could be implemented as loop over bash array.
curl command might be very expensive. You do not check the response from each curl request. Run curl in background and if possible in parallel for all users.
Kind SO members could assist more, if you put sample lines from $AUTHFILE in the questions as sample input line.

Calculate time for each step of a shell script and show total execution time

I have below script and for a requirement I have to place some function for each of these script to get time information for each script and at last show total time.
My main scripts looks like below:
/u01/scripts/stop.sh ${1} | tee ${stop_log}
/u01/scripts/kill_proc.sh ${1} | tee ${kill_log}
/u01/scripts/detach.sh ${1}| tee ${detach_log}
/u01/scripts/copy.sh ${1} | tee ${copy_log}
I want to use something like below function to get every script execution time and at last with a global variable I can show Total time taken by all the scripts.
I created below but unfortunately I could not use properly , if you have something kindly help here.
time_check()
{
export time_log=${log}/time_log_${dts}.log
echo 'StartingTime:'date +%s > ${time_log}
echo 'EndingTime:'date +%s >> ${time_log}
}
I want to use something like above function to get every script execution time and at last with a global variable I can show total time taken by all the scripts . Could anyone please guide how to get the desired result.
If you are OK with the time granularity of seconds, you could simply do this:
start=$SECONDS
/u01/scripts/stop.sh ${1} | tee ${stop_log}
stop=$SECONDS
/u01/scripts/kill_proc.sh ${1} | tee ${kill_log}
kill_proc=$SECONDS
/u01/scripts/detach.sh ${1}| tee ${detach_log}
detach=$SECONDS
/u01/scripts/copy.sh ${1} | tee ${copy_log}
end=$SECONDS
printf "%s\n" "stop=$((stop-start)), kill_proc=$((kill_proc-stop)), detach=$((detach-kill_proc)), copy=$((end-detach)), total=$((end-start))"
You can write a function to do this as well:
time_it() {
local start=$SECONDS rc
echo "$(date): Starting $*"
"$#"; rc=$?
echo "$(date): Finished $*; elapsed = $((SECONDS-start)) seconds"
return $rc
}
With Bash version >= 4.2 you can use printf to print date rather than invoking an external command:
time_it() {
local start=$SECONDS ts rc
printf -v ts '%(%Y-%m-%d_%H:%M:%S)T' -1
printf '%s\n' "$ts Starting $*"
"$#"; rc=$?
printf -v ts '%(%Y-%m-%d_%H:%M:%S)T' -1
printf '%s\n' "$ts Finished $*; elapsed = $((SECONDS-start)) seconds"
return $rc
}
And invoke it as:
start=$SECONDS
time_it /u01/scripts/stop.sh ${1} | tee ${stop_log}
time_it /u01/scripts/kill_proc.sh ${1} | tee ${kill_log}
time_it /u01/scripts/detach.sh ${1}| tee ${detach_log}
time_it /u01/scripts/copy.sh ${1} | tee ${copy_log}
echo "Total time = $((SECONDS-start)) seconds"
Related:
How do I measure duration in seconds in a shell script?

Bash concurrent jobs gets stuck

I've implemented a way to have concurrent jobs in bash, as seen here.
I'm looping through a file with around 13000 lines. I'm just testing and printing each line, as such:
#!/bin/bash
max_bg_procs(){
if [[ $# -eq 0 ]] ; then
echo "Usage: max_bg_procs NUM_PROCS. Will wait until the number of background (&)"
echo " bash processes (as determined by 'jobs -pr') falls below NUM_PROCS"
return
fi
local max_number=$((0 + ${1:-0}))
while true; do
local current_number=$(jobs -pr | wc -l)
if [[ $current_number -lt $max_number ]]; then
echo "success in if"
break
fi
echo "has to wait"
sleep 4
done
}
download_data(){
echo "link #" $2 "["$1"]"
}
mapfile -t myArray < $1
i=1
for url in "${myArray[#]}"
do
max_bg_procs 6
download_data $url $i &
((i++))
done
echo "finito!"
I've also tried other solutions such as this and this, but my issue is persistent:
At a "random" given step, usually between the 2000th and the 5000th iteration, it simply gets stuck. I've put those various echo in the middle of the code to see where it would get stuck but it the last thing it prints is the $url $i.
I've done the simple test to remove any parallelism and just loop the file contents: all went fine and it looped till the end.
So it makes me think I'm missing some limitation on the parallelism, and I wonder if anyone could help me out figuring it out.
Many thanks!
Here, we have up to 6 parallel bash processes calling download_data, each of which is passed up to 16 URLs per invocation. Adjust per your own tuning.
Note that this expects both bash (for exported function support) and GNU xargs.
#!/usr/bin/env bash
# ^^^^- not /bin/sh
download_data() {
echo "link #$2 [$1]" # TODO: replace this with a job that actually takes some time
}
export -f download_data
<input.txt xargs -d $'\n' -P 6 -n 16 -- bash -c 'for arg; do download_data "$arg"; done' _
Using GNU Parallel it looks like this
cat input.txt | parallel echo link '\#{#} [{}]'
{#} = the job number
{} = the argument
It will spawn one process per CPU. If you instead want 6 in parallel use -j:
cat input.txt | parallel -j6 echo link '\#{#} [{}]'
If you prefer running a function:
download_data(){
echo "link #" $2 "["$1"]"
}
export -f download_data
cat input.txt | parallel -j6 download_data {} {#}

Editing a .CSV file through shell automation script

I'm getting an error when trying to execute the script below around the done statement. The point of the code is to have the while statement execute for the duration of files listed in the filenames that log in order to grab the revision number for each file location in my branches folder.
filea=/home/filenames.log
fileb=/home/actions.log
filec=/home/revisions.log
filed=/home/final.log
count=1
while read Path do
Status=`sed -n "$count"p $fileb`
Revision=`svn info ${WORKSPACE}/$Path | grep "Revision" | awk '{print $2}'`
if `echo $Path | grep "UpgradeScript"` then
Results="Reverted - ROkere"
Details="Reverted per process"
else if `echo $Path | grep "tsu_includes/shell_scripts"` then
Results="Reverted - ROkere"
Details="Reverted per process"
else
Results="Verified - ROkere"
Details=""
fi
echo "$Path,$Status,$Revision,$Results,$Details" > $filed
count=`expr $count + 1`
done < $filea
need a semicolon or newline before do and then.
Change else if to elif
change
if `echo $Path | grep "UpgradeScript"` then
to (removing backticks, using "here-string", and -q option for grep)
if grep -q "UpgradeScript" <<< "$Path"; then
"filed" will only ever contain just one line. I assume you want to append >> instead of overwrite >
Actually, a quick rewrite. You're reading corresponding lines from 2 files. Faster to do that completely within the shell instead of invoking sed once for each line in the file.
#!/bin/bash
filea=/home/filenames.log
fileb=/home/actions.log
filec=/home/revisions.log # not used?
filed=/home/final.log
exec 3<"$filea" # open $filea on fd 3
exec 4<"$fileb" # open $fileb on fd 4
while read -u3 Path && read -u4 Status; do
Revision=$(svn info "$WORKSPACE/$Path" | awk '/Revision/ {print $2}')
if [[ "$Path" == *"UpgradeScript"* ]]; then
Results="Reverted - ROkere"
Details="Reverted per process"
elif [[ "$Path" == *"tsu_includes/shell_scripts"* ]]; then
Results="Reverted - ROkere"
Details="Reverted per process"
else
Results="Verified - ROkere"
Details=""
fi
echo "$Path,$Status,$Revision,$Results,$Details"
done > "$filed"
exec 3<&- # close fd 3
exec 4<&- # close fd 4

Shell function to tail a log file for a specific string for a specific time

I need to the following things to make sure my application server is
Tail a log file for a specific string
Remain blocked until that string is printed
However if the string is not printed for about 20 mins quit and throw and exception message like "Server took more that 20 mins to be up"
If string is printed in the log file quit the loop and proceed.
Is there a way to include time outs in a while loop ?
#!/bin/bash
tail -f logfile | grep 'certain_word' | read -t 1200 dummy_var
[ $? -eq 0 ] && echo 'ok' || echo 'server not up'
This reads anything written to logfile, searches for certain_word, echos ok if all is good, otherwise after waiting 1200 seconds (20 minutes) it complains.
You can do it like this:
start_time=$(date +"%s")
while true
do
elapsed_time=$(($(date +"%s") - $start_time))
if [[ "$elapsed_time" -gt 1200 ]]; then
break
fi
sleep 1
if [[ $(grep -c "specific string" /path/to/log/file.log) -ge 1 ]]; then
break
fi
done
You can use signal handlers from shell scripts (see http://www.ibm.com/developerworks/aix/library/au-usingtraps/index.html).
Basically, you'd define a function to be called on, say, signal 17, then put a sub-script in the background that will send that signal at some later time:
timeout(pid) {
sleep 1200
kill -SIGUSR1 $pid
}
watch_for_input() {
tail -f file | grep item
}
trap 'echo "Not found"; exit' SIGUSR1
timeout($$) &
watch_for_input
Then if you reach 1200 seconds, your function is called and you can choose what to do (like signal your tail/grep combo that is watching for your pattern in order to kill it)
time=0
found=0
while [ $time -lt 1200 ]; do
out=$(tail logfile)
if [[ $out =~ specificString ]]; then
found=1
break;
fi
let time++
sleep 1
done
echo $found
The accepted answer doesn't work and will never exit (because althouth read -t exits, the prior pipe commands (tail -f | grep) will only be notified of read -t exit when they try to write to output, which never happens until the string matches).
A one-liner is probably feasible, but here are scripted (working) approaches.
Logic is the same for each one, they use kill to terminate the current script after the timeout.
Perl is probably more widely available than gawk/read -t
#!/bin/bash
FILE="$1"
MATCH="$2"
# Uses read -t, kill after timeout
#tail -f "$FILE" | grep "$MATCH" | (read -t 1 a ; kill $$)
# Uses gawk read timeout ability (not available in awk)
#tail -f "$FILE" | grep "$MATCH" | gawk "BEGIN {PROCINFO[\"/dev/stdin\", \"READ_TIMEOUT\"] = 1000;getline < \"/dev/stdin\"; system(\"kill $$\")}"
# Uses perl & alarm signal
#tail -f "$FILE" | grep "$MATCH" | perl -e "\$SIG{ALRM} = sub { `kill $$`;exit; };alarm(1);<>;"

Resources