Making a bash script multithread - linux

I have this script:
#!/bin/bash
CONTOR=0
total=`grep -c . $1`
for i in `cat $1`
do
CONTOR=`ps x | grep -c bash`
while [ $CONTOR -ge 500 ];do
CONTOR=`ps x | grep -c bash`
sleep 5
done
if [ $CONTOR -le 500 ]; then
./bing-ip2hosts -n $i >> url.txt &
fi
done
The scripts takes an IP from a list then runs ./bing-ip2hosts -n $i[the ip].
How can I make it multi-threaded so it runs faster. Now it opens like 20-30 processes and I would like for it to open 150 maybe even 200.

Try using the parallel command:
cat $1 | parallel -j4 s./bing-ip2hosts -n {} >> url.txt
see http://www.xensoft.com/content/use-multiple-cpu-cores-single-threaded-linux-commands for more examples.

Sure, technically these are processes, and this program should really be called a process spawning manager, but this is only due to the way that BASH works when it forks using the ampersand, it uses the fork() or perhaps clone() system call which clones into a separate memory space, rather than something like pthread_create() which would share memory. If BASH supported the latter, each "sequence of execution" would operate just the same and could be termed to be traditional threads whilst gaining a more efficient memory footprint. Functionally however it works the same, though a bit more difficult since GLOBAL variables are not available in each worker clone hence the use of the inter-process communication file and the rudimentary flock semaphore to manage critical sections. Forking from BASH of course is the basic answer here but I feel as if people know that but are really looking to manage what is spawned rather than just fork it and forget it. This demonstrates a way to manage up to 200 instances of forked processes all accessing a single resource. I hope you enjoy it, I enjoyed writing it.
#!/bin/bash
ME=$(basename $0)
IPC="/tmp/$ME.ipc" #interprocess communication file (global thread accounting stats)
DBG=/tmp/$ME.log
echo 0 > $IPC #initalize counter
F1=thread
SPAWNED=0
COMPLETE=0
SPAWN=10000 #number of jobs to process
SPEEDFACTOR=1 #dynamically compensates for execution time
THREADLIMIT=200 #maximum concurrent threads
TPS=1 #threads per second delay
THREADCOUNT=0 #number of running threads
SCALE="scale=5" #controls bc's precision
START=$(date +%s) #whence we began
MAXTHREADDUR=30 #maximum thread life span - demo mode
LOWER=$[$THREADLIMIT*100*90/10000] #90% worker utilization threshold
UPPER=$[$THREADLIMIT*100*95/10000] #95% worker utilization threshold
DELTA=10 #initial percent speed change
threadspeed() #dynamically adjust spawn rate based on worker utilization
{
#vaguely assumes thread execution average will be consistent
THREADCOUNT=$(threadcount)
if [ $THREADCOUNT -ge $LOWER ] && [ $THREADCOUNT -le $UPPER ] ;then
echo SPEED HOLD >> $DBG
return
elif [ $THREADCOUNT -lt $LOWER ] ;then
#if maxthread is free speed up
SPEEDFACTOR=$(echo "$SCALE;$SPEEDFACTOR*(1-($DELTA/100))"|bc)
echo SPEED UP $DELTA%>> $DBG
elif [ $THREADCOUNT -gt $UPPER ];then
#if maxthread is active then slow down
SPEEDFACTOR=$(echo "$SCALE;$SPEEDFACTOR*(1+($DELTA/100))"|bc)
DELTA=1 #begin fine grain control
echo SLOW DOWN $DELTA%>> $DBG
fi
echo SPEEDFACTOR $SPEEDFACTOR >> $DBG
#average thread duration (total elapsed time / number of threads completed)
#if threads completed is zero (less than 100), default to maxdelay/2 maxthreads
COMPLETE=$(cat $IPC)
if [ -z $COMPLETE ];then
echo BAD IPC READ ============================================== >> $DBG
return
fi
#echo Threads COMPLETE $COMPLETE >> $DBG
if [ $COMPLETE -lt 100 ];then
AVGTHREAD=$(echo "$SCALE;$MAXTHREADDUR/2"|bc)
else
ELAPSED=$[$(date +%s)-$START]
#echo Elapsed Time $ELAPSED >> $DBG
AVGTHREAD=$(echo "$SCALE;$ELAPSED/$COMPLETE*$THREADLIMIT"|bc)
fi
echo AVGTHREAD Duration is $AVGTHREAD >> $DBG
#calculate timing to achieve spawning each workers fast enough
# to utilize threadlimit - average time it takes to complete one thread / max number of threads
TPS=$(echo "$SCALE;($AVGTHREAD/$THREADLIMIT)*$SPEEDFACTOR"|bc)
#TPS=$(echo "$SCALE;$AVGTHREAD/$THREADLIMIT"|bc) # maintains pretty good
#echo TPS $TPS >> $DBG
}
function plot()
{
echo -en \\033[${2}\;${1}H
if [ -n "$3" ];then
if [[ $4 = "good" ]];then
echo -en "\\033[1;32m"
elif [[ $4 = "warn" ]];then
echo -en "\\033[1;33m"
elif [[ $4 = "fail" ]];then
echo -en "\\033[1;31m"
elif [[ $4 = "crit" ]];then
echo -en "\\033[1;31;4m"
fi
fi
echo -n "$3"
echo -en "\\033[0;39m"
}
trackthread() #displays thread status
{
WORKERID=$1
THREADID=$2
ACTION=$3 #setactive | setfree | update
AGE=$4
TS=$(date +%s)
COL=$[(($WORKERID-1)/50)*40]
ROW=$[(($WORKERID-1)%50)+1]
case $ACTION in
"setactive" )
touch /tmp/$ME.$F1$WORKERID #redundant - see main loop
#echo created file $ME.$F1$WORKERID >> $DBG
plot $COL $ROW "Worker$WORKERID: ACTIVE-TID:$THREADID INIT " good
;;
"update" )
plot $COL $ROW "Worker$WORKERID: ACTIVE-TID:$THREADID AGE:$AGE" warn
;;
"setfree" )
plot $COL $ROW "Worker$WORKERID: FREE " fail
rm /tmp/$ME.$F1$WORKERID
;;
* )
;;
esac
}
getfreeworkerid()
{
for i in $(seq 1 $[$THREADLIMIT+1])
do
if [ ! -e /tmp/$ME.$F1$i ];then
#echo "getfreeworkerid returned $i" >> $DBG
break
fi
done
if [ $i -eq $[$THREADLIMIT+1] ];then
#echo "no free threads" >> $DBG
echo 0
#exit
else
echo $i
fi
}
updateIPC()
{
COMPLETE=$(cat $IPC) #read IPC
COMPLETE=$[$COMPLETE+1] #increment IPC
echo $COMPLETE > $IPC #write back to IPC
}
worker()
{
WORKERID=$1
THREADID=$2
#echo "new worker WORKERID:$WORKERID THREADID:$THREADID" >> $DBG
#accessing common terminal requires critical blocking section
(flock -x -w 10 201
trackthread $WORKERID $THREADID setactive
)201>/tmp/$ME.lock
let "RND = $RANDOM % $MAXTHREADDUR +1"
for s in $(seq 1 $RND) #simulate random lifespan
do
sleep 1;
(flock -x -w 10 201
trackthread $WORKERID $THREADID update $s
)201>/tmp/$ME.lock
done
(flock -x -w 10 201
trackthread $WORKERID $THREADID setfree
)201>/tmp/$ME.lock
(flock -x -w 10 201
updateIPC
)201>/tmp/$ME.lock
}
threadcount()
{
TC=$(ls /tmp/$ME.$F1* 2> /dev/null | wc -l)
#echo threadcount is $TC >> $DBG
THREADCOUNT=$TC
echo $TC
}
status()
{
#summary status line
COMPLETE=$(cat $IPC)
plot 1 $[$THREADLIMIT+2] "WORKERS $(threadcount)/$THREADLIMIT SPAWNED $SPAWNED/$SPAWN COMPLETE $COMPLETE/$SPAWN SF=$SPEEDFACTOR TIMING=$TPS"
echo -en '\033[K' #clear to end of line
}
function main()
{
while [ $SPAWNED -lt $SPAWN ]
do
while [ $(threadcount) -lt $THREADLIMIT ] && [ $SPAWNED -lt $SPAWN ]
do
WID=$(getfreeworkerid)
worker $WID $SPAWNED &
touch /tmp/$ME.$F1$WID #if this loops faster than file creation in the worker thread it steps on itself, thread tracking is best in main loop
SPAWNED=$[$SPAWNED+1]
(flock -x -w 10 201
status
)201>/tmp/$ME.lock
sleep $TPS
if ((! $[$SPAWNED%100]));then
#rethink thread timing every 100 threads
threadspeed
fi
done
sleep $TPS
done
while [ "$(threadcount)" -gt 0 ]
do
(flock -x -w 10 201
status
)201>/tmp/$ME.lock
sleep 1;
done
status
}
clear
threadspeed
main
wait
status
echo

Related

How to exit a bash loop and stop the script but only when all the iterations are completed

I have this code, which basically does a loop inside the DF command to see which disks are more than 90% full.
df -H | sed 1d | awk '{ print $5 " " $1 }' | while read -r dfh;
do
#echo "$output"
op=$(echo "$dfh" | awk '{ print $1}' | cut -d'%' -f1 )
partition=$(echo "$dfh" | awk '{ print $2 }' )
if [ $op -ge 90 ];
then
echo ">> ### WARNING! Running out of space on \"$partition ($op%)\" on $(hostname) as on $(date)" >> LOGFILE
echo -e ">> ### WARNING! Running out of space on \"$partition ($op%)\" on $(hostname) as on $(date)"
echo ">> There is not enough left storage in the disk to perform the upgrade, exiting..." >> LOGFILE
echo -e ">> There is not enough left storage in the disk to perform the upgrade, exiting..."
exit 1
elif [ $op -ge 85 ];
then
echo -e ">> ### WARNING! Running out of space on \"$partition ($op%)\" on $(hostname) as on $(date)" >> LOGFILE
echo ">> ### WARNING! Running out of space on \"$partition ($op%)\" on $(hostname) as on $(date)"
echo ">> There enough left storage in the disk to perform the upgrade, but it is recommended to first increase the size of the disk $partition" >> LOGFILE
echo -e ">> There enough left storage in the disk to perform the upgrade, but it is recommended to first increase the size of the disk $partition"
fi
done
if [ "$?" -eq 1 ];
then
exit
else
echo -e ">> There is enough left storage in the disk to continue with the upgrade, OK"
fi
I want it to exit only if at least one disk is more than 90% full, that's the pourpose of the last if statement
The problem here is that the loop exits on the first disk recognised at more than 90%, this is bad because if I have 3 disks at more than 90% it will only report one of them (the first one in the loop) and then exit. Basically I want the script to report all the disks that are at 90% or more (and the ones that are at 85% too but without exiting, as you can read).
Is this possible?
Thank you in advance
Your script looks like it's screaming loudly to be refactored into Awk. But here is a more gentle refactoring.
rc=0
df -H |
awk 'NR>1 { n=$5; sub(/%/, "", n); print n, $1 }' |
while read -r op partition;
do
if [ $op -ge 90 ];
then
tee -a LOGFILE <<-________EOF
>> ### WARNING! Running out of space on "$partition ($op%)" on $(hostname) as on $(date)
>> There is not enough left storage in the disk to perform the upgrade, exiting...
________EOF
rc=1
elif [ $op -ge 85 ];
then
tee -a LOGFILE <<-________EOF
>> ### WARNING! Running out of space on "$partition ($op%)" on $(hostname) as on $(date)
>> There enough left storage in the disk to perform the upgrade, but it is recommended to first increase the size of the disk $partition
________EOF
fi
[ "$rc" -eq 0 ]
done &&
echo ">> There is enough left storage in the disk to continue with the upgrade, OK" ||
exit 1
This keeps track of rc while looping over all the partitions, then proceeds with the final condition only when the loop is done.
Generally, avoid echo -e in favor of printf, though here, since the -e wasn't doing anything useful anyway, a here document worked better.
You could do something like:
#!/bin/sh
warn=${1-85}
fail=${2-90}
test "$warn" -lt "$fail" || { echo Invalid parameters >&2; exit 1; }
check_disk(){
op=${1%%%}
partition=${2}
if test "$op" -ge "$warn"; then
tee -a LOGFILE <<-EOF
>> ### WARNING! Running out of space on "$partition ($op%)" on $(hostname) as on $(date)
>> There is not enough left storage in the disk to perform the upgrade, exiting...
EOF
fi
if test "$op" -ge "$fail"; then
return 1
fi
} 2> /dev/null
rv=0
df -H | awk 'NR > 1{ print $5 " " $1 }' \
| { while read -r op partition; do
if ! check_disk "$op" "$partition"; then rv=1; fi
done
test "$rv" -eq 0
} || exit 1
A few notes:
You must have literal hard tabs as indentation if you want the <<- to suppress the indentation in the output.
You need to put the while/done in the brackets to give the rv variable the full scope to be able to check it. If you just do df ... | while do/done, then the variable will be unset outside of the pipeline.
This is still a bit kludgy, and it would probably be better to do the whole thing in awk instead of having the while/do loop in the shell at all, but these are some ideas. In particular, you definitely do not want to be writing the same echo line 4 times!
Also note that parsing the output of df is really fragile. On my box, there are lines in which the filesystem column contains whitespace, so that the 5th column is not the current usage of the mountpoint. This script will probably not work as desired on such output.
You can do it in one check, like this:
$ a=10 b=5 c=7 ; (( a>=10 && b>=10 && c>=10 )) && echo 'all >= 10'
$ a=10 b=5 c=10; (( a>=10 && b>=10 && c>=10 )) && echo 'all >= 10'
$ a=10 b=10 c=10; (( a>=10 && b>=10 && c>=10 )) && echo 'all >= 10'
all >= 10

Stop a loop process in shell scripting after some time

I have a script that looks like this, this script is checking whether my pods is in Running state or Not by redefining x on every loop.
x=$(/opt/oc get pods --selector app=${bamboo.shortPlanName} -o jsonpath='{range .items[]}{.status.phase}{"\n"}{end}')
until [ "$x" == "Running" ];
do
sleep 5
x=$(/opt/oc get pods --selector app=${bamboo.shortPlanName} -o jsonpath='{range .items[]}{.status.phase}{"\n"}{end}')
echo $x
done
But, I want to modify my current script to also support timeout, I mean after 60 second of looping, then it should be stop, or after 12 times of looping, then it should be stop. Any idea how to do that?
bash provides a special variable SECONDS that can be used as a rough timer.
SECONDS=0
while (( SECONDS < 60)); do
x=$(/opt/oc get pods --selector app=${bamboo.shortPlanName} -o jsonpath='{range .items[]}{.status.phase}{"\n"}{end}')
if [[ $x == Running ]]; then
break
fi
sleep 5
done
The expansion of SECONDS gives you not the assigned value, but the difference between the numbers of seconds since the assignment and the assigned value. The effect is like a variable whose value is incremented by 1 each second.
For timeout after 60 seconds try this Shellcheck-clean code:
#! /bin/bash -p
readonly kOC_SLEEP_SECS=5
readonly kOC_TIMEOUT_SECS=60
initial_secs=$SECONDS
while
status=$(/opt/oc get pods --selector app=bamboo.shortPlanName \
-o jsonpath='{range .items[]}{.status.phase}{"\n"}{end}')
printf '%s\n' "$status"
[[ $status != Running ]]
do
if (( (SECONDS - initial_secs) >= kOC_TIMEOUT_SECS )); then
echo 'ERROR: Timed out' >&2
exit 1
fi
sleep -- "$kOC_SLEEP_SECS"
done
I replaced app=${bamboo.shortPlanName} with app=bamboo.shortPlanName because the old code was causing Bash errors. You'll need to fix it properly.
See Why is printf better than echo? for an explanation of why I replaced echo with printf for printing the status.
The code treats a timeout as an error, and exits with bad status. You might want to do something different.
The actual timeout period will be somewhere between 60 and 65 (or maybe a little more) seconds. You'll need to do something different if you need a more accurate timeout.
For a timeout after 12 iterations try this Shellcheck-clean variation on the code above:
#! /bin/bash -p
readonly kOC_SLEEP_SECS=5
readonly kOC_MAX_ITERS=12
oc_iters=0
while
status=$(/opt/oc get pods --selector app=bamboo.shortPlanName \
-o jsonpath='{range .items[]}{.status.phase}{"\n"}{end}')
printf '%s\n' "$status"
[[ $status != Running ]]
do
if (( ++oc_iters >= kOC_MAX_ITERS )); then
echo 'ERROR: Timed out' >&2
exit 1
fi
sleep -- "$kOC_SLEEP_SECS"
done
I think trap is the easiest way to have an accurate timeout.
Prototype:
#! /bin/bash
HALT=0
TIMEOUT=4
# Trap for SIGALRM
stop_loop() {
HALT=1
}
# Set trap
trap stop_loop SIGALRM
# The timeout goes after $TIMEOUT seconds.
{ sleep $TIMEOUT && kill -SIGALRM $$ & }
# Main loop
until false || [[ $HALT -eq 1 ]]; do
sleep 1
echo 'loop'
done
echo 'out of loop'
exit 0
In your case, this looks a something like:
HALT=0
TIMEOUT=4
stop_loop() {
HALT=1
}
trap stop_loop SIGALRM
x=$(/opt/oc get pods --selector app=${bamboo.shortPlanName} \
-o jsonpath='{range .items[]}{.status.phase}{"\n"}{end}')
{ sleep $TIMEOUT && kill -SIGALRM $$ & }
until [ "$x" == "Running" ] || [[ $HALT -eq 1 ]];
do
sleep 5
x=$(/opt/oc get pods --selector app=${bamboo.shortPlanName} \
-o jsonpath='{range .items[]}{.status.phase}{"\n"}{end}')
echo $x
done
exit 0

Bash script unexpected end of file

I have the following code:
#!/bin/bash -x
# Arguments:
# $1 - command to run
# $2 - time limit (in milliseconds)
# $3 - memory limit (in kilobytes)
# NOTE TO SELF: $! is the pid of last process
# NOTE TO SELF: Appending & creates new process
dir=$(mktemp -d)
ulimit -m $3
{ $1 ; "$?" > "$dir/retc" } &
pid=$!
./sleep.pl $2
if [ ps -p $pid > /dev/null ]
then
kill -9 $pid
echo "0Time Limit Exceeded"
else
echo "NAH"
ret=$(cat "$dir/retc")
if [ $ret = 9 ]
then
echo "0Memory Limit Exceeded"
else
if [ $ret = 0 ]
then
echo "1" # If it only returns one then it must be passed through final phase of verifying if result is correct
else
echo "0Received signal $ret"
fi
fi
fi
rm -r $dir
exit 0
However, it returns an error "Unexpected end of file", without executing anything in the if/else blocks.
You missed the semicolon, the redirection is wrong also:
{ $1 ; "$?"; } >"$dir/retc" &
From bash man: list must be terminated with a newline or semicolon
Also:
if [ ps -p $pid > /dev/null ]
Should be:
if [[ $(ps -p $pid >/dev/null) -eq 0 ]]
I'd recommend using () over {} to explicitly show that you're wanting to summon a subshell. Also I think you missed echo to show the value of $?.
( "$1"; echo "$?" > "$dir/etc"; ) &
With () you wouldn't need to add a semicolon in the end unlike with {} however it's still a good practice.

unix script to collect the thread dump logs of a process

I am trying to generate an issue regarding HashMap.put function.I have written a test code which will run more than 100 threads.. by using jstack or kill I'am able to get the thread dump of a particular thread of my process..The problem is I can not capture the thread dump immediately, I want all the thread dumps to be logged in a file until the process ends.Is there any linux command or shell script that can write to do this?
#!/bin/bash
if [ $# -eq 0 ]; then
echo >&2 "Usage: jstackSeries [ <count> [ <delay> ] ]"
echo >&2 " Defaults: count = 10, delay = 1 (seconds)"
exit 1
fi
pid=$1 # required
count=${2:-10} # defaults to 10 times
delay=${3:-1} # defaults to 1 second
while [ $count -gt 0 ]
do
jstack $pid >jstack.$pid.$(date +%H%M%S.%N)
sleep $delay
let count--
echo -n "."
done
refer here:
http://howtodoinjava.com/2012/12/19/how-to-get-thread-dump-in-linux-using-jstack/

Tailing log until certain time threshold is reached

I'm tailing a log like this:
while [[ ! -n "${ready}" ]]; do
start_info=`grep "Ready" $LOG_FILE`
sleep 10
done
If the log file doesn't have "Ready" inside this goes on forever, how can I make it run for lets say 200 seconds? Like some kind of time threshold.
Thanks
This avoids repeatedly grepping the whole file:
start=$SECONDS
limit=200
while read -r line
do
if [[ $line =~ $ready ]]
then
start_info=$line
break
fi
if (( $SECONDS >= start + limit ))
then
break
fi
done < "$LOG_FILE"
If I got your question correctly (while [[ ! -n "${ready}" ]]; do is confusing) here is an example how you can check time threshold:
#!/bin/sh
...
timelimit=200
pausetime=10
while [[ -z "${start_info}" ]]; do
start_info=`grep "Ready" $LOG_FILE`
sleep $pausetime
timelimit=$((timelimit - $pausetime))
if [ $timelimit -le 0 ]; then
break
fi
done

Resources