Increment variable when matched awk from tail - linux

I'm monitoring from an actively written to file:
My current solution is:
ws_trans=0
sc_trans=0
tail -F /var/log/file.log | \
while read LINE
echo $LINE | grep -q -e "enterpriseID:"
if [ $? = 0 ]
then
((ws_trans++))
fi
echo $LINE | grep -q -e "sc_ID:"
if [ $? = 0 ]
then
((sc_trans++))
fi
printf "\r WSTRANS: $ws_trans \t\t SCTRANS: $sc_trans"
done
However when attempting to do this with AWK I don't get the output - the $ws_trans and $sc_trans remains 0
ws_trans=0
sc_trans=0
tail -F /var/log/file.log | \
while read LINE
echo $LINE | awk '/enterpriseID:/ {++ws_trans} END {print | ws_trans}'
echo $LINE | awk '/sc_ID:/ {++sc_trans} END {print | sc_trans}'
printf "\r WSTRANS: $ws_trans \t\t SCTRANS: $sc_trans"
done
Attempting to do this to reduce load. I understand that AWK doesn't deal with bash variables, and it can get quite confusing, but the only reference I found is a non tail application of AWK.
How can I assign the AWK Variable to the bash ws_trans and sc_trans? Is there a better solution? (There are other search terms being monitored.)

You need to pass the variables using the option -v, for example:
$ var=0
$ printf %d\\n {1..10} | awk -v awk_var=${var} '{++awk_var} {print awk_var}'
To set the variable "back" you could use declare, for example:
$ declare $(printf %d\\n {1..10} | awk -v awk_var=${var} '{++awk_var} END {print "var=" awk_var}')
$ echo $var
$ 10
Your script could be rewritten like this:
ws_trans=0
sc_trans=0
tail -F /var/log/system.log |
while read LINE
do
declare $(echo $LINE | awk -v ws=${ws_trans} '/enterpriseID:/ {++ws} END {print "ws_trans="ws}')
declare $(echo $LINE | awk -v sc=${sc_trans} '/sc_ID:/ {++sc} END {print "sc_trans="sc}')
printf "\r WSTRANS: $ws_trans \t\t SCTRANS: $sc_trans"
done

Related

Difficulty to create .txt file from loop in bash

I've this data :
cat >data1.txt <<'EOF'
2020-01-27-06-00;/dev/hd1;100;/
2020-01-27-12-00;/dev/hd1;100;/
2020-01-27-18-00;/dev/hd1;100;/
2020-01-27-06-00;/dev/hd2;200;/usr
2020-01-27-12-00;/dev/hd2;200;/usr
2020-01-27-18-00;/dev/hd2;200;/usr
EOF
cat >data2.txt <<'EOF'
2020-02-27-06-00;/dev/hd1;120;/
2020-02-27-12-00;/dev/hd1;120;/
2020-02-27-18-00;/dev/hd1;120;/
2020-02-27-06-00;/dev/hd2;230;/usr
2020-02-27-12-00;/dev/hd2;230;/usr
2020-02-27-18-00;/dev/hd2;230;/usr
EOF
cat >data3.txt <<'EOF'
2020-03-27-06-00;/dev/hd1;130;/
2020-03-27-12-00;/dev/hd1;130;/
2020-03-27-18-00;/dev/hd1;130;/
2020-03-27-06-00;/dev/hd2;240;/usr
2020-03-27-12-00;/dev/hd2;240;/usr
2020-03-27-18-00;/dev/hd2;240;/usr
EOF
I would like to create a .txt file for each filesystem ( so hd1.txt, hd2.txt, hd3.txt and hd4.txt ) and put in each .txt file the sum of the value from each FS from each dataX.txt. I've some difficulties to explain in english what I want, so here an example of the result wanted
Expected content for the output file hd1.txt:
2020-01;/dev/hd1;300;/
2020-02;/dev/hd1;360;/
2020-03;/dev/hd1;390:/
Expected content for the file hd2.txt:
2020-01;/dev/hd2;600;/usr
2020-02;/dev/hd2;690;/usr
2020-03;/dev/hd2;720;/usr
The implementation I've currently tried:
for i in $(cat *.txt | awk -F';' '{print $2}' | cut -d '/' -f3| uniq)
do
cat *.txt | grep -w $i | awk -F';' -v date="$(cat *.txt | awk -F';' '{print $1}' | cut -d'-' -f-2 | uniq )" '{sum+=$3} END {print date";"$2";"sum}' >> $i
done
But it doesn't works...
Can you show me how to do that ?
Because the format seems to be so constant, you can delimit the input with multiple separators and parse it easily in awk:
awk -v FS='[;-/]' '
prev != $9 {
if (length(output)) {
print output >> fileoutput
}
prev = $9
sum = 0
}
{
sum += $9
output = sprintf("%s-%s;/%s/%s;%d;/%s", $1, $2, $7, $8, sum, $11)
fileoutput = $8 ".txt"
}
END {
print output >> fileoutput
}
' *.txt
Tested on repl generates:
+ cat hd1.txt
2020-01;/dev/hd1;300;/
2020-02;/dev/hd1;360;/
2020-03;/dev/hd1;390;/
+ cat hd2.txt
2020-01;/dev/hd2;600;/usr
2020-02;/dev/hd2;690;/usr
2020-03;/dev/hd2;720;/usr
Alternatively, you could -v FS=';' and use split to split first and second column to extract the year and month and the hdX number.
If you seek a bash solution, I suggest you invert the loops - first iterate over files, then over identifiers in second column.
for file in *.txt; do
prev=
output=
while IFS=';' read -r date dev num path; do
hd=$(basename "$dev")
if [[ "$hd" != "${prev:-}" ]]; then
if ((${#output})); then
printf "%s\n" "$output" >> "$fileoutput"
fi
sum=0
prev="$hd"
fi
sum=$((sum + num))
output=$(
printf "%s;%s;%d;%s" \
"$(cut -d'-' -f1-2 <<<"$date")" \
"$dev" "$sum" "$path"
)
fileoutput="${hd}.txt"
done < "$file"
printf "%s\n" "$output" >> "$fileoutput"
done
You could also almost translate awk to bash 1:1 by doing IFS='-;/' in while read loop.

Total count of the array values

Here I'm accepting few mount points from the user and using each value to get space available on the host.
./user_input.ksh -string /m01,/m02,/m03
#!/bin/ksh
STR=$2
function showMounts {
echo "$STR"
arr=($(tr ',' ' ' <<< "$STR"))
printf "%s\n" "$(arr[#]}"
for x in "${arr[#]}"
do
free_space=`df -h "$x" | grep -v "Avail" | awk '{print $4}'`
echo "$x": free_space "$free_space"
done
#echo "$total_free_space"
}
Problems:
How can I exit for loop if any of the user input mount not avaialble?
currently it only add error in the log.
How to get total_free_space (i.e. sum of free_space)?
If you want to keep your code , test this (no ksh here). If you don't care, read Ed Morton's answer.
./user_input.ksh -string /m01,/m02,/m03
#!/bin/ksh
STR=$2
function showMounts {
echo "$STR"
arr=($(tr ',' ' ' <<< "$STR"))
printf "%s\n" "${arr[#]}"
for x in "${arr[#]}"; do
free_space=$(df -P "$x" | awk 'NR > 1 && !/Avail/{print $4}')
echo "$x: free_space $free_space"
((total_free_space+=$free_space))
done
echo "$((total_free_space/1024/1000))G"
}
showMounts
Caution:
"${arr[#]}"
not
"$(arr[#]}"
As I said in your last question, you do not need ANY of that, all you need is a one-liner like:
df -h "${STR//,/ }" | awk '/^ /{print $5, $3; sum+=$3} END{print sum}'
I have to say "like" because you haven't shown us the df -h /m01 /m02 /m03 output yet so I don't know exactly how to parse it.

Getting error while running script to find disk space

I am running below script:-
#!/bin/bash
threshold="20"
i=2
result=`df -kh |grep -v “Filesystem” | awk ‘{ print $5 }’ | sed ‘s/%//g’`
for percent in $result; do
if ((percent > threshold))
then
partition=`df -kh | head -$i | tail -1| awk ‘{print $1}’`
echo “$partition at $(hostname -f) is ${percent}% full”
fi
let i=$i+1
done
But I get the following error:
awk: ‘{
awk: ^ invalid char '▒' in expression
sed: -e expression #1, char 1: unknown command: `▒'
Please help me to resolve this.
What awk does not work? (your script does work fine on my Ubuntu)
This line:
result=`df -kh |grep -v "Filesystem" | awk '{ print $5 }' | sed 's/%//g'`
could be changed to:
result=$(df -kh | awk '!/Filesystem/ {print $5+0}')
Avoid using old and outdated backtics if parentheses works like this: var=$(code...)
This:
partition=`df -kh | head -$i | tail -1| awk '{print $1}'`
could be changed to:
partition=$(df -kh | awk -v line="$i" 'NR==line {print $1}')
This
let i=$i+1
could be change to:
((i++))
This would then give some like this:
#!/bin/bash
threshold="20"
i=2
result=$(df -kh | awk '!/Filesystem/ {print $5+0}')
for percent in $result; do
if ((percent > threshold))
then
partition=$(df -kh | awk -v line="$i" 'NR==line {print $1}')
echo "$partition at $(hostname -f) is ${percent}% full"
fi
((i++))
done
You're using ‘ for a single quote not '. Try re-encoding your file with an editor.
You got the answer to your syntax error, now re-write the whole script as just:
#!/bin/bash
df -kh |
awk -v t=20 -v h="$(hostname -f)" '(NR>1)&&($5+0>t){printf "%s at %s is %s full\n",$1,h,$5}'

bash command error

i=0
while read line
do
echo "i is --- $i"
#echo $line "\n"
if (( $i > 0 ))
then
$Eda_package=$(echo $line | awk '{print $1}')
$well_bias=$(echo $line | awk '{print $2}')
$biasmap=$(echo $line | awk '{print $3}')
$unified=$(echo $line | awk '{print $4}')
echo "eda pack --$Eda_package wellbias is --$well_bias biasmap is --$biasmap unified- -- $unified"
fi
i=$((i+1))
done < config.list
In the above bash program I get an error:
./script.sh: line 9: =EDA_7p0: command not found
How do I fix this?
Lines of the form:
$xyzzy=plugh
will have xyzzy substituted before they're executed so that they look like:
=plugh
assuming they're not yet set. If they are set, you'll probably get different behaviour but still almost certainly not what you want.
You should change your lines from (for one example):
$Eda_package=$(echo $line | awk '{print $1}')
to:
Eda_package=$(echo $line | awk '{print $1}')
The $ is not part of the variable name, it's an indication that the following word is a variable that should be substituted.
Let var1=1 and var2=2 now if you simply write $var2=$var1 then it will give you error that 2=1 command not found
When you initialize any variable you have to do it without $ with variable name on left side

How to use sed on Linux to get values from file name?

how to retrive every portion separately from following file name? DSA4020_frontcover_20346501_2011-05.doc
I want to retrieve informations as below;
name = DSA4020
type = frontcover
id = 20346501
date = 2011-05
is it possible to do with sed??
Yes, you can:
pax$ echo 'DSA4020_frontcover_20346501_2011-05.doc' | sed
-e 's/^/name=/'
-e 's/_/\ntype=/'
-e 's/_/\nid=/'
-e 's/_/\ndate=/'
-e 's/\..*//'
name=DSA4020
type=frontcover
id=20346501
date=2011-05
That's all on one line, I've just split it for readability.
You could also do it with awk if you wish:
pax$ echo 'DSA4020_frontcover_20346501_2011-05.doc'
| awk -F_ '{print "name="$1"\ntype="$2"\nid="$3"\ndate="substr($4,1,7)}'
name=DSA4020
type=frontcover
id=20346501
date=2011-05
awk may be a better choice
# f=DSA4020_frontcover_20346501_2011-05.doc
# name=$(echo $f | awk -F_ '{print $1}')
# echo $name
DSA4020
# type=$(echo $f | awk -F_ '{print $2}')
# echo $type
frontcover
In pure bash
FILE="DSA4020_frontcover_20346501_2011-05.doc"
eval $(echo $FILE |(IFS="_";read a b c d; echo "name=$a;type=$b;id=$c;date=${d%.doc}"))
echo Name:$name Type:$type ID:$id DATE:$date

Resources