Unable to get uniq -c in while loop - linux

Question

I'm trying to sort a log file and get a total count of unique IP addresses and do something with the ones that appear > than n. this is my first command:
$ grep -B 1 "foobar" ip.log | grep "IP Address" > ip_count.log
which outputs:
IP Address : 133.55.39.56
IP Address : 116.243.70.151
IP Address : 117.46.13.194
IP Address : 115.179.82.10
IP Address : 13.123.2.123
IP Address : 13.123.2.123
IP Address : 13.123.2.123
IP Address : 13.123.2.123
IP Address : 13.123.2.123
IP Address : 13.123.2.123
IP Address : 13.123.2.123
IP Address : 13.123.2.123
IP Address : 13.123.2.123
IP Address : 33.123.2.123
IP Address : 33.123.2.123
IP Address : 33.123.2.123
IP Address : 33.123.2.123
IP Address : 33.123.2.123
IP Address : 33.123.2.123
if i do this I get a proper count:
$ awk '{print $4}' ip_count.log | uniq -c
1 133.55.39.56
1 116.243.70.151
1 117.46.13.194
1 115.179.82.10
9 13.123.2.123
6 33.123.2.123
but if I do this I don't:
$ while read -r line ; do c=$(echo $line | awk '{print $4}' | \
uniq -c | awk '{print $1}') ; if [[ $c -gt 1 ]]; then echo "$line" ; \
fi ; done < ip_count.log
1 133.55.39.56
1 116.243.70.151
1 117.46.13.194
1 115.179.82.10
1 13.123.2.123
1 13.123.2.123
1 13.123.2.123
1 13.123.2.123
1 13.123.2.123
1 13.123.2.123
1 13.123.2.123
1 13.123.2.123
1 13.123.2.123
1 33.123.2.123
1 33.123.2.123
1 33.123.2.123
1 33.123.2.123
1 33.123.2.123
1 33.123.2.123
i'm not sure what i'm doing wrong within the loop, so maybe someone could tell me. there's probably some better way to combine all this into one command anyway, so any tips would be appreciated, thanks.

Answer 1

It seems no loop is needed :
uniq -c ip_count.log | awk '$1 > 1 { print $1" "$5 }'

Answer 2

When you read the file in the loop, it executes command row by row.
awk '{print $4}' ip_count.log | uniq -c
In the above, all records are read from stdin and process.
when you execute loop while read -r line your variable $line only contains 1 record instead of the whole file.
You can do it in single awk command using an associative array.
An alternative solution using awk
$awk -F':' '{ count=1; a[$2]=a[$2]+count} END {for (i in a) if (a[i]>1 )print i, a[i]} ' ip_count.log
33.123.2.123 6
13.123.2.123 9
$
Explanation:
-F':' --> we are setting custom field seprator as :
count=1; --> Initialize a variable as default value 2
a[$2]=a[$2]+count --> Create a associative array with $2 as index and count as values.
END --> Wait till { } code block execution is complete
{for (i in a) if (a[i]>1 )print i, a[i]} --> Loop through each value in array
Also, it's a good practice to pass sorted input to uniq command.
For example:
$echo -e "a\na\nb\na"
a
a
b
a
$echo -e "a\na\nb\na" | uniq -c
2 a
1 b
1 a
$echo -e "a\na\nb\na" | sort | uniq -c
3 a
1 b
$