Optimizing Bash script, subshell removal - linux

Question

I have a bash script that lists the amount of ip addresses connected on a port. My issue is, is that with large amounts of connections it is slow as poo. I think it is because of the subshells in use, but I am having trouble removing them without borking the rest of the script. Here is the script in its entirety as it is fairly short:
#!/bin/bash
portnumber=80
reversedns_enabled=0
[ ! -z "${1}" ] && portnumber=${1}
[ ! -z "${2}" ] && reversedns_enabled=${2}
#this will hold all of our ip addresses extracted from netstat
ipaddresses=""
#get all of our connected ip addresses
while read line; do
ipaddress=$( echo ${line} | cut -d' ' -f5 | sed s/:[^:]*$// )
ipaddresses="${ipaddresses}${ipaddress}\n"
done < <( netstat -ano | grep -v unix | grep ESTABLISHED | grep \:${portnumber} )
#remove trailing newline
ipaddresses=${ipaddresses%%??}
#output of program
finaloutput=""
#get our ip addresses sorted, uniq counted, and reverse sorted based on amount of uniq
while read line; do
if [[ ${reversedns_enabled} -eq 1 ]]; then
reversednsname=""
#we use justipaddress to do our nslookup(remove the count of uniq)
justipaddress=$( echo ${line} | cut -d' ' -f2 )
reversednsstring=$( host ${justipaddress} )
if echo "${reversednsstring}" | grep -q "domain name pointer"; then
reversednsname=$( echo ${reversednsstring} | grep -o "pointer .*" | cut -d' ' -f2 )
else
reversednsname="reverse-dns-not-found"
fi
finaloutput="${finaloutput}${line} ${reversednsname}\n"
else
finaloutput="${finaloutput}${line}\n"
fi
done < <( echo -e ${ipaddresses} | uniq -c | sort -r )
#tabulate that sheet son
echo -e ${finaloutput} | column -t
The majority of the time spent is doing this operation: echo ${line} | cut -d' ' -f5 | sed s/:[^:]*$// what is the best way to inline this to produce a faster script. It takes well over a second with 1000 concurrent users (which is my base target, although should be able to process more without using up all of my cpu).

Answer 1

You could reduce that with cut -d' ' <<< "$line" | sed .... You could write a more complex sed script and avoid the use of cut.
But the real benefit would be in avoiding the loop so there's only one sed (or awk or perl or …) script involved. I'd probably look to reduce it to ipaddresses=$(netstat -ano | awk '...') so that instead of 3 grep processes, plus one cut and sed per line, there was just a single awk process.
ipaddresses=$(netstat -ano |
awk " /unix/ { next } # grep -v unix
!/ESTABLISHED/ { next } # grep ESTABLISHED
!/:${portnumber}/ { next } # grep :${portnum} "'
{ sub(/:[^:]*$/, "", $5); print $5; }'
)
That's probably rather clumsy, but it is a fairly direct transliteration of the existing code. Watch for the quotes to get ${portnumber} into the regex.
Since you feed the list of IP addresses into uniq -c and sort -r. You probably should use sort -rn, and you could use awk to do the uniq -c, too.
The only bit that you can't readily improve is host; that seems to only take one host or IP address argument at a time, so you have to run it for each name or address.

Answer 2

I'll take a stab at a couple of issues:
The following line from the script which performs incremental string concatenation will not be be efficient without the means to allocate a reasonable buffer:
ipaddresses="${ipaddresses}${ipaddress}\n"
For another, using a while loop with read line when a pipeline will do is significantly worse than the pipeline. Try something like this instead of the first loop:
netstat -ano |
grep -v 'unix' |
grep 'ESTABLISHED' |
grep "\:${portnumber}" |
cut -d' ' -f5 |
sed 's/:[^:]*$//' |
while read line; do ...
Also, try combining at least two of the three sequential grep commands into one invocation of grep.
If nothing else, this will mean you are no longer spawning a pipeline which creates new cut and sed processes for each line of input processed in the first loop.

Answer 3

Here is a whole script optimized & refactored:
#!/bin/bash
portnumber=80
reversedns_enabled=0
[[ $1 ]] && portnumber=$1
[[ $2 ]] && reversedns_enabled=$2
#this will hold all of our ip addresses extracted from netstat
ipaddresses=''
#get all of our connected ip addresses
while IFS=' :' read -r type _ _ _ _ ipaddress port state _; do
if [[ $type != 'unix' && $port == "$portnumber" && $state == 'ESTABLISHED' ]]; then
ipaddresses+="$ipaddress\n"
fi
done < <(netstat -ano)
#remove trailing newline
ipaddresses=${ipaddresses%%??}
#output of program
finalOutput=""
#get our ip addresses sorted, uniq counted, and reverse sorted based on amount of uniq
while read -r line; do
if (( reversedns_enabled == 1 )); then
reverseDnsName=""
#we use justipaddress to do our nslookup(remove the count of uniq)
read -r _ justipaddress _ <<< "$line"
reverseDnsString=$(host "$justipaddress")
if [[ $reverseDnsString == *'domain name pointer'* ]]; then
reverseDnsName=${reverseDnsName##*domain name pointer }
else
reverseDnsName="reverse-dns-not-found"
fi
finalOutput+="$line $reverseDnsName\n"
else
finalOutput+="$line\n"
fi
done < <(echo -e "$ipaddresses" | sort -ur)
#tabulate that sheet son
echo -e "$finalOutput" | column -t
As you can see, there are almost no external tools used (no sed, awk or grep). Awesome!