Sort and counting a column without uniq in bash - linux

Question

I want to add a count of only the first column using bash, without doing uniq, like this:
input:
58311s2727 NC_000082.6 100.00 50
58311s2727 NC_000083.6 100.00 60
58311s2727 NC_000084.6 100.00 70
58310s2691 NC_000080.6 100.00 30
58310s2691 NC_000081.6 100.00 20
58308s2441 NC_000074.6 100.00 50
output:
3 58311s2727 NC_000082.6 100.00 50
3 58311s2727 NC_000083.6 100.00 60
3 58311s2727 NC_000084.6 100.00 70
2 58310s2691 NC_000080.6 100.00 30
2 58310s2691 NC_000081.6 100.00 20
1 58308s2441 NC_000074.6 100.00 50
I tried:
sort input.txt | cut -f1 | uniq -c
but the output is not what I want. I want to know if there will be simple ways to solve this.

Answer 1

With sorted input, you can simply use awk, capturing the set of lines that have the same key and printing the previous set out when the key changes. Handling EOF is a tad messy; you have to repeat the printing. You could write an awk function to do the printing, but it is almost overkill for something this simple.
script.awk
$1 != old_key { if (n_keys > 0) for (i = 0; i < n_keys; i++) print n_keys, saved[i]; n_keys = 0 }
{ saved[n_keys++] = $0; old_key = $1 }
END { if (n_keys > 0) for (i = 0; i < n_keys; i++) print n_keys, saved[i] }
Example runs
For the sample input input.txt (which is already grouped), the output is:
$ awk -f script.awk input.txt
3 58311s2727 NC_000082.6 100.00 50
3 58311s2727 NC_000083.6 100.00 60
3 58311s2727 NC_000084.6 100.00 70
2 58310s2691 NC_000080.6 100.00 30
2 58310s2691 NC_000081.6 100.00 20
1 58308s2441 NC_000074.6 100.00 50
$
If you want it sorted, sort it first:
$ sort input.txt | awk -f script.awk
1 58308s2441 NC_000074.6 100.00 50
2 58310s2691 NC_000080.6 100.00 30
2 58310s2691 NC_000081.6 100.00 20
3 58311s2727 NC_000082.6 100.00 50
3 58311s2727 NC_000083.6 100.00 60
3 58311s2727 NC_000084.6 100.00 70
$
Note that amongst other advantages, this can process data from a pipeline because it doesn't need to process the file twice, unlike at least one of the other solutions, which is currently accepted. It also only keeps as many lines in memory as there are lines in the biggest group of a common key, so even fairly big files are unlikely to stress the memory on the system. (The sort probably imposes more memory load than the awk does.)
script2.awk
Using a function, and some white space, the code becomes:
function dump_keys( i) {
if (n_keys > 0)
{
for (i = 0; i < n_keys; i++)
print n_keys, saved[i]
}
n_keys = 0
}
$1 != old_key { dump_keys() }
{ saved[n_keys++] = $0; old_key = $1 }
END { dump_keys() }
The variable i is local to the function (a quirk of awk). I could simply omit it from the argument list since i is not used elsewhere in the script.
This produces the same output as script.awk.

Answer 2

I would do this in awk. But as Aaron said, it will require reading the input twice, since the first time you hit a particular line, you don't know how many other times you'll hit it.
$ awk 'NR==FNR{a[$1]++;next} {print a[$1],$0}' inputfile inputfile
This goes through the file the first time, populating an array with a counter of the first field. Then it goes through a second time, printing the count along with each line.
You can adjust the print statement to suite your formatting requirements (perhaps replacing it with printf).
If you don't want to use awk, and really want this to work natively in bash, you could use a couple of one-liners with for loops to achieve nearly the same results:
$ declare -A a
$ while read word therest; do ((a[$word]++)); done < inputfile
$ while read word therest; do printf "%5d\t%s\t%s\n" "${a[$word]}" "$word" "$therest"; done < inputfile
The declare -A is required because $a needs to be an associative array, with the first word of each line as the key. awk, on the other hand, treats every array as associative. Note that this solution does not maintain your whitespace.

Answer 3

Without uniq, you'll have to read the input twice. There are ways to do that in pure BASH, but this is when I'd switch to a proper scripting language like Python 2:
import codecs
from collections import Counter
filename='...'
encoding='...' # file encoding
counter = Counter()
with codecs.open(filename, 'r', encoding) as fh:
for line in fh:
parts = line.split(' ')
counter[parts[0]] += 1
with codecs.open(filename, 'r', encoding) as fh:
for line in fh:
parts = line.split(' ')
count = counter[parts[0]]
print '%d%s' % (count, line),