Linux: I want to fetch the top 10 records of the column3. The column has some missing data. I have sorted the file - linux

Question

I am not sure how to ignore the missing data here.
My ; separated file looks like (writing it with spaces so that it is readable):
Col1 Col2 Col3 Col4 Col5
12 a ? ? ?
1 b ? ? ?
45 c 7.22 6.09 2.2
11 d 7.0 3.89 9.7
26 e 6.24 8.2 5.9
and so on....
I want to fetch the records with maximum values of column3
Col1 Col2 Col3 Col4 Col5
45 c 7.22 6.09 2.2
11 d 7.0 3.89 9.7
26 e 6.24 8.2 5.9
I sorted the file on column3 in reverse order. Not sure how to proceed further.
sort -t';' -k3 -r original.txt > newfile.txt

Answer 1

Something great about the command line is that you can easily use the best tool for the proper application, either chaining output with pipes | or by creating temporary files like newfile.txt.
In this case, using sort is the apt choice for sorting your data. Once it's sorted, you can use a separate tool that's very efficient at parsing data, awk.
Starting from your sorted newfile.txt, this awk operation will only print a line with 5 fields (assuming your missing data is actually missing and there are no empty separators, e.g. your line looks like 45;c; rather than 45;c;;;)
awk -F';' 'NF == 5 { print }' newfile.txt
However, in the case that the empty fields are delimited (e.g. 45;c;;;), and assuming that only columns 3 through 5 may have missing data, this will handle it:
awk -F';' 'NF == 5 && $3 && $4 && $5 { print }' newfile.txt
Note that since the default behavior of awk is to print, the above { print } is actually unnecessary, but included pedagogically.
Thus, from start to finish, you can get your desired result with,
sort -t ';' -rk3 original.txt | awk 'NF==5 && $3 && $4 && $5' > result.txt

Answer 2

You can use the following command:
$ head -1 fileToSort; (sort -k3 -n -r <(tail -n +2 fileToSort) | head)
Col1 Col2 Col3 Col4 Col5
45 c 7.22 6.09 2.2
11 d 7.0 3.89 9.7
26 e 6.24 8.2 5.9
1 b ? ? ?
12 a ? ? ?
where fileToSort is
cat fileToSort
Col1 Col2 Col3 Col4 Col5
12 a ? ? ?
1 b ? ? ?
45 c 7.22 6.09 2.2
11 d 7.0 3.89 9.7
26 e 6.24 8.2 5.9
Explanations:
Use -t';' if your field separator is ;
<(tail -n +2 fileToSort) will exclude the header of the input file
You then sort it via the 3rd key in reverse and numeric mode using -n
head will limit the output at the first 10 records
head -1 fileToSort; will print the header line before printing the top 10 records
If you need an awk solution:
awk 'NR==1;NF == 5 && $3~/^[0-9]+(\.[0-9]+)+$/ && $4~/^[0-9]+(\.[0-9]+)+$/ && $5~/^[0-9]+(\.[0-9]+)+$/{buff[$3]=$0}END{n=asorti(buff,out); for (i = n; i >= 1; i--){print buff[out[i]]}}' fileToSort
Col1 Col2 Col3 Col4 Col5
45 c 7.22 6.09 2.2
11 d 7.0 3.89 9.7
26 e 6.24 8.2 5.9
You might need to add -F';' just after the awk command if your file does have ; instead of spaces. Your command will become: awk -F';' ...
NR==1; print the first line
NF == 5 && $3~/^[0-9]+(\.[0-9]+)+$/ && $4~/^[0-9]+(\.[0-9]+)+$/ && $5~/^[0-9]+(\.[0-9]+)+$/ check that you have 5 fields and that the values of the 3 last columns are numerical
{buff[$3]=$0} save each line in a buffer indexed by the col3 value
END{n=asorti(buff,out); for (i = n; i >= 1; i--){print buff[out[i]]}} at the end of the processing just order the array depending on the value of the index and print it in the reverse order.