I have this data to sort. The 1st column is the item ID. The 2nd column is the numerical value. Some items do not have a numerical value.
03875334 -4.27
03860156 -7.27
03830332
19594535 7.87
01542392 -5.74
01481815 11.45
04213946 -10.06
03812865 -8.67
03831625
01552174 -9.28
13540266 -8.27
03927870 -7.25
00968327 -8.09
I want to use the Linux sort command to sort the items numerically in the ascending order of their value, but leave those empty items to the end. So, this is the expected output I want to obtain:
04213946 -10.06
01552174 -9.28
03812865 -8.67
13540266 -8.27
00968327 -8.09
03860156 -7.27
03927870 -7.25
01542392 -5.74
03875334 -4.27
19594535 7.87
01481815 11.45
03830332
03831625
I tried "sort -k2n" and "sort -k2g", but neither yielded the output I want. Any idea?
Here is a simple Schwartzian transform based on the assumption that all actual values are smaller than 123456789.
awk '{ printf "%s\t%s", ($2 || 123456789), $0 }' file |
sort -n | cut -f2- >output
Assuming data is in d.txt and blanks have 4 spaces at the end
egrep " $" d.txt > blanks.txt ; egrep -v " $" d.txt | sort -n -k2 | cat - blanks.txt
This should work:
awk '$2 ~ /[0-9]$/' d.txt | sort -k2g && awk '$2 !~ /[0-9]$/' d.txt
Related
Can you please tell me how to Select rows with min value based on fourth column and group by first column in linux?
Original file
x,y,z,w
1,a,b,0.22
1,a,b,0.35
1,a,b,0.45
2,c,d,0.06
2,c,d,0.20
2,c,d,0.46
3,e,f,0.002
3,e,f,0.98
3,e,f,1.0
The file I want is as below.
x,y,z,w
1,a,b,0.22
2,c,d,0.06
3,e,f,0.002
I tried as below but this does not work.
sort -k1,4 -u original_file.txt | awk '!a[$1] {a[$1] = $4} $4 == a[$1]' >> out.txt
You should just sort by column 4. You need to store the entire line in the array, not just $4. And then print the entire array at the end.
To keep the heading from getting mixed in, I print that separately and then process the rest of the file.
head -n 1 original_file
tail -n +2 original_file | sort -t, -k 4n -u | awk -F, '
!a[$1] { a[$1] = $0 }
END { for (k in a) print a[k] }' | sort -t, -k 1,1n >> out
this is my file: FILEABC.txt
Name|address|age|country
john|london|12|UK
adam|newyork|39|US|X12|123
jake|madrid|45|ESP
ram|delhi
joh|cal|34|US|788
I wanted to find the the header count in the file. so i've this command
cat FILEABC.txt | awk --field-separator='|' '{print NF}' | sort -n |uniq -c
the result i get for this cmd is
cat FILEABC.txt | awk --field-separator='|' '{print NF}' | sort -n |uniq -c
1 2
3 4
1 5
1 6
My requirement is that, how do i find those records that have only 2 fields, 4 fields and so on from my file.
for ex,
if want to see the records having only 2 col:
ram|delhi
if want to see rec's having more than 4 col:
adam|newyork|39|US|X12|123
If you want to only print the records which have 2 fields then following may help you in same.
awk -F"|" 'NF==2' Input_file
For any kind of records if you need a line which has more than 4 fields then change above condition to NF>4 or you need line which have more than 5 fields eg--> NF>5
Explanation: BY doing -F"|" I am making sure field separator is pipe here, then NF is an awk out of the box variable which defines the TOTAL number of fields in a line, so as per your request checking if number of fields are more than 2 here, if this condition is TRUE then print the current line(where I have NOT written print because awk works on method of condition and action, so if condition is TRUE here I am not mentioning any action and by default action print will happen for that line).
Using awk, variable NF gives total number of fields in record/row, by default awk use single space as field separator, if you alter FS, it will calculate NF based on field separator mentioned, so what you can do is
awk -v FS='|' 'NF==2' infile
Which is same as
# Usual Syntax : awk 'condition { action }' infile
awk -v FS='|' 'NF==2{ print }' infile
For more than 4 fields,
awk -v FS='|' 'NF > 4' infile
you can also use grep to filter 2-columed records:
grep '^[^|]*|[^|]*$' FILEABC.txt
It will output:
ram|delhi
How do I use awk on a file that looks like this:
abcd Z
efdg Z
aqbs F
edf F
aasd A
I want to extract the number of times each letter of the alphabet occurs in the second column, so output should be:
Z 2
F 2
A 1
try: If you want the order of output same as Input_file then following may help you.
awk 'FNR==NR{A[$2]++;next} A[$2]{print $2,A[$2];delete A[$2]}' Input_file Input_file
if you don't bother of order of $2 then following may help you.
awk '{A[$2]++} END{for(i in A){print i,A[i]}}' Input_file
In first solution reading the Input_file twice and creating an array A whose index is $2 with it's incrementing value. then when second Input_file is being read then printing the $2 and it's count.
In Second solution creating an array A whose index $2 and incrementing value of it. Then in end section go through the array A and print it's index and array A's value.
I would use sort | uniq for this purpose as these two utils are designed specifically for this kind of task:
cat <<END |
abcd Z
efdg Z
aqbs F
edf F
aasd A
END
awk '{print $2}' | sort -r | uniq -c | awk '{printf "%s %d\n", $2, $1}'
Would produce exactly the desired output
Z 2
F 2
A 1
Here awk '{print $2}' is used to get the second column from a document with fields separated by one or more whitespace characters. If we knew the width of the columns is fixed, we could use a faster cut utility instead.
sort -r | uniq -c is doing the main algorithmic part of the task - sort the letters in reverse order and count the number of occurrences of each letter.
awk '{printf "%s %d\n", $2, $1}' does some reformatting of the uniq -c output to match the required format exactly.
Update: AWK has powerful array support so this can be done with awk alone:
cat <<END |
abcd Z
efdg Z
aqbs F
edf F
aasd A
END
awk '{a[$2]++}
END {n=asorti(a,b,"#ind_str_desc");
for (k=1;k<=n;k++) {printf b[k], a[b[k]]} }'
We use the array a that is indexed with letters found in the input stream, and on each line the element indexed by the corresponding letter gets incremented.
In the END clause we reverse the order of indices and output the array.
I am trying to print only max values of column two for dates in column one.
My file is:
2014-04-09,135303
2014-04-09,416400
2014-04-15,143684
2014-04-15,156011
2014-04-15,184406
2014-04-16,1123083
2014-04-16,167486
2014-04-16,862196
2014-04-17,963023
2014-04-19,583844
Required Output:
2014-04-09,416400
2014-04-15,184406
2014-04-16,1123083
2014-04-17,963023
2014-04-19,583844
I tried sort but not working:
cat file|sort -k2 -r | sort --unique --stable -k1
please suggest how it can be done using awk or sort
kent$ awk -F, '{a[$1]=$2>a[$1]?$2:a[$1]}END{for(x in a)print x "," a[x]}' file
2014-04-15,184406
2014-04-16,1123083
2014-04-17,963023
2014-04-09,416400
2014-04-19,583844
if you want the result ordered by date, pipe the line above to sort:
awk -F, '{a[$1]=$2>a[$1]?$2:a[$1]}END{for(x in a)print x "," a[x]}' f|sort
2014-04-09,416400
2014-04-15,184406
2014-04-16,1123083
2014-04-17,963023
2014-04-19,583844
I have a text file with a large amount of data which is tab delimited. I want to have a look at the data such that I can see the unique values in a column. For example,
Red Ball 1 Sold
Blue Bat 5 OnSale
...............
So, its like the first column has colors, so I want to know how many different unique values are there in that column and I want to be able to do that for each column.
I need to do this in a Linux command line, so probably using some bash script, sed, awk or something.
What if I wanted a count of these unique values as well?
Update: I guess I didn't put the second part clearly enough. What I wanted to do is to have a count of "each" of these unique values not know how many unique values are there. For instance, in the first column I want to know how many Red, Blue, Green etc coloured objects are there.
You can make use of cut, sort and uniq commands as follows:
cat input_file | cut -f 1 | sort | uniq
gets unique values in field 1, replacing 1 by 2 will give you unique values in field 2.
Avoiding UUOC :)
cut -f 1 input_file | sort | uniq
EDIT:
To count the number of unique occurences you can make use of wc command in the chain as:
cut -f 1 input_file | sort | uniq | wc -l
awk -F '\t' '{ a[$1]++ } END { for (n in a) print n, a[n] } ' test.csv
You can use awk, sort & uniq to do this, for example to list all the unique values in the first column
awk < test.txt '{print $1}' | sort | uniq
As posted elsewhere, if you want to count the number of instances of something you can pipe the unique list into wc -l
Assuming the data file is actually Tab separated, not space aligned:
<test.tsv awk '{print $4}' | sort | uniq
Where $4 will be:
$1 - Red
$2 - Ball
$3 - 1
$4 - Sold
# COLUMN is integer column number
# INPUT_FILE is input file name
cut -f ${COLUMN} < ${INPUT_FILE} | sort -u | wc -l
Here is a bash script that fully answers the (revised) original question. That is, given any .tsv file, it provides the synopsis for each of the columns in turn. Apart from bash itself, it only uses standard *ix/Mac tools: sed tr wc cut sort uniq.
#!/bin/bash
# Syntax: $0 filename
# The input is assumed to be a .tsv file
FILE="$1"
cols=$(sed -n 1p $FILE | tr -cd '\t' | wc -c)
cols=$((cols + 2 ))
i=0
for ((i=1; i < $cols; i++))
do
echo Column $i ::
cut -f $i < "$FILE" | sort | uniq -c
echo
done
This script outputs the number of unique values in each column of a given file. It assumes that first line of given file is header line. There is no need for defining number of fields. Simply save the script in a bash file (.sh) and provide the tab delimited file as a parameter to this script.
Code
#!/bin/bash
awk '
(NR==1){
for(fi=1; fi<=NF; fi++)
fname[fi]=$fi;
}
(NR!=1){
for(fi=1; fi<=NF; fi++)
arr[fname[fi]][$fi]++;
}
END{
for(fi=1; fi<=NF; fi++){
out=fname[fi];
for (item in arr[fname[fi]])
out=out"\t"item"_"arr[fname[fi]][item];
print(out);
}
}
' $1
Execution Example:
bash> ./script.sh <path to tab-delimited file>
Output Example
isRef A_15 C_42 G_24 T_18
isCar YEA_10 NO_40 NA_50
isTv FALSE_33 TRUE_66