When I use uniq -u data.txt lists the whole file and when I use sort data.txt | uniq -u it omits repeated lines. Why does this happen?
uniq man says that -u, --unique only prints unique lines. I don't understand why I need to use pipe to get correct output.
uniq removes adjacent duplicates. If you want to omit duplicates that are not adjacent, you'll have to sort the data first.
Linux RHEL5 machine
How can I sort the following input to get 1.0.0.1019 in latest variable? Tried -t, -k and -n but it didn't help or may be I'm missing something.
$ echo '1.0.0
1.0.0.1018
1.0.0.1019
1.0.0.1019
1.0.0.7' | sort -u
Could you please try following and let me know if this helps(tested with GNU sort):
echo "1.0.0
1.0.0.1018
1.0.0.1019
1.0.0.1019
1.0.0.7" | sort --version-sort --field-separator=. --key=4 -r
Above will give 1019 in first place(latest one) in case you want it to last place then remove -r in above code please.
sort -n -t. -k1,4
Sort the input numerically.
Fields are separated by '.'
Only use the first four fields, in that order.
How can I find the files from a folder where a specific word appears on more than 3 lines? I tried using recursive grep for finding that word and then using -c to count the number of lines where the word appears.
This command will recursively list the files in the current directory where word appears on more than 3 lines, along with the matches count for each file:
grep -c -r 'word' . | grep -v -e ':[0123]$' | sort -n -t: -k2
The final sort is not necessary if you don't want the results sorted, but I'd say it's convenient.
The first command in the pipeline (grep -c -r 'word' .) recursively finds every file in the current directory that contains word, and counts the occurrences for each file. The intermediate grep discards every count that is 0, 1, 2 or 3, so you just get counts greater than 3 (this is because -v in grep(1) inverts the sense of matching to select non-matching lines). The final sort step sorts the list according to the occurrences for each file; it sets the field delimiter to : and instructs sort(1) to do a numeric-based sorting using the 2nd field (the count) as the sort key.
Here's a sample output from some tests I ran:
./file1:4
./dir1/dir2/file3:5
./dir1/file2:8
If you just want the filenames without the match counts, you can use sed(1) to discard the :count portions:
grep -m 4 -c -r 'word' . | grep -v -e ':[0123]$' | sed -r 's/:[0-9]+$//'
As noted in the comments, if matches count is not important, in this case we can optimize the first grep with -m 4, which stops reading the file after 4 matching lines.
UPDATE
The solution above works fine up to a certain extent if used with small numbers, but it does not scale well for larger numbers. If you want to filter based on an arbitrary number, you can use awk(1) (and in fact it ends up being much more clean), like so:
grep -c -r 'word' . | awk -F: '$2 > 10'
The -F: argument to awk(1) is necessary; it instructs awk(1) to separate fields by : rather than the default (whitespace and tab). This solution generalizes well to any number.
Again, if matches count doesn't matter and all you want is to get a list of the filenames, do this instead:
grep -c -r 'word' . | awk -F: '$2 > 10 { print $1 }'
I have a shell script which is grepping the results of a file then it calls sort -u to get the unique entries. Is there a way to have sort also tell me how many of each of those entries there are? So the output would be something like:
user1 - 50
user2 - 23
user3 - 40
etc..
Use sort input | uniq -c. uniq does what -u does in sort -u, but also has the additional -c option for counting.
Grep has a -c switch to count the occurrence of each item..
grep -c needle haystack
will give the number of needles which you can sort as needed..
Given a sorted list, uniq -c will show the item, and how many. It will be the first column, so I will often do something like:
sort file.txt | uniq -c |sort -nr
The -n in the sort will parse numbers correctly, like 9 before 11 (though with the '-r', it will reverse the count, since I usually want the higher count lines first).
I want to sort a unix file by the id column but when I use sort -k4,4 or -k4,4n I do not get the expected result.
The column of interest should be sorted like this:
id1
id2
id3
id4
etc.
Instead it is sorted like this when i do sort -k4,4
id1
id10
id100
id1000
id10000
id10001
etc.
My unix version uses the following sort function:
sort --help
Usage: sort [OPTION]... [FILE]...
Write sorted concatenation of all FILE(s) to standard output.
Mandatory arguments to long options are mandatory for short options too.
Ordering options:
-b, --ignore-leading-blanks ignore leading blanks
-d, --dictionary-order consider only blanks and alphanumeric characters
-f, --ignore-case fold lower case to upper case characters
-g, --general-numeric-sort compare according to general numerical value
-i, --ignore-nonprinting consider only printable characters
-M, --month-sort compare (unknown) < `JAN' < ... < `DEC'
-n, --numeric-sort compare according to string numerical value
-r, --reverse reverse the result of comparisons
Other options:
-c, --check check whether input is sorted; do not sort
-k, --key=POS1[,POS2] start a key at POS1, end it at POS2 (origin 1)
-m, --merge merge already sorted files; do not sort
-o, --output=FILE write result to FILE instead of standard output
-s, --stable stabilize sort by disabling last-resort comparison
-S, --buffer-size=SIZE use SIZE for main memory buffer
-t, --field-separator=SEP use SEP instead of non-blank to blank transition
-T, --temporary-directory=DIR use DIR for temporaries, not $TMPDIR or /tmp;
multiple options specify multiple directories
-u, --unique with -c, check for strict ordering;
without -c, output only the first of an equal run
-z, --zero-terminated end lines with 0 byte, not newline
--help display this help and exit
--version output version information and exit
Use the -V or --version-sort option for version sort
sort -V -k4,4 file.txt
Example:
$ cat file.txt
id5
id3
id100
id1
id10
Ouput:
$ sort -V file.txt
id1
id3
id5
id10
id100
EDIT:
If your implementation of sort doesn't have the -V option then a work-around using sed to remove id so a numeric sort -n can be done and then replace id back with sed, like this:
sed -E 's/id([0-9]+)/\1/' file.txt | sort -n -k4,4 | sed -E 's/( *)([0-9]+)( *|$)/\1id\2\3/'
Note: this solution is dependent on the data, only works if no columns containing purely numbers are found before the ID column.
As sudo_o has already mentioned, the easiest would be to use --version-sort which does natural sorting of numbers that occur within text.
If your version of sort does not have that option, a hacky way to approach this would be to temporarily remove the "id" prefix before a sort, then replace them. Here's one way, using awk:
awk 'sub("^id", "", $4)' file.txt | sort -k4,4n | awk 'sub("^", "id", $4)'
If your sort supports it, you can also use the syntax F.C to use specific characters from a field.
This would sort on field 4, from chars 3 to 10, numerical value:
sort -bn -k 4.3,4.10 file
And this would sort on field 4, from chars 3 to end of field, numerical value:
sort -bn -k 4.3,4 file