using grep commands to find a duplicate id within a json file - linux

Question

I am looking for a way to use grep on a linux server to find duplicate json records, is it possible to have a grep to search for duplicate id's in the example below ?
so the grep would return: 01
{
"book": [
{
"id": "01",
"language": "Java",
"edition": "third",
"author": "Herbert Schildt"
},
{
"id": "02",
"language": "Java",
"edition": "third",
"author": "Herbert Schildt"
},
{
"id": "03",
"language": "Java",
"edition": "third",
"author": "Herbert Schildt"
},
{
"id": "01",
"language": "Java",
"edition": "third",
"author": "Herbert Schildt"
},
{
"id": "04",
"language": "C++",
"edition": "second",
"author": "E.Balagurusamy"
}
]
}

Answer 1

use grep along with uniq.
grep '"id":' filename | sort | uniq -d
The -d option only prints duplicates.
However, this depends on the JSON being laid out neatly. To handle more general formatting, I recommend you use the jq utility.

Answer 2

A jq-based approach:
jq -r '.book[].id' < in.json | sort | uniq -d
01
This should work even for minified JSON files with no newlines.

Answer 3

OK, discarding any whitespace from the JSON strings I can offer this if awk is acceptable - hutch being the formatted chunk of JSON above in a file.
I use tr to remove any whitespace, use , as a field separator in awk; iterate over the one long lines elements with a for-loop, do some pattern-matching in awk to isolate ID fields and increment an array for each matched ID. At the end of processing I iterate over the array and print ID's that have more than one match.
Here your data:
$ cat hutch
{
"book": [
{
"id": "01",
"language": "Java",
"edition": "third",
"author": "Herbert Schildt"
},
{
"id": "02",
"language": "Java",
"edition": "third",
"author": "Herbert Schildt"
},
{
"id": "03",
"language": "Java",
"edition": "third",
"author": "Herbert Schildt"
},
{
"id": "01",
"language": "Java",
"edition": "third",
"author": "Herbert Schildt"
},
{
"id": "04",
"language": "C++",
"edition": "second",
"author": "E.Balagurusamy"
}
]
}
And here the finding of dupes:
$ tr -d '[:space:]' <hutch | awk -F, '{for(i=1;i<=NF;i++){if($i~/"id":/){a[gensub(/^.*"id":"([0-9]+)"$/, "\\1","1",$i)]++}}}END{for(i in a){if(a[i]>1){print i}}}'
01

Answer 4

Use a Perl one-liner to extract the numeric ids, then sort | uniq -d to print only the duplicates (as in the answer by Barmar):
This assumes that the id key/value pair is on the same line, but disregards whitespace (or lack of whitespace) anywhere on the line (leading, trailing, and in between):
perl -lne 'print for /"id":\s*"(\d+)"/' in.json | sort | uniq -d
This makes no assumptions (disregards whitespace and newlines). Note that it reads the entire json file into memory (using the -0777 command line switch):
perl -0777 -nE 'say for /"id":\s*"(\d+)"/g' in.json | sort | uniq -d
The Perl one-liner uses these command line flags:
-e : Tells Perl to look for code in-line, instead of in a file.
-E : Tells Perl to look for code in-line, instead of in a file. Also enables all optional features. Here, enables say.
-n : Loop over the input one line at a time, assigning it to $_ by default.
-l : Strip the input line separator ("\n" on *NIX by default) before executing the code in-line, and append it when printing.
-0777 : Slurp files whole.
The regex uses this modifier:
/g : Multiple matches.
SEE ALSO:
perldoc perlrun: how to execute the Perl interpreter: command line switches
perldoc perlre: Perl regular expressions (regexes)
perldoc perlre: Perl regular expressions (regexes): Quantifiers; Character Classes and other Special Escapes; Assertions; Capture groups
perldoc perlrequick: Perl regular expressions quick start