Finding Duplicate rows based on a column in Unix File - linux

Question

I have a file of about 1 Million records. I need to extract the records which have different FName and LName for id.
Input File
Col1,Col2,Col3,Col4,ID,FName,Col5,LName,Col6,Col7,Col8
AP,abc#gmail.com,xyz1,abc1,123,Ram,,Kumar,phn1,fax1,url1
AP,abc2#gmail.com,xyz2,abc2,123,Shyam,,Kumar,phn2,fax2,url1
AP,abc#gmail.com,xyz1,abc1,345,Raman,,Kumar,phn2,fax2,url1
AP,abc#gmail.com,xyz1,abc1,345,Raman,,Kumar,phn2,fax2,url1
AP,abc#gmail.com,xyz1,abc1,567,Alex,,Smith,phn2,fax2,url1
AP,abc#gmail.com,xyz1,abc1,789,Allen,,Prack,phn2,fax2,url1
The result that I want to see
AP,abc#gmail.com,xyz1,abc1,123,Ram,,Kumar,phn1,fax1,url1
AP,abc2#gmail.com,xyz2,abc2,123,Shyam,,Kumar,phn2,fax2,url1
Any AWK or Sed command or script can help? Thanks

Answer 1

You may try this awk:
awk 'BEGIN {FS=OFS=","} {id = $5; name = $6 FS $8} id in map && map[id] != name {if (!done[id]++) print rec[id]; print} {map[id] = name; rec[id] = $0}' file
AP,abc#gmail.com,xyz1,abc1,123,Ram,,Kumar,phn1,fax1,url1
AP,abc2#gmail.com,xyz2,abc2,123,Shyam,,Kumar,phn2,fax2,url1
Or a bit more readable:
awk 'BEGIN {
FS=OFS=","
}
{
id = $5
# name variable to store fname, lname
name = $6 FS $8
}
# if this id is already stored as key in map and if it is there check
# if stored name is different from current name
id in map && map[id] != name {
# print previous record if not already printed
if (!done[id]++)
print rec[id]
# print current record
print
}
{
# store name by key as id in map array
# and store full record by key as id in rec array
map[id] = name
rec[id] = $0
}' file

Answer 2

Using GNU awk for arrays of arrays:
$ awk -F, '
{ vals[$5][$6 FS $8] = $0 }
END {
for ( id in vals ) {
if ( length(vals[id]) > 1 ) {
for (name in vals[id]) {
print vals[id][name]
}
}
}
}
' file
AP,abc#gmail.com,xyz1,abc1,123,Ram,,Kumar,phn1,fax1,url1
AP,abc2#gmail.com,xyz2,abc2,123,Shyam,,Kumar,phn2,fax2,url1
or if your input file is sorted by "id" as shown in your sample input then with any awk and without storing the input file in memory:
$ cat tst.awk
BEGIN { FS=OFS="," }
NR > 1 {
id = $5
name = $6 FS $8
if ( id == prevId ) {
if ( name != prevName ) {
if ( firstRec != "" ) {
print firstRec
firstRec = ""
}
print
}
}
else {
firstRec = $0
}
prevId = id
prevName = name
}
$ awk -f tst.awk file
AP,abc#gmail.com,xyz1,abc1,123,Ram,,Kumar,phn1,fax1,url1
AP,abc2#gmail.com,xyz2,abc2,123,Shyam,,Kumar,phn2,fax2,url1

Answer 3

This one-liner should do the job:
awk -F "," '!a[$5] {a[$5]=$0} a[$5]!=$0{print a[$5]; print $0; a[$5]=$0}' input_file.txt
Output:
AP,abc#gmail.com,xyz1,abc1,123,Ram,,Kumar,phn1,fax1,url1
AP,abc2#gmail.com,xyz2,abc2,123,Shyam,,Kumar,phn2,fax2,url1
Note that the entire lines are compared based on ID.

Answer 4

awk -F, -v id="123" '$1 == id { map[NR]=$0 } END { for(i in map) { print map[i] } }' file
With awk, set the field separator to a comma and pass a variable in called id. When the first field is equal to the passed id, add to an array called map, indexed by the record number and with the line as the value. At the end loop through the array and print the values.