How to combine files and search the result? - linux

Question

I have two sample files as follows:
File1
item name="NY" block="A" idnum="12345"
item name="NJ" block="B" idnum="123456"
item name="CT" block="C" idnum="1234567"
File2
ID_B|ID_C|NY|4|8198|10|2374|127
ID_C|ID_D|NJ|4|8198|10|2374|127
ID_D|ID_E|CT|4|8198|10|2374|127
I would like to be able to generate a file as passing ID as argument and output should look like this
If I am looking for info for ID_B then output should be
ID_B|ID_C|NY|4|8198|10|2374|127 => "NY" block="A" idnum="12345"
If am looking for two ID_C and ID_D together it should be
ID_C|ID_D|NJ|4|8198|10|2374|127 => "NJ" block="B" idnum="123456"
ID_D|ID_E|CT|4|8198|10|2374|127 => "CT" block="C" idnum="1234567"

Answer 1

With bash, join, sort and awk.
script.sh:
#!/bin/bash
file1="File1"
file2="File2"
grep -f <(printf "^%s|\n" "$#") <(join -t '|' -1 1 -2 3 <(awk -F'"' '{OFS="|"; print $2,$4,$6}' "$file1" | sort -t '|' -k 1,1) <(sort -t '|' -k 3,3 "$file2") -o 2.1,2.2,2.3,2.4,2.5,2.6,2.7,2.8,2.3,1.2,1.3 | awk -F'|' '{ printf "%s|%s|%s|%s|%s|%s|%s|%s => \"%s\" block=\"%s\" idnum=\"%s\"\n",$1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11 }')
Example: script.sh ID_C ID_D
Output:
ID_D|ID_E|CT|4|8198|10|2374|127 => "CT" block="C" idnum="1234567"
ID_C|ID_D|NJ|4|8198|10|2374|127 => "NJ" block="B" idnum="123456"

Answer 2

Something like
echo "Please enter the key"
read key
grep "item name=\"$(grep "^${key}|" file2 | cut -d"|" -f3)\"" file1
Edit:
I did not make the literal output, just wanted to show that you could grep the fields. A complete solution should look like:
#!/bin/bash
if [ $# -eq 0 ]; then
echo "Usage $0 id [id2 ..]"
exit 1
fi
for key in $*; do
f2=$(grep "^${key}|" file2 )
f1=$(grep "item name=\"$(echo "^${f2}" | cut -d"|" -f3)\"" file1 )
echo "${f2} => ${f1#item name=}"
done

Answer 3

awk, wrapped in a thin shell script
#!/bin/sh
awk -F '[|"]' -v OFS='"' -v keys="$*" '
BEGIN {
n = split(keys, a, " ")
for (i=1; i <= n; i++) key[a[i]] = 1
}
NR == FNR {
if ($1 in key) line[$3] = $0
next
}
$2 in line {
$1 = line[$2] " => "
print
}
' file2 file1
Then
$ sh script.sh ID_B ID_D
ID_B|ID_C|NY|4|8198|10|2374|127 => "NY" block="A" idnum="12345"
ID_D|ID_E|CT|4|8198|10|2374|127 => "CT" block="C" idnum="1234567"

Answer 4

What you want, is very similar to the JOIN in the SQL.
Unfortunately, implementing this in procedural languages like bash, awk or sed, requires dictionaries, or string-indexed string arrays. (And even with them would be it much more complex as a single-line sql query.) None of them supports it, although there are numerous possibilities to emulate it.
In your place I would use perl or python to solve this problem - on these languages would be your goal so simple, that even with the additional learning effort would it be around so effective as a bash workaround.
But you want a bash-awk-sed solution, so this is what I give you now.
Don't use these languages alone, use them integrated. The most simplest thing what you can do in bash, to use the external command grep.
For example, an integrated awk/bash core which partially does your the task (I didn't tested it):
awk 'BEGIN {FS="[_|]"} { print $2 " " $0}' file1|while read l
do
set $l
echo $2 $(grep 'block="'$1'"' file2)
done
But to understand this code, debug it and extend it, you need to have a big experience in regexps, exact details of escaping, and how is it the fastest to integrate them on both of these languages. On my opinion, if you are learning programming, it is a must-have before you start to learn perl/python, but in everyday work the higher level solutions are better.

Answer 5

You can also write a short bash script that will retrieve the information in a brute-force manner:
#!/bin/bash
[ -z "$1" -o -z "$2" -o -z "$3" ] && {
printf "error: insufficient input. Usage: %s file1 file2 query\n" ${0##*/}
exit 1
}
file1="$1"
file2="$2"
query="$3"
## line from file1
f1=$(grep "$query" "$file1")
[ -z "$f1" ] && {
printf "error: '%s' not found in '%s'\n." "$query" "$file1"
exit 1
}
## line from file2
f2=$(grep $(awk -F '|' '{ print $3 }' <<<$f1) "$file2")
[ -z "$f1" ] && {
printf "error: no related row found in '%s'\n." "$file2"
exit 1
}
printf "%s => %s\n" "$f1" "$f2"
Examples
$ bash joinfiles.sh join1.txt join2.txt ID_B
ID_B|ID_C|NY|4|8198|10|2374|127 => item name="NY" block="A" idnum="12345"
$ bash joinfiles.sh join1.txt join2.txt ID_E
ID_D|ID_E|CT|4|8198|10|2374|127 => item name="CT" block="C" idnum="1234567"
$ bash joinfiles.sh join1.txt join2.txt ID_F
error: 'ID_F' not found in 'join1.txt'
Processing Multiple Input Tags
With small adjustments, you can easily process as many tags as you desire. Below, the code will match all of the tags provided as arguments 3 and beyond. If you want to provide more than a half-dozed tags at a time, then you should adjust the script to read tags from stdin instead of taking them as arguments. Also note, the code matches the tag in the first field of the first file. If you want to match tags in all fields in file one, then f1 will become an array and the remaining part of the code with operate on each element in f1. Let me know what you are thinking. Here is the example for multiple input tags as arguments:
#!/bin/bash
[ -z "$1" -o -z "$2" -o -z "$3" ] && {
printf "error: insufficient input. Usage: %s file1 file2 query\n" ${0##*/}
exit 1
}
[ -f "$1" -a -f "$2" ] || {
printf "error: file not found. '%s' or '%s'\n" "$1" "$2"
exit 1
}
file1="$1"
file2="$2"
for i in "${#:3}"; do
query="$i"
## line from file1
f1=$(grep "^$query" "$file1")
[ -z "$f1" ] && {
printf "error: '%s' not found in '%s'\n." "$query" "$file1"
exit 1
}
## line from file2
f2=$(grep $(awk -F '|' '{ print $3 }' <<<$f1) "$file2")
[ -z "$f1" ] && {
printf "error: no related row found in '%s'\n." "$file2"
exit 1
}
printf "%s => %s\n" "$f1" "$f2"
done
Use/Output
$ bash joinfiles2.sh join1.txt join2.txt ID_B ID_D ID_F
ID_B|ID_C|NY|4|8198|10|2374|127 => item name="NY" block="A" idnum="12345"
ID_D|ID_E|CT|4|8198|10|2374|127 => item name="CT" block="C" idnum="1234567"
error: 'ID_F' not found in 'join1.txt'