Is there such commands to merge multiple files in shell? - linux

Question

For example, there're 5 numbers => [1,2,3,4,5] and 3 groups
File1(Group1):
1
3
5
File2(Group2):
3
4
File3(Group3):
1
5
Output (column1: whether in Group1, column2: whether in Group2, column3: whether in Group3 [NA means not..]):
1 NA 1
3 3 NA
NA 4 NA
5 NA 5
Or something like this (+ means in, - means not):
1 + - +
3 + + -
4 - + -
5 + - +
I tried join and merge, but looks like both of them doesn't work well for multiple files.. (for example, 8 files)

Answer 1

You say there's numbers 1-5, but this is, as far as I can see, irrelevant for the output you want. You only use numbers found in your files in the output. This code will do what you want:
use strict;
use warnings;
use feature 'say';
my #hashes;
my %seen;
local $/; # read entire file at once
while (<>) {
my #nums = split; # split file into elements
$seen{$_}++ for #nums; # dedupe elements
push #hashes, { map { $_ => $_ } #nums }; # map into hash
}
my #all = sort { $a <=> $b } keys %seen; # sort deduped elements
# my #all = 1 .. 5; # OR: provide hard-coded list
for my $num (#all) { # for all unique numbers
my #fields;
for my $href (#hashes) { # check each hash
push #fields, $href->{$num} // "NA"; # enter "NA" if not found
}
say join "\t", #fields; # print the fields
}
You may replace the sorted deduped list in #all with just my #all = 1 .. 5 or any other valid list. It will then add lines for those numbers and print out extra "NA" fields for the missing values.
You should also be aware that this relies on the fact that your file contents are numbers, but only as far as it comes to the sorting of the #all array, so if you replace it with your own list, or your own sorting routine, you can use any value.
This script will take an arbitrary number of files and process them. For example:
$ perl script.pl f1.txt f2.txt f3.txt
1 NA 1
3 3 NA
NA 4 NA
5 NA 5
Credit to Brent Stewart for figuring out what the OP meant.

Answer 2

#!/usr/bin/env perl
use strict;
use warnings;
use autodie;
my #lines;
my $filecount = 0;
# parse
for my $filename (#ARGV){
open my $fh, '<', $filename;
while( my $line = <$fh> ){
chomp($line);
next unless length $line;
$lines[$line][$filecount]++;
}
close $fh;
}continue{
$filecount++;
}
# print
for my $linenum ( 1..$#lines ){
my $line = $lines[$linenum];
next unless $line;
print ' ' x (5-length $linenum), $linenum, ' ';
for my $elem( #$line ){
print $elem ? 'X' : ' '
}
print "\n";
}
1 X X
3 XX
4 X
5 X X

Answer 3

For two files, you can easily use join as shown below (assuming file1 and file2 are sorted):
$ join -e NA -o 1.1,2.1 -a 1 -a 2 file1 file2
1 NA
3 3
NA 4
5 NA
It gets more complicated if you have more than two files though.
Here is a brute force grep solution:
#!/bin/bash
files=(file1 file2 file3)
sort -nu "${files[#]}" | while read line; do
for f in "${files[#]}"; do
if grep -qFx "$line" "$f"; then
printf "${line}\t"
else
printf "NA\t"
fi
done
printf "\n"
done
Output:
1 NA 1
3 3 NA
NA 4 NA
5 NA 5

Answer 4

If your input files are monotonically increasing and just consist of a single integer on each line as your input samples suggest, you could simply pre-process the input files and use paste:
for i in file{1,2,3}; do # List input files
awk '{ a += 1; while( $1 > a ) { print "NA"; a += 1 }} 1' $i > $i.out
done
paste file{1,2,3}.out
This leaves the trailing entries in some columns empty. Fixing that is left as an exercise for the reader.