Split large file based on column value - linux - linux

Question

I wanted to split the large file (185 Million records) to more than one files based on one column value.The file is .dat file and the delimiter used inbetween the columns are ^A (\u0001).
The File content is like this:
194^A1^A091502^APR^AKIMBERLY^APO83^A^A^A^A0183^AUSA^A^A^A^A^A^A^A^A
194^A1^A091502^APR^AJOHN^APO83^A^A^A^A0183^AUSA^A^A^A^A^A^A^A^A
194^A^A091502^APR^AASHLEY^APO83^A^A^A^A0183^AUSA^A^A^A^A^A^A^A^A
194^A3^A091502^APR^APETER^APO83^A^A^A^A0183^AUSA^A^A^A^A^A^A^A^A
194^A4^A091502^APR^AJOE^APO83^A^A^A^A0183^AUSA^A^A^A^A^A^A^A^A
now i wanted to split the file based on second column value, if you see the third row the second column value is empty, so all the empty rows should come one file , remaining all should come one file.
Please help me on this. I tried to google, it seems we should use awk for this.
Regards,
Shankar

Answer 1

With awk:
awk -F '\x01' '$2 == "" { print > "empty.dat"; next } { print > "normal.dat" }' filename
The file names can be chosen arbitrarily, of course. print > "file" prints the current record to a file named "file".
Addendum re: comment: Removing the column is a little trickier but certainly feasible. I'd use
awk -F '\x01' 'BEGIN { OFS = FS } { fname = $2 == "" ? "empty.dat" : "normal.dat"; for(i = 2; i < NF; ++i) $i = $(i + 1); --NF; print > fname }' filename
This works as follows:
BEGIN { # output field separator is
OFS = FS # the same as input field
# separator, so that the
# rebuilt lines are formatted
# just like they came in
}
{
fname = $2 == "" ? "empty.dat" : "normal.dat" # choose file name
for(i = 2; i < NF; ++i) { # set all fields after the
$i = $(i + 1) # second back one position
}
--NF # let awk know the last field
# is not needed in the output
print > fname # then print to file.
}