Adding attributes to large datasets using command line tools - attributes

Question

I have an extremely large dataset (approx. 150 MB; 500 targets; 700,000+ attributes). I need to add one attribute to the end of each file.
The date file that I am working with has the following structure:
#relation 'filename'
#attribute "place" string
#attribute "institution" string
#attribute "food" string
#attribute "book" string
#data
3.8,6,0,0,church
86.3,0,63.1,0,man
0,0,0,37,woman
I need to add one attribute of information to each of the rows following #data.
However, due to its sheer number of attributes, I cannot open and modify the data in a text editor.
The attribute that I need to include I have in a separate tab separated file that has the following structure:
church 1
man 1
woman 0
The desired result would have the data set look like this:
#relation 'filename'
#attribute "place" string
#attribute "institution" string
#attribute "food" string
#attribute "book" string
#data
3.8,6,0,0,church,1
86.3,0,63.1,0,man,1
0,0,0,37,woman,0
Where the command would look to match the end of each line after #data with each line of the second file and if it is a match add the corresponding 0 or 1.
I have been searching for a solution for this and my searches have mostly came up with answers that are pointing to the direction to use a text editor. As I mentioned earlier, the problem with the text editors is not necessarily opening the file (UltraEdit for instance can handle for the most part a file of this size). It is manually inserting one attribute after more than 700,000 attributes, which is an extremely time consuming task.
So, I ask the community if what I need to do is possible using a command line argument (awk/grep, etc.) to achieve the desired result?

Answer 1

Python is great because it's installed by default on a lot of POSIX-based systems :)
Now some caveats:
this is simple python and intended for you to learn as you go, so it could be much more optimized
this will read the entire file into memory while processing, so if your file is in the GB, it's going to hit your computer a bit
I recommend throwing some print statements, or using the python debugger to step through the program if you want to know what's going on.
Here's what I came up with:
lookup = {}
output_list = []
# build a lookup based on the lookup file
with open('lookup.csv', 'rb') as lookup_file:
rows = lookup_file.readlines()
for row in rows:
key, value = row.split()
lookup[key] = value
# loop through the big file and add the values
with open('input-big-data.txt', 'rb') as input_file:
rows = input_file.readlines()
target_zone = False
for row in rows:
# keep a copy of every row
output_for_this_row = row
# skip the normal attribute rows
if row.startswith('#'):
target_zone = False
# check to see if we are in the 'target zone'
if row.startswith('#data'):
target_zone = True
# start parsing the rows, but not if they have the attribute flag
if target_zone and not row.startswith('#'):
# do your data processing here
# strip to clobber the newline, then break it into pieces
row_list = row.strip().split(',')
# grab the last item
lookup_key = row_list[-1].strip()
# grab the value for that last item
row_list.append(lookup[lookup_key])
# put the row back in it's original state
output_for_this_row = ",".join(row_list) + "\n"
output_list.append(output_for_this_row)
with open('output-big-data.txt', 'wb') as output_file:
for line in output_list:
output_file.write("{}".format(line))
I've commented pretty thoroughly throughout, so it should be pretty self-explanatory.
From the files in your question, I've named them in order: input-big-data.txt, lookup.csv, and output-big-data.csv.
Here's the output from my example:
#relation 'filename'
#attribute "place" string
#attribute "institution" string
#attribute "food" string
#attribute "book" string
#data
3.8,6,0,0,church,1
86.3,0,63.1,0,man,1
0,0,0,37,woman,0
Hth,
Aaron

Answer 2

As commented below, python can solve this problem quite simply as demonstrated by the solution that I found and used on this blog: http://margerytech.blogspot.it/2011/03/python-appending-column-to-end-of-tab.html.
It is not a command line argument (as I indicated I wanted to use in the question), but it solves the problem all the same.