Compare two files containing IP's with Python - python-3.x

Question

I am having a little trouble finding an efficient way to compare two files in order to create a third file.
I'm using Python 3.6
The first file is a list of IP addresses that I want to delete. The second file contains all of the DNS records associated with that IP address targeted for deletion.
If I find DNS record in the second file, I want to add the entire line to a third file.
This is sample of file 1:
IP
10.10.10.234
10.34.76.4
This is sample of file 2:
DNS Record Type,DNS Record,DNS Response,View
PTR,10.10.10.234,testing.example.com,internal
A,testing.example.com,10.10.10.234,internal
A,dns.google.com,8.8.8.8,external
This is what I'm trying to do. It is accurate, however it is taking forever. There are ~2 million lines in file 2 and 150K lines in file 1.
def create_final_stale_ip_file():
PD = set()
with open(stale_file) as f1:
reader1 = csv.DictReader(f1)
for row1 in reader1:
with open(prod_dns) as f2:
reader2 = csv.DictReader(f2)
for row2 in reader2:
if row2['DNS Record Type'] == 'A':
if row1['IP'] == row2['DNS Response']:
PD.update([row2['View']+'del,'+row2['DNS Record Type']+','+row2['DNS Record']+','+row2['DNS Response']])
if row2['DNS Record Type'] == 'PTR':
if row1['IP'] == row2['DNS Record']:
PD.update([row2['View']+'del,'+row2['DNS Record Type']+','+row2['DNS Response']+','+row2['DNS Record']])
o1 = open(delete_file,'a')
for i in PD:
o1.write(i+'\n')
o1.close()
Thanks in advance!

Answer 1

You should read the whole IP file into a set first, and then check if the IPs on the second file are found in that set, since checking if an element exists in a set is very fast:
def create_final_stale_ip_file():
PD = set()
# It's much prettier and easier to manage the strings in one place
# and without using the + operator. Read about `str.format()`
# to understand how these work. They will be used later in the code
A_string = '{View}del,{DNS Record Type},{DNS Record},{DNS Response}'
PTR_string = '{View}del,{DNS Record Type},{DNS Response},{DNS Record}'
# We can open and create readers for both files at once
with open(stale_file) as f1, open(prod_dns) as f2:
reader1, reader2 = csv.DictReader(f1), csv.DictReader(f2)
# Read all IPs into a python set, they're fast!
ips = {row['IP'] for row in reader1}
# Now go through every line and simply check if the IP
# exists in the `ips` set we created above
for row in reader2:
if (row['DNS Record Type'] == 'A'
and row['DNS Response'] in ips):
PD.add(A_string.format(**row))
elif (row['DNS Record Type'] == 'PTR'
and row2['DNS Record'] in ips):
PD.add(PTR_string.format(**row))
# Finally, write all the lines to the file using `writelines()`.
# Also, it's always better to use `with open()`
with open(delete_file, 'a') as f:
f.writelines(PD)
As you see, I also changed some minor stuff, like:
write to a file using writelines()
open the last file using with open() for safety
we're only adding one element to our set, so use PD.add() instead of PD.update()
use Python's awesome str.format() to create much cleaner string formatting
Last but not least, I would actually split this into multiple functions, one for reading the files, one for going through the read dictionaries, etc, and each function taking the proper arguments instead of using global variable names like stale_file and prod_dns as you seem to be using. But that's up to you.

Answer 2

You can do it using grep very easily:
grep -xf file1 file2
This will give you a file with the lines of file2 which match lines in file1. From there it should be much easier to manipulate the text to the final form you need.