Removing XML Elements Based On Attribute Value Using Python Element Tree - python-3.x

Question

I've been working with this much of the day but haven't been able to find a solution. I have a fairly large XML file that I need to strip of some data. The 'fields' are annotated using attributes by Id (field name) and Num (unique number for the field name).
<?xml version="1.0" encoding="UTF-8"?>
<Data>
<Item Id="Type" Num="30">3</Item>
<Item Id="Version" Num="50">180</Item>
<Owner>
<Item Id="IdNumber" Num="20">00000000</Item>
<Item Id="race1" Num="160">01</Item>
<Item Id="race2" Num="161">88</Item>
<Item Id="race3" Num="162">88</Item>
<Dog>
<Item Id="Breed" Num="77">Mutt</Item>
<Item Id="Weight" Num="88">88</Item>
</Dog>
<Dog>
<Item Id="Breed" Num="77">Retriever</Item>
<Item Id="Weight" Num="88">77</Item>
</Dog>
</Owner>
<Owner>
<Item Id="IdNumber" Num="20">00033000</Item>
<Item Id="race1" Num="160">03</Item>
<Item Id="race2" Num="161">88</Item>
<Item Id="race3" Num="162">88</Item>
<Dog>
<Item Id="Breed" Num="77">Poodle</Item>
<Item Id="Weight" Num="88">21</Item>
</Dog>
</Owner>
</Data>
Here's the pretty simple python code that I assumed would do the trick, but it's stripping out the data as expected.
#!/usr/bin/env python3
import xml.etree.ElementTree as ET
# list of "Nums" to drop from each Owner and Dog
drops = ['160', '161', '162', '88']
# read in XML file
with open('dogowners.xml') as xmlin:
tree = ET.parse(xmlin)
root = tree.getroot()
for x in root:
for y in x:
# checking to make sure it's an Owner
if y.attrib:
# if the value for attribute Num is in the list of drops then remove it
if y.attrib["Num"] in drops:
x.remove(y)
# finally output new tree
tree.write('output.xml')
output.xml
The issue I'm running into is it's not removing all the drops/Nums listed. In the case with this small XML, it's only doing the first and third value on the Owner level, which is consistent with my large file because it appears to be only removing every other one.
<Data>
<Item Id="Type" Num="30">3</Item>
<Item Id="Version" Num="50">180</Item>
<Owner>
<Item Id="IdNumber" Num="20">00000000</Item>
<Item Id="race2" Num="161">88</Item>
<Dog>
<Item Id="Breed" Num="77">Mutt</Item>
<Item Id="Weight" Num="88">88</Item>
</Dog>
<Dog>
<Item Id="Breed" Num="77">Retriever</Item>
<Item Id="Weight" Num="88">77</Item>
</Dog>
</Owner>
<Owner>
<Item Id="IdNumber" Num="20">00033000</Item>
<Item Id="race2" Num="161">88</Item>
<Dog>
<Item Id="Breed" Num="77">Poodle</Item>
<Item Id="Weight" Num="88">21</Item>
</Dog>
</Owner>
</Data>
I feel like I might be missing something kind of obvious, but XML parsing is not my forte and I've been fighting with this for a couple of hours. Any help is greatly appreciated.

Answer 1

This seems like a good candidate for the identiy transform pattern. The following will copy the xml document, but will exclude the Item elements that match the empty template at the end of the xsl string.
owner-dog.py
#!/usr/bin/env python3
from lxml import etree
# list of "Nums" to drop from each Owner and Dog
drops = ('160', '161', '162', '88')
# we turn it into an xsl attribute pattern:
# #Num = '160' or #Num = '161' or #Num = '162' or #Num = '88'
attr_vals = list(map(lambda n: f'#Num = \'{n}\'', drops))
attr_expr = ' or '.join(attr_vals)
xsl = etree.XML('''<xsl:stylesheet version="1.0"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output indent='yes'/>
<xsl:strip-space elements="*" />
<!-- copy all nodes ... -->
<xsl:template match="#*|node()">
<xsl:copy>
<xsl:apply-templates select="#*|node()"/>
</xsl:copy>
</xsl:template>
<!-- ... except for any elements matching the following template -->
<xsl:template match="//Item[ {attr_vals} ]" />
</xsl:stylesheet>'''.format(attr_vals=attr_expr))
transform = etree.XSLT(xsl)
with open('owner-dog.xml') as xml:
print(transform(etree.parse(xml)))
Output
<?xml version="1.0"?>
<Data>
<Item Id="Type" Num="30">3</Item>
<Item Id="Version" Num="50">180</Item>
<Owner>
<Item Id="IdNumber" Num="20">00000000</Item>
<Dog>
<Item Id="Breed" Num="77">Mutt</Item>
</Dog>
<Dog>
<Item Id="Breed" Num="77">Retriever</Item>
</Dog>
</Owner>
<Owner>
<Item Id="IdNumber" Num="20">00033000</Item>
<Dog>
<Item Id="Breed" Num="77">Poodle</Item>
</Dog>
</Owner>
</Data>
Comparing the original xml with the ouput
diff <(xmllint --format owner-dog.xml) <(./owner-dog.py)
1c1
< <?xml version="1.0" encoding="UTF-8"?>
---
> <?xml version="1.0"?>
7,9d6
< <Item Id="race1" Num="160">01</Item>
< <Item Id="race2" Num="161">88</Item>
< <Item Id="race3" Num="162">88</Item>
12d8
< <Item Id="Weight" Num="88">88</Item>
16d11
< <Item Id="Weight" Num="88">77</Item>
21,23d15
< <Item Id="race1" Num="160">03</Item>
< <Item Id="race2" Num="161">88</Item>
< <Item Id="race3" Num="162">88</Item>
26d17
< <Item Id="Weight" Num="88">21</Item>
29a21
>