How to delete lines in xml using logstash - logstash

Question

everybody.
I am new to logstash. I've been studying filters in logstash for a week, but no result.
I want to parse xml. I have the following xml:
<?xml version="1.0" encoding="utf-8" standalone="no"?>
<ContinentLogs>
<Cryptogateways>
<cgw id="1" cssid="0" ip="0.0.0.0" tz="RCPTZ -3:00 RCPTZS -4:00,M0.0.0/00:00:00,M0.0.0/00:00:00">ЦУС</cgw>
<cgw id="2" cssid="50556" ip="0.0.0.0" tz="RCPTZ -3:00 RCPTZS -4:00,M0.0.0/00:00:00,M0.0.0/00:00:00">DP.152FZ.DA01</cgw>
<cgw id="3" cssid="50557" ip="0.0.0.0" tz="RCPTZ -3:00 RCPTZS -4:00,M0.0.0/00:00:00,M0.0.0/00:00:00">M1.152FZ.DA01</cgw>
<cgw id="4" cssid="51358" ip="0.0.0.0" tz="RCPTZ 00:00 RCPTZS 00:00,M0.0.0/00:00:00,M0.0.0/00:00:00">3D Get</cgw>
<cgw id="5" cssid="51491" ip="0.0.0.0" tz="RCPTZ -3:00 RCPTZS -4:00,M0.0.0/00:00:00,M0.0.0/00:00:00">М1.152FZ.CUS02</cgw>
<cgw id="6" cssid="51845" ip="0.0.0.0" tz="RCPTZ -3:00 RCPTZS -4:00,M0.0.0/00:00:00,M0.0.0/00:00:00">DP.152FZ.FW01</cgw>
<cgw id="7" cssid="51847" ip="0.0.0.0" tz="RCPTZ -3:00 RCPTZS -4:00,M0.0.0/00:00:00,M0.0.0/00:00:00">M1.152FZ.FW01</cgw>
<cgw id="8" cssid="53840" ip="0.0.0.0" tz="RCPTZ -3:00 RCPTZS -4:00,M0.0.0/00:00:00,M0.0.0/00:00:00">M1.152FZ.FW02</cgw>
<cgw id="9" cssid="68604" ip="0.0.0.0" tz="RCPTZ -3:00 RCPTZS -4:00,M0.0.0/00:00:00,M0.0.0/00:00:00">TNC MIG</cgw>
<cgw id="10" cssid="69007" ip="0.0.0.0" tz="RCPTZ -3:00 RCPTZS -4:00,M0.0.0/00:00:00,M0.0.0/00:00:00">HAKR</cgw>
<cgw id="11" cssid="111846" ip="0.0.0.0" tz="RCPTZ -3:00 RCPTZS -4:00,M0.0.0/00:00:00,M0.0.0/00:00:00">ALT MOT</cgw>
<cgw id="12" cssid="111978" ip="0.0.0.0" tz="RCPTZ -3:00 RCPTZS -4:00,M0.0.0/00:00:00,M0.0.0/00:00:00">GPB RIM</cgw>
</Cryptogateways>
<FilterRules>
<rule id="-8" deleted="false">Pravil</rule>
<rule id="-5" deleted="false">(?) Neopredelen</rule>
<rule id="-3" deleted="false">Udalenni pravila</rule>
<rule id="-2" deleted="false">SD prav</rule>
<rule id="-1" deleted="false">Slygeb prav</rule>
<rule id="0" deleted="false">Ne sootvetsvyet</rule>
<rule id="17" deleted="false">Mon => Mail Server</rule>
</FilterRules>
</ContinentLogs>
I want to remove lines that have «cgw id="3"» и «cgw id="7"» in the section.
And remove the lines that have «rule id="-5"» и «rule id="0"» in the section.
Write everything back to xml without changing anything else.
Help how to do this through Logstash?
Thank you in advance.

Answer 1

if your logstash is feeding the ELK, then your solution should be in Prune blacklist
first, add the item you want in the xpath
for example
let my Xml file be like
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN">
<Appenders>
<Console name="Console" target="SYSTEM_OUT">
<PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n" />
</Console>
</Appenders>
<Loggers>
<Logger name="com.foo.Bar" level="trace">
<AppenderRef ref="Console" />
</Logger>
<Logger name="com.foo.Bar.test.2.error" level="error">
<AppenderRef ref="Console" />
</Logger>
<Root level="error">
<AppenderRef ref="Console" />
</Root>
</Loggers>
</Configuration>
and now in the configuration file use the filter as the following.
filter {
xml {
source => "message"
store_xml => false
target => "rec"
xpath => [
"/Configuration /#status", "Configuration_LEVEL" // defining my item to delete
]
}
prune {
blacklist_names => ["Configuration_LEVEL"] // delete the item using prune blacklist
}
}
I hope it helped you.