Grok logstash pipeline not filtering texts - logstash-grok

Question

I am new to Elasticstack. I am trying to implement a logstash pipeline in which the a file would be processed and it would filter and output if the line of file contains following keyword -
java.lang.Exception - Any line of file containing Exception should be filtered and be available on Kibana
XYZ process completed.
I tried following but it seems to outputting all the contents that do not match the Exception too-
input {
beats {
port => 5044
tags => "exception"
}
}
filter{
if "exception" in [tags]{
grok {
match => { message => "Exception"
}
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "%{[#metadata][beat]}-%{[#metadata][version]}"
}
}
Please help and advise

Answer 1

If this is the only filter you are applying in the pipeline, everything that does not match your GROK filter will have a _grokparsefailure tag added.
You can use it to drop the events. using below after all the options in your filter are done.
if "_grokparsefailure" in [tags] {
drop{}
}
So a sample filter can be
filter {
grok {
match => { message => "Exception" }
}
if "_grokparsefailure" in [tags] {
drop{}
}
}
Another way to do this can be, add a tag when match happens and then only ingest events that have the tag.
Example
input {
beats {
port => 5044
tags => "exception"
}
}
filter{
if "exception" in [tags]{
grok {
match => { message => "Exception"}
add_tag => [ "exception_match" ]
}
}
output {
if"exception_match" in [tags]
{
elasticsearch {
hosts => ["http://localhost:9200"]
index => "%{[#metadata][beat]}-%{[#metadata][version]}"
}
}
}