Logstash: grok expression for multiline data - logstash

Question

I am new to ELK stack. I am trying to write one grok expression for the following log statement
2017-10-26 19:20:28.538 ERROR --- [logAppenderService] [Serv01] [restartedMain] ns.pcs.log.appender.LogAppender : [1234] doStuff Some statement here - {}
java.lang.Exception: Hello World
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.devtools.restart.RestartLauncher.run(RestartLauncher.java:49)
I have written the following logstash configuration:
input{
kafka {
type => "mylog"
topic_id => 'mylog'
}
}
filter{
if [type] == "mylog" {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} \[%{DATA:serviceName}] \[%{DATA:nodeName}] \[%{DATA:trName}] %{NOTSPACE:className} %{NOTSPACE:':'} \[%{DATA:refName}] %{GREEDYDATA:msg}" }
}
}
}
output{
if [type] == "mylog" {
elasticsearch {
hosts => ["101.18.19.89:9200"]
index => "logstash-%{+YYYY-MM-dd}"
}
}
stdout {
codec => rubydebug
}
}
When I am trying to run the same I am getting json parse exception. Not sure if I am missing something or not. I am really stuck at this stage.

Answer 1

Your problem is that the input is not matching in your pattern
Please try this
%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} --- \[%{DATA:serviceName}\] \[%{DATA:nodeName}\] \[%{DATA:trName}\] %{NOTSPACE:className} %{NOTSPACE:':'} \[%{DATA:refName}] %{GREEDYDATA:msg}
you missing the
---
]
pattern
if you still can't please check the mutiline is sending into your log
you could add the mutiline into your input if needed
codec => multiline {
pattern => "^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}[\.,][0-9]{3,7} "
negate => true
what => "previous"
}