How to use NiFi ExtractGrok properly - logstash

Question

I'm using: NiFi v1.8.0 and Logstash v7.1.1
I'm tasked to move all our Logstash configurations over to NiFi. I am trying to understand how the NiFi ExtractGrok works, but I can't find any examples. How is this intended to be used? And how can you set a NiFi attribute with this grok processor? And when I mean examples, I mean actual examples that show you a before and after so people can understand whats going on. I've read the NiFi ExtractGrok documentation, but its very limited and seems to assume you understand how it works.
This is the only example I've been able to find: How to fetch multiline with ExtractGrok processor in ApacheNifi?

Answer 1

According to what you are saying, the processor you need, is rather ConvertRecord than ExtractGrok. ExtractGrok will only extract certain fields into FlowFile attributes or content.
If you want to format your log files into a workable format(like JSON, if you want to send those files to ElasticSearch), then you would use GrokReader as Record Reader and Record Writer as JsonRecordSetWriter.
Then, you would configure your Schema Text (or use a Schema Registry) in both RecordReader and RecordWriter to be your schema, and set Grok Expression to be your grok expression in your GrokReader.
For example:
my log messages log like this:
2019-12-09 07:59:59,136 this is the first log message
2019-12-09 09:59:59,136 this is the first log message with a stack trace: org.springframework.boot.actuate.jdbc.DataSourceHealthIndicator - DataSource health check failed
org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.)
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:81)......
So, my grok would be:
%{TIMESTAMP_ISO8601:timestamp}\s+%{GREEDYDATA:log_message}
and my schema would be:
{
"name": "MyClass",
"type": "record",
"namespace": "com.acme.avro",
"fields": [
{
"name": "timestamp",
"type": "string"
},
{
"name": "log_message",
"type": "string"
},
{
"name": "stackTrace",
"type": "string"
}
]
}
Note the stackTrace field I've added to the schema. The GrokReader automatically maps stack traces into their own field. So you have to add stackTrace field if you want to map it too. Then, you can put it into the log_message field if you want, using Jolt.
The output of this ConvertRecord would be:
[ {
"timestamp" : "2019-12-09 07:59:59,136",
"log_message" : "this is the first log message",
"stackTrace" : null
}, {
"timestamp" : "2019-12-09 09:59:59,136",
"log_message" : "this is the first log message with a stack trace: org.springframework.boot.actuate.jdbc.DataSourceHealthIndicator - DataSource health check failed",
"stackTrace" : "org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.)\nat org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:81)......"
} ]