How can I extract key word value from a string - logstash-grok

Question

I am using this tool https://grokdebug.herokuapp.com/ to test my grok parser. The origin string I have is something like:
2020-05-01 01:59:10 server1 17.5.36.8 POST /v1/user.aspx r=1000&11:59:11.219&Method=Start&Credentials=xxxxxx:++http://localhost/v1/user.aspx&Reque
I'd like to parse the data to:
{
Method: Start,
r: 1000
Credential: xxxxx
}
I am looking at the parser https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns but I can't find a good one to use.

Answer 1

I don't know how to achieve this using solely grok patterns. If you're also using Logstash, you can try the following solution:
The pattern:
%{TIMESTAMP_ISO8601:timestamp}\s%{GREEDYDATA:server}\s%{IP:ip}\s%{GREEDYDATA:request_type}\s%{PATH:path}\sr=%{NUMBER:r}&%{TIME:some_time}&Method=%{GREEDYDATA:method}&Credentials=%{GREEDYDATA:username}:%{GREEDYDATA:password}%{MY_URI:uri}
It works with the custom pattern:
MY_URI http://(?:%{USER}(?::[^#]*)?#)?(?:%{URIHOST})?(?:%{URIPATHPARAM})?
I used the custom pattern because the %{URIPROTO} from the original %{URI} pattern won't separate credentials from the uri properly. I assumed that credentials are given in the following format:
username:password
In case I'm wrong, please replace:
Credentials=%{GREEDYDATA:username}:%{GREEDYDATA:password}
with:
Credentials=%{GREEDYDATA:credentials}
If the :++ string separates credentials from uri, you can use it to apply the %{URI} instead of %{MY_URI}.
The pattern works in the Grok Debugger. However the output is more readable in the Grok Constructor Matcher.
Since you're only interested in some fields, use the remove_field plugin. Use the add_field plugin to enclose the result in a new field.
Your logstash.conf file may look like this (if you place the file with the custom pattern in the patterns directory alongside the config file):
# logstash.conf
…
filter {
grok {
patterns_dir => ["./patterns"]
match => {
"message" => "%{TIMESTAMP_ISO8601:timestamp}\s%{GREEDYDATA:server}\s%{IP:ip}\s%{GREEDYDATA:request_type}\s%{PATH:path}\sr=%{NUMBER:r}&%{TIME:some_time}&Method=%{GREEDYDATA:method}&Credentials=%{GREEDYDATA:username}:%{GREEDYDATA:password}%{MY_URI:uri}"
}
mutate {
add_field => { "result" => "Method: %{method}, r: %{r} Credential: %{username}}
remove_field => ["timestamp", "server", "ip", "request_type", "path", "some_time", "password", "uri", "method", "r", "username"]
}
}
…