automatically map fields in syslog "message" section - logstash

Question

Is it possible to automatically map fields for events I would receive by syslog, if they follow a format field1=value1 field2=value2 ... ? An example would be
name=john age=15
age=29 name=jane
name=mark car=porshe
(note that the fields are different and not always there)
One of the solutions I am considering is to send the syslog "message" part as JSON but I am not sure if it possible to automatically parse it (when the rest of the log is in syslog format). My current approach fails with _jsonparsefailure but I will keep trying
input {
tcp
{
port => 5514
type => "syslogandjson"
codec => json
}
}
filter{
json{
source => "message"
}
}
output ...

Answer 1

Fields with a key=value format can be parsed with the kv filter, but it doesn't support fields with double-quoted values, i.e.
key1=value1 key2="value2 with spaces" key3=value3
or (even worse)
key1=value1 key2=value2 with spaces key3=value3
won't turn out good.
Sending the message as JSON is way better, but as you've discovered you can't use the json codec since the codec applies to the whole message (timestamp and all) and not just the message part where your serialized JSON string can be found. You're on the right track with the json filter though. Just make sure you have that filter after the grok filter that parses the raw syslog message to extract timestamp, severity, and so on. You'll want something like this:
filter {
grok {
match => [...]
# Allow replacement of the original message field
overwrite => ["message"]
}
date {
...
}
json {
source => "message"
}
}
Since presumably not all messages you pick up are JSON messages you might want a conditional around the json filter. Or, attempt the JSON parsing of all messages but remove any _jsonparsefailure tag that the filter adds for messages it couldn't parse.