Logstash - how to split array by comma? - logstash

My message looks like this
[Metric][methodName: someName][methodParams: [ClassName{field1="val1", field2="val2", field3="val3"}, ClassName{field1="val1", field2="val2", field3="val3"}, ClassName{field1="val1", field2="val2", field3="val3"}]]
Is there a way to separate this log in more smaller ones and filter them separately?
If the first option isn't possible, how can I parse to get all elements of the array?
(?<nameOfClass>[A-Za-z]+)\{field1='%{DATA:textfield1}',\sfield2='%{DATA:textfield2}',\sfield3='%{DATA:textfield3}'\}


Since everything after methodParams: looks to be JSON you could use a JSON filter to parse it. Something like:
filter{
# Parse your JSON out here using GROK to a field called myjson
grok {
match => {
"message" => "methodParams: %{GREEDYDATA:myjson}"
}
}
#
json{
source => "myjson"
}
}

Related

grok/kv filter audit logs logstash

I'm learning how to use the grok plugin. I have a message string like so
"type=CRYPTO_SESSION msg=audit(111111.111:111111): pid=22730 uid=0 auid=123 ses=123 subj=system_u:system_r:sshd_t:a1-a1:a1.a1234 msg='op=xx=xx cipher=xx ksize=111 mac=xx pfs=xxx spid=111 suid=000 rport=000 laddr=11.11.111.111 lport=123 exe=\"/usr/sbin/sshd\" hostname=? addr=11.111.111.11 terminal=? res=success'"
I'd like to extract the fields laddr, addr, and lport. I created a patterns directory with the following structure
patterns
|
-- laddr
|
-- addr
My filter is written like so
filter {
grok {
patterns_dir => ["./patterns"]
match => { "messaage" => "%{LADDR:laddr} %{ADDR:addr}"}
}
}
I was expecting to extract at least laddr and addr. I get matches using https://grokdebug.herokuapp.com/. With these patterns
(?<laddr>\b(laddr=\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\b)
(?<addr>\b(addr=\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\b)
but the configuration fails to compile. I'm just going off of these docs: https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html. I've also tried using a kv filter the issue that I run into when I try to use something like
filter{
kv {
value_split => "="
}
}
I end up with msg field showing up twice. I'd really like to figure out how to get the properties from this string. Any help would be greatly appreciated.

I think its field split :
filter {
kv {
field_split => "&?"
}
}
Did you try this, are you getting any error messages?

Grok Pattern not working in Logstash

After parsing logs I am find there are some new lines at the end of the message
Sample message
ts:2016-04-26 05-02-16-018
CDT|ll:TRACE|tid:10000.140|scf:xxxxxxxxxxxxxxxxxxxxxxxxxxx.pc|mn:null|fn:xxxxxxxxxxxxxxxxxxxxxxxxxxx|ln:749|auid:xxxxxxxxxxxxxxxxxxxxxxxxxxx|eid:xxx.xxx.xxx.xxx-58261618-1-1461664935955-139|cid:900009865|ml:null|mid:-99|uip:xxx.xxx.xxx.xxx|hip:xxx.xxx.xxx.xxx|pli:null|msg:
xxxxxxxxxxxxxxxxxxxxxxxxxxx|pl: xxxxxxxxxxxxxxxxxxxxxxxxxxx
TAKE 1 xxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxx
I am using the regex pattern below as suggested below as answers
ts:(?(([0-9]+)-)+ ([0-9]+-)+ [A-Z]+)\|ll:%{WORD:ll}\|tid:%{NUMBER:tid}\|scf:%{DATA:scf}\|mn:%{WORD:mn}\|fn:%{WORD:fn}\|ln:%{WORD:ln}\|auid:%{WORD:auid}\|eid:%{DATA:eid}\|cid:%{WORD:cid}\|ml:%{WORD:ml}\|mid:%{NUMBER:mid}\|uip:%{DATA:uip}\|hip:%{DATA:hip}\|pli:%{WORD:pli}\|\smsg:%{GREEDYDATA:msg}(\|pl:(?(.|\r|\n)))
But unfortunately it is not working properly when the last part of the log is not present
ts:2016-04-26 05-02-16-018
CDT|ll:TRACE|tid:10000.140|scf:xxxxxxxxxxxxxxxxxxxxxxxxxxx.pc|mn:null|fn:xxxxxxxxxxxxxxxxxxxxxxxxxxx|ln:749|auid:xxxxxxxxxxxxxxxxxxxxxxxxxxx|eid:xxx.xxx.xxx.xxx-58261618-1-1461664935955-139|cid:900009865|ml:null|mid:-99|uip:xxx.xxx.xxx.xxx|hip:xxx.xxx.xxx.xxx
What should be the correct pattern?
-------------------Previous Question --------------------------------------
I am trying to parse log line such as this one.
ts:2016-04-26 05-02-16-018 CDT|ll:TRACE|tid:10000.140|scf:xxxxxxxxxxxxxxxxxxxxxxxxxxx.pc|mn:null|fn:xxxxxxxxxxxxxxxxxxxxxxxxxxx|ln:749|auid:xxxxxxxxxxxxxxxxxxxxxxxxxxx|eid:xxx.xxx.xxx.xxx-58261618-1-1461664935955-139|cid:900009865|ml:null|mid:-99|uip:xxx.xxx.xxx.xxx|hip:xxx.xxx.xxx.xxx|pli:null|msg: xxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxx
Below is my logstash filter
filter {
grok {
match => ["mesage", "ts:(?<date>(([0-9]+)-*)+ ([0-9]+-*)+ [A-Z]+)\|ll:%{WORD:ll}\|tid:%{WORD:tid}\|scf:%{WORD:scf}\|mn:%{WORD:mn}\|fn:%{WORD:fn}\|ln:%{WORD:ln}\|auid:%{WORD:auid}\|eid:%{WORD:eid}\|cid:%{WORD:cid}\|ml:%{WORD:ml}\|mid:%{WORD:mid}\|uip:%{WORD:uip}\|hip:%{WORD:hip}\|pli:%{WORD:pli}\|msg:%{WORD:msg}"]
}
date {
match => ["ts","yyyy-MM-dd HH-mm-ss-SSS ZZZ"]
target => "#timestamp"
}
}
I am getting "_grokparsefailure"

I have tested the configuration from #HAL, there was a few things to change:
In the grok filter mesage => message
In the date filter ts => date so the date parsing is on the right field
The CDT is a time zone name, it is captured by z in the date syntax.
So the right configuration would look like this :
filter{
grok {
match => ["message", "ts:(?<date>(([0-9]+)-*)+ ([0-9]+-*)+ [A-Z]+)\|ll:%{WORD:ll}\|tid:%{NUMBER:tid}\|scf:%{DATA:scf}\|mn:%{WORD:mn}\|fn:%{WORD:fn}\|ln:%{WORD:ln}\|auid:%{WORD:auid}\|eid:%{DATA:eid}\|cid:%{WORD:cid}\|ml:%{WORD:ml}\|mid:%{NUMBER:mid}\|uip:%{DATA:uip}\|hip:%{DATA:hip}\|pli:%{WORD:pli}\|\s*msg:%{GREEDYDATA:msg}"]
}
date {
match => ["date","yyyy-MM-dd HH-mm-ss-SSS z"]
target => "#timestamp"
}
}

Tried to parse your input via grokdebug with your expression but it failed to read out any fields. Managed to get it to work by changing the expression to:
ts:(?<date>(([0-9]+)-*)+ ([0-9]+-*)+ [A-Z]+)\|ll:%{WORD:ll}\|tid:%{NUMBER:tid}\|scf:%{DATA:scf}\|mn:%{WORD:mn}\|fn:%{WORD:fn}\|ln:%{WORD:ln}\|auid:%{WORD:auid}\|eid:%{DATA:eid}\|cid:%{WORD:cid}\|ml:%{WORD:ml}\|mid:%{NUMBER:mid}\|uip:%{DATA:uip}\|hip:%{DATA:hip}\|pli:%{WORD:pli}\|\s*msg:%{GREEDYDATA:msg}
I also think that you need to change the name of the column that logstash shall parse from mesage to message.
Also, the date parsing pattern should match the format of the date in the input. There is no timezone identity (ZZZ) in your input data (at least not in the example).
Something like this should work better (not tested though):
filter {
grok {
match => ["mesage", "ts:(?<date>(([0-9]+)-*)+ ([0-9]+-*)+ [A-Z]+)\|ll:%{WORD:ll}\|tid:%{NUMBER:tid}\|scf:%{DATA:scf}\|mn:%{WORD:mn}\|fn:%{WORD:fn}\|ln:%{WORD:ln}\|auid:%{WORD:auid}\|eid:%{DATA:eid}\|cid:%{WORD:cid}\|ml:%{WORD:ml}\|mid:%{NUMBER:mid}\|uip:%{DATA:uip}\|hip:%{DATA:hip}\|pli:%{WORD:pli}\|\s*msg:%{GREEDYDATA:msg}"]
}
date {
match => ["ts","yyyy-MM-dd HH-mm-ss-SSS"]
target => "#timestamp"
}
}

Match multiple field names in logstash mutate filter

I would like to convert all metrics* fields into floats for logstash. For a structure like
{
"metric1":"1",
"metric2":"2"
}
I'd like to do something like
mutate {
convert => {"metric*" => "float" }
}
Is that possible?

It's not possible without using a ruby filter like this:
ruby {
code => "
event.to_hash.keys.each { |k|
if k.start_with?('metric') and event[k].is_a?(String)
event[k] = event[k].to_float
end
}
"
}
So basically look at all of the keys in the event, and if they start with metric, covert them to a float. The is_a?(String) is there just in case you get an array field (because .to_float won't work on it)

Parse a log using Losgtash

I am using Logstash to parse a log file. A sample log line is shown below.
2011/08/10 09:51:34.450457,1.048908,tcp,213.200.244.217,47908, ->,147.32.84.59,6881,S_RA,0,0,4,244,124,flow=Background-Established-cmpgw-CVUT
I am using following filter in my confguration file.
grok {
match => ["message","%{DATESTAMP:timestamp},%{BASE16FLOAT:value},%{WORD:protocol},%{IP:ip},%{NUMBER:port},%{GREEDYDATA:direction},%{IP:ip2},%{NUMBER:port2},%{WORD:status},%{NUMBER:port3},%{NUMBER:port4},%{NUMBER:port5},%{NUMBER:port6},%{NUMBER:port7},%{WORD:flow}" ]
}
It works well for error-free log lines. But when I have a line like below, it fails. Note that the second field is missing.
2011/08/10 09:51:34.450457,,tcp,213.200.244.217,47908, ->,147.32.84.59,6881,S_RA,0,0,4,244,124,flow=Background-Established-cmpgw-CVUT
I want to put a default value in there in my output Json object, if a value is missing. how can I do that?

Use (%{BASE16FLOAT:value})? for second field to make it optional - ie. regex ()? .
Even if the second field is null the grok will work.
So entire grok look like this:
%{DATESTAMP:timestamp},(%{BASE16FLOAT:value})?,%{WORD:protocol},%{IP:ip},%{NUMBER:port},%{GREEDYDATA:direction},%{IP:ip2},%{NUMBER:port2},%{WORD:status},%{NUMBER:port3},%{NUMBER:port4},%{NUMBER:port5},%{NUMBER:port6},%{NUMBER:port7},%{WORD:flow}

Use it in your conf file. Now, if value field is empty it will omit it in response.
input {
stdin{
}
}
filter {
grok {
match => ["message","%{DATESTAMP:timestamp},%{DATA:value},%{WORD:protocol},%{IP:ip},%{NUMBER:port},%{GREEDYDATA:direction},%{IP:ip2},%{NUMBER:port2},%{WORD:status},%{NUMBER:port3},%{NUMBER:port4},%{NUMBER:port5},%{NUMBER:port6},%{NUMBER:port7},%{WORD:flow}" ]
}
}
output {
stdout {
codec => rubydebug
}
}

logstash mutate filter always stringifies hash and array

I have a json log file, which I am taking as a input with this config
input {
file { filename }
codec { json_lines }
}
Each line is a deeply nested JSON.
In the filters,
When I say
mutate { add_field => { "new_field_name" => "%{old_field_name}"}
if the old_field is a nested hash/array it is converted to a string and then added. Is there anyway I can preserve the type instead of stringifying it ?

You might consider using the ruby filter to duplicate the array:
filter {
ruby { code => "event['newhash'] = event['myhash']" }
}
I don't think there is a cleaner solution.
Apart from that, do you really use the json_lines codec with the file input? From logstash doc:
Do not use this codec if your source input is line-oriented JSON, for
example, redis or file inputs. Rather, use the json codec.

Resources