I am facing problem with Logstash KV filter:
Below is sample event:
2016-08-15T12:43:04.478Z 103.240.35.216 <190>date=2016-08-15 time=18:13:16 timezone="IST" device_name="CR25iNG" device_id=C2222-123 log_id=010302602002 log_type="Firewall" log_component="Appliance Access" log_subtype="Denied" status="Deny" priority=Information duration=0 fw_rule_id=0 user_name="" user_gp="" iap=0 ips_policy_id=0 appfilter_policy_id=0 application="" application_risk=0 application_technology="" application_category="" in_interface="PortA" out_interface="" src_mac=44:d9:e7:ba:5b:6c src_ip=172.16.16.19 src_country_code= dst_ip=255.255.255.255 dst_country_code= protocol="UDP" src_port=45541 dst_port=10001 sent_pkts=0 recv_pkts=0 sent_bytes=0 recv_bytes=0 tran_src_ip= tran_src_port=0 tran_dst_ip= tran_dst_port=0 srczonetype="" srczone="" dstzonetype="" dstzone="" dir_disp="" connid="" vconnid=""
Below is the KV filter output:
"#version" => "1",
"#timestamp" => "2016-08-16T13:48:30.602Z",
"type" => "cyberoam.input",
"host" => "ip-172-31-6-249",
"time" => "18:13:16",
"timezone" => "IST",
"status" => "Deny",
"priority" => "Information",
"duration" => "0",
"iap" => "0",
"application" => "",
"application_risk" => "0",
"application_technology" => "",
"application_category" => "",
"dst_country_code" => "protocol=UDP",
"recv_pkts" => "0",
"tran_src_ip" => "tran_src_port=0",
"tran_dst_ip" => "tran_dst_port=0",
"srczonetype" => "",
"srczone" => "",
"dstzonetype" => "",
"dstzone" => "",
"dir_disp" => "",
"syslog_severity_code" => 5,
"syslog_facility_code" => 1,
"syslog_facility" => "user-level",
"syslog_severity" => "notice",
"date" => "2016-08-15",
Problem:
"dst_country_code" => "protocol=UDP",
"tran_src_ip" => "tran_src_port=0",
"tran_dst_ip" => "tran_dst_port=0",
Above is due to empty keys "dst_country_code", "tran_src_ip" and "tran_dst_ip".
I was suggested to use mutate gsub to add default value to empty field by substituting =\w with ="".
But this never worked.
Pleas help.
I got response from Logstash community and that worked.
mutate {
gsub => [ 'message', '= ', '="" ' ]
}
Thanks.
Related
I am unsuccessfully trying to drop logs based on the value of the kv value field.
filter {
if [type] == "cef" {
mutate {
add_field => { "tmp_message" => "%{message}" }
split => ["message", "|"]
add_field => { "version" => "%{message[0]}" }
add_field => { "device_vendor" => "%{message[1]}" }
add_field => { "device_product" => "%{message[2]}" }
add_field => { "device_version" => "%{message[3]}" }
add_field => { "sig_id" => "%{message[4]}" }
add_field => { "sig_name" => "%{message[5]}" }
add_field => { "sig_severity" => "%{message[6]}" }
}
kv {
field_split => " "
trim_value => "<>\[\],"
}
mutate {
replace => { "message" => "%{tmp_message}" }
remove_field => [ "tmp_message" ]
}
}
if [FTNTFGTsrcintfrole_s] == "wan" {
drop { }
}
[FTNTFGTsrcintfrole_s] is one of the keys that are parsed out by kv. If the value of the key is "wan", it should drop the log. That's not happening.
How can I filter out those logs?
Edit: Here is an example of the parsed data
{
"dst" => "xxx.xxx.xxx.xxx",
"FTNTFGTtz" => "+0000",
"FTNTFGTsubtype" => "forward",
"message" => "%{tmp_message}",
"host" => "xxx.xxx.xxx.xxx",
"spt" => "59975",
"type" => "cef",
"deviceInboundInterface" => "ssl.root",
"FTNTFGTdstintfrole" => "wan",
"FTNTFGTduration" => "180",
"FTNTFGTdstcountry" => "United",
"FTNTFGTpolicyid" => "47",
"FTNTFGTpolicytype" => "policy",
"FTNTFGTpoluuid" => "801d40c2-3b60-51ea-d66a-293bf886d27e",
"FTNTFGTeventtime" => "1633506791693710149",
"sourceTranslatedAddress" => "xxx.xxx.xxx.xxx",
"dpt" => "8253",
"app" => "udp/8253",
"FTNTFGTpolicyname" => "xxxxxxxx",
"tags" => [
[0] "fortigate",
[1] "_mutate_error"
],
"act" => "accept",
"FTNTFGTlogid" => "0000000013",
"in" => "64",
"sourceTranslatedPort" => "59975",
"FTNTFGTsentpkt" => "1",
"FTNTFGTtrandisp" => "snat",
"FTNTFGTsrcintfrole" => "wan",
"#version" => "1",
"FTNTFGTrcvdpkt" => "1",
"deviceExternalId" => "xxxxx",
"FTNTFGTauthserver" => "xxxxx",
"#timestamp" => 2021-10-06T07:53:11.729Z,
"FTNTFGTsrccountry" => "Reserved",
"deviceOutboundInterface" => "wan1",
"proto" => "17",
"out" => "48",
"src" => "xxx.xxx.xxx.xxx",
"externalId" => "870512",
"FTNTFGTlevel" => "notice",
"FTNTFGTvd" => "root",
"duser" => "xxxxx",
"cat" => "traffic:forward",
"FTNTFGTappcat" => "unscanned"
}
I found the answer thanks to #YLR and #Filip. The SIEM was adding "_s" to the key name when creating the field leading me to believe that that was the original key name and in turn what I was filtering for. After seeing the log output and realizing that wasn't the case, I corrected the filter and it worked.
I have log files that I am able to get fields based on two different if/grok statements and patterns. The output from the two are like below;
{
timestamp" => 2021-06-09T03:08:30.943Z,
"Loc" => "91340",
"#version" => "1",
"#timestamp" => 2021-07-17T04:09:36.438Z,
"location" => 274.05292,
"speed" => 2.6279999999999997,
"target_location" => 261.11999999999995,
"host" => "AUDPRWL00192",
"path" => "C:/ELK/LOGS/91340____________090621_021536_2653_ATO_B.txt",
}
{
"ID" => "066",
"host" => "AUDPRWL00192",
"MESSAGE" => "0560BFC0BC00C8005023AE00164260BFC0BC6B5DDC5B",
"timestamp" => 2021-06-09T03:08:27.540Z,
"path" => "C:/ELK/LOGS/91340____________090621_021536_2653_ATO_B.txt",
"Loc" => "91340",
"#version" => "1",
"#timestamp" => 2021-07-17T04:09:36.428Z
I am trying to aggregate so that my end goal is to get the following i.e pick values from the previous event i.e speed and location so that the output that i can send to Elastic is;
{
"ID" => "066",
"host" => "AUDPRWL00192",
"MESSAGE" => "0560BFC0BC00C8005023AE00164260BFC0BC6B5DDC5B",
"timestamp" => 2021-06-09T03:08:27.540Z,
"path" => "C:/ELK/LOGS/91340____________090621_021536_2653_ATO_B.txt",
"Loc" => "91340",
"speed" => 2.6279999999999997,
"location" => 274.05292,
"#version" => "1",
"#timestamp" => 2021-07-17T04:09:36.428Z
}
The aggregation filter i am trying is;
aggregate {
task_id => "%{host}%{path}"
code => "map['location'] = event.get('[location]')"
map_action => "create"}
I am working in the Elapsed filter. I read the guide of Elapsed filter in logstash. then i made a sample config file and csv to test the working of Elapsed filter. But it seems to be not working. There is no change in uploading the data to ES. i have attached the csv file and config code. Can you give some examples for how to use the elapsed filter.
Here's my csv data:
here's my config file:
input {
file {
path => "/home/paulsteven/log_cars/aggreagate.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
csv {
separator => ","
quote_char => "%"
columns => ["state","city","haps","ads","num_id","serial"]
}
elapsed {
start_tag => "taskStarted"
end_tag => "taskEnded"
unique_id_field => "num_id"
}
}
output {
elasticsearch {
hosts => "localhost:9200"
index => "el03"
document_type => "details"
}
stdout{}
}
Output in ES:
{
"city" => "tirunelveli",
"path" => "/home/paulsteven/log_cars/aggreagate.csv",
"num_id" => "2345-1002-4501",
"message" => "tamil nadu,tirunelveli,hap0,ad1,2345-1002-4501,1",
"#version" => "1",
"serial" => "1",
"haps" => "hap0",
"state" => "tamil nadu",
"host" => "smackcoders",
"ads" => "ad1",
"#timestamp" => 2019-05-06T10:03:51.443Z
}
{
"city" => "chennai",
"path" => "/home/paulsteven/log_cars/aggreagate.csv",
"num_id" => "2345-1002-4501",
"message" => "tamil nadu,chennai,hap0,ad1,2345-1002-4501,5",
"#version" => "1",
"serial" => "5",
"haps" => "hap0",
"state" => "tamil nadu",
"host" => "smackcoders",
"ads" => "ad1",
"#timestamp" => 2019-05-06T10:03:51.447Z
}
{
"city" => "kottayam",
"path" => "/home/paulsteven/log_cars/aggreagate.csv",
"num_id" => "2345-1002-4501",
"message" => "kerala,kottayam,hap1,ad2,2345-1002-4501,9",
"#version" => "1",
"serial" => "9",
"haps" => "hap1",
"state" => "kerala",
"host" => "smackcoders",
"ads" => "ad2",
"#timestamp" => 2019-05-06T10:03:51.449Z
}
{
"city" => "Jalna",
"path" => "/home/paulsteven/log_cars/aggreagate.csv",
"num_id" => "2345-1002-4501",
"message" => "mumbai,Jalna,hap2,ad3,2345-1002-4501,13",
"#version" => "1",
"serial" => "13",
"haps" => "hap2",
"state" => "mumbai",
"host" => "smackcoders",
"ads" => "ad3",
"#timestamp" => 2019-05-06T10:03:51.452Z
}
You have to tag your events in order Logstash could find the start / end tags.
Basically you have to know when an event is considered a start event and when it's an end event.
Elapsed filter plugin works only for two events (for example a request event and a response event in order to get the latency between them)
Both these two kinds of event need to own an ID field which identify uniquely that particular task. The name of this field is stored in unique_id_field.
For your example you have to identify a pattern for start and end event, let's say that you have in your csv a column type (see the code below) when type contains "START", the line is considered start event and if it contains "END" it's an end event, pretty straightforward, and a columnn id that stores the unique identifier.
filter {
csv {
separator => ","
quote_char => "%"
columns => ["state","city","haps","ads","num_id","serial", "type", "id"]
}
grok {
match => { "type" => ".*START.*" }
add_tag => [ "taskStarted" ]
}grok {
match => { "type" => ".*END*" }
add_tag => [ "taskTerminated" ]
} elapsed {
start_tag => "taskStarted"
end_tag => "taskTerminated"
unique_id_field => "id"
}
}
I feel like your need is different.
If you want to aggregate more than two events, all the events with the same value for column state for example, please check out this plugin
from my output as below,
"message" =>"<....... ",
"#version" => "1",
"#timestamp" => "2016-04-29T02:33:34.586Z",
"timestamp" => "Apr 29 10:30:37",
"syslog_severity_code" => 5,
"syslog_facility_code" => 1,
"syslog_facility" => "user-level",
"syslog_severity" => "notice"
i try to get the field value
filter
{
mutate {
add_field => {"newfield"=> "timestamp"}
}
but still cann't get the timestamp value to newfield
it's will get
"newfield" => "newfield",
Is anyone is having the same problem or find a solution?
Any help is welcome to resolve this.
That`s easy,do it like this:
filter
{
mutate {
add_field => ["newfield"=> "%{timestamp}"]
}
You can get :
"message" => "test it \b\b",
"#version" => "1",
"#timestamp" => "2016-04-28T09:16:56.934Z",
"host" => "bag",
"timestamp" => "Apr 29 10:30:37",
"new_field" => "Apr 29 10:30:37"
i have a log print as follows,
"message" => "....",
"host" => "10.10.12.13",
"#version" => "1",
"#timestamp" => "2016-04-13T01:52:43.535Z",
"DISMAN-EVENT-MIB::sysUpTimeInstance" => "22 days, 16:33:23.24",
"SNMP-MIB::OID_0" => "example::bgpPeerState",
"source_ip" => "10.10.12.13"
I want to parse the string that is based on the prefix "specific" and add a field for this and remove the original
"SNMP-MIB::OID_0" => "example::bgpPeerState"
it's should looks like as below ,
"message" => "....",
"host" => "10.10.12.13",
"#version" => "1",
"#timestamp" => "2016-04-13T01:52:43.535Z",
"type" => "snmptrap",
"DISMAN-EVENT-MIB::sysUpTimeInstance" => "22 days, 16:33:23.24",
"example" => "bgpPeerState",
"source_ip" => "10.10.12.13"
my conf,
filter
{
if "example" in [SNMP-MIB::OID_0] {
# I don't how to parse it and add a field ???
}
else
{
.......
}
}
As always, many thanks for your help!
Use kv filter:
filter {
if "example" in [SNMP-MIB::OID_0] {
kv {
source => "SNMP-MIB::OID_0"
value_split => ":"
trim => ":"
remove_field => "SNMP-MIB::OID_0"
}
}
}
}