So I have the following logstash filter config file.
filter {
if [type] == "syslog" and [message] =~ "diskstation" {
grok {
match => { "message" => "<%{POSINT:syslog_pri}>%{INT:version} %{TIMESTAMP_ISO8601:timestamp} %{HOSTNAME:hostname} %{DATA:syslog_program} - - (?:\[meta sequenceId="%{POSINT:message_id}"])? %{GREEDYDATA:syslog_message}" }
}
} else if [type] == "syslog" {
grok {
match => { "message" => "<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
}
}
}
Using the Grok Debugger the filters work fine, but when starting logstash I get the following error:
Jun 03 11:49:38 nuc logstash[27352]: Error: Expected one of #, {, } at line 16, column 170 (byte 348) after filter {
Jun 03 11:49:38 nuc logstash[27352]: if [type] == "syslog" and [message] =~ "ds02" {
Jun 03 11:49:38 nuc logstash[27352]: grok {
Jun 03 11:49:38 nuc logstash[27352]: match => { "message" => "<%{POSINT:syslog_pri}>%{INT:version} %{TIMESTAMP_ISO8601:timestamp} %{HOSTNAME:hostname} %{DATA:syslog_program} - - (?:\[meta sequenceId="
Jun 03 11:49:38 nuc logstash[27352]: You may be interested in the '--configtest' flag which you can
Jun 03 11:49:38 nuc logstash[27352]: use to validate logstash's configuration before you choose
Jun 03 11:49:38 nuc logstash[27352]: to restart a running system.
I cant for the life of me figure out what the syntax error is. Could anyone point me in the right direction?
The solution was staring me in the face the entire time of course. The problem is the double quotes in the filter. Can be solved by using single quotes.
filter {
if [type] == "syslog" and [message] =~ "diskstation" {
grok {
match => { "message" => '<%{POSINT:syslog_pri}>%{INT:version} %{TIMESTAMP_ISO8601:timestamp} %{HOSTNAME:hostname} %{DATA:syslog_program} - - (?:[meta sequenceId="%{POSINT:message_id}"])? %{GREEDYDATA:syslog_message}' }
}
} else if [type] == "syslog" {
grok {
match => { "message" => "<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" }
}
}
}
Related
I have the following in my logstash pipeline (which receives multiline logs from Filebeat):
filter {
if [type] == "oracle" {
grok {
match => { "message" => "(?<day>[A-Za-z]{3})(\s*)(?<month>[A-Za-z]{3})(\s*)(?<monthday>[0-9]{1,2})(\s*)(?<hour>[0-9]{1,2}):(?<min>[0-9]{1,2}):(?<sec>[0-9]{2})(\s*)(?<year>[0-9]{4})(\s*)%{GREEDYDATA:audit_message}" }
add_tag => [ "oracle_audit" ]
}
grok {
match => { "audit_message" => "ACTION :\[[0-9]*\] '(?<ora_audit_action>.*)'.*DATABASE USER:\[[0-9]*\] '(?<ora_audit_dbuser>.*)'.*PRIVILEGE :\[[0-9]*\] '(?<ora_audit_priv>.*)'.*CLIENT USER:\[[0-9]*\] '(?<ora_audit_osuser>.*)'.*CLIENT TERMINAL:\[[0-9]*\] '(?<ora_audit_term>.*)'.*STATUS:\[[0-9]*\] '(?<ora_audit_status>.*)'.*DBID:\[[0-9]*\] '(?<ora_audit_dbid>.*)'.*SESSIONID:\[[0-9]*\] '(?<ora_audit_sessionid>.*)'.*USERHOST:\[[0-9]*\] '(?<ora_audit_dbhost>.*)'.*CLIENT ADDRESS:\[[0-9]*\] '(?<ora_audit_clientaddr>.*)'.*ACTION NUMBER:\[[0-9]*\] '(?<ora_audit_actionnum>.*)'" }
}
grok {
match => { "source" => [ ".*/[a-zA-Z0-9_#$]*_[a-z0-9]*_(?<ora_audit_derived_pid>[0-9]*)_[0-9]*\.aud" ] }
}
mutate {
add_field => { "ts" => "%{year}-%{month}-%{monthday} %{hour}:%{min}:%{sec}" }
}
date {
locale => "en"
match => [ "ts", "YYYY-MMM-dd HH:mm:ss", "YYYY-MMM-d HH:mm:ss" ]
}
mutate {
remove_field => [ "ts", "year", "month", "day" , "monthday", "hour", "min", "sec", "audit_message" ]
}
}
}
Sample log (coming from Filebeat and I can confirm they have been chunked correctly) :
Audit file /u01/app/oracle/admin/DEVINST/adump/DEVINST_ora_43619_20200913121607479069143795.aud
Oracle Database 12c Standard Edition Release 12.2.0.1.0 - 64bit Production
Build label: RDBMS_12.2.0.1.0_LINUX.X64_170125
ORACLE_HOME: /u01/app/oracle/product/12.2.0/dbhome_1
System name: Linux
Node name: testserver
Release: 3.10.0-862.14.4.el7.x86_64
Version: #1 SMP Fri Sep 21 09:07:21 UTC 2018
Machine: x86_64
Instance name: DEVINST
Redo thread mounted by this instance: 1
Oracle process number: 55
Unix process pid: 43619, image: oracle#testserver (TNS V1-V3)
Sun Sep 13 12:16:07 2020 +00:00
LENGTH : '275'
ACTION :[7] 'CONNECT'
DATABASE USER:[1] '/'
PRIVILEGE :[6] 'SYSDBA'
CLIENT USER:[9] 'testuser'
CLIENT TERMINAL:[5] 'pts/0'
STATUS:[1] '0'
DBID:[10] '1762369616'
SESSIONID:[10] '4294967295'
USERHOST:[21] 'testserver'
CLIENT ADDRESS:[0] ''
ACTION NUMBER:[3] '100'
However although the logs went thru to Kibana, it is not showing me the "right" data, complaining of grokparse and dateparse error (even though the grok rule tested fine in the Kibana debugger!):
Message shown in Kibana:
Audit file /u01/app/oracle/admin/DEVINST/adump/DEVINST_ora_43619_20200913121607479069143795.aud
Oracle Database 12c Standard Edition Release 12.2.0.1.0 - 64bit Production
Build label: RDBMS_12.2.0.1.0_LINUX.X64_170125
ORACLE_HOME: /u01/app/oracle/product/12.2.0/dbhome_1
System name: Linux
Node name: testserver
Release: 3.10.0-862.14.4.el7.x86_64
Version: #1 SMP Fri Sep 21 09:07:21 UTC 2018
Machine: x86_64
Instance name: DEVINST
Redo thread mounted by this instance: 1
Oracle process number: 55
Unix process pid: 43619, image: oracle#testserver (TNS V1-V3)
Message expected:
+00:00
LENGTH : '275'
ACTION :[7] 'CONNECT'
DATABASE USER:[1] '/'
PRIVILEGE :[6] 'SYSDBA'
CLIENT USER:[9] 'testuser'
CLIENT TERMINAL:[5] 'pts/0'
STATUS:[1] '0'
DBID:[10] '1762369616'
SESSIONID:[10] '4294967295'
USERHOST:[21] 'testserver'
CLIENT ADDRESS:[0] ''
ACTION NUMBER:[3] '100'
Due to these the fields werent parsed properly either.
What am I doing wrong? Why is it not parsing the message and the date correctly even though the debugger shows the right output?
EDIT:
As per baudsp suggestion I have overwritten my message like so:
filter {
if [type] == "oracle" {
grok {
match => { "message" => "(?<day>[A-Za-z]{3})(\s*)(?<month>[A-Za-z]{3})(\s*)(?<monthday>[0-9]{1,2})(\s*)(?<hour>[0-9]{1,2}):(?<min>[0-9]{1,2}):(?<sec>[0-9]{2})(\s*)(?<year>[0-9]{4})(\s*)(?<message>[\S\s]*)" }
overwrite => [ "message" ]
}
.....
However Kibana is still showing me grokparse and dateparse errors :(
Thanks
J
I having been trying since long time to extract and mark data from my customized log using logstash, but not getting anywhere, I having a customized haproxy log like below:
Feb 22 21:17:32 ap haproxy[1235]: 10.172.80.45:32071 10.31.33.34:44541 10.31.33.34:32772 13.127.229.72:443 [22/Feb/2020:21:17:32.006] this_machine~ backend_test-tui/test-tui_32772 40/0/5/1/836 200 701381 - - ---- 0/0/0/0/0 0/0 {testtui.net} {cache_hit} "GET /ob/720/output00007.ts HTTP/1.1"
I want to extract and mark specific content in kibana dashboard from log, like:
from "40/0/5/1/836" section i want to mark the only the last section digit (836) as "response_time"
"701381" as "response_bytes"
"/ob/720/output00007.ts" as "content_url"
And want to use the timestamp in the log file and not the default one
I have created a grok filter using https://grokdebug.herokuapp.com/ but whenever i apply it i m seeing "_grokparsefailure" message and the kibana dashboard stops getting populated
Below is the logstash debug log
{
"#version" => "1",
"message" => "Mar 8 13:53:59 ap haproxy[22158]: 10.172.80.45:30835 10.31.33.34:57886 10.31.33.34:32771 43.252.91.147:443 [08/Mar/2020:13:53:59.827] this_machine~ backend_noida/noida_32771 55/0/1/0/145 200 2146931 - - ---- 0/0/0/0/0 0/0 {testalef1.adcontentamtsolutions.} {cache_hit} \"GET /felaapp/virtual_videos/og/1080/output00006.ts HTTP/1.1\"",
"#timestamp" => 2020-03-08T10:24:07.348Z,
"path" => "/home/alef/haproxy.log",
"host" => "com1",
"tags" => [
[0] "_grokparsefailure"
]
}
Below is the Filter which i have created
%{MONTH:[Month]} %{MONTHDAY:[date]} %{TIME:[time]} %{WORD:[source]} %{WORD:[app]}\[%{DATA:[class]}\]: %{IPORHOST:[UE_IP]}:%{NUMBER:[UE_Port]} %{IPORHOST:[NATTED_IP]}:%{NUMBER:[NATTED_Source_Port]} %{IPORHOST:[NATTED_IP]}:%{NUMBER:[NATTED_Destination_Port]} %{IPORHOST:[WAN_IP]}:%{NUMBER:[WAN_Port]} \[%{HAPROXYDATE:[accept_date]}\] %{NOTSPACE:[frontend_name]}~ %{NOTSPACE:[backend_name]} %{NOTSPACE:[ty_name]}/%{NUMBER:[response_time]} %{NUMBER:[http_status_code]} %{INT:[response_bytes]} - - ---- %{NOTSPACE:[df]} %{NOTSPACE:[df]} %{DATA:[domain_name]} %{DATA:[cache_status]} %{DATA:[domain_name]} %{NOTSPACE:[content]} HTTP/%{NUMBER:[http_version]}
Below is my logstash conf file:
input {
beats {
port => 5044
}
}
filter {
grok {
match => { "message" => "%{MONTH:[Month]} %{MONTHDAY:[date]} %{TIME:[time]} %{WORD:[source]} %{WORD:[app]}\[%{DATA:[class]}\]: %{IPORHOST:[UE_IP]}:%{NUMBER:[UE_Port]} %{IPORHOST:[NATTED_IP]}:%{NUMBER:[NATTED_Source_Port]} %{IPORHOST:[NATTED_IP]}:%{NUMBER:[NATTED_Destination_Port]} %{IPORHOST:[WAN_IP]}:%{NUMBER:[WAN_Port]} \[%{HAPROXYDATE:[accept_date]}\] %{NOTSPACE:[frontend_name]}~ %{NOTSPACE:[backend_name]} %{NOTSPACE:[ty_name]}/%{NUMBER:[response_time]} %{NUMBER:[http_status_code]} %{INT:[response_bytes]} - - ---- %{NOTSPACE:[df]} %{NOTSPACE:[df]} %{DATA:[domain_name]} %{DATA:[cache_status]} %{DATA:[domain_name]} %{NOTSPACE:[content]} HTTP/%{NUMBER:[http_version]} " }
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}
output {
elasticsearch { hosts => ["localhost:9200"] }
}
Using the below filter resolved my issue had to do debugging in the logstash itself to get proper filter:
input { beats {
port => 5044 } }
filter { grok {
match => { "message" => "%{MONTH:month} %{MONTHDAY:date} %{TIME:time} %{WORD:[source]} %{WORD:[app]}[%{DATA:[class]}]:
%{IPORHOST:[UE_IP]}:%{NUMBER:[UE_Port]}
%{IPORHOST:[NATTED_IP]}:%{NUMBER:[NATTED_Source_Port]}
%{IPORHOST:[NATTED_IP]}:%{NUMBER:[NATTED_Destination_Port]}
%{IPORHOST:[WAN_IP]}:%{NUMBER:[WAN_Port]}
[%{HAPROXYDATE:[accept_date]}] %{NOTSPACE:[frontend_name]}~
%{NOTSPACE:[backend_name]}
%{NOTSPACE:[ty_name]}/%{NUMBER:[response_time]:int}
%{NUMBER:[http_status_code]} %{NUMBER:[response_bytes]:int} - - ----
%{NOTSPACE:[df]} %{NOTSPACE:[df]} %{DATA:[domain_name]}
%{DATA:[cache_status]} %{DATA:[domain_name]} %{URIPATHPARAM:[content]}
HTTP/%{NUMBER:[http_version]}" }
add_tag => [ "response_time", "response_time" ]
} date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ] } }
output { elasticsearch { hosts => ["localhost:9200"] }
stdout {
codec => rubydebug
} }
I am new to logstash.When I am trying to put a if statement in logstash config file it gives me error
if statement used is:
if {await} > 10
{ mutate {add_field => {"RULE_DATA" => "Value is above threshold"}
add_field => {"ACTUAL_DATA" => "%{await}"}
}
}
the error faced is given below:
[ERROR] 2018-07-20 16:52:21.327 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, => at line 18, column 10 (byte 729) after filter{\n grok {\n patterns_dir => [\"./patterns\"]\n match => { \"message\" => [\"%{TIME:time}%{SPACE}%{USERNAME:device}%{SPACE}%{USERNAME:tps}%{SPACE}%{SYSLOGPROG:rd_sec/s}%{SPACE}%{SYSLOGPROG:wr_sec/s}%{SPACE}%{SYSLOGPROG:avgrq-sz}%{SPACE}%{SYSLOGPROG:avgqu-sz}%{SPACE}%{NUMBER:await}%{SPACE}%{SYSLOGPROG:svctm}%{SPACE}%{SYSLOGPROG:%util}\"]\n }\n overwrite => [\"message\"]\n } \n if \"_grokparsefailure\" in [tags] {\n drop { }\n }\nif {await", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:incompile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in block in compile_sources'", "org/jruby/RubyArray.java:2486:inmap'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in compile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:51:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:169:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:315:in block in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:inwith_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:312:in block in converge_state'", "org/jruby/RubyArray.java:1734:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:299:in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:inblock in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:inconverge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:348:inblock in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}
Please suggest what has caused this error.
You have a syntax error. If you have a field as name it await. Like output of grok parse etc.
use the below
if [await] > 10
{
mutate {
add_field => {"RULE_DATA" => "Value is above threshold"}
add_field => {"ACTUAL_DATA" => "%{await}"}
}
}
Logstash conditional's expression enclosed in [] not {}, have a look at the following example from conditional documentation,
filter {
if [action] == "login" {
mutate { remove_field => "secret" }
}
}
I'm trying to replace 10.100.251.98 with another IP 10.100.240.199 in my logstash config, I have tried using filter with mutate function, yet, I'm unable to get the syntax wrtie
Sep 25 15:50:57 10.100.251.98 mail_logs: Info: New SMTP DCID 13417989 interface 172.30.75.10 address 172.30.75.12 port 25
Sep 25 15:50:57 10.100.251.98 local_mail_logs: Info: New SMTP DCID 13417989 interface 172.30.75.10 address 172.30.75.12 port 25
Sep 25 15:51:04 10.100.251.98 cli_logs: Info: PID 35559: User smaduser login from 10.217.3.22 on 172.30.75.10
Sep 25 15:51:22 10.100.251.98 cli_logs: Info: PID 35596: User smaduser login from 10.217.3.22 on 172.30.75.10
Here is my code:
input { file { path => "/data/collected" } }
filter {
if [type] == "syslog" {
mutate {
replace => [ "#source_host", "10.100.251.99" ]
}
}
}
output {
syslog {
facility => "kernel"
host => "10.100.250.199"
port => 514
}
}
I'm noticing a few things about your config. First, you don't have any log parsing. You won't be able to replace a field if it doesn't yet exist. To do this, you can use a codec in your input block or a grok filter. I added a simple grok filter.
You also check if [type] == "syslog". You never set the type, so that check will always fail. If you want to set a type, you can do that in your input block input { file { path => "/data/collected" type => "syslog} }
Here is the sample config I used for testing the grok pattern and replacement of the IP.
input { tcp { port => 5544 } }
filter {
grok { match => { "message" => "%{CISCOTIMESTAMP:log_time} %{IP:#source_host} %{DATA:log_type}: %{DATA:log_level}: %{GREEDYDATA:log_message}" } }
mutate {
replace => [ "#source_host", "10.100.251.199" ]
}
}
output {
stdout { codec => rubydebug }
}
which outputs this:
{
"message" => "Sep 25 15:50:57 10.100.251.98 mail_logs: Info: New SMTP DCID 13417989 interface 172.30.75.10 address 172.30.75.12 port 25",
"#version" => "1",
"#timestamp" => "2016-09-25T14:03:20.332Z",
"host" => "0:0:0:0:0:0:0:1",
"port" => 52175,
"log_time" => "Sep 25 15:50:57",
"#source_host" => "10.100.251.199",
"log_type" => "mail_logs",
"log_level" => "Info",
"log_message" => "New SMTP DCID 13417989 interface 172.30.75.10 address 172.30.75.12 port 25"
}
I'm a new user to ELK stack. I'm using UWSGI as my server. I need to parse my uwsgi logs using Grok and then analyze them.
Here is the format of my logs:-
[pid: 7731|app: 0|req: 357299/357299] ClientIP () {26 vars in 511 bytes} [Sun Mar 1 07:47:32 2015] GET /?file_name=123&start=0&end=30&device_id=abcd&verif_id=xyzsghg => generated 28 bytes in 1 msecs (HTTP/1.0 200) 2 headers in 79 bytes (1 switches on core 0)
I used this link to generate my filter, but it didn't parse much of the information.
The filter generated by the above link is
%{SYSLOG5424SD} %{IP} () {26 vars in 511 bytes} %{SYSLOG5424SD} GET %{URIPATHPARAM} => generated 28 bytes in 1 msecs (HTTP%{URIPATHPARAM} 200) 2 headers in 79 bytes (1 switches on core 0)
Here is my logstash-conf file.
input { stdin { } }
filter {
grok {
match => { "message" => "%{SYSLOG5424SD} %{IP} () {26 vars in 511 bytes} %{SYSLOG5424SD} GET %{URIPATHPARAM} => generated 28 bytes in 1 msecs (HTTP%{URIPATHPARAM} 200) 2 headers in 79 bytes (1 switches on core 0)" }
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}
output {
stdout { codec => rubydebug }
}
Upon running logstash with this conf file, I get an error message saying:-
{
"message" => "[pid: 7731|app: 0|req: 357299/357299] ClientIP () {26 vars in 511 bytes} [Sun Mar 1 07:47:32 2015] GET /?file_name=123&start=0&end=30&device_id=abcd&verif_id=xyzsghg => generated 28 bytes in 1 msecs (HTTP/1.0 200) 2 headers in 79 bytes (1 switches on core 0)",
"#version" => "1",
"#timestamp" => "2015-03-01T07:57:02.291Z",
"host" => "cube26-Inspiron-3542",
"tags" => [
[0] "_grokparsefailure"
]
}
The date has been properly formatted. How do I extract other information from my logs, such as my query parameters(filename, start,end, deviceid etc) and ClientIP , Response code etc.
Also, is there any built-in UWSGI log parser which can be used, such as the one built for apache and syslog?
EDIT
I wrote this on my own, but it throws the same error:
%{SYSLOG5424SD} %{IP:client_ip} () {%{NUMBER:vars} vars in %{NUMBER:bytes} bytes} %{SYSLOGTIMESTAMP:date} %{WORD:method} %{URIPATHPARAM:request} => generated %{NUMBER:generated_bytes} bytes in {NUMBER:secs} msecs (HTTP/1.0 %{NUMBER:response_code}) %{NUMBER:headers} headers in %{NUMBER:header_bytes} (1 switches on core 0)
EDIT 2
I'm finally able to crack it myself. The GROK filter for the above log will be:
\[pid: %{NUMBER:pid}\|app: %{NUMBER:app}\|req: %{NUMBER:req_num1}/%{NUMBER:req_num2}\] %{IP:client_ip} \(\) \{%{NUMBER:vars} vars in %{NUMBER:bytes} bytes\} %{SYSLOG5424SD} %{WORD:method} /\?file_name\=%{NUMBER:file_name}\&start\=%{NUMBER:start}\&end\=%{NUMBER:end} \=\> generated %{NUMBER:generated_bytes} bytes in %{NUMBER:secs} msecs \(HTTP/1.0 %{NUMBER:response_code}\) %{NUMBER:headers} headers in %{NUMBER:header_bytes}
But my questions still remain:
is there any default uwsgi log filter in grop??**
I've been applying different matches for different query parameters. Is there anything in grok that fetches the different query parameters by itself??
I found the solution for extracting the query parameters:-
Here is my final configuration:-
For log line
[pid: 7731|app: 0|req: 426435/426435] clientIP () {28 vars in 594 bytes} [Mon Mar 2 06:43:08 2015] GET /?file_name=wqvqwv&start=0&end=30&device_id=asdvqw&verif_id=qwevqwr&lang=English&country=in => generated 11018 bytes in 25 msecs (HTTP/1.0 200) 2 headers in 82 bytes (1 switches on core 0)
the configuration is
input { stdin { } }
filter {
grok {
match => { "message" => "\[pid: %{NUMBER}\|app: %{NUMBER}\|req: %{NUMBER}/%{NUMBER}\] %{IP} \(\) \{%{NUMBER} vars in %{NUMBER} bytes\} %{SYSLOG5424SD:DATE} %{WORD} %{URIPATHPARAM} \=\> generated %{NUMBER} bytes in %{NUMBER} msecs \(HTTP/1.0 %{NUMBER}\) %{NUMBER} headers in %{NUMBER}" }
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
kv {
field_split => "&? "
include_keys => [ "file_name", "device_id", "lang", "country"]
}
}
output {
stdout { codec => rubydebug }
elasticsearch { host => localhost }
}
I found your solution did't support HTTP/1.1. I fixed it and also add variables name. Ref
Here's my grok config:
grok {
match => { "message" => "\[pid: %{NUMBER:pid}\|app: %{NUMBER:id}\|req: %{NUMBER:currentReq}/%{NUMBER:totalReq}\] %{IP:remoteAddr} \(%{WORD:remoteUser}?\) \{%{NUMBER:CGIVar} vars in %{NUMBER:CGISize} bytes\} %{SYSLOG5424SD:timestamp} %{WORD:method} %{URIPATHPARAM:uri} \=\> generated %{NUMBER:resSize} bytes in %{NUMBER:resTime} msecs \(HTTP/%{NUMBER:httpVer} %{NUMBER:status}\) %{NUMBER:headers} headers in %{NUMBER:headersSize} bytes %{GREEDYDATA:coreInfo}" }
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}