I filter some "status code" as output attribute from each line. like, code = 100, code = 100, code = 200, code = 200,code = 200,code = 300
Here, I'm trying to get the count of each type of status codes like 100 = 2, 200 = 3, 300 = 1
The config am working on looks like,
filter {
grok {
patterns_dir => ["./patterns"]
match => [ "message", ["%{CUSTOMPATTERN:code}" ]]
}
mutate {
remove_field => ["host", "path", "#version", "#timestamp"]
}
}
output {
statsd {
increment => "%{code}"
count => ["code", "%{code}"]
}
stdout {
codec => json
}
}
I know the statsd config is wrong, but couldn't fix it.
Related
Here is my sample config LS 7.9.
input {
jdbc { ... }
}
filter {
json {
#It's JSON field from DB, including only two for reference.
source => "tags_json"
#Need it as Sub-field like tags.companies, tags.geographies in ES
add_field => {
"[tags][companies]" => "%{companies}"
"[tags][geographies]" => "%{geographies}"
}
output {
elasticsearch { ... }
}
JSON structure in DB field tags_json
{"companies": ["ABC","XYZ"],
"geographies": [{"Market": "Group Market", "Region": "Group Region", "Country": "my_country"}],
"xyz":[]...
}
Logstash prints root geographies field correctly, this is what I need as sub-field under tags.
"geographies" => [
[0] {
"Market" => "Group Market",
"Region" => "Group Region"
},
## But as sub-field under the tags, only geographies is nil
"tags" => {
"companies" => [
[0] "ABC",
[1] "XYZ"
],
"geographies" => nil
}
I tried below copy, ruby, but doesn't seem to fix it :
mutate { copy => { "%{geographies}" => "[tags][geographies]"} }
Also tried Ruby
ruby { code => " event.set('[tags][geographies]', event.get('%{geographies}')) " }
Any help please. Thanks.
Resolved it with ruby event.
ruby {
code => 'event.set("[tags][geographies]", event.get("geographies"))'
}
I want to import aws cloudtrail eventTime through logstash.
Works well but fails to get eventTime.
my logstash.conf
input {
s3 {
bucket => "xxxxx"
prefix => "xxxxx"
sincedb_path => "/etc/logstash/sincedb/cloudtrail"
temporary_directory => "/etc/logstash/tmp"
region => "xxxxx"
type => "cloudtrail"
codec => "cloudtrail"
}
}
filter {
if [type] == "cloudtrail" {
mutate {
gsub => [ "eventSource", "\.amazonaws\.com$", "" ]
}
if [eventSource] == "elasticloadbalancing" and [eventName] == "describeInstanceHealth" and [userIdentity.userName] == "secret_username" {
drop {}
}
}
date {
match => ["eventTime", "ISO8601"]
}
}
In Kibana, other tables can be checked, but eventTime cannot be found.
The cloudtrail codec removes eventTime from the event and assigns this to the events #timestamp. See:
https://github.com/logstash-plugins/logstash-codec-cloudtrail/blob/4486ce2f986bc4778562060bbaf9d6dfd99ab84e/lib/logstash/codecs/cloudtrail.rb#L22
I am trying to parse
[7/1/05 13:41:00:516 PDT]
This is the configuration grok I have written for the same :
\[%{DD/MM/YY HH:MM:SS:S Z}\]
With the date filter :
input {
file {
path => "logstash-5.0.0/bin/sta.log"
start_position => "beginning"
}
}
filter {
grok {
match =>" \[%{DATA:timestamp}\] "
}
date {
match => ["timestamp","DD/MM/YY HH:MM:SS:S ZZZ"]
}
}
output {
stdout{codec => "json"}
}
above is the configuration I have used.
And consider this as my sta.log file content:
[7/1/05 13:41:00:516 PDT]
Getting this error :
[2017-01-31T12:37:47,444][ERROR][logstash.agent ] fetched an invalid config {:config=>"input {\nfile {\npath => \"logstash-5.0.0/bin/sta.log\"\nstart_position => \"beginning\"\n}\n}\nfilter {\ngrok {\nmatch =>\"\\[%{DATA:timestamp}\\]\"\n}\ndate {\nmatch => [\"timestamp\"=>\"DD/MM/YY HH:MM:SS:S ZZZ\"]\n}\n}\noutput {\nstdout{codec => \"json\"}\n}\n\n", :reason=>"Expected one of #, {, ,, ] at line 12, column 22 (byte 184) after filter {\ngrok {\nmatch =>\"\\[%{DATA:timestamp}\\]\"\n}\ndate {\nmatch => [\"timestamp\""}
Can anyone help here?
You forgot to specify the input for your grokfilter. A correct configuration would look like this:
input {
file {
path => "logstash-5.0.0/bin/sta.log"
start_position => "beginning"
}
}
filter {
grok {
match => {"message" => "\[%{DATA:timestamp} PDT\]"}
}
date {
match => ["timestamp","dd/MM/yy HH:mm:ss:SSS"]
}
}
output {
stdout{codec => "json"}
}
For further reference check out the grok documentation here.
I have tab separated string and I want to extract each fields using grok plugin.
The tab separated string is like
http://www.allaboutpc.co.kr 2016110913 d6123c6caa12f08852c82b876bdd3ceceb166d5e 0 0 1 0 /Event/QuizChoice.asp?IdxEvent=3141
I would like to get each fields as url, datetime, hashvalue, count1, count2, count3, count4, path.
I used %{DATA:hashvalue} for 3rd field to extract hashvalue but logstash didn't print hashvalue
Here is my conf file
input {
stdin { }
file {
path => "/Users/Projects/webmastermrinput/20161021/17/*"
codec => plain
}
}
filter {
# tab to space
mutate {
gsub => ["message", "\t", " "]
}
grok {
match => {
'message' => "%{DATA:url} %{NUMBER:datetime2} %{DATA:hashvalue} % {NUMBER:count1} %{NUMBER:count2} %{NUMBER:count3} %{NUMBER:count4} % {URIPATHPARAM:path}'
}
}
}
output {
stdout { codec => rubydebug }
}
Logstash output for input : "http://www.allaboutpc.co.kr 2016110913 d6123c6caa12f08852c82b876bdd3ceceb166d5e 0 0 1 0 /Event/QuizChoice.asp?IdxEvent=3141"
{
"#timestamp" => 2016-11-11T02:26:01.828Z,
"#version" => "1",
"host" => "MacBook-Air-10.local",
"datetime" => "2016110913",
"message" => "http://www.allaboutpc.co.kr 2016110913 d6123c6caa12f08852c82b876bdd3ceceb166d5e 0 0 1 0 /Event/QuizChoice.asp?IdxEvent=3141",
"url" => "http://www.allaboutpc.co.kr"
}
Your grok works perfectly well, you just need to remove the spaces between %and { in % {NUMBER:count1} and % {URIPATHPARAM:path}
'message' => "%{DATA:url} %{NUMBER:datetime2} %{DATA:hashvalue} % {NUMBER:count1} %{NUMBER:count2} %{NUMBER:count3} %{NUMBER:count4} % {URIPATHPARAM:path}'
^ ^
| |
here and here
I am trying to get logstash to parse key-value pairs in an HTTP get request from my ELB log files.
the request field looks like
http://aaa.bbb/get?a=1&b=2
I'd like there to be a field for a and b in the log line above, and I am having trouble figuring it out.
My logstash conf (formatted for clarity) is below which does not load any additional key fields. I assume that I need to split off the address portion of the URI, but have not figured that out.
input {
file {
path => "/home/ubuntu/logs/**/*.log"
type => "elb"
start_position => "beginning"
sincedb_path => "log_sincedb"
}
}
filter {
if [type] == "elb" {
grok {
match => [ "message", "%{TIMESTAMP_ISO8601:timestamp}
%{NOTSPACE:loadbalancer} %{IP:client_ip}:%{NUMBER:client_port:int}
%{IP:backend_ip}:%{NUMBER:backend_port:int}
%{NUMBER:request_processing_time:float}
%{NUMBER:backend_processing_time:float}
%{NUMBER:response_processing_time:float}
%{NUMBER:elb_status_code:int}
%{NUMBER:backend_status_code:int}
%{NUMBER:received_bytes:int} %{NUMBER:sent_bytes:int}
%{QS:request}" ]
}
date {
match => [ "timestamp", "ISO8601" ]
}
kv {
field_split => "&?"
source => "request"
exclude_keys => ["callback"]
}
}
}
output {
elasticsearch { host => localhost }
}
kv will take a URL and split out the params. This config works:
input {
stdin { }
}
filter {
mutate {
add_field => { "request" => "http://aaa.bbb/get?a=1&b=2" }
}
kv {
field_split => "&?"
source => "request"
}
}
output {
stdout {
codec => rubydebug
}
}
stdout shows:
{
"request" => "http://aaa.bbb/get?a=1&b=2",
"a" => "1",
"b" => "2"
}
That said, I would encourage you to create your own versions of the default URI patterns so that they set fields. You can then pass the querystring field off to kv. It's cleaner that way.
UPDATE:
For "make your own patterns", I meant to take the existing ones and modify them as needed. In logstash 1.4, installing them was as easy as putting them in a new file the 'patterns' directory; I don't know about patterns for >1.4 yet.
MY_URIPATHPARAM %{URIPATH}(?:%{URIPARAM:myuriparams})?
MY_URI %{URIPROTO}://(?:%{USER}(?::[^#]*)?#)?(?:%{URIHOST})?(?:%{MY_URIPATHPARAM})?
Then you could use MY_URI in your grok{} pattern and it would create a field called myuriparams that you could feed to kv{}.