I use logstash as a syslog relay, it forwards the data to a graylog and writes data to a file.
I use the dns filter module to replace the IP with the FQDN and after this I can't write raw content to file, the IP is "json-ed".
What I get :
2022-05-17T15:17:01.580175Z {ip=vm2345.lab.com} <86>1 2022-05-17T17:17:01.579496+02:00 vm2345 CRON 2057538 - - pam_unix(cron:session): session closed for user root
What I want to get :
2022-05-17T15:17:01.580175Z vm2345.lab.com <86>1 2022-05-17T17:17:01.579496+02:00 vm2345 CRON 2057538 - - pam_unix(cron:session): session closed for user root
My config :
input {
syslog {
port => 514
type => "rsyslog"
}
}
filter {
if [type] == "rsyslog" {
dns {
reverse => [ "[host][ip]" ]
action => "replace"
}
}
}
output {
if [type] == "rsyslog" {
gelf {
host => "graylog.lab.com"
port => 5516
}
file {
path => "/data/%{+YYYY}/%{+MM}/%{+dd}/%{[host][ip]}/%{[host][ip]}_%{{yyyy_MM_dd}}.log"
codec => "line"
}
stdout { }
}
}
What's the best way to handle this ?
When you use codec => line, there is no default setting for the #format option, so the codec calls, .to_s on the event. The toString method for an event concatenates the #timestamp, the [host] field, and [message] field. You want the [host][ip] field, not the [host] field (which is an object) so tell the codec that
codec => line { format => "%{#timestamp} %{[host][ip]} %{message}" }
I've tried to parse it using the json, json_lines and even the multiline input plugin, yet to no avail. The multiline works well on my local machine but doesn't seem to work on my s3 and ec2 instance.
How would I write the grok filter to parse this?
This is what my JSON file looks like
{
"sourceId":"94:54:93:3B:81:6F1",
"machineId":"c1VR21A0GoCBgU6EMJ78d3CL",
"columnsCSV":"timestamp,state,0001,0002,0003,0004",
"tenantId":"iugcp",
"valuesCSV":"1557920277890,1,98.66,0.07,0.1,0.17 ",
"timestamp":"2019-05-15T11:37:57.890Z"
}
This is my config -
input {
file{
codec => multiline
{
pattern => '^\{'
negate => true
what => previous
}
path => "/home/*myusername*/Desktop/data/*.json"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
mutate
{
replace => [ "message", "%{message}}" ]
gsub => [ 'message','\n','']
}
if [message] =~ /^{.*}$/
{
json { source => message }
}
}
//Output tag is correct, haven't included it here
The results I get is just the json file present in the "message" field.
What I wanted is for every json tag, there should be a separate field in the document.
Logstash v2.4.1.
I'm sending JSON formatted logs to my Logstash server via UDP packet. The logs look something similar to this.
{
"key1":"value1",
"key2":"value2",
"msg":"2017-03-02 INFO [com.company.app] Hello world"
}
This is my output filter
output {
stdout {
codec => rubydebug
}
file {
path => "/var/log/trm/debug.log"
codec => line { format => "%{msg}" }
}
}
The rubydebug output codec shows the log like this
{
"message" => {\"key1\":\"value1\", "key2\":\"value2\", \"msg\":\"2017-03-02 INFO [com.company.app] Hello world\"
}
and the file output filter also shows the JSON log correctly, like this
{"key1":"value1", "key2":"value2", "msg":"2017-03-02 INFO [com.company.app] Hello world"}
When I use the JSON code in the input filter, I get _jsonparsefailures from Logstash on "some" logs, even though different online JSON parsers parse the JSON correctly, meaning my logs are in a valid JSON format.
input {
udp => {
port => 5555
codec => json
}
}
Therefore, I'm trying to use the json filter instead, like this
filter {
json => {
source => "message"
}
}
Using the json filter, how can I extract the "key1", "key2", and the "msg" fields in the "message?"
I tried this to no avail, that is, I don't see the "key1" field in my rubydebug output.
filter {
json => {
source => "message"
add_field => {
"key1" => "%{[message][key1]}"
}
}
}
I would suggest you to start with one of the two configuration below (I use the multiline codec to concatenate the input into a json, because otherwise logstash will read line by line, and one line of a json is not a valid json), then either filter the json, or use the json codec, and then output it to wherever it is needed. You will still have some configuration to do, but I believe it might help you to get started:
input{
file {
path => "/an/absolute/path/tt2.json" #It really has to be absolute!
start_position => beginning
sincedb_path => "/another/absolute/path" #Not mandatory, just for ease of testing
codec => multiline{
pattern => "\n"
what => "next"
}
}
}
filter{
json {
source => "multiline"
}
}
output {
file {
path => "data/log/trm/debug.log"
}
stdout{codec => json}
}
Second possibility:
input{
file {
path => "/an/absolute/path/tt2.json" #It really has to be absolute!
start_position => beginning
sincedb_path => "/another/absolute/path" #Not mandatory, just for ease of testing
codec => multiline{
pattern => "\n"
what => "next"
}
codec => json{}
}
}
output {
file {
path => "data/log/trm/debug.log"
}
stdout{codec => json}
}
Edit With the udp input I guess it should be (not tested):
input {
udp => {
port => 5555
codec => multiline{ #not tested this part
pattern => "^}"
what => "previous"
}
codec => json{}
}
}
I have the following json object logs as following in a log file
{"con":"us","sl":[[1,2]],"respstats_1":"t:2,ts:140,m:192.168.7.5,p:|mobfox:1,P,E,0,0.4025:0.0:-:-,0-98;appnexus-marimedia:2,P,L,140,0.038:0.0:-:-,-;","rid":"AKRXRWLYCZIDFM","stats":"t:2,h:2,ts:140,mobfox:0,appnexus-marimedia:140,;m:192.168.7.5;p:","resp_count":0,"client_id":"15397682","err_stats":"mobfox:0-98,"}
{"con":"br","sl":[[1,2,3,4]],"respstats_1":"t:4,ts:285,m:192.168.7.5,p:|smaato:1,P,M,143,0.079:0.0:-:-,-;vserv-specialbuy:2,P,W,285,0.0028:0.0:-:-,-;mobfox:3,P,E,42,0.077:0.0:-:-,0-98;inmobi-pre7:4,P,H,100,0.0796:0.0:-:-,-;","rid":"AKRXRWLYCY4DOU","stats":"t:4,h:4,ts:285,smaato:143,vserv-specialbuy:285,mobfox:42,inmobi-pre7:100,;m:192.168.7.5;p:","resp_count":1,"client_id":"15397682","err_stats":"mobfox:0-98,","ads":[{"pricing":{"price":"0","type":"cpc"},"rank":2,"resp_json":{"img_url":"http://img.vserv.mobi/i/320x50_7/7bfffd967a91e0e38ee06ffcee1a75e5.jpg?108236_283989_c46e3f74","cli_url":"http://c.vserv.mobi/delivery/ck.php?p=2__b=283989__zoneid=108236__OXLCA=1__cb=c46e3f74__dc=1800__cd=usw3_uswest2a-1416567600__c=37742__rs=0a587520_15397682__mser=cdn__dat=3__dacp=12__zt=s__r=http%3A%2F%2Fyeahmobi.go2cloud.org%2Faff_c%3Foffer_id%3D28007%26aff_id%3D10070%26aff_sub%3D108236_283989_c46e3f74","beacons":["http://img.vserv.mobi/b.gif"],"ad_type":"image"},"resp_code":200,"resp_html":"<a href=\"http://c.vserv.mobi/delivery/ck.php?p=2__b=283989__zoneid=108236__OXLCA=1__cb=c46e3f74__dc=1800__cd=usw3_uswest2a-1416567600__c=37742__rs=0a587520_15397682__mser=cdn__dat=3__dacp=12__zt=s__r=http%3A%2F%2Fyeahmobi.go2cloud.org%2Faff_c%3Foffer_id%3D28007%26aff_id%3D10070%26aff_sub%3D108236_283989_c46e3f74\"><img src=\"http://img.vserv.mobi/i/320x50_7/7bfffd967a91e0e38ee06ffcee1a75e5.jpg?108236_283989_c46e3f74\" alt=\"\" /> <\/a><img src=\"http://img.vserv.mobi/b.gif\" alt=\"\" />","tid":"vserv-specialbuy","bid":"576111"}]}
How ever I am not able to figure out whether they are multiline or single line but I have used as following configuration
input {
file {
codec => multiline {
pattern => '^{'
negate => true
what => previous
}
path => ['/home/pp38/fetcher.log']
}
}
filter {
json {
source => message
remove_field => message
}
}
output { stdout { codec => rubydebug } }
I am not able to see any kind of output or error when it is started
edited:
I have used the following config which had generated output.
input {
file {
codec => "json"
type => "json"
path => "/home/pp38/fetcher.log"
sincedb_path => "/home/pp38/tmp/logstash/sincedb"
}
}
filter {
json {
source => "message"
target => "message"
}
}
output {
stdout { codec => rubydebug }
elasticsearch {
hosts => ["localhost:9200"]
}
}
But i am getting the output where each field is indexed by elasticsearch
how can i append the entire json message to new field as message:jsonContent ?
You can handle this with the plain multiline, but for you situation there is a better codec plugin called json_lines.
The json_lines will input a source with multiple jsons(one in each line) and handle each json out of the box.
This codec will decode streamed JSON that is newline delimited. Encoding will emit a single JSON string ending in a \n NOTE: Do not use this codec if your source input is line-oriented JSON, for example, redis or file inputs. Rather, use the json codec. More info: This codec is expecting to receive a stream (string) of newline terminated lines. The file input will produce a line string without a newline. Therefore this codec cannot work with line oriented inputs.
I am failing to understand how to print the metric.
With following logstash config
input {
generator {
type => "generated"
}
}
filter {
metrics {
type => "generated"
meter => "events"
add_tag => "metric"
}
}
output {
stdout {
tags => "metric"
message => "rate: %{events.rate_1m}"
}
}
all I see is
rate: %{events.rate_1m}
rate: %{events.rate_1m}
instead of actual value.
When I enable debug in stdout I see that #fileds have
the data that metric is support to print.
"#fields" => {
"events.count" => 114175,
"events.rate_1m" => 6478.26368594885,
"events.rate_5m" => 5803.767865770155,
"events.rate_15m" => 5686.915084346328
},
How do I access #fields.events.count?
logstash version = 1.1.13
It looks like a known issue in logstash 1.1.13 and lower.
One need to escape '.' in %{events.rate_1m} as %{events\.rate_1m}
Details are in this logstash JIRA