In Logstash I have a JSON payload which I've decoded like so:
filter
{
json
{
source => "payload"
target => "payloadData"
}
}
Now I want to check if payloadData._extraData exists, and then deserialise that too.
I've tried this method, working from the example given in this related question:
filter
{
json
{
source => "payload"
target => "payloadData"
}
if [payloadData._extraData] =~ /.+/
{
sourcce => "payloadData._extraData"
target => "payloadData.extraData"
}
}
But it doesn't do anything (no crash, no error message, just doesn't do anything)
The correct syntax is:
if [payloadData][_extraData] =~ /.+/ { }
Example input:
{"foo":"bar","spam":"eggs","abc":"xyz","one":"two","three":"four","five":"six","seven":{"eight":"nine"}}
Config:
filter {
json { source => message }
if [seven][eight] =~ /.+/ {
# do something
}
}
Apart from that, the code inside your if statement doesn't do anything. You need to specify a filter that should be executed. e.g.:
if [payloadData][_extraData] =~ /.+/ {
json {
source => "payloadData[_extraData]"
target => "payloadData[extraData]"
}
}
What do you want to deserialize in your if statement? The first json filter should recognize nested objects.
Related
I have a json event coming into logstash that looks like (values are different in each event):
{
"metadata":{
"app_type":"Foo",
"app_namespace":"System.Bar"
"json_string":"{\"test\":true}"
}
}
I would like my logstash filter to output the following to elasticsearch (i'm not worried about trimming the existing fields):
{
"app_event":{
"foo":{
"system_bar":{
"test":true
}
}
}
}
I cannot figure out the syntax for how to use field values as the names of new fields.
I tried a few different ways, all resulted in a config error when starting my logstash instance. I'm only including the filter block of logstash.conf in my examples, the rest of the config is working as expected.
1.
# note this one does not have replacement and lowercase values I want
filter {
json {
source => "[metadata][json_string]"
target => "[app_event][%{[metadata][app_type]}][%{[metadata][app_namespace]}]"
}
}
I tried to create variables with values I need and then use those in the target
filter {
appType = "%{[metadata][instrument_type]}".downcase
appNamespace = "%{[metadata][app_namespace]}".downcase
appNamespace.gsub(".", "_")
json {
source => "[metadata][json_string]"
target => "[app_event][#{appType}][#{appNamespace}]"
}
}
Just to confirm that everything else was set up correctly this does work, but does not include the dynamically generated field structure I'm looking for.
filter {
json {
source => "[metadata][json_string]"
target => "[app_event]"
}
}
The json filter does not sprintf the value of target, so you cannot use a json filter in those ways. To make the field name variable you must use a ruby filter. Try
json { source => "message" target => "[#metadata][json1]" remove_field => [ "message" ] }
json { source => "[#metadata][json1][metadata][json_string]" target => "[#metadata][json2]" }
ruby {
code => '
namespace = event.get("[#metadata][json1][metadata][app_namespace]")
if namespace
namespace = namespace.downcase.sub("\.", "_")
end
type = event.get("[#metadata][json1][metadata][app_type]")
if type
type = type.downcase
end
value = event.get("[#metadata][json2]")
if type and namespace and value
event.set("app_event", { "#{type}" => { "#{namespace}" => value } } )
end
'
}
I got a filename in the format <key>:<value>-<key>:<value>.log like e.g. pr:64-author:mxinden-platform:aws.log containing logs of a test run.
I want to stream each line of the file to elasticsearch via logstash. Each line should be treated as a separate document. Each document should get the fields according to the filename. So e.g. for the above example let's say log-line 17-12-07 foo something happened bar would get the fields: pr with value 64, author with value mxinden and platform with value aws.
At the point in time, where I write the logstash configuration I do not know the names of the fields.
How do I dynamically add fields to each line based on the fields contained in the filename?
The static approach so far is:
filter {
mutate { add_field => { "file" => "%{[#metadata][s3][key]}"} }
else {
grok { match => { "file" => "pr:%{NUMBER:pr}-" } }
grok { match => { "file" => "author:%{USERNAME:author}-" } }
grok { match => { "file" => "platform:%{USERNAME:platform}-" } }
}
}
Changes to the filename structure are fine.
Answering my own question based on #dan-griffiths comment:
Solution for a file like pr=64,author=mxinden,platform=aws.log is to use the Elasticsearch kv filter like e.g.:
filter {
kv {
source => "file"
field_split => ","
}
}
where file is a field extracted from the filename via the AWS S3 input plugin.
I have a probleme to create a list of json object. this is my config:
input {
file {#settings...}
}
filter {
mutate {
add_field => {"book_title" => "%{Book_title}"}
add_field => {"book_price" => "%{Book_price}"}
}
mutate {
rename => { "book_title" => "[book_market][book_title]"
"book_price" => "[book_market][book_price]" }
}
}
I want book_market be list for some reason even it has one object
ruby {
code => "
event['book_market'] = [event['book_market']]
"
}
}
the result here after execution:
"book_market" : [ {
"book_title":"title",
"book_price":"price"
}]
it's that I want !
But in a second time a new book arrives from another file... the probleme here with the same config I lost the first json object because book_market is overwritten.. I want to insert the new object in book_market list.
Thank you for help !
I'm building out logstash and would like to build functionality to anonymize fields as specified in the message.
Given the message below, the field fta is a list of fields to anonymize. I would like to just use %{fta} and pass it through to the anonymize filter, but that doesn't seem to work.
{ "containsPII":"True", "fta":["f1","f2"], "f1":"test", "f2":"5551212" }
My config is as follows
input {
stdin { codec => json }
}
filter {
if [containsPII] {
anonymize {
algorithm => "SHA1"
key => "123456789"
fields => %{fta}
}
}
}
output {
stdout {
codec => rubydebug
}
}
The output is
{
"containsPII" => "True",
"fta" => [
[0] "f1",
[1] "f2"
],
"f1" => "test",
"f2" => "5551212",
"#version" => "1",
"#timestamp" => "2016-07-13T22:07:04.036Z",
"host" => "..."
}
Does anyone have any thoughts? I have tried several permutations at this point with no luck.
Thanks,
-D
EDIT:
After posting in the Elastic forums, I found out that this is not possible using base logstash functionality. I will try using the ruby filter instead. So, to ammend my question, How do I call another filter from within the ruby filter? I tried the following with no luck and honestly can't even figure out where to look. I'm very new to ruby.
filter {
if [containsPII] {
ruby {
code => "event['fta'].each { |item| event[item] = LogStash::Filters::Anonymize.execute(event[item],'12345','SHA1') }"
add_tag => ["Rubyrun"]
}
}
}
You can execute the filters from ruby script. Steps will be:
Create the required filter instance in the init block of inline ruby script.
For every event call the filter method of the filter instance.
Following is the example for above problem statement. It will replace my_ip field in event with its SHA1.
Same can be achieved using ruby script file.
Following is the sample config file.
input { stdin { codec => json_lines } }
filter {
ruby {
init => "
require 'logstash/filters/anonymize'
# Create instance of filter with applicable parameters
#anonymize = LogStash::Filters::Anonymize.new({'algorithm' => 'SHA1',
'key' => '123456789',
'fields' => ['my_ip']})
# Make sure to call register
#anonymize.register
"
code => "
# Invoke the filter
#anonymize.filter(event)
"
}
}
output { stdout { codec => rubydebug {metadata => true} } }
Well, I wasn't able to figure out how to call another filter from within a ruby filter, but I did get to the functional goal.
filter {
if [fta] {
ruby {
init => "require 'openssl'"
code => "event['fta'].each { |item| event[item] = OpenSSL::HMAC.hexdigest(OpenSSL::Digest::SHA256.new, '123456789', event[item] ) }"
}
}
}
If the field FTA exists, it will SHA2 encode each of the fields listed in that array.
I want to put a value into part of a nested hash, but name that part depending on upstream filters. This is to refactor and reduce overall code size as currently each of the 20+ incoming event types have their own section like this with 18 lines in the logstash file (but currently the %{detail_part} bit is hard-coded).
# Working code
filter {
if [result] == 0 {
# Success
mutate {
add_field => {
"[Thing][ThingDetail][OtherThing][MoreDetail]" => "true"
}
}
}
else {
# Failed
mutate {
add_field => {
"[Thing][ThingDetail][OtherThing][MoreDetail]" => "false"
}
}
}
}
Above is hard-coded to "OtherThing". Below has a variable, but doesn't work.
# Non-Working code
filter {
if [result] == 0 {
# Success
mutate {
add_field => {
"[Thing][ThingDetail][%{detail_part}][MoreDetail]" => "true"
}
}
}
else {
# Failed
mutate {
add_field => {
"[Thing][ThingDetail][%{detail_part}][MoreDetail]" => "false"
}
}
}
}
In the above (non-Working code), detail_part is set in an upstream filter to a string value like "OtherThing". This currently compiles and runs, but no XML is output from it, so I don't think anything is set into the hash as a result of these statements.
I know it can be done with embedded Ruby code, but I would like a way that is as simple as possible. The output of this process is going to XML so I am constrained to use this kind of nested hash.
Is this possible with Logstash?
It turns out that yes, Logstash supports this, just the syntax was wrong. So here is the fix:
filter {
# This field could be set conditionally in an if statement ...
mutate { add_field => { "[detail_part]" => "Result" } }
if [result] == 0 {
# Success
mutate {
add_field => {
"[Thing][ThingDetail][%{[detail_part]}][MoreDetail]" => "true"
}
}
}
else {
# Failed
mutate {
add_field => {
"[Thing][ThingDetail][%{[detail_part]}][MoreDetail]" => "false"
}
}
}
}
I just couldn't find any non-trivial examples that did things like this.