Using Logstash 1.4.2, I have a field myfield that is a boolean value in my JSON document.
To check if it exists (don't care about the boolean value) I used:
if[myfield] { ...exists... } else { ...doesn't exist... }
The results from testing this conditional statement are:
[myfield] does not exist --> false
[myfield] exists, is true --> true
[myfield] exists, is false --> false //expected true because the field exists
It is checking the boolean value, not its existence.
How can I check that a boolean field exists?
Kind of lame that it doesn't work right out of the box, but you can hack it like this -- add a string representation of the boolean, compare against the string, and then remove the added field:
filter {
mutate {
add_field => { "test" => "%{boolean}" }
}
if [test] == 'true' or [test] == 'false' {
// field is present and set right
} else {
// field isn't present or set to something other than true/false
}
mutate {
remove_field => [ "test" ]
}
}
you can use logstash ruby plugin.
filter {
ruby {
code => '
myfield= event.get("[myfield]")
if !myfield.nil?
event.set("[myfield]", "some_value_1")
else
event.set("[myfield]", "some_value_2")
end
'
}
}
Related
When I know name of one of fields in expected object, I can use
if [check_if_this_is_an_object][known_field_name] { # object } else { # not an object }
But I would like a way to do a general check, without known field name.
You will need to use a ruby filter is you do not know the name of any of the fields within the object. If you just need to check one field you could use
ruby {
code => '
field = event.get("foo")
if field.respond_to? "each"
answer = true
elsif field.is_a? LogStash::Timestamp
answer = true
else
answer = false
end
event.set("[#metadata][fooObject]", answer)
'
}
If you need to check multiple fields then use a script file. The script would be
def register(params)
#fieldName = params["field"]
end
def filter(event)
field = event.get(#fieldName)
if field.respond_to? "each" # Array or Hash
answer = true
elsif field.is_a? LogStash::Timestamp
answer = true
else # boolean, integer, float or string
answer = false
end
event.set("[#metadata][#{#fieldName}Object]", answer)
[event]
end
and then call it using
ruby {
path => "/home/user/isObject.rb"
script_params => { "field" => "foo" }
}
ruby {
path => "/home/user/isObject.rb"
script_params => { "field" => "bar" }
}
and then you can write a conditional using [#metadata][fooObject] or [#metadata][barObject]
I have a json event coming into logstash that looks like (values are different in each event):
{
"metadata":{
"app_type":"Foo",
"app_namespace":"System.Bar"
"json_string":"{\"test\":true}"
}
}
I would like my logstash filter to output the following to elasticsearch (i'm not worried about trimming the existing fields):
{
"app_event":{
"foo":{
"system_bar":{
"test":true
}
}
}
}
I cannot figure out the syntax for how to use field values as the names of new fields.
I tried a few different ways, all resulted in a config error when starting my logstash instance. I'm only including the filter block of logstash.conf in my examples, the rest of the config is working as expected.
1.
# note this one does not have replacement and lowercase values I want
filter {
json {
source => "[metadata][json_string]"
target => "[app_event][%{[metadata][app_type]}][%{[metadata][app_namespace]}]"
}
}
I tried to create variables with values I need and then use those in the target
filter {
appType = "%{[metadata][instrument_type]}".downcase
appNamespace = "%{[metadata][app_namespace]}".downcase
appNamespace.gsub(".", "_")
json {
source => "[metadata][json_string]"
target => "[app_event][#{appType}][#{appNamespace}]"
}
}
Just to confirm that everything else was set up correctly this does work, but does not include the dynamically generated field structure I'm looking for.
filter {
json {
source => "[metadata][json_string]"
target => "[app_event]"
}
}
The json filter does not sprintf the value of target, so you cannot use a json filter in those ways. To make the field name variable you must use a ruby filter. Try
json { source => "message" target => "[#metadata][json1]" remove_field => [ "message" ] }
json { source => "[#metadata][json1][metadata][json_string]" target => "[#metadata][json2]" }
ruby {
code => '
namespace = event.get("[#metadata][json1][metadata][app_namespace]")
if namespace
namespace = namespace.downcase.sub("\.", "_")
end
type = event.get("[#metadata][json1][metadata][app_type]")
if type
type = type.downcase
end
value = event.get("[#metadata][json2]")
if type and namespace and value
event.set("app_event", { "#{type}" => { "#{namespace}" => value } } )
end
'
}
I want to put a value into part of a nested hash, but name that part depending on upstream filters. This is to refactor and reduce overall code size as currently each of the 20+ incoming event types have their own section like this with 18 lines in the logstash file (but currently the %{detail_part} bit is hard-coded).
# Working code
filter {
if [result] == 0 {
# Success
mutate {
add_field => {
"[Thing][ThingDetail][OtherThing][MoreDetail]" => "true"
}
}
}
else {
# Failed
mutate {
add_field => {
"[Thing][ThingDetail][OtherThing][MoreDetail]" => "false"
}
}
}
}
Above is hard-coded to "OtherThing". Below has a variable, but doesn't work.
# Non-Working code
filter {
if [result] == 0 {
# Success
mutate {
add_field => {
"[Thing][ThingDetail][%{detail_part}][MoreDetail]" => "true"
}
}
}
else {
# Failed
mutate {
add_field => {
"[Thing][ThingDetail][%{detail_part}][MoreDetail]" => "false"
}
}
}
}
In the above (non-Working code), detail_part is set in an upstream filter to a string value like "OtherThing". This currently compiles and runs, but no XML is output from it, so I don't think anything is set into the hash as a result of these statements.
I know it can be done with embedded Ruby code, but I would like a way that is as simple as possible. The output of this process is going to XML so I am constrained to use this kind of nested hash.
Is this possible with Logstash?
It turns out that yes, Logstash supports this, just the syntax was wrong. So here is the fix:
filter {
# This field could be set conditionally in an if statement ...
mutate { add_field => { "[detail_part]" => "Result" } }
if [result] == 0 {
# Success
mutate {
add_field => {
"[Thing][ThingDetail][%{[detail_part]}][MoreDetail]" => "true"
}
}
}
else {
# Failed
mutate {
add_field => {
"[Thing][ThingDetail][%{[detail_part]}][MoreDetail]" => "false"
}
}
}
}
I just couldn't find any non-trivial examples that did things like this.
In Logstash I have a JSON payload which I've decoded like so:
filter
{
json
{
source => "payload"
target => "payloadData"
}
}
Now I want to check if payloadData._extraData exists, and then deserialise that too.
I've tried this method, working from the example given in this related question:
filter
{
json
{
source => "payload"
target => "payloadData"
}
if [payloadData._extraData] =~ /.+/
{
sourcce => "payloadData._extraData"
target => "payloadData.extraData"
}
}
But it doesn't do anything (no crash, no error message, just doesn't do anything)
The correct syntax is:
if [payloadData][_extraData] =~ /.+/ { }
Example input:
{"foo":"bar","spam":"eggs","abc":"xyz","one":"two","three":"four","five":"six","seven":{"eight":"nine"}}
Config:
filter {
json { source => message }
if [seven][eight] =~ /.+/ {
# do something
}
}
Apart from that, the code inside your if statement doesn't do anything. You need to specify a filter that should be executed. e.g.:
if [payloadData][_extraData] =~ /.+/ {
json {
source => "payloadData[_extraData]"
target => "payloadData[extraData]"
}
}
What do you want to deserialize in your if statement? The first json filter should recognize nested objects.
I have log files coming in to an ELK stack. I want to copy a field (foo) in order to perform various mutations on it, However the field (foo) isn't always present.
If foo doesn't exist, then bar still gets created, but is assigned the literal string "%{foo}"
How can I perform a mutation only if a field exists?
I'm trying to do something like this.
if ["foo"] {
mutate {
add_field => "bar" => "%{foo}
}
}
To check if field foo exists:
1) For numeric type fields use:
if ([foo]) {
...
}
2) For types other than numeric like boolean, string use:
if ("" in [foo]) {
...
}
"foo" is a literal string.
[foo] is a field.
# technically anything that returns 'true', so good for numbers and basic strings:
if [foo] {
}
# contains a value
if [foo] =~ /.+/ {
}
On Logstash 2.2.2, the ("" in [field]) construct does not appear to work for me.
if ![field] { }
does, for a non-numerical field.
It's 2020 and none of the above answers are quite correct. I've been working with logstash since 2014 and expressions in filter were, are and will be a thing...
For example, you may have a boolean field with false value and with the above solutions you may not know if false is the value of the field or the resulting value of the expression because the field doesn't exists.
Workaround for checking if a field exists in all versions
I think all versions of logstash supports [#metadata] field. That is, a field that will not be visible for output plugins and lives only in the filtering state. So this is what I have to workaround:
filter {
mutate {
# we use a "temporal" field with a predefined arbitrary known value that
# lives only in filtering stage.
add_field => { "[#metadata][testField_check]" => "unknown arbitrary value" }
# we copy the field of interest into that temporal field.
# If the field doesn't exist, copy is not executed.
copy => { "testField" => "[#metadata][testField_check]" }
}
# now we now if testField didn't exists, our field will have
# the initial arbitrary value
if [#metadata][testField_check] == "unknown arbitrary value" {
# just for debugging purpouses...
mutate { add_field => { "FIELD_DID_NOT_EXISTED" => true }}
} else {
# just for debugging purpouses...
mutate { add_field => { "FIELD_DID_ALREADY_EXISTED" => true }}
}
}
Old solution for logstash prior version 7.0.0
Check my issue in github.
I've been struggling a lot with expressions in logstash. My old solution worked until version 7. This was for boolean fields, for instance:
filter {
# if the field does not exists, `convert` will create it with "false" string. If
# the field exists, it will be the boolean value converted into string.
mutate { convert => { "field" => "string" } }
# This condition breaks on logstash > 7 (see my bug report). Before version 7,
# this condition will be true if a boolean field didn't exists.
if ![field] {
mutate { add_field => { "field" => false } }
}
# at this stage, we are sure field exists, so make it boolean again
mutate { convert => { "field" => "boolean" } }
}