When I know name of one of fields in expected object, I can use
if [check_if_this_is_an_object][known_field_name] { # object } else { # not an object }
But I would like a way to do a general check, without known field name.
You will need to use a ruby filter is you do not know the name of any of the fields within the object. If you just need to check one field you could use
ruby {
code => '
field = event.get("foo")
if field.respond_to? "each"
answer = true
elsif field.is_a? LogStash::Timestamp
answer = true
else
answer = false
end
event.set("[#metadata][fooObject]", answer)
'
}
If you need to check multiple fields then use a script file. The script would be
def register(params)
#fieldName = params["field"]
end
def filter(event)
field = event.get(#fieldName)
if field.respond_to? "each" # Array or Hash
answer = true
elsif field.is_a? LogStash::Timestamp
answer = true
else # boolean, integer, float or string
answer = false
end
event.set("[#metadata][#{#fieldName}Object]", answer)
[event]
end
and then call it using
ruby {
path => "/home/user/isObject.rb"
script_params => { "field" => "foo" }
}
ruby {
path => "/home/user/isObject.rb"
script_params => { "field" => "bar" }
}
and then you can write a conditional using [#metadata][fooObject] or [#metadata][barObject]
Related
I have a json event coming into logstash that looks like (values are different in each event):
{
"metadata":{
"app_type":"Foo",
"app_namespace":"System.Bar"
"json_string":"{\"test\":true}"
}
}
I would like my logstash filter to output the following to elasticsearch (i'm not worried about trimming the existing fields):
{
"app_event":{
"foo":{
"system_bar":{
"test":true
}
}
}
}
I cannot figure out the syntax for how to use field values as the names of new fields.
I tried a few different ways, all resulted in a config error when starting my logstash instance. I'm only including the filter block of logstash.conf in my examples, the rest of the config is working as expected.
1.
# note this one does not have replacement and lowercase values I want
filter {
json {
source => "[metadata][json_string]"
target => "[app_event][%{[metadata][app_type]}][%{[metadata][app_namespace]}]"
}
}
I tried to create variables with values I need and then use those in the target
filter {
appType = "%{[metadata][instrument_type]}".downcase
appNamespace = "%{[metadata][app_namespace]}".downcase
appNamespace.gsub(".", "_")
json {
source => "[metadata][json_string]"
target => "[app_event][#{appType}][#{appNamespace}]"
}
}
Just to confirm that everything else was set up correctly this does work, but does not include the dynamically generated field structure I'm looking for.
filter {
json {
source => "[metadata][json_string]"
target => "[app_event]"
}
}
The json filter does not sprintf the value of target, so you cannot use a json filter in those ways. To make the field name variable you must use a ruby filter. Try
json { source => "message" target => "[#metadata][json1]" remove_field => [ "message" ] }
json { source => "[#metadata][json1][metadata][json_string]" target => "[#metadata][json2]" }
ruby {
code => '
namespace = event.get("[#metadata][json1][metadata][app_namespace]")
if namespace
namespace = namespace.downcase.sub("\.", "_")
end
type = event.get("[#metadata][json1][metadata][app_type]")
if type
type = type.downcase
end
value = event.get("[#metadata][json2]")
if type and namespace and value
event.set("app_event", { "#{type}" => { "#{namespace}" => value } } )
end
'
}
I am using node-config in basically all my projects and most of the time I come across the problem of parsing booleans and numbers which are set as environment variables.
E.g.
default.js
module.exports = {
myNumber = 10,
myBool = true
}
custom-environment-variables.js
module.exports = {
myNumber = "MY_NUMBER",
myBool = "MY_BOOL"
}
Now, the obvious problem is that if I override the default values with custom values set as environment variables they will be a string value instead of a number or boolean value. So now, to make sure in my code that the types are correct. I always have to do type conversion and for booleans use a proper library e.g. yn. The problem is I have to do this conversion every time I use config.get() for example +config.get("myNumber") or yn(config.get("myBool")).
Is there a better and more elegant way to do this?
One solution I see would be to add a type property to an environment variable as it is done here with format. This would allow to do something like this...
custom-environment-variables.js
module.exports = {
myNumber = {
name: "MY_NUMBER",
type: "number"
},
myBool = {
name: "MY_BOOL",
type: "boolean"
}
}
node-config would handle the type conversions and there would be no need to do it all the time in the code when getting it. Of course there would be the requirement to implement a proper parser for booleans but those already exist and could be used here.
By default, environment variables will be parsed as string.
In node-config, we could override this behaviour with __format as shown below.
We don't need any additional libraries. Normal json datatypes like boolean, number, nested json etc., should work well.
Taking an easy to relate example.
config/default.json
{
"service": {
"autostart": false
}
}
custom-environment-variables.json
{
"service": {
"autostart": {
"__name": "AUTOSTART",
"__format": "json"
}
}
}
Now we can pass environment variables when we like to override and no type conversation should be needed for basic types.
This feature is now supported in node-config v3.3.2, see changelog
I use this method:
const toBoolean = (dataStr) => {
return !!(dataStr?.toLowerCase?.() === 'true' || dataStr === true);
};
You can add cases if you want 0 to resolve to true as well:
const toBoolean = (dataStr) => {
return !!(dataStr?.toLowerCase?.() === 'true' || dataStr === true || Number.parseInt(dataStr, 10) === 0);
};
Currently I am doing something like this in my logstash config file :
filter {
...
mutate {
...
convert => {
"blahId" => "integer"
"blahblahId" => "integer"
...
...
"b...blahId" => "integer"
}
...
}
...
}
So basically I want to convert all the fields ending with "Id" to type integer. Is there a way to do that in one line? Something like "*Id" => "integer" which does that ?
Edit : I tried
convert => {
"*Id" => "integer"
}
As I expected, didn't work.
Using ruby filter perhaps ?
Not specifically an answer to this but this is what I end up doing :
ruby {
code => "
fieldArray = event['kvmess'].split('|');
for field in fieldArray
name = field.split('=')[0];
value = field.split('=')[1];
if value =~ /\A\d+\Z/
event[name] = value.to_i
else
event[name] = value
end
end
"
}
My kvmess was like "blahId=123|blahblahId=456|some=thing|b..blahId=789".
So this converted all keys having numeric values to type integer.
There is a plugin - kv meant specifically for this but it does not have the functionality to change datatype to int so I ended up using this ruby plugin instead.
I have log files coming in to an ELK stack. I want to copy a field (foo) in order to perform various mutations on it, However the field (foo) isn't always present.
If foo doesn't exist, then bar still gets created, but is assigned the literal string "%{foo}"
How can I perform a mutation only if a field exists?
I'm trying to do something like this.
if ["foo"] {
mutate {
add_field => "bar" => "%{foo}
}
}
To check if field foo exists:
1) For numeric type fields use:
if ([foo]) {
...
}
2) For types other than numeric like boolean, string use:
if ("" in [foo]) {
...
}
"foo" is a literal string.
[foo] is a field.
# technically anything that returns 'true', so good for numbers and basic strings:
if [foo] {
}
# contains a value
if [foo] =~ /.+/ {
}
On Logstash 2.2.2, the ("" in [field]) construct does not appear to work for me.
if ![field] { }
does, for a non-numerical field.
It's 2020 and none of the above answers are quite correct. I've been working with logstash since 2014 and expressions in filter were, are and will be a thing...
For example, you may have a boolean field with false value and with the above solutions you may not know if false is the value of the field or the resulting value of the expression because the field doesn't exists.
Workaround for checking if a field exists in all versions
I think all versions of logstash supports [#metadata] field. That is, a field that will not be visible for output plugins and lives only in the filtering state. So this is what I have to workaround:
filter {
mutate {
# we use a "temporal" field with a predefined arbitrary known value that
# lives only in filtering stage.
add_field => { "[#metadata][testField_check]" => "unknown arbitrary value" }
# we copy the field of interest into that temporal field.
# If the field doesn't exist, copy is not executed.
copy => { "testField" => "[#metadata][testField_check]" }
}
# now we now if testField didn't exists, our field will have
# the initial arbitrary value
if [#metadata][testField_check] == "unknown arbitrary value" {
# just for debugging purpouses...
mutate { add_field => { "FIELD_DID_NOT_EXISTED" => true }}
} else {
# just for debugging purpouses...
mutate { add_field => { "FIELD_DID_ALREADY_EXISTED" => true }}
}
}
Old solution for logstash prior version 7.0.0
Check my issue in github.
I've been struggling a lot with expressions in logstash. My old solution worked until version 7. This was for boolean fields, for instance:
filter {
# if the field does not exists, `convert` will create it with "false" string. If
# the field exists, it will be the boolean value converted into string.
mutate { convert => { "field" => "string" } }
# This condition breaks on logstash > 7 (see my bug report). Before version 7,
# this condition will be true if a boolean field didn't exists.
if ![field] {
mutate { add_field => { "field" => false } }
}
# at this stage, we are sure field exists, so make it boolean again
mutate { convert => { "field" => "boolean" } }
}
Using Logstash 1.4.2, I have a field myfield that is a boolean value in my JSON document.
To check if it exists (don't care about the boolean value) I used:
if[myfield] { ...exists... } else { ...doesn't exist... }
The results from testing this conditional statement are:
[myfield] does not exist --> false
[myfield] exists, is true --> true
[myfield] exists, is false --> false //expected true because the field exists
It is checking the boolean value, not its existence.
How can I check that a boolean field exists?
Kind of lame that it doesn't work right out of the box, but you can hack it like this -- add a string representation of the boolean, compare against the string, and then remove the added field:
filter {
mutate {
add_field => { "test" => "%{boolean}" }
}
if [test] == 'true' or [test] == 'false' {
// field is present and set right
} else {
// field isn't present or set to something other than true/false
}
mutate {
remove_field => [ "test" ]
}
}
you can use logstash ruby plugin.
filter {
ruby {
code => '
myfield= event.get("[myfield]")
if !myfield.nil?
event.set("[myfield]", "some_value_1")
else
event.set("[myfield]", "some_value_2")
end
'
}
}