Currently I am doing something like this in my logstash config file :
filter {
...
mutate {
...
convert => {
"blahId" => "integer"
"blahblahId" => "integer"
...
...
"b...blahId" => "integer"
}
...
}
...
}
So basically I want to convert all the fields ending with "Id" to type integer. Is there a way to do that in one line? Something like "*Id" => "integer" which does that ?
Edit : I tried
convert => {
"*Id" => "integer"
}
As I expected, didn't work.
Using ruby filter perhaps ?
Not specifically an answer to this but this is what I end up doing :
ruby {
code => "
fieldArray = event['kvmess'].split('|');
for field in fieldArray
name = field.split('=')[0];
value = field.split('=')[1];
if value =~ /\A\d+\Z/
event[name] = value.to_i
else
event[name] = value
end
end
"
}
My kvmess was like "blahId=123|blahblahId=456|some=thing|b..blahId=789".
So this converted all keys having numeric values to type integer.
There is a plugin - kv meant specifically for this but it does not have the functionality to change datatype to int so I ended up using this ruby plugin instead.
Related
I have a json event coming into logstash that looks like (values are different in each event):
{
"metadata":{
"app_type":"Foo",
"app_namespace":"System.Bar"
"json_string":"{\"test\":true}"
}
}
I would like my logstash filter to output the following to elasticsearch (i'm not worried about trimming the existing fields):
{
"app_event":{
"foo":{
"system_bar":{
"test":true
}
}
}
}
I cannot figure out the syntax for how to use field values as the names of new fields.
I tried a few different ways, all resulted in a config error when starting my logstash instance. I'm only including the filter block of logstash.conf in my examples, the rest of the config is working as expected.
1.
# note this one does not have replacement and lowercase values I want
filter {
json {
source => "[metadata][json_string]"
target => "[app_event][%{[metadata][app_type]}][%{[metadata][app_namespace]}]"
}
}
I tried to create variables with values I need and then use those in the target
filter {
appType = "%{[metadata][instrument_type]}".downcase
appNamespace = "%{[metadata][app_namespace]}".downcase
appNamespace.gsub(".", "_")
json {
source => "[metadata][json_string]"
target => "[app_event][#{appType}][#{appNamespace}]"
}
}
Just to confirm that everything else was set up correctly this does work, but does not include the dynamically generated field structure I'm looking for.
filter {
json {
source => "[metadata][json_string]"
target => "[app_event]"
}
}
The json filter does not sprintf the value of target, so you cannot use a json filter in those ways. To make the field name variable you must use a ruby filter. Try
json { source => "message" target => "[#metadata][json1]" remove_field => [ "message" ] }
json { source => "[#metadata][json1][metadata][json_string]" target => "[#metadata][json2]" }
ruby {
code => '
namespace = event.get("[#metadata][json1][metadata][app_namespace]")
if namespace
namespace = namespace.downcase.sub("\.", "_")
end
type = event.get("[#metadata][json1][metadata][app_type]")
if type
type = type.downcase
end
value = event.get("[#metadata][json2]")
if type and namespace and value
event.set("app_event", { "#{type}" => { "#{namespace}" => value } } )
end
'
}
I'm trying to add "type" field that i would be able to use as tag when filtering to my input plugin.
For example, I can add "type" in the 'file' input plugin, and filter it later.
Code:
input {
file {
path => "/var/log/%{type}.%{+yyyy.MM.dd.HH}"
type => "myType"
}
}
filter {
if [type] == "myType"
...
}
output {
...
}
I looked at the file input plugin implementation, and did not see any field related to the "type" field.
So i just tried to add it like this:
input {
myPlugin {
path => "/var/log/%{type}.%{+yyyy.MM.dd.HH}"
type => "myType"
}
}
But the filtering did not work at all.
How can i add this "tag" (or whatever it is) to my plugin?
The functionalitiy of type is to use it for filtering:
Types are used mainly for filter activation.
So you can't actually see a new field being created unless you have created a new field under your filter, something like this:
input {
file {
path => "/var/log/%{type}.%{+yyyy.MM.dd.HH}"
type => "myType"
}
}
filter {
if [type] == "myType"{
add_field => {
"somefield" => "this is a sample" <--- this will actually create the field
}
add_tag => "myType" <--- this would actually create a tag for your es record if it matches the filter
}
}
You could maybe check out this, in order to add a new field and adding tags as well. Make sure your plugin supports type. Hope this helps!
I would like to convert all metrics* fields into floats for logstash. For a structure like
{
"metric1":"1",
"metric2":"2"
}
I'd like to do something like
mutate {
convert => {"metric*" => "float" }
}
Is that possible?
It's not possible without using a ruby filter like this:
ruby {
code => "
event.to_hash.keys.each { |k|
if k.start_with?('metric') and event[k].is_a?(String)
event[k] = event[k].to_float
end
}
"
}
So basically look at all of the keys in the event, and if they start with metric, covert them to a float. The is_a?(String) is there just in case you get an array field (because .to_float won't work on it)
I have log files coming in to an ELK stack. I want to copy a field (foo) in order to perform various mutations on it, However the field (foo) isn't always present.
If foo doesn't exist, then bar still gets created, but is assigned the literal string "%{foo}"
How can I perform a mutation only if a field exists?
I'm trying to do something like this.
if ["foo"] {
mutate {
add_field => "bar" => "%{foo}
}
}
To check if field foo exists:
1) For numeric type fields use:
if ([foo]) {
...
}
2) For types other than numeric like boolean, string use:
if ("" in [foo]) {
...
}
"foo" is a literal string.
[foo] is a field.
# technically anything that returns 'true', so good for numbers and basic strings:
if [foo] {
}
# contains a value
if [foo] =~ /.+/ {
}
On Logstash 2.2.2, the ("" in [field]) construct does not appear to work for me.
if ![field] { }
does, for a non-numerical field.
It's 2020 and none of the above answers are quite correct. I've been working with logstash since 2014 and expressions in filter were, are and will be a thing...
For example, you may have a boolean field with false value and with the above solutions you may not know if false is the value of the field or the resulting value of the expression because the field doesn't exists.
Workaround for checking if a field exists in all versions
I think all versions of logstash supports [#metadata] field. That is, a field that will not be visible for output plugins and lives only in the filtering state. So this is what I have to workaround:
filter {
mutate {
# we use a "temporal" field with a predefined arbitrary known value that
# lives only in filtering stage.
add_field => { "[#metadata][testField_check]" => "unknown arbitrary value" }
# we copy the field of interest into that temporal field.
# If the field doesn't exist, copy is not executed.
copy => { "testField" => "[#metadata][testField_check]" }
}
# now we now if testField didn't exists, our field will have
# the initial arbitrary value
if [#metadata][testField_check] == "unknown arbitrary value" {
# just for debugging purpouses...
mutate { add_field => { "FIELD_DID_NOT_EXISTED" => true }}
} else {
# just for debugging purpouses...
mutate { add_field => { "FIELD_DID_ALREADY_EXISTED" => true }}
}
}
Old solution for logstash prior version 7.0.0
Check my issue in github.
I've been struggling a lot with expressions in logstash. My old solution worked until version 7. This was for boolean fields, for instance:
filter {
# if the field does not exists, `convert` will create it with "false" string. If
# the field exists, it will be the boolean value converted into string.
mutate { convert => { "field" => "string" } }
# This condition breaks on logstash > 7 (see my bug report). Before version 7,
# this condition will be true if a boolean field didn't exists.
if ![field] {
mutate { add_field => { "field" => false } }
}
# at this stage, we are sure field exists, so make it boolean again
mutate { convert => { "field" => "boolean" } }
}
After changing my mapping in ElasticSearch to more definitively type the data I am inputting into the system, I have unwittingly made my new variables a nested object. Upon thinking about it more, I actually like the idea of those fields being nested objects because that way I can explicitly know if that src_port statistic is from netflow or from the ASA logs, as an example.
I'd like to use a mutate (gsub, perhaps?) to cause all of my fieldnames for a given type to be renamed to newtype.fieldname. I see that there is gsub which uses a regexp, and rename which takes the literal field name, but I would like to prevent having 30 distinct gsub/rename statements when I will be replacing all of the fields in that type with the "newtype" prefix.
Is there a way to do this?
Here is an example for your reference.
input {
stdin{
type => 'netflow'
}
}
filter {
mutate {
add_field => {"%{type}.message" => "%{message}"}
remove_field => ["message"]
}
}
output {
stdout{
codec => rubydebug
}
}
In this example I have change the message field name to type.message, then delete the origin message field. I think you can use this sample to do what you want.
Hope this can help you.
I have updated my answer!
Use the ruby plugin to do what you want!
Please notice that elasticsearch uses #timestamp field to do index, so I recommend do not change the field name.
input {
stdin{
type => 'netflow'
}
}
filter {
ruby {
code => "
data = event.clone.to_hash;
type = event['type']
data.each do |k,v|
if k != '#timestamp'
newFieldName = type +'.'+ k
event[newFieldName] = v
event.remove(k)
end
end
"
}
}
output {
stdout{
codec => rubydebug
}
}