I am using Logstash and one of my applications is sending me fields like:
[message][UrlVisited]
[message][TotalDuration]
[message][AccountsProcessed]
I'd like to be able to collapse these fields, removing the top level message altogether. So the above fields will become:
[UrlVisited]
[TotalDuration]
[AccountsProcessed]
Is there a way to do this in Logstash?
Assuming the names of all such subfields are known in advance you can use the mutate filter:
filter {
mutate {
rename => ["[message][UrlVisited]", "UrlVisited"]
}
mutate {
rename => ["[message][TotalDuration]", "TotalDuration"]
}
mutate {
rename => ["[message][AccountsProcessed]", "AccountsProcessed"]
}
mutate {
remove_field => ["message"]
}
}
Alternatively, use a ruby filter (which works even if you don't know the field names):
filter {
ruby {
code => "
event.get('message').each {|k, v|
event.set(k, v)
}
event.remove('message')
"
}
}
This example works on Logstash 2.4 and later. For earlier versions use event['message'].each ... and event[k] = v instead.
Related
I have a logstash configuration. With that configuration logstash do some operation in filter and send outputs. Everything works well.
I want one more elasticsearch output in same configuration file. I mean after parsing the logs, logstash
send results to one index after that removing some fields and send them to another index. My aim is in below example.
output {
elasticsearch {
..index1..
}
mutate {
remove_field ...
}
mutate {
add_field ..
}
elasticsearch {
... index2 ...
}
}
But i can not work with mutate plugin in output ? How can i achieve this ?
Thanks for answering
mutate is a filter plugin, so it will only work inside the filter block.
What you can do maybe is use the clone filter to duplicate your event, apply different filters to the original and cloned event and then deliver the two copies to different outputs accordingly.
It could look roughly like this
filter {
clone {
clones => ["cloned"]
}
# apply here filters that should be applied to both the primary and secondary event
if [type] == "cloned" {
# apply here filters that should be applied to the secondary event
}
}
output {
if [type] == "cloned" {
elasticsearch {
... index2 ...
}
} else {
elasticsearch {
... index1 ...
}
}
}
See https://discuss.elastic.co/t/can-i-filter-the-output-data-separately/36064/5
I was wondering what will be the best way to implement the following task in logstash :
I have the following field that contains multiple paths divided by ':' :
my_field : "/var/log/my_custom_file.txt:/var/log/otherfile.log/:/root/aaa.jar
I want to add a new field called "first_file" that will contain only the file_name(without suffix) of the first path :
first_file : my_custom_file
I implemented it with the following ruby code ;
code => 'event.set("first_file",event.get("[my_field]").split(":")[0].split("/")[-1].split(".")[0])'
How can I use logstash filters (add_field,split,grok) to do the same task ? I feel like using ruby code should be my last option.
You could do it using just grok, but I think it would be clearer to use mutate to pull out the first value
mutate { split => { "my_field" => ":" } }
mutate { replace => "{ "my_field" => "[my_field][0]" } }
grok { match => { "my_field" => "/(?<my_field>[^/]+)\.%{WORD}$" } overwrite => [ "my_field" ] }
rather than
grok { match => { "my_field" => "/(?<my_field>[^/]+)\.%{WORD}:" } overwrite => [ "my_field" ] }
The (?<my_field>[^/]+) is a custom pattern (documented here) which creates a field called [my_field] from a sequence of one or more (+) characters which are not /
Yes with a basic grok you could match every field in the value.
This kind of filter must work (put it in your logstash configuration file), this one extract the "basename" of the file (filename without extension and path) :
filter{
grok {
match => { "my_field" => "%{GREEDYDATA}/%{WORD:filename}.%{WORD}:%{GREEDYDATA}/%{WORD:filename2}.%{WORD}:%{GREEDYDATA}/%{WORD:filename3}.%{WORD}" }
}
}
You could be more strict in grok with use of PATH in place of GREYDATA, I let you determine your best approach that works in your context.
You could debug the grok pattern with the online tool grokdebug.
I wonder if there is a way of using a substring when using add_field in mutate filter, maybe with %{field} syntax.
For example:
filter {
mutate {
add_field => { "shorter_field" => "%{field[0:4]}" }
}
}
I saw solutions using ruby filters, but I prefer only using mutate because I have a sequence of operation I'm doing using the filter and I prefer keeping it simple
Try this:
(assuming that you need first 4 characters of the field.)
CODE:
filter {
mutate {
gsub => ["shorter_field", "(?<=^....)(.*)", ""]
}
}
In here all characters except first 4 characters will be removed from shorter_field.
example:
INPUT:
shorter_field = example_value
OUTPUT:
shorter_field = exam
I'm trying to add "type" field that i would be able to use as tag when filtering to my input plugin.
For example, I can add "type" in the 'file' input plugin, and filter it later.
Code:
input {
file {
path => "/var/log/%{type}.%{+yyyy.MM.dd.HH}"
type => "myType"
}
}
filter {
if [type] == "myType"
...
}
output {
...
}
I looked at the file input plugin implementation, and did not see any field related to the "type" field.
So i just tried to add it like this:
input {
myPlugin {
path => "/var/log/%{type}.%{+yyyy.MM.dd.HH}"
type => "myType"
}
}
But the filtering did not work at all.
How can i add this "tag" (or whatever it is) to my plugin?
The functionalitiy of type is to use it for filtering:
Types are used mainly for filter activation.
So you can't actually see a new field being created unless you have created a new field under your filter, something like this:
input {
file {
path => "/var/log/%{type}.%{+yyyy.MM.dd.HH}"
type => "myType"
}
}
filter {
if [type] == "myType"{
add_field => {
"somefield" => "this is a sample" <--- this will actually create the field
}
add_tag => "myType" <--- this would actually create a tag for your es record if it matches the filter
}
}
You could maybe check out this, in order to add a new field and adding tags as well. Make sure your plugin supports type. Hope this helps!
After changing my mapping in ElasticSearch to more definitively type the data I am inputting into the system, I have unwittingly made my new variables a nested object. Upon thinking about it more, I actually like the idea of those fields being nested objects because that way I can explicitly know if that src_port statistic is from netflow or from the ASA logs, as an example.
I'd like to use a mutate (gsub, perhaps?) to cause all of my fieldnames for a given type to be renamed to newtype.fieldname. I see that there is gsub which uses a regexp, and rename which takes the literal field name, but I would like to prevent having 30 distinct gsub/rename statements when I will be replacing all of the fields in that type with the "newtype" prefix.
Is there a way to do this?
Here is an example for your reference.
input {
stdin{
type => 'netflow'
}
}
filter {
mutate {
add_field => {"%{type}.message" => "%{message}"}
remove_field => ["message"]
}
}
output {
stdout{
codec => rubydebug
}
}
In this example I have change the message field name to type.message, then delete the origin message field. I think you can use this sample to do what you want.
Hope this can help you.
I have updated my answer!
Use the ruby plugin to do what you want!
Please notice that elasticsearch uses #timestamp field to do index, so I recommend do not change the field name.
input {
stdin{
type => 'netflow'
}
}
filter {
ruby {
code => "
data = event.clone.to_hash;
type = event['type']
data.each do |k,v|
if k != '#timestamp'
newFieldName = type +'.'+ k
event[newFieldName] = v
event.remove(k)
end
end
"
}
}
output {
stdout{
codec => rubydebug
}
}