I'm trying to write a new logstash filter for events coming from Wazuh. Generally all events set a "%{[rule][description]}" variable and I write this into my alert field. I'm finding that one event is not populating this variable so when I write it to my alert field, I just get %{[rule][description]} instead of the contents.
Does anyone know how to check if a variable exists in a logstash filter? It's fairly easy for fields but not for a variable from what I can gather so far. I'd like to be able to say if the variable doesn't exist, then set it to a string of my choosing.
It's pretty strange getting a rule with the rule.description field empty, could you share with me the rule's id?
Anyway, the filter you are looking for is the following one:
filter{
if [rule][description] == '' {
mutate {
add_field => [ "rule.description", "VALUE" ]
}
}
}
Where you can fill VALUE with the string you wish.
Hope it helps.
Related
I have a file with the following format:
10302\t<document>.....</document>
12303\t<document>.....</document>
10054\t<document>.....</document>
10034\t<document>.....</document>
as you can see there are two values separated by a tab char. I need to
index the first token (e.g. 10302, 12303...) as ID
extract (and then index) some information from the second token (the XML document). In other words, the second token would be used with the xml filter for extracting some information
Is it possibile to do that separating the two values using the kv filter? Ideally I should end, for each line, with a document like this:
id:10302
msg:<document>....</document>
I could use a grok filter but I'd like to avoid any regex as the field detection is very easy and can be accomplished with a simple key-value logic. However, using a plain kv detection I'm ending with the following:
"10302": <document>.....</document>
"12303": <document>.....</document>
"10054": <document>.....</document>
"10034": <document>.....</document>
and this is not want I need.
It is not possible to use kv for the job you want to do, as far as I know, since there are no possible key for the id (10302, 10303, 10304...). There are no possible key since there is nothing before the id.
This grok configuration would work, assuming each id + document is on the same line :
grok {
match => { "message" => "^%{INT:ID}\t%{GREEDYDATA:msg}"}
}
Here's what I want, it's a bit the opposite of incremental data.
some data's are logs with a specific token, and I want to be able to keep (or to show in Elasticsearch) only the first submitted data, the oldiest information of each token.
I want to ignore any new log of the same token ?
How can I do that ? is it in logstash or elasticsearch ?
Thanks
Updates 2016-05-31
I think we can see that in different perspective. but globally what I want is the table like in the picture, but without the red lines, I want them to be ignored by logstash, or not display in ES queries.
I know it can be done, if I was able to add any flag in those lines I want to delete, but it's not possible, the only fact that tell us they can be removed is because we already have a key first-AAA that has been logged before.
At the logging process, we don't have this information.
You can achieve this using the elasticsearch filter. The filter would check in ES if the record already exists and if it is the case, we ask Logstash to just drop the line.
Note that I'm making the assumption that the Id field (AAA) is used as the document _id and is also present in the document as the Id field. Feel free to change whatever needs to, but this will work.
input {
...
}
filter {
elasticsearch {
hosts => ["localhost:9200"]
query => "_type:your_type AND _id:%{[Id]}"
fields => {"Id" => "found"}
}
if [found] {
drop {}
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
...
}
}
I try to convert a field "tmp_reponse" in integer in the file "conf" with logstash as follows :
mutate {
convert => {"TMP_REPONSE" => "integer"}
}
,but on Kibana it shows me that he is still string. I do not understand how I can make a convertion to use my fields "tmp_response" to use it like as a metric fields on kibana
thank you help me please and if there is anyone who can explain to me how I can master the metrics on Kibana and use fields as being of metrics fields
mutate{} will change the type of the field in logstash. If you added a stdout{} output stanza, you would see that it's an integer at that point.
How elasticsearch treats it is another problem entirely. Elasticsearch usually sets the type of a field based on the first input received, so if you sent documents in before you added the mutate to your logstash config, they would have been strings and the elasticsearch index will always consider that field to be a string.
The type may also have been defined in an elasticsearch template or mapping.
The good news is that your mutate will probably set the type when a new index is created. If you're using daily indexes (the default in logstash), you can just wait a day. Or you can delete the index (losing any data so far) and let a new one be created. Or you could rebuild the index.
Good luck.
I'm new at ELK-stack and want to add a field in kibana(discover) interface that matches a specific part of the message text (one word or a sentence).
for example:
I want to have a field in the left side that matches the word 'installed' in the message text.
Which filter in logstash should I use and how does it look like?
How about grok{}, which applies a regular expression to your input message and can make new fields?
Thanks for the answer. I used grok as following to match how many users created new accounts.
grok {
match => [ "message", "(?<user_created>(user_created))"]
break_on_match => false
}
Anyway I found out the problem is that Kibana is showing old logs and doesn't care what I do in the logstash config file! still can't find out why!
I have many messages like this:
Error GetMilesFromLocationService(Eastvale, CA,Yorkshire, NY,1561517,19406,True.)
The problem is that they are unique because of the city names. In a Kibana Visualization, is it possible group these into "Error GetMilesFromLocationService" messages? Here's an example of my metrics visual. Ideally, they would all be in one row.
These could be easily grouped by a regex match.
Of course, I could add a new field with Logstash, but if Kibana is able to do this, I'll be happy.
Thanks!
Use a grok filter to parse the message and extract fields from it. At the very least you'll want to extract "Error GetMilesFromLocationService" into a separate field (perhaps error_type?) to allow aggregation. Or perhaps it would be better to extract "GetMilesFromLocationService" into a function field? Without knowing the structure of your log messages giving firm advice is hard.
This grok filter extracts an error_type field:
filter {
grok {
match => [
"message",
"^(?<error_type>Error %{WORD})"
]
}
}