Logstash - match filename - linux

I have two servers. The first one hosts elastic stack. Both servers have a file /var/log/commands.log which is configured in the same way and are being shipped with filebeat to logstash.
Using grok, I tried parsing the data into custom fields using this statement:
if [log][file][path] == "/var/log/commands.log" {
grok{
match => { "message" => "*some grok stuff*"
}
}
}
Problem is, even though on both servers the file is /var/log/commands.log & they're configured the same - it skips the if statement as if it's false.
I've noticed that if I ship the logs locally (without filebeat - just do input{file{input => "/var/log/commands.log}} ) it works for the local "/var/log/commands.log" file on that machine that hosts logstash.
For reference, this is the full .conf file for logstash: https://pastebin.com/1QbnAG7G
This is how elastic sees the file path: https://i.imgur.com/5h9HXf2.png
Does anyone why it skips the "if" statement? How to make it filter by name. Thanks ahead!

So, it looks like you're using =~ in your pastebin code. If that's the case, you'll be matching a regex. Is that what you meant to do?
If you're intending to use a regex, then you'd want something like this, probably:
if [log][file][path] =~ /\/var\/log\/commands\.log/
Example: https://regex101.com/r/nqGg9G/1

Related

Creating custom grok filters

I need to find grok pattern for files where the lines are of the format :
3 dbm.kfa 0 340220 7766754 93.9
3 swapper/3 0 340220 7766754 93.9
This is the grok pattern that I have done so far.
\s*%{INT:no1}\s*%{USERNAME:name}\s*%{INT:no2}\s*%{INT:no3}\s*%{INT:no4}\s*%{GREEDYDATA:ans}
The field USERNAME works for dbm.kfa but not for swapper/3 as USERNAME does not include \ character. I would like to create some custom filter for this purpose, but have no idea how to create one.
Any help would be really appreciated. Thanks a lot !
To create a custom pattern you need to use an external file in the following format and put that file in a directory the will be used only for pattern files.
PATTERN_NAME [regex for your pattern]
Then you will need to change your grok config to point to the pattern files directory.
grok {
patterns_dir => ["/path/to/patterns/dir"]
match => { "message" => "%{PATTERN_NAME:fieldName}" }
}
But in your specific case if you change %{USERNAME:name} to %{DATA:name} it should work.
For a better explanation about the custom patterns you should read this part of the documentation.
You also can find all the core grok patterns that ships with logstash in this github repository, the most used are in the grok-patterns file.

Is It necessary to restart logstash after modifying a dictionary_path file?

I used the following translate filter in logstash
translate {
field => "countries"
destination => "cities"
dictionary_path => "/home/rrr/cities.yml"
}
And I started logstash this way
/usr/share/logstash/bin/logstash -f $directory --path.settings=/etc/logstash -t
Everything went well and good.
My question is :
Would logstash will take into account any modification that I may do in the dictionary_path file ?
I means do I need to restart logstash after any edition on this file or not ?
It should not be necessary to restart logstash. There is a parameter in the configuration of the translate plugin, refresh interval:
refresh_interval
Value type is number
Default value is 300
When using a dictionary file, this setting will indicate how frequently (in seconds) logstash will check the dictionary file for updates.

Parsing postfix events with grok

I'm trying to figure out how it works logstash and grok to parse messages. I have found that example ftp://ftp.linux-magazine.com/pub/listings/magazine/185/ELKstack/configfiles/etc_logstash/conf.d/5003-postfix-filter.conf
which start like this:
filter {
# grok log lines by program name (listed alpabetically)
if [program] =~ /^postfix.*\/anvil$/ {
grok{...
But don't understand where [program] is parsed. I'm using logstash 2.2
That example are not working in my logstash installation, nothing is parsed.
I answer myself.
The example assumes that the events come from syslog (in that case the field "program" are present), instead filebeats which is what I'm using to send the events to logstash.
To fix-it:
https://github.com/whyscream/postfix-grok-patterns/blob/master/ALTERNATIVE-INPUTS.md

Debugging new logstash grok filters before full use

I have been following this guide:
http://deviantony.wordpress.com/2014/06/04/logstash-debug-configuration/
Which I'm hoping will help me test my logstash filters to see if I get the desired output before using them full time.
As part of the guide it tells you to set up an input and output and then a filter file. the input seems to work fine:
input {
stdin { }
}
The output is this:
output {
stdout {
codec => json
}
file {
codec => json
path => /tmp/debug-filters.json
}
}
I am getting the following error when I try to run the logstash process (here I've run it with --configtest as the error advises me to try that, but it doesn't give any more information):
# /opt/logstash/bin/logstash -f /etc/logstash/debug.d -l /var/log/logstash/logstash-debug.log --configtest
Sending logstash logs to /var/log/logstash/logstash-debug.log.
Error: Expected one of #, ", ', -, [, { at line 21, column 17 (byte 288) after output {
stdout {
codec => json
}
file {
codec => json
path =>
I have tried removing the file section in my output and I can get the logstash process running, but when I paste my log line in to the shell I don't see the log entry broken down in to the components I am expecting the grok filter to break it in to. All I get when I do that is:
Oct 30 08:57:01 VERBOSE[1447] logger.c: == Manager 'sendcron' logged off from 127.0.0.1
{"message":"Oct 30 08:57:01 VERBOSE[1447] logger.c: == Manager 'sendcron' logged off from 127.0.0.1","#version":"1","#timestamp":"2014-10-31T16:09:35.205Z","host":"lumberjack.domain.com"}
Initially I was having a problem with a new grok filter, so I have now tried with an existing filter that I know works (as shown above it is an Asterisk 1.2 filter) and has been generating entries in to elasticsearch for some time.
I have tried touching the json file mentioned in the output, but that hasn't helped.
When I tail the logstash-debug.log now I just see the error that is also being written to my shell.
Any suggestions on debugging grok filters would be appreciated, if I have missed something blindingly obvious, apologies, I've only been working with ELK & grok for a couple of weeks and I might not be doing this in the most sensible way. I was hoping to be able to drop example log entries in to the shell and get the JSON formatted logstash entry to my console so I could see if my filter was working as I hoped, and tagging them up as they will be displayed in kibana at the end. If there is a better way to do this please let me know.
I am using logstash 1.4.2
As far as debugging a grok filter goes, you can use this link (http://grokdebug.herokuapp.com/) It has a very comprehensive pattern detector which is a good start.
As far your file output, you need "" around your path. Here is the example i use in production. Here is the documentation on file output http://logstash.net/docs/1.4.2/outputs/file#path
output {
stdout {
codec => rubydebug
}
file {
codec => "plain"
path => "./logs/logs-%{+YYYY-MM-dd}.txt"
}
}
The Grokconstructor is a similar Grok debugger to Grokdebug which #user3195649 mentioned. I like it's random examples.

Extracting fields in Logstash

I am using Logstash (with Kibana as the UI). I would like to extract some fields from my logs so that I can filter by them on the LHS of the UI.
A sample line from my log looks like this:
2013-07-04 00:27:16.341 -0700 [Comp40_db40_3720_18_25] client_login=C-316fff97-5a19-44f1-9d87-003ae0e36ac9 ip_address=192.168.4.1
In my logstash conf file, I put this:
filter {
grok {
type => "mylog"
pattern => "(?<CLIENT_NAME>Comp\d+_db\d+_\d+_\d+_\d+)"
}
}
Ideally, I would like to extract Comp40_db40_3720_18_25 (the number of digits can vary, but will always be at least 1 in each section separated by _) and client_login (can also be client_logout). Then, I can search for CLIENT_NAME=Comp40... CLIENT_NAME=Comp55, etc.
Am I missing something in my config to make this a field that I can use in Kibana?
Thanks!
If you are having any difficulty getting the pattern to match correctly, using the Grok Debugger is a great solution.
For your given problem you could just separate out your search data into another variable, and save the additional varying digits in another (trash) variable.
For example:
(?<SEARCH_FIELD>Comp\d+)%{GREEDYDATA:trash_variable}]
(Please use the Grok Debugger on the above pattern)

Resources