Filter logs with grok in logstash - logstash

Question

I am trying to filter out logs received with the help of grok. Below is the sample log
INFO | jvm 1 | main | 2013/04/05 01:08:47.048 | [m[32mINFO [TaskExecutor-master-2443-ProcessTask [31111111112]] [b2cConfirmationAction] CRON JOB ID : 101AA1C, ACTION : ConfirmationAction , CUSTOMER ID : 000001111111 , EMAIL ADDRESS : abc#gmail.com , SCHEDULE : Every 1 week , MESSAGE : Execution started for action ConfirmationAction
I am using grok debugger (https://grokdebug.herokuapp.com/) to test before updating logstash conf file.
Below is my filter code :
%{LOGLEVEL:level}%{GREEDYDATA:greedydata}%{SPACE}%{YEAR}[/-]%{MONTHNUM}[/-]%{MONTHDAY}%{SPACE}%{HOUR}:%{MINUTE}:%{SECOND}%{GREEDYDATA:gd} \[(?:%{WORD:action})\]%{GREEDYDATA:cronjobresult}
Here I am getting outpout as
"level": [ [ "INFO" ] ], "greedydata": [ [ " | jvm 1 | main | 20" ] ], "SPACE": [ [ "", " " ] ], "YEAR": [ [ "13" ] ], "MONTHNUM": [ [ "04" ] ], "MONTHDAY": [ [ "05" ] ], "HOUR": [ [ "01" ] ], "MINUTE": [ [ "08" ] ], "SECOND": [ [ "47.048" ] ], "gd": [ [ " | \u001b[m\u001b[32mINFO [TaskExecutor-master-2443-ProcessTask [31111111112]]" ] ], "action": [ [ "b2cConfirmationAction" ] ], "cronjobresult": [ [ " CRON JOB ID : 101AA4A , ACTION : ConfirmationAction , CUSTOMER ID : 000001111111 , EMAIL ADDRESS : abc#gmail.com , SCHEDULE : Every 1 week , MESSAGE : Execution started for action ConfirmationAction" ] ] }
My requirement is to get values under cronjobresult like cron job iD customer id with different and independent field so that I can use these values in kibana. Right now I am not able to get it. Also I have used greedyData twice, better approach for this log would be appreciable.

Answer 1

You can simply extend your filter further and match it explicitly. For instance, to match cron job id, you can write CRON JOB ID : %{BASE16NUM:Cron_job_id} in your filter.
If you do not need any data from log then you can simply write .* instead of GREEDYDATA and it will be skipped.
Here is the complete filter for your log,
%{LOGLEVEL:level}%{GREEDYDATA:greedydata}%{SPACE}%{YEAR}[/-]%{MONTHNUM}[/-]%{MONTHDAY}%{SPACE}%{HOUR}:%{MINUTE}:%{SECOND}%{GREEDYDATA:gd} \[(?:%{WORD:action})\] CRON JOB ID : %{BASE16NUM:Cron_job_id},.*CUSTOMER ID : %{NUMBER:Customer_id}.*EMAIL ADDRESS : %{EMAILADDRESS}.*SCHEDULE : %{GREEDYDATA:schedule}.*, MESSAGE : %{GREEDYDATA:Message}
Output:
{
"level": [
[
"INFO"
]
],
"greedydata": [
[
" | jvm 1 | main | 20"
]
],
"SPACE": [
[
"",
" "
]
],
"YEAR": [
[
"13"
]
],
"MONTHNUM": [
[
"04"
]
],
"MONTHDAY": [
[
"05"
]
],
"HOUR": [
[
"01"
]
],
"MINUTE": [
[
"08"
]
],
"SECOND": [
[
"47.048"
]
],
"gd": [
[
" | [m[32mINFO [TaskExecutor-master-2443-ProcessTask [31111111112]]"
]
],
"action": [
[
"b2cConfirmationAction"
]
],
"Cron_job_id": [
[
"101AA1C"
]
],
"Customer_id": [
[
"000001111111"
]
],
"BASE10NUM": [
[
"000001111111"
]
],
"EMAILADDRESS": [
[
"abc#gmail.com"
]
],
"local": [
[
"abc"
]
],
"remote": [
[
"gmail.com"
]
],
"schedule": [
[
"Every 1 week "
]
],
"Message": [
[
"Execution started for action"
]
]
}
Please note that I have used EMAILADDRESS pattern from, https://github.com/rgevaert/grok-patterns/blob/master/grok.d/postfix_patterns
If you want to test it on https://grokdebug.herokuapp.com, you need to add,
EMAILADDRESSPART [a-zA-Z0-9_.+-=:]+
EMAILADDRESS %{EMAILADDRESSPART:local}#%{EMAILADDRESSPART:remote}
as cusomtom patterns by checking add custom patterns