Grok filter for selecting and formatting certain logs lines - logstash

Question

I am writing up a grok filter for parsing my application log which is unstructured. What i need is to look for certain lines and generate output in a specific format. e.g below are my logs
2018-05-07 01:19:40 M :Memory (xivr = 513.2 Mb, system = 3502.0 Mb, physical = 5386.7 Mb), CpuLoad (sys = 0%, xivr = 0%)
2018-05-07 01:29:40 M :Memory (xivr = 513.2 Mb, system = 3495.3 Mb, physical = 5370.1 Mb), CpuLoad (sys = 0%, xivr = 0%)
2018-05-07 05:51:19 1 :Hangup call
***2018-05-07 05:51:22 24 :Answer call from 71840746 for 91783028 [C:\xivr\es\IVR-Dialin.dtx***]
2018-05-07 05:51:30 24 :Hangup call
***2018-05-07 05:51:34 24 :Answer call from 71840746 for 91783028 [C:\xivr\es\IVR-Dialin.dtx]***
2018-05-07 00:31:21 45 :Device Dialogic Digital dxxxB12C1 [gc60.dev - Dialogic (SDK 6.0) ver 3.0.702:11646] (ThreadID: 1FF0, DriverChannel: 44)
2018-05-07 00:31:22 40 :Device Dialogic Digital dxxxB10C4 [gc60.dev - Dialogic (SDK 6.0) ver 3.0.702:11646] (ThreadID: 1B2C, DriverChannel: 39)
I need to enter only lines highlighted with *** in below format in my Kibana: Other lines should be simply ignored
Logtimestamp: 2018-05-07 05:51:22
Channel_id: 24
Source_number:
71840746
Destination_Number: 91783028
How can this be achieved?

Answer 1

You can explicitly write whatever is unique about that particular pattern, and use pre-defined grok patterns for the rest.
In your case, the grok pattern would be,
%{TIMESTAMP_ISO8601:Logtimestamp} %{NUMBER:Channel_id} :Answer call from %{NUMBER:Source_number} for %{NUMBER:Destination_Number} %{GREEDYDATA:etc}
It will only match following pattern,
2018-05-07 05:51:34 24 :Answer call from 71840746 for 91783028 [C:\xivr\es\IVR-Dialin.dtx]
Explanation
The syntax for a grok pattern is %{SYNTAX:SEMANTIC}.
In your filter,
%{TIMESTAMP_ISO8601:Logtimestamp} matches 2018-05-07 05:51:34
%{NUMBER:Channel_id} match 24
:Answer call from matches the string literally
%{NUMBER:Source_number} matches 71840746
%{NUMBER:Destination_Number} matches 91783028
%{GREEDYDATA:etc} matches rest of the data i.e. [C:\xivr\es\IVR-Dialin.dtx]
in that order.
Output:
{
"Logtimestamp": [
[
"2018-05-07 05:51:22"
]
],
"Channel_id": [
[
"24"
]
],
"Source_number": [
[
"71840746"
]
],
"Destination_Number": [
[
"91783028"
]
],
"etc": [
[
"[C:\\xivr\\es\\IVR-Dialin.dtx***]"
]
]
}
You can test it here.
Hope it helps.