Grok pattern for creating separate section in Kibana dashboard - logstash

Question

I having been trying since long time to extract and mark data from my customized log using logstash, but not getting anywhere, I having a customized haproxy log like below:
Feb 22 21:17:32 ap haproxy[1235]: 10.172.80.45:32071 10.31.33.34:44541 10.31.33.34:32772 13.127.229.72:443 [22/Feb/2020:21:17:32.006] this_machine~ backend_test-tui/test-tui_32772 40/0/5/1/836 200 701381 - - ---- 0/0/0/0/0 0/0 {testtui.net} {cache_hit} "GET /ob/720/output00007.ts HTTP/1.1"
I want to extract and mark specific content in kibana dashboard from log, like:
from "40/0/5/1/836" section i want to mark the only the last section digit (836) as "response_time"
"701381" as "response_bytes"
"/ob/720/output00007.ts" as "content_url"
And want to use the timestamp in the log file and not the default one
I have created a grok filter using https://grokdebug.herokuapp.com/ but whenever i apply it i m seeing "_grokparsefailure" message and the kibana dashboard stops getting populated
Below is the logstash debug log
{
"#version" => "1",
"message" => "Mar 8 13:53:59 ap haproxy[22158]: 10.172.80.45:30835 10.31.33.34:57886 10.31.33.34:32771 43.252.91.147:443 [08/Mar/2020:13:53:59.827] this_machine~ backend_noida/noida_32771 55/0/1/0/145 200 2146931 - - ---- 0/0/0/0/0 0/0 {testalef1.adcontentamtsolutions.} {cache_hit} \"GET /felaapp/virtual_videos/og/1080/output00006.ts HTTP/1.1\"",
"#timestamp" => 2020-03-08T10:24:07.348Z,
"path" => "/home/alef/haproxy.log",
"host" => "com1",
"tags" => [
[0] "_grokparsefailure"
]
}
Below is the Filter which i have created
%{MONTH:[Month]} %{MONTHDAY:[date]} %{TIME:[time]} %{WORD:[source]} %{WORD:[app]}\[%{DATA:[class]}\]: %{IPORHOST:[UE_IP]}:%{NUMBER:[UE_Port]} %{IPORHOST:[NATTED_IP]}:%{NUMBER:[NATTED_Source_Port]} %{IPORHOST:[NATTED_IP]}:%{NUMBER:[NATTED_Destination_Port]} %{IPORHOST:[WAN_IP]}:%{NUMBER:[WAN_Port]} \[%{HAPROXYDATE:[accept_date]}\] %{NOTSPACE:[frontend_name]}~ %{NOTSPACE:[backend_name]} %{NOTSPACE:[ty_name]}/%{NUMBER:[response_time]} %{NUMBER:[http_status_code]} %{INT:[response_bytes]} - - ---- %{NOTSPACE:[df]} %{NOTSPACE:[df]} %{DATA:[domain_name]} %{DATA:[cache_status]} %{DATA:[domain_name]} %{NOTSPACE:[content]} HTTP/%{NUMBER:[http_version]}
Below is my logstash conf file:
input {
beats {
port => 5044
}
}
filter {
grok {
match => { "message" => "%{MONTH:[Month]} %{MONTHDAY:[date]} %{TIME:[time]} %{WORD:[source]} %{WORD:[app]}\[%{DATA:[class]}\]: %{IPORHOST:[UE_IP]}:%{NUMBER:[UE_Port]} %{IPORHOST:[NATTED_IP]}:%{NUMBER:[NATTED_Source_Port]} %{IPORHOST:[NATTED_IP]}:%{NUMBER:[NATTED_Destination_Port]} %{IPORHOST:[WAN_IP]}:%{NUMBER:[WAN_Port]} \[%{HAPROXYDATE:[accept_date]}\] %{NOTSPACE:[frontend_name]}~ %{NOTSPACE:[backend_name]} %{NOTSPACE:[ty_name]}/%{NUMBER:[response_time]} %{NUMBER:[http_status_code]} %{INT:[response_bytes]} - - ---- %{NOTSPACE:[df]} %{NOTSPACE:[df]} %{DATA:[domain_name]} %{DATA:[cache_status]} %{DATA:[domain_name]} %{NOTSPACE:[content]} HTTP/%{NUMBER:[http_version]} " }
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}
output {
elasticsearch { hosts => ["localhost:9200"] }
}

Answer 1

Using the below filter resolved my issue had to do debugging in the logstash itself to get proper filter:
input { beats {
port => 5044 } }
filter { grok {
match => { "message" => "%{MONTH:month} %{MONTHDAY:date} %{TIME:time} %{WORD:[source]} %{WORD:[app]}[%{DATA:[class]}]:
%{IPORHOST:[UE_IP]}:%{NUMBER:[UE_Port]}
%{IPORHOST:[NATTED_IP]}:%{NUMBER:[NATTED_Source_Port]}
%{IPORHOST:[NATTED_IP]}:%{NUMBER:[NATTED_Destination_Port]}
%{IPORHOST:[WAN_IP]}:%{NUMBER:[WAN_Port]}
[%{HAPROXYDATE:[accept_date]}] %{NOTSPACE:[frontend_name]}~
%{NOTSPACE:[backend_name]}
%{NOTSPACE:[ty_name]}/%{NUMBER:[response_time]:int}
%{NUMBER:[http_status_code]} %{NUMBER:[response_bytes]:int} - - ----
%{NOTSPACE:[df]} %{NOTSPACE:[df]} %{DATA:[domain_name]}
%{DATA:[cache_status]} %{DATA:[domain_name]} %{URIPATHPARAM:[content]}
HTTP/%{NUMBER:[http_version]}" }
add_tag => [ "response_time", "response_time" ]
} date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ] } }
output { elasticsearch { hosts => ["localhost:9200"] }
stdout {
codec => rubydebug
} }