Impossible to use grok date match correctly - logstash

Question

I have this message
2016/02/22 08:40:10 [error] 2127#0: *193 open()
"/etc/nginx/nginx/html/static-cdn.arte.tv/resize-preprod/nQa5oWnNDknADSxe0mPEMd5McUA=/940x530/smart/default/prog_img/IMG_APIOS/051000/051700/051757-001_1137283_32_202.jpg"
failed (2: No such file or directory), client: 192.168.144.95, server:
api.magritte.arte.tv, request: "GET
/static-cdn.arte.tv/resize-preprod/nQa5oWnNDknADSxe0mPEMd5McUA=/940x530/smart/default/prog_img/IMG_APIOS/051000/051700/051757-001_1137283_32_202.jpg
HTTP/1.1", host: "api.magritte.arte.tv", referrer:
"https://api.magritte.arte.tv/api/oauth/user/documentation/opa/endpoint/27/-api-opa-v2-videos"
And I parse it this way
grok {
match => { "message" => "(?<timestamp>%{YEAR}/%{MONTHNUM2}/%{MONTHDAY} %{TIME}) \[%{LOGLEVEL:severity}\] %{POSINT:pid}#%{NUMBER:tid}:( \*%{NUMBER:cid})? %{GREEDYDATA:errormessage}(?:, client: (?<client>%{IP}|%{HOSTNAME}))(?:, server: %{IPORHOST:server})(?:, request: %{QS:request})?(?:, upstream: \"%{URI:upstream}\")?(?:, host: %{QS:host})?(?:, referrer: \"%{URI:referrer}\")?"}
}
date {
match => [ "timestamp" , "YYYY/MM/dd HH:mm:ss" ]
}
When a new message arrives the following behaviour occurs
message sent to rabbit : OK
message read from rabbit : OK
problem when logstash read message
"reason"=>"failed to parse [timestamp]",
"caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid
format: \"2016/02/22 08:40:10\" is malformed at \"/02/22
08:40:10\""}}}}, :level=>:warn}
But I have no idea where my error is. using http://grokconstructor.appspot.com/do/match#result all seems OK
The full log in the logstash is
{:timestamp=>"2016-02-22T08:43:29.968000+0100", :message=>"Failed action. ", :status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-2016.02.22", :_type=>"nginx_error", :_routing=>nil}, #<LogStash::Event:0x75f8f9a0 #metadata_accessors=#<LogStash::Util::Accessors:0x402f1514 #store={}, #lut={}>, #cancelled=false, #data={"message"=>"2016/02/22 08:40:10 [error] 2127#0: *193 open() \"/etc/nginx/nginx/html/static-cdn.arte.tv/resize-preprod/nQa5oWnNDknADSxe0mPEMd5McUA=/940x530/smart/default/prog_img/IMG_APIOS/051000/051700/051757-001_1137283_32_202.jpg\" failed (2: No such file or directory), client: 192.168.144.95, server: api.magritte.arte.tv, request: \"GET /static-cdn.arte.tv/resize-preprod/nQa5oWnNDknADSxe0mPEMd5McUA=/940x530/smart/default/prog_img/IMG_APIOS/051000/051700/051757-001_1137283_32_202.jpg HTTP/1.1\", host: \"api.magritte.arte.tv\", referrer: \"https://api.magritte.arte.tv/api/oauth/user/documentation/opa/endpoint/27/-api-opa-v2-videos\"", "#version"=>"1", "#timestamp"=>"2016-02-22T07:40:10.000Z", "path"=>"/var/log/nginx/api.magritte.arte.tv_error.log", "host"=>["magritte.arte.tv", "\"api.magritte.arte.tv\""], "type"=>"nginx_error", "application"=>"api", "timestamp"=>"2016/02/22 08:40:10", "severity"=>"error", "pid"=>2127, "tid"=>0, "cid"=>193, "errormessage"=>"open() \"/etc/nginx/nginx/html/static-cdn.arte.tv/resize-preprod/nQa5oWnNDknADSxe0mPEMd5McUA=/940x530/smart/default/prog_img/IMG_APIOS/051000/051700/051757-001_1137283_32_202.jpg\" failed (2: No such file or directory)", "client"=>"192.168.144.95", "server"=>"api.magritte.arte.tv", "request"=>"\"GET /static-cdn.arte.tv/resize-preprod/nQa5oWnNDknADSxe0mPEMd5McUA=/940x530/smart/default/prog_img/IMG_APIOS/051000/051700/051757-001_1137283_32_202.jpg HTTP/1.1\"", "referrer"=>"https://api.magritte.arte.tv/api/oauth/user/documentation/opa/endpoint/27/-api-opa-v2-videos"}, #metadata={}, #accessors=#<LogStash::Util::Accessors:0x27ca0e3f #store={"message"=>"2016/02/22 08:40:10 [error] 2127#0: *193 open() \"/etc/nginx/nginx/html/static-cdn.arte.tv/resize-preprod/nQa5oWnNDknADSxe0mPEMd5McUA=/940x530/smart/default/prog_img/IMG_APIOS/051000/051700/051757-001_1137283_32_202.jpg\" failed (2: No such file or directory), client: 192.168.144.95, server: api.magritte.arte.tv, request: \"GET /static-cdn.arte.tv/resize-preprod/nQa5oWnNDknADSxe0mPEMd5McUA=/940x530/smart/default/prog_img/IMG_APIOS/051000/051700/051757-001_1137283_32_202.jpg HTTP/1.1\", host: \"api.magritte.arte.tv\", referrer: \"https://api.magritte.arte.tv/api/oauth/user/documentation/opa/endpoint/27/-api-opa-v2-videos\"", "#version"=>"1", "#timestamp"=>"2016-02-22T07:40:10.000Z", "path"=>"/var/log/nginx/api.magritte.arte.tv_error.log", "host"=>["magritte.arte.tv", "\"api.magritte.arte.tv\""], "type"=>"nginx_error", "application"=>"api", "timestamp"=>"2016/02/22 08:40:10", "severity"=>"error", "pid"=>2127, "tid"=>0, "cid"=>193, "errormessage"=>"open() \"/etc/nginx/nginx/html/static-cdn.arte.tv/resize-preprod/nQa5oWnNDknADSxe0mPEMd5McUA=/940x530/smart/default/prog_img/IMG_APIOS/051000/051700/051757-001_1137283_32_202.jpg\" failed (2: No such file or directory)", "client"=>"192.168.144.95", "server"=>"api.magritte.arte.tv", "request"=>"\"GET /static-cdn.arte.tv/resize-preprod/nQa5oWnNDknADSxe0mPEMd5McUA=/940x530/smart/default/prog_img/IMG_APIOS/051000/051700/051757-001_1137283_32_202.jpg HTTP/1.1\"", "referrer"=>"https://api.magritte.arte.tv/api/oauth/user/documentation/opa/endpoint/27/-api-opa-v2-videos"}, #lut={"type"=>[{"message"=>"2016/02/22 08:40:10 [error] 2127#0: *193 open() \"/etc/nginx/nginx/html/static-cdn.arte.tv/resize-preprod/nQa5oWnNDknADSxe0mPEMd5McUA=/940x530/smart/default/prog_img/IMG_APIOS/051000/051700/051757-001_1137283_32_202.jpg\" failed (2: No such file or directory), client: 192.168.144.95, server: api.magritte.arte.tv, request: \"GET /static-cdn.arte.tv/resize-preprod/nQa5oWnNDknADSxe0mPEMd5McUA=/940x530/smart/default/prog_img/IMG_APIOS/051000/051700/051757-001_1137283_32_202.jpg HTTP/1.1\", host: \"api.magritte.arte.tv\", referrer: \"https://api.magritte.arte.tv/api/oauth/user/documentation/opa/endpoint/27/-api-opa-v2-videos\"", "#version"=>"1", "#timestamp"=>"2016-02-22T07:40:10.000Z", "path"=>"/var/log/nginx/api.magritte.arte.tv_error.log", "host"=>["magritte.arte.tv", "\"api.magritte.arte.tv\""], "type"=>"nginx_error", "application"=>"api", "timestamp"=>"2016/02/22 08:40:10", "severity"=>"error", "pid"=>2127, "tid"=>0, "cid"=>193, "errormessage"=>"open() \"/etc/nginx/nginx/html/static-cdn.arte.tv/resize-preprod/nQa5oWnNDknADSxe0mPEMd5McUA=/940x530/smart/default/prog_img/IMG_APIOS/051000/051700/051757-001_1137283_32_202.jpg\" failed (2: No such file or directory)", "client"=>"192.168.144.95", "server"=>"api.magritte.arte.tv", "request"=>"\"GET /static-cdn.arte.tv/resize-preprod/nQa5oWnNDknADSxe0mPEMd5McUA=/940x530/smart/default/prog_img/IMG_APIOS/051000/051700/051757-001_1137283_32_202.jpg HTTP/1.1\"", "referrer"=>"https://api.magritte.arte.tv/api/oauth/user/documentation/opa/endpoint/27/-api-opa-v2-videos"}, "type"], "[type]"=>[{"message"=>"2016/02/22 08:40:10 [error] 2127#0: *193 open() \"/etc/nginx/nginx/html/static-cdn.arte.tv/resize-preprod/nQa5oWnNDknADSxe0mPEMd5McUA=/940x530/smart/default/prog_img/IMG_APIOS/051000/051700/051757-001_1137283_32_202.jpg\" failed (2: No such file or directory), client: 192.168.144.95, server: api.magritte.arte.tv, request: \"GET /static-cdn.arte.tv/resize-preprod/nQa5oWnNDknADSxe0mPEMd5McUA=/940x530/smart/default/prog_img/IMG_APIOS/051000/051700/051757-001_1137283_32_202.jpg HTTP/1.1\", host: \"api.magritte.arte.tv\", referrer: \"https://api.magritte.arte.tv/api/oauth/user/documentation/opa/endpoint/27/-api-opa-v2-videos\"", "#version"=>"1", "#timestamp"=>"2016-02-22T07:40:10.000Z", "path"=>"/var/log/nginx/api.magritte.arte.tv_error.log", "host"=>["magritte.arte.tv", "\"api.magritte.arte.tv\""], "type"=>"nginx_error", "application"=>"api", "timestamp"=>"2016/02/22 08:40:10", "severity"=>"error", "pid"=>2127, "tid"=>0, "cid"=>193, "errormessage"=>"open() \"/etc/nginx/nginx/html/static-cdn.arte.tv/resize-preprod/nQa5oWnNDknADSxe0mPEMd5McUA=/940x530/smart/default/prog_img/IMG_APIOS/051000/051700/051757-001_1137283_32_202.jpg\" failed (2: No such file or directory)", "client"=>"192.168.144.95", "server"=>"api.magritte.arte.tv", "request"=>"\"GET /static-cdn.arte.tv/resize-preprod/nQa5oWnNDknADSxe0mPEMd5McUA=/940x530/smart/default/prog_img/IMG_APIOS/051000/051700/051757-001_1137283_32_202.jpg HTTP/1.1\"", "referrer"=>"https://api.magritte.arte.tv/api/oauth/user/documentation/opa/endpoint/27/-api-opa-v2-videos"}, "type"]}>>], :response=>{"create"=>{"_index"=>"logstash-2016.02.22", "_type"=>"nginx_error", "_id"=>"AVMH7uSoo1ZDC2Pzezhl", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [timestamp]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: \"2016/02/22 08:40:10\" is malformed at \"/02/22 08:40:10\""}}}}, :level=>:warn}
I think it's a quote issue ...
1
Message in nginx logfile 2016/02/22 08:40:10 [error] 2127#0: *193
open()
"/etc/nginx/nginx/html/static-cdn.arte.tv/resize-preprod/nQa5oWnNDknADSxe0mPEMd5McUA=/940x530/smart/default/prog_img/IMG_APIOS/051000/051700/051757-001_1137283_32_202.jpg"
failed (2: No such file or directory), client: 192.168.144.95, server:
api.magritte.arte.tv, request: "GET
/static-cdn.arte.tv/resize-preprod/nQa5oWnNDknADSxe0mPEMd5McUA=/940x530/smart/default/prog_img/IMG_APIOS/051000/051700/051757-001_1137283_32_202.jpg
HTTP/1.1", host: "api.magritte.arte.tv", referrer:
"https://api.magritte.arte.tv/api/oauth/user/documentation/opa/endpoint/27/-api-opa-v2-videos"
It's the one I use to do my grok parsing (logstash sourceside)
rabbitmq message payload
{"message":"2016/02/22 08:40:10 [error] 2127#0: *193 open()
\"/etc/nginx/nginx/html/static-cdn.arte.tv/resize-preprod/nQa5oWnNDknADSxe0mPEMd5McUA=/940x530/smart/default/prog_img/IMG_APIOS/051000/051700/051757-001_1137283_32_202.jpg\"
failed (2: No such file or directory), client: 192.168.144.95, server:
api.magritte.arte.tv, request: \"GET
/static-cdn.arte.tv/resize-preprod/nQa5oWnNDknADSxe0mPEMd5McUA=/940x530/smart/default/prog_img/IMG_APIOS/051000/051700/051757-001_1137283_32_202.jpg
HTTP/1.1\", host: \"api.magritte.arte.tv\", referrer:
\"https://api.magritte.arte.tv/api/oauth/user/documentation/opa/endpoint/27/-api-opa-v2-videos\"","#version":"1","#timestamp":"2016-02-22T07:40:10.000Z","path":"/var/log/nginx/api.magritte.arte.tv_error.log","host":["magritte.arte.tv","\"api.magritte.arte.tv\""],"type":"nginx_error","application":"api","timestamp":"2016/02/22
08:40:10","severity":"error","pid":2127,"tid":0,"cid":193,"errormessage":"open()
\"/etc/nginx/nginx/html/static-cdn.arte.tv/resize-preprod/nQa5oWnNDknADSxe0mPEMd5McUA=/940x530/smart/default/prog_img/IMG_APIOS/051000/051700/051757-001_1137283_32_202.jpg\"
failed (2: No such file or
directory)","client":"192.168.144.95","server":"api.magritte.arte.tv","request":"\"GET
/static-cdn.arte.tv/resize-preprod/nQa5oWnNDknADSxe0mPEMd5McUA=/940x530/smart/default/prog_img/IMG_APIOS/051000/051700/051757-001_1137283_32_202.jpg
HTTP/1.1\"","referrer":"https://api.magritte.arte.tv/api/oauth/user/documentation/opa/endpoint/27/-api-opa-v2-videos"}
Some backslashes are added
This backslash prevent logstash (target side) to correctly handle the messages.

Answer 1

With this, it works :)
grok {
match => { "message" => "(?<timestamp>%{YEAR}/%{MONTHNUM2}/%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND}) \[%{LOGLEVEL:severity}\] %{POSINT:p_id}#%{NUMBER:t_id}:( \*%{NUMBER:c_id})? %{GREEDYDATA:errormessage}(?:, client: (?<client>%{IP}|%{HOSTNAME}))(?:, server: %{IPORHOST:server})(?:, request: %{QS:request})?(?:, upstream: %{QS:upstream})?(?:, host: %{QS:vhost})?(?:, referrer: \"%{URI:referrer}\")?"}
}
date {
match => [ "timestamp" , "yyyy/MM/dd HH:mm:ss" ]
}

Answer 2

A grok pattern is available, that can solve your problem.
grok {
match => ["message","%{DATESTAMP:timestamp}" ]
}

Answer 3

Try y instead of Y in date filter.
joda.time.format.DateTimeFormat
date {
match => [ "timestamp" , "yyyy/MM/dd HH:mm:ss" ]
}