_dateparsefailure when the date already has a match - logstash

Question

I'm trying to config logstash to process some test log, but I keep getting a dateparsefailure and I don't understand why. My input is
2016-09-18 00:00:02,013 UTC, idf="639b26a731284b43beac8b26f829bcab"
And my config (I've also tried including the timezone into the pattern):
input {
file {
path => "/tmp/test.log"
start_position => "beginning"
}
}
filter {
date {
match => ["message", "yyyy-MM-dd HH:mm:ss,SSS"]
timezone => "UTC"
add_field => { "debug" => "timestampMatched"}
}
grok {
match => { "message" => "%{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:second},%{NUMBER:milis} UTC, idf=\"%{WORD:idf}\""}
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
}
stdout {
codec => rubydebug
}
}
Finaly, the error:
{:timestamp=>"2016-09-21T10:04:32.060000+0000", :message=>"Failed parsing date from field", :field=>"message", :value=>"2016-09-18 00:00:02,013 UTC, idf=\"639b26a731284b43beac8b26f829bcab\"", :exception=>"Invalid format: \"2016-09-18 00:00:02,013 UTC, idf=\"639b26a731284b4...\" is malformed at \" UTC, idf=\"639b26a731284b4...\"", :config_parsers=>"yyyy-MM-dd HH:mm:ss,SSS", :config_locale=>"default=en_US", :level=>:warn, :file=>"logstash/filters/date.rb", :line=>"354", :method=>"filter"}
It says that the date it is malformed after the end of it. Why does this happen, shouldn't it 'stop searching' since the date has already a match?

Answer 1

Before you can use the date filter, you first have to use grok to separate the date and the rest of the message. The date filter only accepts a timestamp. If you have any other information in the field the error you are describing will occur.
Using your provided logline I would recommend this:
filter {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:timedate} %{GREEDYDATA}"}
}
date {
match => [ "timedate" => "yyyy-MM-dd HH:mm:ss,SSS"]
}
}
In this minimal example I match the timestamp in the timedate field and then crunch it trough the date filter.