How to modify #timestamp with an entry in logs using logstash - logstash

Question

I have some logs which has only time as its entries
1. 17:20:45.331|ERR|....
2. 17:20:54.715|SYS|.....Logging started for [....] (Date=[07/28/2014], ...
3. 17:20:54.716|SYS....
and so on
I have the date in only one line of the logs. based on that i want to create a timestamp such as that logging date in logs + the time in each entry
Iam able to get the time in each entry. i can get the log_message => " Logging started for [....] (Date=[07/28/2014], ..." as one entry.
Is it possible to get the date from this entry and modify all other entry's timestamp?
how can I add time and the date and modify the timestamp?
Any help will be appreciated as iam new to logstash
My filter in logstash conf
filter {
grok { match => [ "message", "%{TIME:time}\|%{WORD:Message_type}\|%{GREEDYDATA:Component}\|%{NUMBER:line_number}\| %{GREEDYDATA:log_message}"]
}
date {
match => ["timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ] => need to modify this as date+%{time}
}
}
time field has milliseconds also.

Answer 1

Your options are:
Change how things are logged to get the date included
Write something to fix the logs before they are picked up by logstash (ie something that looks for the entry any modifies the log)
use the memorize plugin that I wrote (and I submitted a pull request for to try and get it in a future version).
The plugin is detailed in this answer. The caveat with this solution is that if the plugin misses the line that has the date, you'll have issues with the remainder of the file. This could happen if you restart logstash, so you'll need to add in some logic to handle this -- in this case below, I assume that if it hasn't seen the date, it's today.
An implementation using the memorize plugin would look like this:
filter {
if ([message] =~ /Date=/) {
grok { match => [ "message", "Date=%{DATE:date}" ] }
}
# either add the field date to the saved date or pull the date from the saved data
memorize { fields => ["date"] }
# if we still don't have a date, lets just assume it's today
if ([date] == '') {
ruby {
code => 'event["date"]=ime.now.strftime("%m/%d/%Y")'
}
}
if ([message] !~ /Date=/) {
# grok to parse message
grok { match => [ "message", "%{TIME:time}\|%{WORD:Message_type}\|%{GREEDYDATA:Component}\|%{NUMBER:line_number}\| %{GREEDYDATA:log_message}"]
# now add in date
mutate {
add_field => {
datetime => "%{date} %{time}"
}
}
}
}
(This example has not been tested, so there may be syntax/logic errors, but it should get you down the right path).