logstash date format, not getting any data - logstash

Question

Hi following are my configuration in centos6 logstash server. I am using logstash 1.4.2 and elasticsearch 1.2.1. I am forwarding logs from /var/log/messages and /var/log/secure and there time format is "Sep 1 22:15:34"
1. input.conf
input {
lumberjack {
port => 5000
type => "logs"
ssl_certificate => "certs/logstash-forwarder.crt"
ssl_key => "private/logstash-forwarder.key"
}
}
2. filter.conf
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{#timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
locale => "en" // possibly this didn't work in logstash 1.4.2
match => ["syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss", "ISO8601"]
add_field => { "debug" => "timestampMatched"}
timezone => "UTC"
}
ruby { code => "event['#timestamp'] = event['#timestamp'].getlocal"} //I saw somewhere instead of "locale => en " we have to use this in logstash 1.4.2
mutate { replace => [ "syslog_timestamp", "%{syslog_timestamp} +0545" ] } //this probably won't work and give date parsing error
}
}
3. output.conf
output {
elasticsearch { host => "logstash_server_ip" }
stdout { codec => rubydebug }
}
Below is logstash-forwarder conf in all client server
{
"network": {
"servers": [ "logstash_server_ip:5000" ],
"timeout": 15,
"ssl ca": "certs/logstash-forwarder.crt"
},
"files": [
{
"paths": [
"/var/log/messages",
"/var/log/secure"
],
"fields": { "type": "syslog" }
}
]
}
Here is the problem. I am forwarding logs from 5 servers with different timezone eg: EDT, NDT, NST, NPT. The logstash_server timezone is in NPT (Nepali time) [UTC + 5:45]
All server giving following
2014/09/02 08:09:02.204882 Setting trusted CA from file: certs/logstash-forwarder.crt
2014/09/02 08:09:02.205372 Connecting to logstash_server_ip:5000 (logstash_server_ip)
2014/09/02 08:09:02.205600 Launching harvester on new file: /var/log/secure
2014/09/02 08:09:02.205615 Starting harvester at position 5426763: /var/log/messages
2014/09/02 08:09:02.205742 Current file offset: 5426763
2014/09/02 08:09:02.279715 Starting harvester: /var/log/secure
2014/09/02 08:09:02.279756 Current file offset: 12841221
2014/09/02 08:09:02.638448 Connected to logstash_server_ip
2014/09/02 08:09:09.998098 Registrar received 1024 events
2014/09/02 08:09:15.189079 Registrar received 1024 events
which I hope is good but only one with timezone NPT is forwarding the logs and I am able to see it in kibana, all others gave me above logs but I am not able to see it in kibana. I think the problem is in DATE since it is not able to parse the date from different server. Also there is no log showing error in logstash.
How do I solve the problem in this case?

Answer 1

In logstash-forwarder config change
"fields": { "type": "syslog" }
to
"fields": { "type": "syslog", "syslog_timezone": "Asia/Kathmandu" }
And change filter.conf to
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{#timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
# Set timezone appropriately
if [syslog_timezone] in [ "Asia/Kathmandu" ] {
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
remove_field => [ "syslog_timezone" ]
timezone => "Asia/Kathmandu"
}
} else if [syslog_timezone] in [ "America/Chicago", "US/Central" ] {
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
remove_field => [ "syslog_timezone" ]
timezone => "America/Chicago"
}
} else if [syslog_timezone] =~ /.+/ {
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
add_tag => [ "unknown_timezone" ]
timezone => "Etc/UTC"
}
} else {
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
timezone => "Etc/UTC"
}
}
}
}