Logstash not running with filebeat - logstash

Question

I'm following the digital ocean guide to installing the ELK stack. It was running, but now I am running into problems with logstash.
When I run to debug, I get this::
tail -f /var/log/logstash/logstash.log
{:timestamp=>"2017-02-10T10:38:05.169000-0500", :message=>"Error: Expected one of #, input, filter, output at line 1, column 1 (byte 1) after "}
{:timestamp=>"2017-02-10T10:38:05.201000-0500", :message=>"You may be interested in the '--configtest' flag which you can\nuse to validate logstash's configuration before you choose\nto restart a running system."}
{:timestamp=>"2017-02-14T10:51:46.921000-0500", :message=>"fetched an invalid config", :config=>"input {\n lumberjack {\n port => 5044\n type => syslog\n ssl_certificate => \"/etc/pki/tls/certs/logstash-forwarder.crt\"\n ssl_key => \"/etc/pki/tls/private/logstash-forwarder.key\"\n }\n}\n\ninput {\n beats {\n port => 5044\n ssl => true\n ssl_certificate => \"/etc/pki/tls/certs/logstash-forwarder.crt\"\n ssl_key => \"/etc/pki/tls/private/logstash-forwarder.key\"\n }\n}\n\nfilter {\n if [type] == \"syslog\" {\n grok {\n match => { \"message\" => \"%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\\[%{POSINT:syslog_pid}\\])?: %{GREEDYDATA:syslog_message}\" }\n add_field => [ \"received_at\", \"%{#timestamp}\" ]\n add_field => [ \"received_from\", \"%{host}\" ]\n }\n syslog_pri { }\n date {\n match => [ \"syslog_timestamp\", \"MMM d HH:mm:ss\", \"MMM dd HH:mm:ss\" ]\n }\n }\n}\n\ninput{\n beats {\n port => 5044\n }\n}\n\noutput {\n elasticsearch {\n hosts => \"10.84.234.224:9200\"\n manage_template => false\n index => \"%{[#metadata][beat]}-%{+YYYY.MM.dd}\"\n document_type => \"%{[metadata][type]}\"\n }\n}\n\noutput {\n elasticsearch {\n hosts => [\"localhost:9200\"]\n sniffing => true\n manage_template => false\n index => \"%{[#metadata][beat]}-%{+YYYY.MM.dd}\"\n document_type => \"%{[#metadata][type]}\"\n }\n\u007Fstdout { codec => rubydebug }\n}\n\noutput {\n elasticsearch { host => localhost }\n stdout { codec => rubydebug }\n}\n\ninput {\n tcp {\n port => 5400\n type => syslog\n }\n udp {\n port => 5400\n type => syslog\n }\n}\n\nfilter {\n if [type] == \"syslog\" {\n grok {\n match => { \"message\" => \"%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\\[%{POSINT:syslog_pid}\\])?: %{GREEDYDATA:syslog_message}\" }\n add_field => [ \"received_at\", \"%{#timestamp}\" ]\n add_field => [ \"received_from\", \"%{host}\" ]\n }\n date {\n match => [ \"syslog_timestamp\", \"MMM d HH:mm:ss\", \"MMM dd HH:mm:ss\" ]\n }\n }\n}\n\noutput {\n elasticsearch { hosts => [\"localhost:9200\"] }\n stdout { codec => rubydebug }\n}\n\n\n", :reason=>"Expected one of #, if, \", ', } at line 56, column 1 (byte 1278) after output {\n elasticsearch {\n hosts => [\"localhost:9200\"]\n sniffing => true\n manage_template => false\n index => \"%{[#metadata][beat]}-%{+YYYY.MM.dd}\"\n document_type => \"%{[#metadata][type]}\"\n }\n", :level=>:error}
However, I tried to delete my lumberjack confs and I can't find a problem in any of my configs. Does anyone know if there is a way to do a fresh reinstall of logstash. I think I've messed with too many confs, logstash, logstash-forwarder, etc.
Conf file:
input {
beats {
port => 5044
ssl => true
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{#timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
output {
elasticsearch {
host => ["localhost:9200"]
sniffing => true
manage_template => false
index => "%{[#metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[#metadata][type]}"
}
}
UPDATE
When I run a configtest, I get this error:
sudo service logstash configtest
/etc/init.d/logstash: 156: /etc/init.d/logstash: /opt/logstash/bin/logstash: not found

Answer 1

Note that logstash as a service is reading all files from /etc/logstash/conf.d, so if you edit just one, that doesn't mean it'll reload and fix the other files.
I'm not sure what file you are editing, but you have many files that logstash is appending together.
For example, one file has this
input {
tcp {
port => 5400
type => syslog
}
udp {
port => 5400
type => syslog
}
}
Another has this
input{
beats {
port => 5044
}
}
And so on... so, check all files in /etc/logstash/conf.d for errors individually using logstash --configtest, as the error says.
If you look at the end of your error message...
:reason=>"Expected one of #, if, \", ', } at line 56, column 1 (byte 1278) after ...
output {
elasticsearch {
hosts => [\"localhost:9200\"]
sniffing => true
manage_template => false
index => \"%{[#metadata][beat]}-%{+YYYY.MM.dd}\"
document_type => \"%{[#metadata][type]}\"
}
Looks like you are missing a closing bracket for output {}, or something on line 56 is bad / missing, it thinks.
I'm not sure what is wrong with configtest, but you do not need a service for that. The command logstash --configtest can test your files itself.