how to implement the unit or integration tests for logstash configuration? - logstash

Question

With the logstash 1.2.1 one can now have conditional to do various stuff. Even the earlier version's conf file can get complicated if one is managing many log files and implement metric extraction.
After looking at this comprehensive example, I really wondered my self, how can I detect any breakages in this configuration?
Any ideas.

Answer 1

For a syntax check, there is --configtest:
java -jar logstash.jar agent --configtest --config <yourconfigfile>
To test the logic of the configuration you can write rspec tests. This is an example rspec file to test a haproxy log filter:
require "test_utils"
describe "haproxy logs" do
extend LogStash::RSpec
config <<-CONFIG
filter {
grok {
type => "haproxy"
add_tag => [ "HTTP_REQUEST" ]
pattern => "%{HAPROXYHTTP}"
}
date {
type => 'haproxy'
match => [ 'accept_date', 'dd/MMM/yyyy:HH:mm:ss.SSS' ]
}
}
CONFIG
sample({'#message' => '<150>Oct 8 08:46:47 syslog.host.net haproxy[13262]: 10.0.1.2:44799 [08/Oct/2013:08:46:44.256] frontend-name backend-name/server.host.net 0/0/0/1/2 404 1147 - - ---- 0/0/0/0/0 0/0 {client.host.net||||Apache-HttpClient/4.1.2 (java 1. 5)} {text/html;charset=utf-8|||} "GET /app/status HTTP/1.1"',
'#source_host' => '127.0.0.1',
'#type' => 'haproxy',
'#source' => 'tcp://127.0.0.1:60207/',
}) do
insist { subject["#fields"]["backend_name"] } == [ "backend-name" ]
insist { subject["#fields"]["http_request"] } == [ "/app/status" ]
insist { subject["tags"].include?("HTTP_REQUEST") }
insist { subject["#timestamp"] } == "2013-10-08T06:46:44.256Z"
reject { subject["#timestamp"] } == "2013-10-08T06:46:47Z"
end
end
This will, based on a given filter configuration, run input samples and test if the expected output is produced.
To run the test, save the test as haproxy_spec.rb and run `logstash rspec:
java -jar logstash.jar rspec haproxy_spec.rb
There are lots of spec examples in the Logstash source repository.

Answer 2

since logstash has been upgraded and now the command will be something like (give the folder)
/opt/logstash/bin/logstash agent --configtest -f /etc/logstash/logstash-indexer/conf.d
If you see some warning, but the error message is mixed together, and you didn't know which one have issue. You have to check its file one by one