How to reference environment variables in logstash configuration file? - logstash

Question

Is it possible to reference environment variables in logstash configuration?
In my case, i want to make my elasticsearch address configurable that i have set in the environment.

Answer 1

With logstash 2.3, you can set environment variable references into Logstash plugins configuration using ${var} or $var.
https://www.elastic.co/guide/en/logstash/current/environment-variables.html
Before logstash 2.3, you can use the "environment" filter plugin which is community maintained.
Documentation at : https://www.elastic.co/guide/en/logstash/current/plugins-filters-environment.html#plugins-filters-environment-add_field_from_env
How to install this plugin:
$LOGSTASH_HOME/bin/plugin install logstash-filter-environment
Source code at : https://github.com/logstash-plugins/logstash-filter-environment
The main part is:
# encoding: utf-8
require "logstash/filters/base"
require "logstash/namespace"
# Set fields from environment variables
class LogStash::Filters::Environment < LogStash::Filters::Base
config_name "environment"
# Specify a hash of fields to the environment variable
# A hash of matches of `field => environment` variable
config :add_field_from_env, :validate => :hash, :default => {}
public
def register
# Nothing
end # def register
public
def filter(event)
return unless filter?(event)
#add_field_from_env.each do |field, env|
event[field] = ENV[env]
end
filter_matched(event)
end # def filter
end # class LogStash::Filters::Environment

Answer 2

I can hardly believe, that these are the only solutions left: Hacking logstash or using some kind of templating system to re-write the config.
Actually, I do not want to touch or tweak the config for different deployment-scenarios: All I want is to pass in some parameters to connect logstash to the outside world (e.g. where elasticsearch is located, usernames/credentials to connect to other systems). I googled for an hour now an all I could find were these awkwardly complicated solutions for this very simple and common problem.
I sincerely hope, that someone comes up with a better idea like
%{ENV[ELASTICSEARCH_HOST]}}

Answer 3

That's not directly supported, no.
However, if you're running a version later than 1.4.0, it would be pretty trivial to edit elasticsearch.rb to add this feature. Around line 183:
client_settings["network.host"] = #bind_host if #bind_host
You could tweak it to read an environment variable:
if ENV["ESHOST"].nil? then
client_settings["network.host"] = ENV["ESHOST"]
else
client_settings["network.host"] = #bind_host if #bind_host
end
If you prefer, you can run Logstash with the -e command-line option to pass config via STDIN. You could cat in some file with special tokens that you've replaced with your environment variable(s).

Answer 4

The logstash configuration as of this writing is just a configuration file, but it's not a programing language. Thus, it has a few reasonable "limitations", for example, it cannot reference environment variables, cannot pass parameters it, hard to reuse other configuration file. Those limitations would make the logstash configuration file hard to maintain when the configuration file grows or you want to adjust its behavior on the fly.
My approach is to use template engine to generate the logstash configuration file. I used Jinja2 in Python.
For example, the elastic search output could be templated as
output {
elasticsearch {
index => {{ es_index_name }}
host => {{ es_hostname }}
}
}
Then I wrote a simple python code using Jinja2 to generate the logstash configuration file, and the value of es_index_name and es_hostname could be passed via the Python script argument. See here for Jiaja2 tutorial: http://kagerato.net/articles/software/libraries/jinja-quickstart.html
In this way, a big logstash configuration could be splitted into reusable pieces and its behavior can be adjusted on the fly.

Answer 5

As explained in logstash-issues
Connections are established at plugin registration time (during initialization, as they almost certainly should be), but field interpolation (like %{escluster}) is an event-processing-time operation. As such, host isn't really eligible for this behavior.
So unless input or output plugin will natively supports %{foo} syntax, doing any environment variable evaluation at the stage of event filtering is too late for the input and output plugin to take advantage of it.

Answer 6

.conf file support environment variables.
you just have to export the environment variable:
export EXAMPLE_VAR=123
and use it in the configuration file this way:
${EXAMPLE_VAR}