I have following values defined in hieradata in puppet
globals::myservers:
- fqdn: "host1.example.com"
port: 22
protocol: "ssh"
- fqdn: "host2.example.com"
port: 22
protocol: "ssh"
and I would like it to print the following values with above data
my_servers = host1.example.com host2.example.com
Your hiera data looks like you have an array called myservers with each element containing a hash.
# Build a =data structure the same as you have in hiera.
$myservers = [ {fqdn => 'host1', port => '22' }, {fqdn => 'host2', port => '22' } ]
# Use the map function to extract all the values for fqdn into an array.
$fqdnarray = $myservers.map |$item| { $item[fqdn] }
# use the join function to turn the array into a string.
$fqdnstring = join($fqdnarray,',')
# Print the output
notify { 'Hosts message':
message => "my_servers = ${fqdnstring}"
}
You should be able to drop the above code straight into a file, let's call it test.pp and then run puppet apply test.pp on a machine with a Puppet agent on to see what it does.
The first thing to do is to load the data into Puppet. The specified Hiera key is appropriate for automatically binding the data to a parameter $myservers of a class named globals:
class globals(
Array[Hash[String, String]] $myservers
) {
# ... do something with $myservers ...
}
If you do not already have such a class, however, then you can alternatively look up the data from within any other class or any defined type:
class mymodule::any_class {
$myservers = lookup('globals::myservers', Array[Hash[String, String]])
# ... do something with $myservers ...
}
Having obtained the data, the question is how to write an ERB template that iterates over it and formats it as desired. This is not hard, provided that you know some Ruby (that's what the 'R' stands for in ERB, after all). The wanted template might be something like this ...
my_servers = <%= #myservers.map { |server| server['fqdn'] } .join(' ') %>
Inside the template, the Ruby #myservers variable is bound to the $myservers Puppet variable from the local scope from which the template is evaluated. The scriptlet extracts the 'fqdn' members of all the array elements and joins their string representations together with space separators.
As for invoking the template evaluation, you have your choice of putting the template into a separate file and using the template() function, or putting it directly into your manifest and using inline_template(). For example,
$formatted_server_list = inline_template(
"my_servers = <%= #myservers.map { |server| server['fqdn'] } .join(' ') %>")
notify { "${formatted_server_list}": }
Related
I'm struggling with setting dynamic private dns zone records for multiple private endpoints for same resource in Azure (ACR - Azure Container Registry).
So I have currently setup this simple example. Basically it simulate creation of ACR and building a map of records which need to be registered in the DNS so ACR could be used correclty. The thing here is that it is needed to somehow extract value from inner field of each resource/object (here it is endpoints variable) and then transform it correctly so it would match expected output. This is needed as in my case it is integral part of a module and it needs to be built dynamically based on values of real resources created.
Following example should help anyone who'd like to check this problem out and help a bit.
To verify if you've got correct result just type "terraform refresh" and the output should be printed.
terraform {
backend local {}
}
# simulates creation of multiple Azure Container Registry private endpoints
variable endpoints {
type = map
default = {
"registry" = {
custom_dns_configs = [
{
fqdn = "something.westeurope.data.azurecr.io"
ip_addresses = ["1.1.1.1",]
},
{
fqdn = "something.azurecr.io"
ip_addresses = ["1.1.1.2",]
}
]
},
"registry2" = {
custom_dns_configs = [
{
fqdn = "something.westeurope.data.azurecr.io"
ip_addresses = ["1.1.2.1",]
},
{
fqdn = "something.azurecr.io"
ip_addresses = ["1.1.2.2",]
},
]
},
#...
# "registryN" = {...}
}
}
# Question: How to write for block here to get out of it exactly what's in expected
# result having in mind that there can be multiple of "registry" blocks?
# Below line produce only single result which doesn't match expected result.
output result {
value = {for pe in var.endpoints:
pe.custom_dns_configs[0].fqdn => pe.custom_dns_configs[0].ip_addresses
}
}
# Expected result:
# result = {
# "something.westeurope.data.azurecr.io" = [
# "1.1.1.1",
# "1.1.2.1",
# ]
# "something.azurecr.io" = [
# "1.1.1.2",
# "1.1.2.2",
# ]
# }
Answering my own question:
output result {
value = {for k,v in {
for cdc in flatten(
[for pe in var.endpoints: pe.custom_dns_configs]
):
cdc.fqdn => cdc.ip_addresses...
}:
k => flatten(v)
}
}
Now few words of explanation:
[] and {} - [] produce a tuple which strip "registry" key from incomming map whereas {} would require to produce some kind of dynamic keys
[for pe in var.endpoints: pe.custom_dns_configs] just extract internal field from var.environments for each element of the map. Then flatten is used just to make thing simpler without the need to dig into different levels of nested lists
next for is to build new map for fqdn -> list(ip addresses)
the "..." after cdc.ip_addresses are required. That's notation to allow grouping values by keys. In this case we have at least 2 times same fqdn and by normal means terraform would complain that it cannot create such map when keys are non-uniqe. Adding those 3 dots there enable this grouping.
then top-most for block is used just to flatten whole value output.
Now the problem would be if we would still want to keep "registry", "registry2", "registryN" grouping. For that I haven't found solution yet.
I need to build a list of templatefile's like this:
templatefile("${path.module}/assets/files_eth0.nmconnection.yaml", {
interface-name = "eth0",
addresses = element(values(var.virtual_machines), count.index),
gateway = element(var.gateway, count.index % length(var.gateway)),
dns = join(";", var.dns_servers),
dns-search = var.domain,
}),
templatefile("${path.module}/assets/files_etc_hostname.yaml", {
hostname = element(keys(var.virtual_machines), count.index),
}),
by iterating over a map of maps like the following:
variable templatefiles {
default = {
"files_eth0.nmconnection.yaml" = {
"interface-name" = "eth0",
"addresses" = "element(values(var.virtual_machines), count.index)",
"gateway" = "element(var.gateway, count.index % length(var.gateway))",
"dns" = "join(";", var.dns_servers)",
"dns-search" = "var.domain",
},
"files_etc_hostname.yaml" = {
"hostname" = "host1"
}
}
}
I've done something similar with a list of files:
file("${path.module}/assets/files_90-disable-console-logs.yaml"),
file("${path.module}/assets/files_90-disable-auto-updates.yaml"),
...but would like to expand this to templatefiles (above).
Here's the code I've done for the list of files:
main.tf
variable files {
default = [
"files_90-disable-auto-updates.yaml",
"files_90-disable-console-logs.yaml",
]
}
output "snippets" {
value = flatten(module.ingition_snippets.files)
}
modules/main.tf
variable files {}
resource "null_resource" "files" {
for_each = toset(var.files)
triggers = {
snippet = file("${path.module}/assets/${each.value}")
}
}
output "files" {
value = [for s in null_resource.files: s.triggers.*.snippet]
}
Appreciate any help!
Both of these use-cases can be met without using any resource blocks at all, because the necessary features are built in to the Terraform language.
Here is a shorter way to write the example with static files:
variable "files" {
type = set(string)
}
output "files" {
value = tomap({
for fn in var.files : fn => file("${path.module}/assets/${fn}")
})
}
The above would produce a map from filenames to file contents, so the calling module can more easily access the individual file contents.
We can adapt that for templatefile like this:
variable "template_files" {
# We can't write down a type constraint for this case
# because each file might have a different set of
# template variables, but our later code will expect
# this to be a mapping type, like the default value
# you shared in your comment, and will fail if not.
type = any
}
output "files" {
value = tomap({
for fn, vars in var.template_files : fn => templatefile("${path.module}/assets/${fn}", vars)
})
}
Again, the result will be a map from filename to the result of rendering the template with the given variables.
If your goal is to build a module for rendering templates from a source directory to publish somewhere, you might find the module hashicorp/dir/template useful. It combines fileset, file, and templatefile in a way that is hopefully convenient for static website publishing and similar use-cases. (At the time I write this the module is transitioning from being in my personal GitHub account to being in the HashiCorp organization, so if you look at it soon after you may see some turbulence as the docs get updated, etc.)
In Python 3, I use serialize data to JSON like so:
import json
config = {
'server' : {
'host' : '127.0.0.1',
'port' : 12345
}
}
serialized_config = json.dumps(config)
Due to json.dumps and the fact that the format of the JSON is fixed, I cannot alter the struct
of the python data arbitralily, e.g. host needs to be a string when config is passed to json.dumps. I cannot wrap it in a list like 'host' : ['1.1.1.1]' e.g.
Now the data is way more complex than in this example, but mostly redundant (e.g. there is only one 'host' that is constant but appears in various places throughout the config.
Now I like to define some template like so:
template = {
'server' : {
'host' : SERVER_IP,
'port' : 12345
}
}
And afterwards replace all occurences of e.g. SERVER_IP with the actual server ip etc.
The point is that I don't know the value of SERVER_IP when I define the template.
The easy way would be to just shift the atomic string into a non-atomic python struct, e.g.:
server = {
'host' : '1.1.1.1',
'port' : 12345
}
template = {
'server' : server
}
...
# Here the actual value of 'host' has been set
server['host'] = '192.168.0.1'
print(json.dumps(template))
This dumps
'{"server": {"host": "192.168.0.1", "port": 12345}}'
As wanted.
However, since port etc. also change, I would prefer something like (pseudo-code):
server_ip = '1.1.1.1' # not the actual value
template = {
'server' : {
'host' : ref(server_ip), # Don't copy the string, but either delay the evaluation or somehow create a ref to the string
'port' : 12345
}
}
...
server_ip = '192.168.0.1'
I am aware that I could write code that just traverses the python data and 'manually' replaces certain strings with different values.
However, I am curious if there is some better solution.
E.g. I played around with facilitating __getattribute__ like so:
class DelayedEval:
def __init__(self, d):
self.data = d
def __getattribute__(self, key):
# Recursive matters let aside for sake of simplicity...
data = object.__getattribute__(self, 'data')
v = data[key]
try:
v = v()
except:
pass
return v
# possibly overwrite __getitem__() etc as well ...
server_ip = '1.1.1.1'
template = {
'server' : {
'host' : lambda: server_ip, # prevents 'premature evalutation'
'port' : 12345
}
}
...
server_ip = '192.168.0.1'
print(json.dumps(DelayedEval(template))) # Does not work
I am not asking for fixing this code, I am asking for any solution that solves my issue, no matter how, as long as You deem it 'better' than the bute-force approach of traverse and replace manually.
Fixing this code might solve my problem, but there might be better solutions (or none?).
I am aware that this is a rather complicated question to pose, and probably for my usecase overly complex, however, I am curious whether there would be a viable solution to this problem apart from the brute-force 'search-and-replace' approach...
Is there some way to define 'self-completing template structs' like that in Python?
One easy approach is to make the template itself a lambda function:
template = lambda: {
'server' : {
'host' : server_ip,
'port' : 12345
}
}
...
server_ip = '192.168.0.1'
print(json.dumps(template()))
I'm trying to automate the Prometheus node_exporter and my Prometheus Server.
For the node_exporter I've written a module to install all the needed packages, set the $::ipaddress based on facter and some more..
Now I'd like to make sure that the collected informations ($hostname, $job_name, [...]) from the applying node are exported into the respective remote Prometheus configfile, but I want to have this step done asynchronously, so for example with a puppet agent run afterwards on the Prometheus Server.
I've tried to orientate the classes towards the puppetlabs/logrotate module, which is basically doing the following:
logrotate/init.pp
class logrotate (
String $ensure = present,
Boolean $hieramerge = false,
Boolean $manage_cron_daily = true,
Boolean $create_base_rules = true,
Boolean $purge_configdir = false,
String $package = 'logrotate',
Hash $rules = {},
) {
do some stuff
}
logrotate/rules.pp
class logrotate::rules ($rules = $::logrotate::rules){
#assert_private()
create_resources('logrotate::rule', $rules)
}
logrotate/rule.pp
define logrotate::rule(
Pattern[/^[a-zA-Z0-9\._-]+$/] $rulename = $title,
Enum['present','absent'] $ensure = 'present',
Optional[Logrotate::Path] $path = undef,
(...)
) {
do some stuff
}
Shortened my ni_trending (node_exporter) & ni_prometheus modules currently look very similar to logrotate:
ni_trending/init.pp
class ni_trending (
$hostname = $::fqdn,
$listen_address = $::ipaddress,
$listen_port = 51118,
) {
) inherits ni_trending::params {
anchor { 'ni_trending::start': }
->class { 'ni_trending::package': }
->class { 'ni_trending::config':
(...)
listen_address => $listen_address,
listen_port => $listen_port,
(...)
}
->class { 'ni_trending::service': }
->class { ' ni_trending::prometheus':
(...)
hostname => $hostname,
listen_port => $listen_port,
(...)
}
->anchor { 'ni_trending::end': }
}
ni_trending/prometheus.pp
class ni_trending::prometheus (
Hash $options = {},
) {
ni_prometheus::nodeexporterrule { 'node_exporter' :
ensure => pick_default($options['ensure'], 'present'),
hostname => pick_default($options['hostname'], $ni_trending::hostname),
listen_port => pick_default($options['hostname'], $ni_trending::listen_port),
}
}
ni_prometheus/nodeexporterrules.pp
class ni_prometheus::nodeexporterrules ($rules = $::ni_prometheus::nodeexporterrules) {
create_resources('ni_prometheus::nodeexporterrule', $nodeexporterrules)
}
ni_prometheus/nodeexporterrule.pp
define ni_prometheus::nodeexporterrule (
$job_name = $title,
Enum['present','absent'] $ensure = 'present',
$hostname = $hostname,
$listen_port = $listen_port,
) {
file_line { "prometheus-${job_name}" :
path => "/etc/prometheus/${job_name}.list",
after => 'hosts:',
line => "${hostname}:${listen_port}",
}
}
But this will just work when I apply the node_exporter locally on the Prometheus Master - not in the case that an external machine has the ni_trending::prometheus class included, which makes sense to me - because it clearly feels that something is missing. :-) How can I get this working?
Thanks!
This sounds like a job for exported resources (that makes two in one day!). This is a facility for one node's catalog building to generate resources that can be applied to other nodes (and also, optionally, to the exporting node itself). I'm still not tracking the details of what you want to manage where, so here's a more generic example: maintaining a local hosts file.
Generic example
Suppose we want to automatically manage a hosts file listing all our nodes under management. Puppet has a built-in resource, Host, representing one entry in a hosts file. We make use of that by having every node under management export an appropriate host resource. Something like this would go inside a class included on every node:
##host { "$hostname": ip => $ipaddress; }
The ## prefix marks the resource as exported. It is not applied to the current target node, unless by the mechanism I will describe in a moment. the $hostname and $ipaddress are just facts presented by the target node, and they are resolved in that context. Note, too, that the resource title is globally unique: each target node has a different hostname, therefore all the exported Host resources that apply to different target nodes will have distinct titles.
Then, separately, every node that wants all those Host entries applied to it will import them in its own catalog by using an exported resource collector:
<<|Host|>>
The nodes that export those resources can also collect some or all of them. Additionally, there are ways to be more selective about which resources are collected; see the link above.
I have a global string variable that's actually an array of names:
"mongo1,mongo2,mongo3"
What I'm doing here is splitting them into an array using the "," as a delimiter and then feeding that array into a define to create all instances I need.
Problem is, every instance has a different port. I made a new stdlib function to get the index of a name in an array, and am feeding that to the port parameter.
This seems bad and I don't like having to alter stdlib.
So I'm wondering how I could do this using something like a nx2 array?
"mongo1,port1;mongo2,port2;mongo3,port3"
or two arrays
"mongo1,mongo2,mongo3" and "port1,port2,port3"
class site::mongomodule {
class { 'mongodb':
package_ensure => '2.4.12',
logdir => '/var/log/mongodb/'
}
define mongoconf () {
$index = array_index($::site::mongomodule::mongoReplSetName_array, $name)
mongodb::mongod { "mongod_${name}":
mongod_instance => $name,
mongod_port => 27017 + $index,
mongod_replSet => 'Shard1',
mongod_shardsvr => 'true',
}
}
$mongoReplSetName_array = split(hiera('site::mongomodule::instances', undef), ',')
mongoconf { $mongoReplSetName_array: }
}
the module I'm using is this one:
https://github.com/echocat/puppet-mongodb
using puppet 3.8.0
Hiera can give you a hash when you lookup a key, so you can have something like this in hiera:
mongoinstances:
mongo1:
port: 1000
mongo2:
port: 1234
Then you lookup the key in hiera to get the hash, and pass it to the create_resources function which will create one instance of a resource per entry in the hash.
$mongoinstances = hiera('mongoinstances')
create_resources('mongoconf', $mongoinstances)
You will need to change mongoconf for this to work by adding a $port parameter. Each time you want to pass an additional value from hiera, just add it as a parameter to your defined type.
If you are using puppet >= 4.0, use puppet hashes with each function.
Define hash e.g:
$my_hash = { mongo1 => port1,
mongo2 => port2, }
Next use each function on it e.g:
$my_hash.each |$key, $val| { some code }.
More about iteration in puppet here.