I have a global string variable that's actually an array of names:
"mongo1,mongo2,mongo3"
What I'm doing here is splitting them into an array using the "," as a delimiter and then feeding that array into a define to create all instances I need.
Problem is, every instance has a different port. I made a new stdlib function to get the index of a name in an array, and am feeding that to the port parameter.
This seems bad and I don't like having to alter stdlib.
So I'm wondering how I could do this using something like a nx2 array?
"mongo1,port1;mongo2,port2;mongo3,port3"
or two arrays
"mongo1,mongo2,mongo3" and "port1,port2,port3"
class site::mongomodule {
class { 'mongodb':
package_ensure => '2.4.12',
logdir => '/var/log/mongodb/'
}
define mongoconf () {
$index = array_index($::site::mongomodule::mongoReplSetName_array, $name)
mongodb::mongod { "mongod_${name}":
mongod_instance => $name,
mongod_port => 27017 + $index,
mongod_replSet => 'Shard1',
mongod_shardsvr => 'true',
}
}
$mongoReplSetName_array = split(hiera('site::mongomodule::instances', undef), ',')
mongoconf { $mongoReplSetName_array: }
}
the module I'm using is this one:
https://github.com/echocat/puppet-mongodb
using puppet 3.8.0
Hiera can give you a hash when you lookup a key, so you can have something like this in hiera:
mongoinstances:
mongo1:
port: 1000
mongo2:
port: 1234
Then you lookup the key in hiera to get the hash, and pass it to the create_resources function which will create one instance of a resource per entry in the hash.
$mongoinstances = hiera('mongoinstances')
create_resources('mongoconf', $mongoinstances)
You will need to change mongoconf for this to work by adding a $port parameter. Each time you want to pass an additional value from hiera, just add it as a parameter to your defined type.
If you are using puppet >= 4.0, use puppet hashes with each function.
Define hash e.g:
$my_hash = { mongo1 => port1,
mongo2 => port2, }
Next use each function on it e.g:
$my_hash.each |$key, $val| { some code }.
More about iteration in puppet here.
Related
I am creating a class which creates a ssh keys pair for users.
In Hiera it looks like this:
ssh::default_user_keys::entries:
root:
privkey: ENC[PKCS7,MIIH/QYJKoZIhvcNAQcDoIIH7jCCB+oCAQAx...]
pubkey: BiQYJKoZIhvcNAQcDoIIBejCCAXYC...
user1:
privkey: ENC[PKCS7,MIIH/Bf0mv0aa7JRbEWWONfVMJBOX2I7x...]
keydir: /srv/data/user1/.ssh
user2:
privkey: ENC[PKCS7,MIIBiQYJKoZIhvcNAQcDoIIBejCCAXYCAQAx...]
keytype: id_ecdsa
If some values are not specified in Hiera, they must be picked up from default values in the manifest.
So my question is how to write the manifest correctly?
This is my pseudo code I try to make working:
class ssh::default_user_keys_refact (
Hash $entries = lookup('ssh::default_user_keys_refact::entries', "merge" => 'hash'),
) {
$entries.each |String $user, Hash $item = {} | {
set_default { $item:
privkey => undef,
pubkey => undef,
keydir => "/home/${user}/.ssh",
keytype => id_rsa,
comment => undef,
}
$sshpriv = $item[privkey]
file { "${keydir}/$[keytype}":
ensure => file,
owner => $user,
mode => '0600',
content => $sshpriv,
}
}
}
Do you have ideas how to make this pseudo code working? Or in other words, how to set up default values for nested hiera hash in puppet manifest?
puppet versions is 6.
Thank you in advance.
Or in other words, how to set up default values for nested hiera hash in puppet manifest?
The fact that the hashes for which you want to provide defaults are nested inside others in the Hiera data isn't much relevant, because at the point of use they have been plucked out. With that being the case, you can accomplish this fairly easily by leveraging Puppet (not Hiera) hash merging. It might look something like this:
$entries.each |String $user, Hash $props| {
# Merge with default values
$full_props = {
'keydir' => "/home/${user}/.ssh",
'keytype' => 'id_rsa'
} + $props
file { "${full_props['keydir']}/${full_props['keytype']}":
ensure => 'file',
owner => $user,
mode => '0600',
content => $full_props['privkey'],
}
}
That's by no means the only way, but it's clean and clear, and it scales reasonably well.
I have following values defined in hieradata in puppet
globals::myservers:
- fqdn: "host1.example.com"
port: 22
protocol: "ssh"
- fqdn: "host2.example.com"
port: 22
protocol: "ssh"
and I would like it to print the following values with above data
my_servers = host1.example.com host2.example.com
Your hiera data looks like you have an array called myservers with each element containing a hash.
# Build a =data structure the same as you have in hiera.
$myservers = [ {fqdn => 'host1', port => '22' }, {fqdn => 'host2', port => '22' } ]
# Use the map function to extract all the values for fqdn into an array.
$fqdnarray = $myservers.map |$item| { $item[fqdn] }
# use the join function to turn the array into a string.
$fqdnstring = join($fqdnarray,',')
# Print the output
notify { 'Hosts message':
message => "my_servers = ${fqdnstring}"
}
You should be able to drop the above code straight into a file, let's call it test.pp and then run puppet apply test.pp on a machine with a Puppet agent on to see what it does.
The first thing to do is to load the data into Puppet. The specified Hiera key is appropriate for automatically binding the data to a parameter $myservers of a class named globals:
class globals(
Array[Hash[String, String]] $myservers
) {
# ... do something with $myservers ...
}
If you do not already have such a class, however, then you can alternatively look up the data from within any other class or any defined type:
class mymodule::any_class {
$myservers = lookup('globals::myservers', Array[Hash[String, String]])
# ... do something with $myservers ...
}
Having obtained the data, the question is how to write an ERB template that iterates over it and formats it as desired. This is not hard, provided that you know some Ruby (that's what the 'R' stands for in ERB, after all). The wanted template might be something like this ...
my_servers = <%= #myservers.map { |server| server['fqdn'] } .join(' ') %>
Inside the template, the Ruby #myservers variable is bound to the $myservers Puppet variable from the local scope from which the template is evaluated. The scriptlet extracts the 'fqdn' members of all the array elements and joins their string representations together with space separators.
As for invoking the template evaluation, you have your choice of putting the template into a separate file and using the template() function, or putting it directly into your manifest and using inline_template(). For example,
$formatted_server_list = inline_template(
"my_servers = <%= #myservers.map { |server| server['fqdn'] } .join(' ') %>")
notify { "${formatted_server_list}": }
In Python 3, I use serialize data to JSON like so:
import json
config = {
'server' : {
'host' : '127.0.0.1',
'port' : 12345
}
}
serialized_config = json.dumps(config)
Due to json.dumps and the fact that the format of the JSON is fixed, I cannot alter the struct
of the python data arbitralily, e.g. host needs to be a string when config is passed to json.dumps. I cannot wrap it in a list like 'host' : ['1.1.1.1]' e.g.
Now the data is way more complex than in this example, but mostly redundant (e.g. there is only one 'host' that is constant but appears in various places throughout the config.
Now I like to define some template like so:
template = {
'server' : {
'host' : SERVER_IP,
'port' : 12345
}
}
And afterwards replace all occurences of e.g. SERVER_IP with the actual server ip etc.
The point is that I don't know the value of SERVER_IP when I define the template.
The easy way would be to just shift the atomic string into a non-atomic python struct, e.g.:
server = {
'host' : '1.1.1.1',
'port' : 12345
}
template = {
'server' : server
}
...
# Here the actual value of 'host' has been set
server['host'] = '192.168.0.1'
print(json.dumps(template))
This dumps
'{"server": {"host": "192.168.0.1", "port": 12345}}'
As wanted.
However, since port etc. also change, I would prefer something like (pseudo-code):
server_ip = '1.1.1.1' # not the actual value
template = {
'server' : {
'host' : ref(server_ip), # Don't copy the string, but either delay the evaluation or somehow create a ref to the string
'port' : 12345
}
}
...
server_ip = '192.168.0.1'
I am aware that I could write code that just traverses the python data and 'manually' replaces certain strings with different values.
However, I am curious if there is some better solution.
E.g. I played around with facilitating __getattribute__ like so:
class DelayedEval:
def __init__(self, d):
self.data = d
def __getattribute__(self, key):
# Recursive matters let aside for sake of simplicity...
data = object.__getattribute__(self, 'data')
v = data[key]
try:
v = v()
except:
pass
return v
# possibly overwrite __getitem__() etc as well ...
server_ip = '1.1.1.1'
template = {
'server' : {
'host' : lambda: server_ip, # prevents 'premature evalutation'
'port' : 12345
}
}
...
server_ip = '192.168.0.1'
print(json.dumps(DelayedEval(template))) # Does not work
I am not asking for fixing this code, I am asking for any solution that solves my issue, no matter how, as long as You deem it 'better' than the bute-force approach of traverse and replace manually.
Fixing this code might solve my problem, but there might be better solutions (or none?).
I am aware that this is a rather complicated question to pose, and probably for my usecase overly complex, however, I am curious whether there would be a viable solution to this problem apart from the brute-force 'search-and-replace' approach...
Is there some way to define 'self-completing template structs' like that in Python?
One easy approach is to make the template itself a lambda function:
template = lambda: {
'server' : {
'host' : server_ip,
'port' : 12345
}
}
...
server_ip = '192.168.0.1'
print(json.dumps(template()))
I'm trying to automate the Prometheus node_exporter and my Prometheus Server.
For the node_exporter I've written a module to install all the needed packages, set the $::ipaddress based on facter and some more..
Now I'd like to make sure that the collected informations ($hostname, $job_name, [...]) from the applying node are exported into the respective remote Prometheus configfile, but I want to have this step done asynchronously, so for example with a puppet agent run afterwards on the Prometheus Server.
I've tried to orientate the classes towards the puppetlabs/logrotate module, which is basically doing the following:
logrotate/init.pp
class logrotate (
String $ensure = present,
Boolean $hieramerge = false,
Boolean $manage_cron_daily = true,
Boolean $create_base_rules = true,
Boolean $purge_configdir = false,
String $package = 'logrotate',
Hash $rules = {},
) {
do some stuff
}
logrotate/rules.pp
class logrotate::rules ($rules = $::logrotate::rules){
#assert_private()
create_resources('logrotate::rule', $rules)
}
logrotate/rule.pp
define logrotate::rule(
Pattern[/^[a-zA-Z0-9\._-]+$/] $rulename = $title,
Enum['present','absent'] $ensure = 'present',
Optional[Logrotate::Path] $path = undef,
(...)
) {
do some stuff
}
Shortened my ni_trending (node_exporter) & ni_prometheus modules currently look very similar to logrotate:
ni_trending/init.pp
class ni_trending (
$hostname = $::fqdn,
$listen_address = $::ipaddress,
$listen_port = 51118,
) {
) inherits ni_trending::params {
anchor { 'ni_trending::start': }
->class { 'ni_trending::package': }
->class { 'ni_trending::config':
(...)
listen_address => $listen_address,
listen_port => $listen_port,
(...)
}
->class { 'ni_trending::service': }
->class { ' ni_trending::prometheus':
(...)
hostname => $hostname,
listen_port => $listen_port,
(...)
}
->anchor { 'ni_trending::end': }
}
ni_trending/prometheus.pp
class ni_trending::prometheus (
Hash $options = {},
) {
ni_prometheus::nodeexporterrule { 'node_exporter' :
ensure => pick_default($options['ensure'], 'present'),
hostname => pick_default($options['hostname'], $ni_trending::hostname),
listen_port => pick_default($options['hostname'], $ni_trending::listen_port),
}
}
ni_prometheus/nodeexporterrules.pp
class ni_prometheus::nodeexporterrules ($rules = $::ni_prometheus::nodeexporterrules) {
create_resources('ni_prometheus::nodeexporterrule', $nodeexporterrules)
}
ni_prometheus/nodeexporterrule.pp
define ni_prometheus::nodeexporterrule (
$job_name = $title,
Enum['present','absent'] $ensure = 'present',
$hostname = $hostname,
$listen_port = $listen_port,
) {
file_line { "prometheus-${job_name}" :
path => "/etc/prometheus/${job_name}.list",
after => 'hosts:',
line => "${hostname}:${listen_port}",
}
}
But this will just work when I apply the node_exporter locally on the Prometheus Master - not in the case that an external machine has the ni_trending::prometheus class included, which makes sense to me - because it clearly feels that something is missing. :-) How can I get this working?
Thanks!
This sounds like a job for exported resources (that makes two in one day!). This is a facility for one node's catalog building to generate resources that can be applied to other nodes (and also, optionally, to the exporting node itself). I'm still not tracking the details of what you want to manage where, so here's a more generic example: maintaining a local hosts file.
Generic example
Suppose we want to automatically manage a hosts file listing all our nodes under management. Puppet has a built-in resource, Host, representing one entry in a hosts file. We make use of that by having every node under management export an appropriate host resource. Something like this would go inside a class included on every node:
##host { "$hostname": ip => $ipaddress; }
The ## prefix marks the resource as exported. It is not applied to the current target node, unless by the mechanism I will describe in a moment. the $hostname and $ipaddress are just facts presented by the target node, and they are resolved in that context. Note, too, that the resource title is globally unique: each target node has a different hostname, therefore all the exported Host resources that apply to different target nodes will have distinct titles.
Then, separately, every node that wants all those Host entries applied to it will import them in its own catalog by using an exported resource collector:
<<|Host|>>
The nodes that export those resources can also collect some or all of them. Additionally, there are ways to be more selective about which resources are collected; see the link above.
I would like to run the following code in a sequential order so that the servers_string variable is computed before the script execution.
Unfortunately puppet failed with the following error :
Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Evaluation Error: Illegal relationship operand, can not form a relationship with a Hash. A Catalog type is required.
The code snippet :
$servers = [{ name => 'toto', ip => '10.0.0.1'}, { name => 'titi', ip => '10.0.0.2' }]
$servers.each | Hash $server | {
if $servers_string != "" {
$servers_string = "${servers_string},"
}
$name = $server['name']
$servers_string = "${servers_string}${name}"
}->
file { '/my/path/myscript.sh':
ensure => file,
mode => '0700',
owner => 'root',
group => 'root',
source => "puppet:///modules/${module_name}/install.sh --servers '${servers_string}'"
}
Any idea ? Thanks
Resource relationships in general and the chain operators in particular are about the order in which resources are applied to the node. They have nothing whatever to do with the order in which the catalog builder evaluates manifest files.
Manifests are always evaluated in order, left-to-right, top-to-bottom. You do not need to use chain operators to ensure that, nor can you use them to change it. Just drop the chain operator, and you'll be fine (at least in this regard).