I am using a Hiera hash to hold some token values which are host-specific. the keys within the hash correspond to the hostname/certname of the node(s) that'll be classified with the profile module that calls the hash value. However, when I apply the module, the value which corresponds to the hash key for the host is always null. Here's the code I'm working with.
in hiera-file.yaml
token_lookup:
host-name1: 'abcdef123'
host-name2: 'abbcde456'
and in profile.pp
$_tokens = hiera_hash('token_lookup', undef, 'hiera-file')
$_specific_token = $_tokens["${::hostname}"] <== never gets a value
I'm certain that the hostname matches the key in the hash. The question is, what's the right syntax here for getting the value from the hiera-file to populate properly? Thank you in advance for the advice.
edit: I believe I have discovered an issue when the hash key has a literal '-' character in it, as many hostnames do. I updated the hash to show keys with dashes in it, and should now ask a more specific question: I see dozens of articles about how to escape characters in the values of hashes by using double quotes, but I don't see anything - even on yaml.org - about how to escape the character if it appears as part of the key. Any tips on this issue? A YAML parser shows that this is valid syntactically, but I believe it is treating the '-' as a collection marker instead of a literal character.
Your code is correct, I test it as below, seems it didn't target the right yaml file in your environment. Check the hierarchy setting, and put the token key-value in the right place.
If I put the yaml file to global.yaml(If hiera can't find the key, it will always go to the last one in my hiera.yaml setting)
I rebuilt it with simplest setting:
$ cat /etc/hiera.yaml:
---
:backends:
- yaml
:hierarchy:
- defaults
- "%{clientcert}"
- "%{environment}"
- global
:yaml:
# datadir is empty here, so hiera uses its defaults:
# - /var/lib/hiera on *nix
# - %CommonAppData%\PuppetLabs\hiera\var on Windows
# When specifying a datadir, make sure the directory exists.
:datadir:
$ cat /var/lib/hiera/global.yaml
token_lookup:
host-name1: 'abcdef123'
host-name2: 'abbcde456'
$ cat profile.pp
$_tokens = hiera_hash('token_lookup', undef, 'hiera-file')
notice ("tokens is $_tokens")
$_specific_token = $_tokens["${::hostname}"]
notice ("token is $_specific_token ")
Then I run puppet apply, I can see the result
$ FACTER_hostname='host-name1' puppet apply profile.pp --hiera_config /etc/hiera.yaml
Notice: Scope(Class[main]): tokens is host-name1abcdef123host-name2abbcde456
Notice: Scope(Class[main]): token is abcdef123
Notice: Compiled catalog for host-name1 in environment production in 0.04 seconds
Notice: Finished catalog run in 0.07 seconds
$ FACTER_hostname='host-name2' puppet apply profile.pp --hiera_config /etc/hiera.yaml
Notice: Scope(Class[main]): tokens is host-name1abcdef123host-name2abbcde456
Notice: Scope(Class[main]): token is abbcde456
Notice: Compiled catalog for host-name2 in environment production in 0.04 seconds
Notice: Finished catalog run in 0.02 seconds
root#ac976d6d79fb:~#
I think hiera_hash is not what you want.
hiera_hash:
Uses a hash merge lookup. Expects every value in the hierarchy for a given key to be a hash, and merges the top-level keys in each hash into a single hash. Note that this does not do a deep-merge in the case of nested structures.
Change:
$_tokens = hiera_hash('token_lookup', undef, 'hiera-file')
to
$_tokens = hiera('token_lookup', {}) #it will create empty hash if it couldn't find 'token_lookup' variable.
Please also check the following example:
# /etc/puppet/hieradata/appservers.yaml
---
proxies:
- hostname: lb01.example.com
ipaddress: 192.168.22.21
- hostname: lb02.example.com
ipaddress: 192.168.22.28
# Get the structured data:
$proxies = hiera('proxies')
# Index into the structure:
$use_ip = $proxies[1]['ipaddress'] # will be 192.168.22.28
Related
I wanted to ask if there is a way to ignore whitespace changes when creating a terraform plan.
This question is related to this one, I created a new one because I wanted to give a new example of the issue.
Terraform shows unnecessary changes due to whitespace
For example, when running
terraform plan
I get the following change for a helm provider resource
# helm_release.cert-manager will be updated in-place
~ resource "helm_release" "cert-manager" {
id = "cert-manager"
name = "cert-manager"
~ values = [
- <<-EOT
installCRDs: true
EOT,
+ <<-EOT
installCRDs: true
EOT,
]
# (27 unchanged attributes hidden)
}
I found out that the change was due to line endings. Deployed was CRLF and my local source file had LF as line ending.
Is there an option to ignore whitespaces and/or line ending characters?
It's typically the responsibility of the provider itself to determine whether the prior value and the new value are equivalent despite not being exactly equal, and so making this work automatically would require a change to the provider itself to notice that this argument is defined as being YAML and YAML doesn't ascribe any meaning to the decision between CRLF and just LF. The provider would ideally perform this check itself and thus avoid you needing to worry about it, and I would suggest opening a feature request with the provider developer to see if they would be interested in handling that.
However, if a provider isn't performing that job correctly itself then you can potentially work around it by doing your own normalization of the value using Terraform language features, so that the value passed to the provider is always the same when the meaning is the same.
One straightforward way to achieve that in this case would be to round-trip the value through both yamldecode and yamlencode, thereby normalizing the input to be in the style that yamlencode produces:
values = [yamlencode(yamldecode(var.something))]
If you want to be more surgical about it and only normalize the line endings, you could use replace to remove the CR character from any CRLF pair:
values = [replace(var.something, "\r\n", "\n")]
The above solution assumes that the difference in whitespace is being caused by something in your module, such as if you're storing your Terraform configuration in a misconfigured Git repository that's rewriting LF to CRLF when you clone it on a Windows system. This config-based normalization can undo that sort of transformation so that the provider will always see the value in the same way.
This solution cannot address problems that are caused by the provider itself misbehaving. Unfortunately some providers have bugs where they will silently rewrite the stored values for some arguments during the "refresh" step, regardless of how you wrote it in the configuration. In that case the only recourse is to fix the provider, because that incorrect value is originating inside the provider itself and isn't under the control of the module author.
I have array of domains like this:
'us1.domain.com', 'us2.domain.com', 'us3.domain.com', 'anotherdomain.com', 'yet.third.com'
I would split or truncate these domain names to:
domain.com
anotherdomain.com
third.com
Could anybody prompt me, please?
This new array will use for certificate file name.
Thanks in advance
You can solve that problem like this:
$array = [
'us1.domain.com', 'us2.domain.com', 'us3.domain.com',
'anotherdomain.com', 'yet.third.com'
]
notice($array.map |$x| { $y=$x.split(/\./); [$y[-2], $y[-1]].join('.') }.unique)
Testing:
▶ puppet apply test.pp
Notice: Scope(Class[main]): [domain.com, anotherdomain.com, third.com]
Notice: Compiled catalog for 192-168-1-103.tpgi.com.au in environment production in 0.05 seconds
Notice: Applied catalog in 0.01 seconds
Key insights there:
You can split each element on a period using using the split function.
You can take the last and second last elements of an array using $arr[-1] and $arr[-2].
You can join it all back together again using the join function.
You can transform the list into a new list using the map function.
You can remove the duplicates using the unique function.
I'm attempting to implement encrypted values in yaml in Hiera 5 to inject passwords securely into Puppet (enterprise) 5.3 via automatic lookup. There's excellent guidance from the Puppet blog and PUP-7284 on the necessary setup.
However, I can't seem to get lookup_options correct to ensure conversion to a Sensitive type (to match the class parameters).
Asserting with the puppet lookup command fails with:
[user#rhel7 ~]$ puppet lookup my_module::db_pass --environment test --type Sensitive[String]
Error: Could not run: Found value has wrong type, expects a Sensitive value, got String
It also appears the lookup_options are being found and they look sensible:
[user#rhel7 ~]$ puppet lookup my_module::db_pass --environment test --explain-options
Hierarchy entry "Passwords"
Path "/etc/puppetlabs/code/environments/test/modules/my_module/data/secrets.eyaml"
Original path: "secrets.eyaml"
Found key: "lookup_options" value: {
"^my_module::.*pass$" => {
"convert_to" => "Sensitive"
}
}
Decryption works just fine (unfortunately to cleartext -- not sure if that's expected?)
[user#rhel7 ~]$ puppet lookup my_module::db_pass --environment test
Found key: "my_module::db_pass" value: "password_is_taco"
The setup is as follows:
[user#rhel7 /etc/puppetlabs/puppet/environment/test/modules/my_module]$ cat hiera.eyaml
---
version: 5
defaults:
data_hash: yaml_data
datadir: data
hierarchy:
- name: "Passwords"
lookup_key: eyaml_lookup_key
paths:
- "secrets.eyaml"
options:
pkcs7_private_key: "/etc/puppetlabs/puppet/keys/private_key.pkcs7.pem"
pkcs7_public_key: "/etc/puppetlabs/puppet/keys/public_key.pkcs7.pem"
[user#rhel7 /etc/puppetlabs/puppet/environment/test/modules/my_module]$ cat ./data/secrets.eyaml
---
lookup_options:
'^my_module::.*pass$':
convert_to: "Sensitive"
my_module::db_pass: >
ENC[PKCS7,MIIBqQYJKoZ...snip]
I've also been unsuccessful with different regexes and/or just using keys directly:
lookup_options:
my_module::db_pass:
convert_to: "Sensitive"
Apologies in advance for any minor copy-paste issues with obfuscated code :)
I never quite figured out why the specific test setup above I tried never worked, but here's what I ultimately ended up implementing:
---
lookup_options:
"^my_module::.*(password|token)$":
convert_to: Sensitive
The pattern match will appropriately cast any of the following to Sensitive[String]:
my_module::password
my_module::service_password
my_module::api_token
my_module::any_number::of_subclasses::token_or_password
If you're considering going through this same process, you might consider:
instead of hard-coding your passwords, try leveraging new Puppet6+ features that avoid obfuscation altogether (by NOT putting credentials in catalogs):
https://puppet.com/blog/agent-side-functions-in-puppet-6
https://www.hashicorp.com/resources/agent-side-lookups-with-hashicorp-vault-puppet-6
If that's not feasible, follow the recommended process (and don't forget
to restart the Puppet Server)
make liberal use of the puppet lookup utility, especially with the --explain-options and type Sensitive[String]
I am creating a manifest to get an Agent's partition name.
The fact $partitions shows the detail of the partition info.
{"xvda1"=>{"uuid"=>"d1697425-49d0-4c9f-9901-5f9260be8196", "size"=>"83859300", "mount"=>"/", "label"=>"cloudimg-rootfs", "filesystem"=>"ext4"}}
But, I just want to get the name part (xvda1) and use it as a variable for a configuration file.
Is there any way to filter the output in Puppet?
The fastest way to solve this would be to use the keys function from puppetlabs/stdlib: https://forge.puppet.com/puppetlabs/stdlib/readme.
keys()
Returns the keys of a hash as an array. Type: rvalue.
With that function, we can transform the output hash from Facter into an array of the keys and access its elements normally. Assuming that xvda1 is the 0th element,
Facter 2:
$variable = keys($::partitions)[0]
Facter 3:
$variable = keys($facts['partitions'])[0]
I would like to create a default users for all servers, but in addition of this default uses, only for specific servers I want to create in addition of the default users a specifics ones.
My problem is that when I run puppet agent -t, puppet only create the users for the first match. If the server match in - node/%{::fqdn} create only the specific users but not the default ones.
in /etc/puppet/hiera.yaml I have the follow:
:backends:
- yaml
:yaml:
:datadir: "/etc/puppet/hieradata"
:hierarchy:
- node/%{::fqdn}
- common
How I can set up hiera in order to always run the common file?
Please use hiera hash merge. Define merge behaviour in hiera.yaml, possible values are native, deep, deeper e.g:
:merge_behavior: deeper
And than just use hiera. According to documentation:
In a deeper hash merge, Hiera recursively merges keys and values in each source hash.
Here you have merge behaviour in examples.
UPDATE:
I have setup the following simple example:
hiera.yaml:
:hierarchy:
- apps
- common
:merge_behavior: deeper
apps.yaml:
test_hash:
abc1:
value: apps
abc2:
value: apps
common.yaml:
test_hash:
abc1:
value: comm
abc3:
value: comm
test_hash.pp
class test_hash
{
$normal_hash = hiera('test_hash')
$hiera_hash = hiera_hash('test_hash')
notify{ " normal: ${normal_hash}":}
notify{ " hiera : ${hiera_hash}":}
}
include test_hash
Next call puppet script puppet apply test_hash.pp
In result:
Notice: normal: {"abc1"=>{"value"=>"apps"}, "abc2"=>{"value"=>"apps"}}
Notice: hiera : {"abc1"=>{"value"=>"apps"}, "abc3"=>{"value"=>"comm"}, "abc2"=>{"value"=>"apps"}}}
UPDATE2:
You can also consider using merge function from stdlib. But probably to use it you will have to change a bit your architecture e.g:
In common define common values, in node/%{::fqdn} define node specific values, and than use it as in example:
$common_hash = hiera('something_from_common')
$node_hash = hiera('something_from_fqdn')
$merged_hash = merge($node_hash, $common_hash)
(Yes it is a bit ugly :))