I'm attempting to implement encrypted values in yaml in Hiera 5 to inject passwords securely into Puppet (enterprise) 5.3 via automatic lookup. There's excellent guidance from the Puppet blog and PUP-7284 on the necessary setup.
However, I can't seem to get lookup_options correct to ensure conversion to a Sensitive type (to match the class parameters).
Asserting with the puppet lookup command fails with:
[user#rhel7 ~]$ puppet lookup my_module::db_pass --environment test --type Sensitive[String]
Error: Could not run: Found value has wrong type, expects a Sensitive value, got String
It also appears the lookup_options are being found and they look sensible:
[user#rhel7 ~]$ puppet lookup my_module::db_pass --environment test --explain-options
Hierarchy entry "Passwords"
Path "/etc/puppetlabs/code/environments/test/modules/my_module/data/secrets.eyaml"
Original path: "secrets.eyaml"
Found key: "lookup_options" value: {
"^my_module::.*pass$" => {
"convert_to" => "Sensitive"
}
}
Decryption works just fine (unfortunately to cleartext -- not sure if that's expected?)
[user#rhel7 ~]$ puppet lookup my_module::db_pass --environment test
Found key: "my_module::db_pass" value: "password_is_taco"
The setup is as follows:
[user#rhel7 /etc/puppetlabs/puppet/environment/test/modules/my_module]$ cat hiera.eyaml
---
version: 5
defaults:
data_hash: yaml_data
datadir: data
hierarchy:
- name: "Passwords"
lookup_key: eyaml_lookup_key
paths:
- "secrets.eyaml"
options:
pkcs7_private_key: "/etc/puppetlabs/puppet/keys/private_key.pkcs7.pem"
pkcs7_public_key: "/etc/puppetlabs/puppet/keys/public_key.pkcs7.pem"
[user#rhel7 /etc/puppetlabs/puppet/environment/test/modules/my_module]$ cat ./data/secrets.eyaml
---
lookup_options:
'^my_module::.*pass$':
convert_to: "Sensitive"
my_module::db_pass: >
ENC[PKCS7,MIIBqQYJKoZ...snip]
I've also been unsuccessful with different regexes and/or just using keys directly:
lookup_options:
my_module::db_pass:
convert_to: "Sensitive"
Apologies in advance for any minor copy-paste issues with obfuscated code :)
I never quite figured out why the specific test setup above I tried never worked, but here's what I ultimately ended up implementing:
---
lookup_options:
"^my_module::.*(password|token)$":
convert_to: Sensitive
The pattern match will appropriately cast any of the following to Sensitive[String]:
my_module::password
my_module::service_password
my_module::api_token
my_module::any_number::of_subclasses::token_or_password
If you're considering going through this same process, you might consider:
instead of hard-coding your passwords, try leveraging new Puppet6+ features that avoid obfuscation altogether (by NOT putting credentials in catalogs):
https://puppet.com/blog/agent-side-functions-in-puppet-6
https://www.hashicorp.com/resources/agent-side-lookups-with-hashicorp-vault-puppet-6
If that's not feasible, follow the recommended process (and don't forget
to restart the Puppet Server)
make liberal use of the puppet lookup utility, especially with the --explain-options and type Sensitive[String]
Related
hiera.yaml
---
:hierarchy:
- node/%{host_fqdn}
- site_config/%{host_site_name}
- site_config/perf_%{host_performance_class}
- site_config/%{host_type}_v%{host_type_version}
- site/%{host_site_name}
- environments/%{site_environment}
- types/%{host_type}_v%{host_type_version}
- hosts
- sites
- users
- common
# options are native, deep, deeper
:merge_behavior: deeper
We currently have this hiera config. So the config gets merged in the following sequence common.yaml > users.yaml > sites.yaml > hosts.yaml > types/xxx_vxxx.yaml > etc. For the variable top hierarchies, it gets overwritten only if that file exists.
eg:
common.yaml
server:
instance_type: m3.medium
site_config/mysite.yaml
server:
instance_type: m4.large
So for all other sites, the instance type will be m3.medium, but only for mysite it will be m4.large.
How can I achieve the same in Ansible?
I think that #Xiong is right that you should go the variables way in Ansible.
You can set up flexible inventory with vars precedence from general to specific.
But you can try this snippet if it helps:
---
- hosts: loc-test
tasks:
- include_vars: hiera/{{ item }}
with_items:
- common.yml
- "node/{{ ansible_fqdn }}/users.yml"
- "node/{{ ansible_fqdn }}/sites.yml"
- "node/{{ ansible_fqdn }}/types/{{ host_type }}_v{{ host_type_version }}.yml"
failed_when: false
- debug: var=server
This will try to load variables from files with structure similar to your question.
Nonexistent files are ignored (because of failed_when: false).
Files are loaded in order of this list (from top to bottom), overwriting previous values.
Gotchas:
all variables that you use in the list must be defined (e.g. host_type in this example can't be defined in common.yml), because list of items to iterate is templated before the whole loop is executed (see update for workaround).
Ansible overwrite(replace) dicts by default, I guess your use case expects merging behavior. This can be achieved with hash_behavior setting – but this is unusual for Ansible playbooks.
P.S. You may alter top-to-bottom-merge behavior by changing with_items to with_first_found and reverse the list (from specific to general). In this case Ansible will load variables from first file found.
Update: use variables from previous includes in file path.
You can split the loop into multiple tasks, so Ansible will evaluate each task's result before templating next file's include path.
Make hiera_inc.yml:
- include_vars: hiera/common.yml
failed_when: false
- include_vars: hiera/node/{{ ansible_fqdn }}/users.yml
failed_when: false
- include_vars: hiera/node/{{ ansible_fqdn }}/sites.yml
failed_when: false
- include_vars: hiera/node/{{ ansible_fqdn }}/types/{{ host_type | default('none') }}_v{{ host_type_version | default('none') }}.yml
failed_when: false
And in your main playbook:
- include: hiera_inc.yml
This looks a bit clumsy, but this way you can define host_type in common.yaml and it will be honored in the path templating for next tasks.
With Ansible 2.2 it will be possible to include_vars into named variable (not global host space), so you can include_vars into hiera_facts and use combine filter to merge them without altering global hash behavior.
I'm not familiar with Puppet, so this may not be a direct mapping. But what I understand your question to be is "how do I use values in one shared location but override their definitions for different servers?". In Ansible, you do this with variables.
You can define variables directly in your inventory. You can define variables in host- and group-specific files. You can define variables at a playbook level. You can define variables at a role level. Heck, you can even define variables with command-line switches.
Between all of these places, you should be able to define overrides to suit your situation. You'll probably want to take a look at the documentation section on how to decide where to define a variable for more info.
It seems a little more basic than Hiera, but somebody has created a basic ansible lookup plugin with similar syntax
https://github.com/sailthru/ansible-oss/tree/master/tools/echelon
I would like to create a default users for all servers, but in addition of this default uses, only for specific servers I want to create in addition of the default users a specifics ones.
My problem is that when I run puppet agent -t, puppet only create the users for the first match. If the server match in - node/%{::fqdn} create only the specific users but not the default ones.
in /etc/puppet/hiera.yaml I have the follow:
:backends:
- yaml
:yaml:
:datadir: "/etc/puppet/hieradata"
:hierarchy:
- node/%{::fqdn}
- common
How I can set up hiera in order to always run the common file?
Please use hiera hash merge. Define merge behaviour in hiera.yaml, possible values are native, deep, deeper e.g:
:merge_behavior: deeper
And than just use hiera. According to documentation:
In a deeper hash merge, Hiera recursively merges keys and values in each source hash.
Here you have merge behaviour in examples.
UPDATE:
I have setup the following simple example:
hiera.yaml:
:hierarchy:
- apps
- common
:merge_behavior: deeper
apps.yaml:
test_hash:
abc1:
value: apps
abc2:
value: apps
common.yaml:
test_hash:
abc1:
value: comm
abc3:
value: comm
test_hash.pp
class test_hash
{
$normal_hash = hiera('test_hash')
$hiera_hash = hiera_hash('test_hash')
notify{ " normal: ${normal_hash}":}
notify{ " hiera : ${hiera_hash}":}
}
include test_hash
Next call puppet script puppet apply test_hash.pp
In result:
Notice: normal: {"abc1"=>{"value"=>"apps"}, "abc2"=>{"value"=>"apps"}}
Notice: hiera : {"abc1"=>{"value"=>"apps"}, "abc3"=>{"value"=>"comm"}, "abc2"=>{"value"=>"apps"}}}
UPDATE2:
You can also consider using merge function from stdlib. But probably to use it you will have to change a bit your architecture e.g:
In common define common values, in node/%{::fqdn} define node specific values, and than use it as in example:
$common_hash = hiera('something_from_common')
$node_hash = hiera('something_from_fqdn')
$merged_hash = merge($node_hash, $common_hash)
(Yes it is a bit ugly :))
I am using a Hiera hash to hold some token values which are host-specific. the keys within the hash correspond to the hostname/certname of the node(s) that'll be classified with the profile module that calls the hash value. However, when I apply the module, the value which corresponds to the hash key for the host is always null. Here's the code I'm working with.
in hiera-file.yaml
token_lookup:
host-name1: 'abcdef123'
host-name2: 'abbcde456'
and in profile.pp
$_tokens = hiera_hash('token_lookup', undef, 'hiera-file')
$_specific_token = $_tokens["${::hostname}"] <== never gets a value
I'm certain that the hostname matches the key in the hash. The question is, what's the right syntax here for getting the value from the hiera-file to populate properly? Thank you in advance for the advice.
edit: I believe I have discovered an issue when the hash key has a literal '-' character in it, as many hostnames do. I updated the hash to show keys with dashes in it, and should now ask a more specific question: I see dozens of articles about how to escape characters in the values of hashes by using double quotes, but I don't see anything - even on yaml.org - about how to escape the character if it appears as part of the key. Any tips on this issue? A YAML parser shows that this is valid syntactically, but I believe it is treating the '-' as a collection marker instead of a literal character.
Your code is correct, I test it as below, seems it didn't target the right yaml file in your environment. Check the hierarchy setting, and put the token key-value in the right place.
If I put the yaml file to global.yaml(If hiera can't find the key, it will always go to the last one in my hiera.yaml setting)
I rebuilt it with simplest setting:
$ cat /etc/hiera.yaml:
---
:backends:
- yaml
:hierarchy:
- defaults
- "%{clientcert}"
- "%{environment}"
- global
:yaml:
# datadir is empty here, so hiera uses its defaults:
# - /var/lib/hiera on *nix
# - %CommonAppData%\PuppetLabs\hiera\var on Windows
# When specifying a datadir, make sure the directory exists.
:datadir:
$ cat /var/lib/hiera/global.yaml
token_lookup:
host-name1: 'abcdef123'
host-name2: 'abbcde456'
$ cat profile.pp
$_tokens = hiera_hash('token_lookup', undef, 'hiera-file')
notice ("tokens is $_tokens")
$_specific_token = $_tokens["${::hostname}"]
notice ("token is $_specific_token ")
Then I run puppet apply, I can see the result
$ FACTER_hostname='host-name1' puppet apply profile.pp --hiera_config /etc/hiera.yaml
Notice: Scope(Class[main]): tokens is host-name1abcdef123host-name2abbcde456
Notice: Scope(Class[main]): token is abcdef123
Notice: Compiled catalog for host-name1 in environment production in 0.04 seconds
Notice: Finished catalog run in 0.07 seconds
$ FACTER_hostname='host-name2' puppet apply profile.pp --hiera_config /etc/hiera.yaml
Notice: Scope(Class[main]): tokens is host-name1abcdef123host-name2abbcde456
Notice: Scope(Class[main]): token is abbcde456
Notice: Compiled catalog for host-name2 in environment production in 0.04 seconds
Notice: Finished catalog run in 0.02 seconds
root#ac976d6d79fb:~#
I think hiera_hash is not what you want.
hiera_hash:
Uses a hash merge lookup. Expects every value in the hierarchy for a given key to be a hash, and merges the top-level keys in each hash into a single hash. Note that this does not do a deep-merge in the case of nested structures.
Change:
$_tokens = hiera_hash('token_lookup', undef, 'hiera-file')
to
$_tokens = hiera('token_lookup', {}) #it will create empty hash if it couldn't find 'token_lookup' variable.
Please also check the following example:
# /etc/puppet/hieradata/appservers.yaml
---
proxies:
- hostname: lb01.example.com
ipaddress: 192.168.22.21
- hostname: lb02.example.com
ipaddress: 192.168.22.28
# Get the structured data:
$proxies = hiera('proxies')
# Index into the structure:
$use_ip = $proxies[1]['ipaddress'] # will be 192.168.22.28
Is there a better way to format my hiera data?
I want to avoid the "write everything twice" problem.
Here is what I have now:
[root#puppet-el7-001 ~]# cat example.yaml
---
controller_ips:
- 10.0.0.51
- 10.0.0.52
- 10.0.0.53
controller::horizon_cache_server_ip:
- 10.0.0.51:11211
- 10.0.0.52:11211
- 10.0.0.53:11211
I was wondering if there is functionality avaialble in hiera that is like Perl's map function.
If so then I could do something like:
controller::horizon_cache_server_ip: "%{hiera_map( {"$_:11211"}, %{hiera('controller_ips')})}"
Thanks
It depends on which puppet version you are using. I puppet 3.x, you can do the following:
common::test::var1: a
common::test::var2: b
common::test::variable:
- "%{hiera('common::test::var1')}"
- "%{hiera('common::test::var2')}"
common::test::variable2:
- "%{hiera('common::test::var1')}:1"
- "%{hiera('common::test::var2')}:2"
In puppet 4.0 you can try using a combination of zip, hash functions from stdlib, with built in function map.
Something like:
$array3 = zip($array1, $array2)
$my_hash = hash($array3)
$my_hash.map |$key,$val|{ "${key}:${val}" }
The mutation is a problem. It is simpler with identical data thanks to YAML's referencing capability.
controller_ips: &CONTROLLERS
- 10.0.0.51
- 10.0.0.52
- 10.0.0.53
controller::horizon_cache_server_ip: *CONTROLLERS
You will need more logic so that the port can be stored independently.
controller::horizon_cache_server_port: 11211
The manifest needs to be structured in a way that allows you to combine the IPs with the port.
I've seen someone doing a check on whether an agent's MAC address is on a specific regular expression before it runs the specified stuff below. The example is something like this:
if $is_virtual == "true" and $kernel == "Linux" and $macaddress =~ /^02:00:0A/ {
include nmonitor
include rootsh
include checkmk-agent
include backuppcacc
include onecontext
include sysstatpkg
include ensurekvmsudo
include cronntpdate
}
That's just it in that particular manifest file. Similarly another manifest example but via regular expression below:
node /^mi-cloud-(dev|stg|prd)-host/ {
if $is_virtual == 'false' {
include etchosts
include checkmk-agent
include nmonitor
include rootsh
include sysstatpkg
include cronntpdate
include fstab-ds-dev
}
}
I've been asked of whether can that similar concept be applied upon checking the agent's hostname with a master file of hostnames allowed to be run or otherwise.
I am not sure whether it can be done, but the rough idea goes around something like:
file { 'hostmasterfile.ini'
ensure => present,
source => puppet:///test/hostmaster.ini,
content => $hostname
}
$coname = content
#Usually the start / head of the manifest
if $hostname == $coname {
include <a>
include <b>
}
Note: $fqdn is out of the question.
To my knowledge, I have not seen any such sample manifest that matches the request. Whats more, it goes against a standard practice of keeping things easier to manage and not putting all eggs in a basket.
An ex-colleague of mine claims that idea above is about self-provisioning. However that concept is non-existent in Puppet (he posed that question at a workshop a few months back). I am not sure how true is that though.
If that thing above can be done, any suggestion of how can it be done? Or is it best to go back to the standard one manifest per node for easy maintenance?
Thanks very much.
M
Well, you can replace your node blocks with if constructs.
if $hostname == 'host1' {
# manifest for host1 here
}
You can combine this with some sort of inifile (e.g., using the generate) function. If the <a> and <b> for the include statements are then fetched from your ini file as well, you have constructed a crude ENC.
Note that this has security implications - any agent can claim to have any host name. It's even very simple to do:
FACTER_hostname=kerberos01 puppet agent --test
Any node can receive the catalog for kerberos01 this way. (node blocks rely on $certname instead, which cannot be forged.)
I could not decipher your precise intent from your question, but I suspect that you really want an ENC or a Hiera based approach.
Edit after feedback from your first comment:
To make the master read contents from local files, you should
get rid of the file { 'hostmasterfile.ini': } - it only allows you to set contents, not retrieve them
initialize the variable content using the file function (this will make all nodes fail if the file is not readable)
The code could look like this (assuming that there can be multiple host names in the ini file).
$ini_data = file('/etc/puppet/files/test/hostmaster.ini')
Next step would be a regex lookup like this:
if $ini_data =~ /name=$hostname/ {
Unfortunately, this does not work! Puppet will not expand variable values in regular expressions, apparently.
You can use this (kind of silly) workaround:
$ini_lookup = regsubst($ini_data, "name=$hostname", '__FOUND__')
if $ini_lookup =~ /__FOUND__/ {
...
}
Final remark about security: If your team is adamant about not using $certname for this lookup (although it should be easy to map host names to cert names), you should consider adding the host name to your trusted facts.