Is there a better way to format my hiera data?
I want to avoid the "write everything twice" problem.
Here is what I have now:
[root#puppet-el7-001 ~]# cat example.yaml
---
controller_ips:
- 10.0.0.51
- 10.0.0.52
- 10.0.0.53
controller::horizon_cache_server_ip:
- 10.0.0.51:11211
- 10.0.0.52:11211
- 10.0.0.53:11211
I was wondering if there is functionality avaialble in hiera that is like Perl's map function.
If so then I could do something like:
controller::horizon_cache_server_ip: "%{hiera_map( {"$_:11211"}, %{hiera('controller_ips')})}"
Thanks
It depends on which puppet version you are using. I puppet 3.x, you can do the following:
common::test::var1: a
common::test::var2: b
common::test::variable:
- "%{hiera('common::test::var1')}"
- "%{hiera('common::test::var2')}"
common::test::variable2:
- "%{hiera('common::test::var1')}:1"
- "%{hiera('common::test::var2')}:2"
In puppet 4.0 you can try using a combination of zip, hash functions from stdlib, with built in function map.
Something like:
$array3 = zip($array1, $array2)
$my_hash = hash($array3)
$my_hash.map |$key,$val|{ "${key}:${val}" }
The mutation is a problem. It is simpler with identical data thanks to YAML's referencing capability.
controller_ips: &CONTROLLERS
- 10.0.0.51
- 10.0.0.52
- 10.0.0.53
controller::horizon_cache_server_ip: *CONTROLLERS
You will need more logic so that the port can be stored independently.
controller::horizon_cache_server_port: 11211
The manifest needs to be structured in a way that allows you to combine the IPs with the port.
Related
I'm attempting to implement encrypted values in yaml in Hiera 5 to inject passwords securely into Puppet (enterprise) 5.3 via automatic lookup. There's excellent guidance from the Puppet blog and PUP-7284 on the necessary setup.
However, I can't seem to get lookup_options correct to ensure conversion to a Sensitive type (to match the class parameters).
Asserting with the puppet lookup command fails with:
[user#rhel7 ~]$ puppet lookup my_module::db_pass --environment test --type Sensitive[String]
Error: Could not run: Found value has wrong type, expects a Sensitive value, got String
It also appears the lookup_options are being found and they look sensible:
[user#rhel7 ~]$ puppet lookup my_module::db_pass --environment test --explain-options
Hierarchy entry "Passwords"
Path "/etc/puppetlabs/code/environments/test/modules/my_module/data/secrets.eyaml"
Original path: "secrets.eyaml"
Found key: "lookup_options" value: {
"^my_module::.*pass$" => {
"convert_to" => "Sensitive"
}
}
Decryption works just fine (unfortunately to cleartext -- not sure if that's expected?)
[user#rhel7 ~]$ puppet lookup my_module::db_pass --environment test
Found key: "my_module::db_pass" value: "password_is_taco"
The setup is as follows:
[user#rhel7 /etc/puppetlabs/puppet/environment/test/modules/my_module]$ cat hiera.eyaml
---
version: 5
defaults:
data_hash: yaml_data
datadir: data
hierarchy:
- name: "Passwords"
lookup_key: eyaml_lookup_key
paths:
- "secrets.eyaml"
options:
pkcs7_private_key: "/etc/puppetlabs/puppet/keys/private_key.pkcs7.pem"
pkcs7_public_key: "/etc/puppetlabs/puppet/keys/public_key.pkcs7.pem"
[user#rhel7 /etc/puppetlabs/puppet/environment/test/modules/my_module]$ cat ./data/secrets.eyaml
---
lookup_options:
'^my_module::.*pass$':
convert_to: "Sensitive"
my_module::db_pass: >
ENC[PKCS7,MIIBqQYJKoZ...snip]
I've also been unsuccessful with different regexes and/or just using keys directly:
lookup_options:
my_module::db_pass:
convert_to: "Sensitive"
Apologies in advance for any minor copy-paste issues with obfuscated code :)
I never quite figured out why the specific test setup above I tried never worked, but here's what I ultimately ended up implementing:
---
lookup_options:
"^my_module::.*(password|token)$":
convert_to: Sensitive
The pattern match will appropriately cast any of the following to Sensitive[String]:
my_module::password
my_module::service_password
my_module::api_token
my_module::any_number::of_subclasses::token_or_password
If you're considering going through this same process, you might consider:
instead of hard-coding your passwords, try leveraging new Puppet6+ features that avoid obfuscation altogether (by NOT putting credentials in catalogs):
https://puppet.com/blog/agent-side-functions-in-puppet-6
https://www.hashicorp.com/resources/agent-side-lookups-with-hashicorp-vault-puppet-6
If that's not feasible, follow the recommended process (and don't forget
to restart the Puppet Server)
make liberal use of the puppet lookup utility, especially with the --explain-options and type Sensitive[String]
hiera.yaml
---
:hierarchy:
- node/%{host_fqdn}
- site_config/%{host_site_name}
- site_config/perf_%{host_performance_class}
- site_config/%{host_type}_v%{host_type_version}
- site/%{host_site_name}
- environments/%{site_environment}
- types/%{host_type}_v%{host_type_version}
- hosts
- sites
- users
- common
# options are native, deep, deeper
:merge_behavior: deeper
We currently have this hiera config. So the config gets merged in the following sequence common.yaml > users.yaml > sites.yaml > hosts.yaml > types/xxx_vxxx.yaml > etc. For the variable top hierarchies, it gets overwritten only if that file exists.
eg:
common.yaml
server:
instance_type: m3.medium
site_config/mysite.yaml
server:
instance_type: m4.large
So for all other sites, the instance type will be m3.medium, but only for mysite it will be m4.large.
How can I achieve the same in Ansible?
I think that #Xiong is right that you should go the variables way in Ansible.
You can set up flexible inventory with vars precedence from general to specific.
But you can try this snippet if it helps:
---
- hosts: loc-test
tasks:
- include_vars: hiera/{{ item }}
with_items:
- common.yml
- "node/{{ ansible_fqdn }}/users.yml"
- "node/{{ ansible_fqdn }}/sites.yml"
- "node/{{ ansible_fqdn }}/types/{{ host_type }}_v{{ host_type_version }}.yml"
failed_when: false
- debug: var=server
This will try to load variables from files with structure similar to your question.
Nonexistent files are ignored (because of failed_when: false).
Files are loaded in order of this list (from top to bottom), overwriting previous values.
Gotchas:
all variables that you use in the list must be defined (e.g. host_type in this example can't be defined in common.yml), because list of items to iterate is templated before the whole loop is executed (see update for workaround).
Ansible overwrite(replace) dicts by default, I guess your use case expects merging behavior. This can be achieved with hash_behavior setting – but this is unusual for Ansible playbooks.
P.S. You may alter top-to-bottom-merge behavior by changing with_items to with_first_found and reverse the list (from specific to general). In this case Ansible will load variables from first file found.
Update: use variables from previous includes in file path.
You can split the loop into multiple tasks, so Ansible will evaluate each task's result before templating next file's include path.
Make hiera_inc.yml:
- include_vars: hiera/common.yml
failed_when: false
- include_vars: hiera/node/{{ ansible_fqdn }}/users.yml
failed_when: false
- include_vars: hiera/node/{{ ansible_fqdn }}/sites.yml
failed_when: false
- include_vars: hiera/node/{{ ansible_fqdn }}/types/{{ host_type | default('none') }}_v{{ host_type_version | default('none') }}.yml
failed_when: false
And in your main playbook:
- include: hiera_inc.yml
This looks a bit clumsy, but this way you can define host_type in common.yaml and it will be honored in the path templating for next tasks.
With Ansible 2.2 it will be possible to include_vars into named variable (not global host space), so you can include_vars into hiera_facts and use combine filter to merge them without altering global hash behavior.
I'm not familiar with Puppet, so this may not be a direct mapping. But what I understand your question to be is "how do I use values in one shared location but override their definitions for different servers?". In Ansible, you do this with variables.
You can define variables directly in your inventory. You can define variables in host- and group-specific files. You can define variables at a playbook level. You can define variables at a role level. Heck, you can even define variables with command-line switches.
Between all of these places, you should be able to define overrides to suit your situation. You'll probably want to take a look at the documentation section on how to decide where to define a variable for more info.
It seems a little more basic than Hiera, but somebody has created a basic ansible lookup plugin with similar syntax
https://github.com/sailthru/ansible-oss/tree/master/tools/echelon
I want Puppet to create a different variable name depending on the hiera file associated with the environment. I want to do this because I want Puppet to use the ip address associated with a specific network interface. Ideally, the network interface will be in the hiera file. That way you could concatenate the ip_address variable name with the network interface defined in the hiera file, which would look something like.
::ipaddress_{$network_interface_from_hiera_file}
Is this possible?
Right now I have an the following, but I think there is a better implementation. If the network interfaces change I would have to add another case.
if $environment == 'production' {
$client_address = $::ipaddress_enp130s0f0
} else {
$client_address = $::ipaddress_eth2
}
It sounds like you're after an eval in Puppet, like you have in shell and Perl other languages, and as far as I know, there isn't one.
I would probably just use a custom fact that always returns the IP address I care about. Of course, then you need to solve the problem of how to get the custom facts out to your fleet.
Another solution might be to use Hiera's hierarchical lookup:
In hiera.yaml:
:hierarchy:
- %{::node_environment}
- common
In common.yaml:
---
myclass::client_address: "%{::ipaddress_eth2}"
In production.yaml:
---
myclass::client_address: "%{::ipaddress_enp130s0f0}"
Finally, be aware that you can look up values from within Hiera, see here. Possibly that could be helpful.
I would like to create a default users for all servers, but in addition of this default uses, only for specific servers I want to create in addition of the default users a specifics ones.
My problem is that when I run puppet agent -t, puppet only create the users for the first match. If the server match in - node/%{::fqdn} create only the specific users but not the default ones.
in /etc/puppet/hiera.yaml I have the follow:
:backends:
- yaml
:yaml:
:datadir: "/etc/puppet/hieradata"
:hierarchy:
- node/%{::fqdn}
- common
How I can set up hiera in order to always run the common file?
Please use hiera hash merge. Define merge behaviour in hiera.yaml, possible values are native, deep, deeper e.g:
:merge_behavior: deeper
And than just use hiera. According to documentation:
In a deeper hash merge, Hiera recursively merges keys and values in each source hash.
Here you have merge behaviour in examples.
UPDATE:
I have setup the following simple example:
hiera.yaml:
:hierarchy:
- apps
- common
:merge_behavior: deeper
apps.yaml:
test_hash:
abc1:
value: apps
abc2:
value: apps
common.yaml:
test_hash:
abc1:
value: comm
abc3:
value: comm
test_hash.pp
class test_hash
{
$normal_hash = hiera('test_hash')
$hiera_hash = hiera_hash('test_hash')
notify{ " normal: ${normal_hash}":}
notify{ " hiera : ${hiera_hash}":}
}
include test_hash
Next call puppet script puppet apply test_hash.pp
In result:
Notice: normal: {"abc1"=>{"value"=>"apps"}, "abc2"=>{"value"=>"apps"}}
Notice: hiera : {"abc1"=>{"value"=>"apps"}, "abc3"=>{"value"=>"comm"}, "abc2"=>{"value"=>"apps"}}}
UPDATE2:
You can also consider using merge function from stdlib. But probably to use it you will have to change a bit your architecture e.g:
In common define common values, in node/%{::fqdn} define node specific values, and than use it as in example:
$common_hash = hiera('something_from_common')
$node_hash = hiera('something_from_fqdn')
$merged_hash = merge($node_hash, $common_hash)
(Yes it is a bit ugly :))
I have a hierachy like this:
- "nodes/%{::certname}"
- (what's here is my question)
- common
I'd like to assign a group to my nodes in their individual configuration in hiera, like this in nodes/hostname.yaml :
---
group: alpha
Now, I'd like to have a file alpha.yaml, where I state group-specific settings.
So my question is how do I write the hierachy to ask hiera for the filename of the group definition?
Is there another way to achieve this?
You can. Make sure you have the group defined in Facts.
- "nodes/%{::certname}"
- "%{::group}"
- common
So you can test with below command
FACTER_group=alpha puppet apply your.pp
For custom facts, you can go through this document: Custom Facts Walkthrough