I would like to create a default users for all servers, but in addition of this default uses, only for specific servers I want to create in addition of the default users a specifics ones.
My problem is that when I run puppet agent -t, puppet only create the users for the first match. If the server match in - node/%{::fqdn} create only the specific users but not the default ones.
in /etc/puppet/hiera.yaml I have the follow:
:backends:
- yaml
:yaml:
:datadir: "/etc/puppet/hieradata"
:hierarchy:
- node/%{::fqdn}
- common
How I can set up hiera in order to always run the common file?
Please use hiera hash merge. Define merge behaviour in hiera.yaml, possible values are native, deep, deeper e.g:
:merge_behavior: deeper
And than just use hiera. According to documentation:
In a deeper hash merge, Hiera recursively merges keys and values in each source hash.
Here you have merge behaviour in examples.
UPDATE:
I have setup the following simple example:
hiera.yaml:
:hierarchy:
- apps
- common
:merge_behavior: deeper
apps.yaml:
test_hash:
abc1:
value: apps
abc2:
value: apps
common.yaml:
test_hash:
abc1:
value: comm
abc3:
value: comm
test_hash.pp
class test_hash
{
$normal_hash = hiera('test_hash')
$hiera_hash = hiera_hash('test_hash')
notify{ " normal: ${normal_hash}":}
notify{ " hiera : ${hiera_hash}":}
}
include test_hash
Next call puppet script puppet apply test_hash.pp
In result:
Notice: normal: {"abc1"=>{"value"=>"apps"}, "abc2"=>{"value"=>"apps"}}
Notice: hiera : {"abc1"=>{"value"=>"apps"}, "abc3"=>{"value"=>"comm"}, "abc2"=>{"value"=>"apps"}}}
UPDATE2:
You can also consider using merge function from stdlib. But probably to use it you will have to change a bit your architecture e.g:
In common define common values, in node/%{::fqdn} define node specific values, and than use it as in example:
$common_hash = hiera('something_from_common')
$node_hash = hiera('something_from_fqdn')
$merged_hash = merge($node_hash, $common_hash)
(Yes it is a bit ugly :))
Related
I'm attempting to implement encrypted values in yaml in Hiera 5 to inject passwords securely into Puppet (enterprise) 5.3 via automatic lookup. There's excellent guidance from the Puppet blog and PUP-7284 on the necessary setup.
However, I can't seem to get lookup_options correct to ensure conversion to a Sensitive type (to match the class parameters).
Asserting with the puppet lookup command fails with:
[user#rhel7 ~]$ puppet lookup my_module::db_pass --environment test --type Sensitive[String]
Error: Could not run: Found value has wrong type, expects a Sensitive value, got String
It also appears the lookup_options are being found and they look sensible:
[user#rhel7 ~]$ puppet lookup my_module::db_pass --environment test --explain-options
Hierarchy entry "Passwords"
Path "/etc/puppetlabs/code/environments/test/modules/my_module/data/secrets.eyaml"
Original path: "secrets.eyaml"
Found key: "lookup_options" value: {
"^my_module::.*pass$" => {
"convert_to" => "Sensitive"
}
}
Decryption works just fine (unfortunately to cleartext -- not sure if that's expected?)
[user#rhel7 ~]$ puppet lookup my_module::db_pass --environment test
Found key: "my_module::db_pass" value: "password_is_taco"
The setup is as follows:
[user#rhel7 /etc/puppetlabs/puppet/environment/test/modules/my_module]$ cat hiera.eyaml
---
version: 5
defaults:
data_hash: yaml_data
datadir: data
hierarchy:
- name: "Passwords"
lookup_key: eyaml_lookup_key
paths:
- "secrets.eyaml"
options:
pkcs7_private_key: "/etc/puppetlabs/puppet/keys/private_key.pkcs7.pem"
pkcs7_public_key: "/etc/puppetlabs/puppet/keys/public_key.pkcs7.pem"
[user#rhel7 /etc/puppetlabs/puppet/environment/test/modules/my_module]$ cat ./data/secrets.eyaml
---
lookup_options:
'^my_module::.*pass$':
convert_to: "Sensitive"
my_module::db_pass: >
ENC[PKCS7,MIIBqQYJKoZ...snip]
I've also been unsuccessful with different regexes and/or just using keys directly:
lookup_options:
my_module::db_pass:
convert_to: "Sensitive"
Apologies in advance for any minor copy-paste issues with obfuscated code :)
I never quite figured out why the specific test setup above I tried never worked, but here's what I ultimately ended up implementing:
---
lookup_options:
"^my_module::.*(password|token)$":
convert_to: Sensitive
The pattern match will appropriately cast any of the following to Sensitive[String]:
my_module::password
my_module::service_password
my_module::api_token
my_module::any_number::of_subclasses::token_or_password
If you're considering going through this same process, you might consider:
instead of hard-coding your passwords, try leveraging new Puppet6+ features that avoid obfuscation altogether (by NOT putting credentials in catalogs):
https://puppet.com/blog/agent-side-functions-in-puppet-6
https://www.hashicorp.com/resources/agent-side-lookups-with-hashicorp-vault-puppet-6
If that's not feasible, follow the recommended process (and don't forget
to restart the Puppet Server)
make liberal use of the puppet lookup utility, especially with the --explain-options and type Sensitive[String]
hiera.yaml
---
:hierarchy:
- node/%{host_fqdn}
- site_config/%{host_site_name}
- site_config/perf_%{host_performance_class}
- site_config/%{host_type}_v%{host_type_version}
- site/%{host_site_name}
- environments/%{site_environment}
- types/%{host_type}_v%{host_type_version}
- hosts
- sites
- users
- common
# options are native, deep, deeper
:merge_behavior: deeper
We currently have this hiera config. So the config gets merged in the following sequence common.yaml > users.yaml > sites.yaml > hosts.yaml > types/xxx_vxxx.yaml > etc. For the variable top hierarchies, it gets overwritten only if that file exists.
eg:
common.yaml
server:
instance_type: m3.medium
site_config/mysite.yaml
server:
instance_type: m4.large
So for all other sites, the instance type will be m3.medium, but only for mysite it will be m4.large.
How can I achieve the same in Ansible?
I think that #Xiong is right that you should go the variables way in Ansible.
You can set up flexible inventory with vars precedence from general to specific.
But you can try this snippet if it helps:
---
- hosts: loc-test
tasks:
- include_vars: hiera/{{ item }}
with_items:
- common.yml
- "node/{{ ansible_fqdn }}/users.yml"
- "node/{{ ansible_fqdn }}/sites.yml"
- "node/{{ ansible_fqdn }}/types/{{ host_type }}_v{{ host_type_version }}.yml"
failed_when: false
- debug: var=server
This will try to load variables from files with structure similar to your question.
Nonexistent files are ignored (because of failed_when: false).
Files are loaded in order of this list (from top to bottom), overwriting previous values.
Gotchas:
all variables that you use in the list must be defined (e.g. host_type in this example can't be defined in common.yml), because list of items to iterate is templated before the whole loop is executed (see update for workaround).
Ansible overwrite(replace) dicts by default, I guess your use case expects merging behavior. This can be achieved with hash_behavior setting – but this is unusual for Ansible playbooks.
P.S. You may alter top-to-bottom-merge behavior by changing with_items to with_first_found and reverse the list (from specific to general). In this case Ansible will load variables from first file found.
Update: use variables from previous includes in file path.
You can split the loop into multiple tasks, so Ansible will evaluate each task's result before templating next file's include path.
Make hiera_inc.yml:
- include_vars: hiera/common.yml
failed_when: false
- include_vars: hiera/node/{{ ansible_fqdn }}/users.yml
failed_when: false
- include_vars: hiera/node/{{ ansible_fqdn }}/sites.yml
failed_when: false
- include_vars: hiera/node/{{ ansible_fqdn }}/types/{{ host_type | default('none') }}_v{{ host_type_version | default('none') }}.yml
failed_when: false
And in your main playbook:
- include: hiera_inc.yml
This looks a bit clumsy, but this way you can define host_type in common.yaml and it will be honored in the path templating for next tasks.
With Ansible 2.2 it will be possible to include_vars into named variable (not global host space), so you can include_vars into hiera_facts and use combine filter to merge them without altering global hash behavior.
I'm not familiar with Puppet, so this may not be a direct mapping. But what I understand your question to be is "how do I use values in one shared location but override their definitions for different servers?". In Ansible, you do this with variables.
You can define variables directly in your inventory. You can define variables in host- and group-specific files. You can define variables at a playbook level. You can define variables at a role level. Heck, you can even define variables with command-line switches.
Between all of these places, you should be able to define overrides to suit your situation. You'll probably want to take a look at the documentation section on how to decide where to define a variable for more info.
It seems a little more basic than Hiera, but somebody has created a basic ansible lookup plugin with similar syntax
https://github.com/sailthru/ansible-oss/tree/master/tools/echelon
I want Puppet to create a different variable name depending on the hiera file associated with the environment. I want to do this because I want Puppet to use the ip address associated with a specific network interface. Ideally, the network interface will be in the hiera file. That way you could concatenate the ip_address variable name with the network interface defined in the hiera file, which would look something like.
::ipaddress_{$network_interface_from_hiera_file}
Is this possible?
Right now I have an the following, but I think there is a better implementation. If the network interfaces change I would have to add another case.
if $environment == 'production' {
$client_address = $::ipaddress_enp130s0f0
} else {
$client_address = $::ipaddress_eth2
}
It sounds like you're after an eval in Puppet, like you have in shell and Perl other languages, and as far as I know, there isn't one.
I would probably just use a custom fact that always returns the IP address I care about. Of course, then you need to solve the problem of how to get the custom facts out to your fleet.
Another solution might be to use Hiera's hierarchical lookup:
In hiera.yaml:
:hierarchy:
- %{::node_environment}
- common
In common.yaml:
---
myclass::client_address: "%{::ipaddress_eth2}"
In production.yaml:
---
myclass::client_address: "%{::ipaddress_enp130s0f0}"
Finally, be aware that you can look up values from within Hiera, see here. Possibly that could be helpful.
Is there a better way to format my hiera data?
I want to avoid the "write everything twice" problem.
Here is what I have now:
[root#puppet-el7-001 ~]# cat example.yaml
---
controller_ips:
- 10.0.0.51
- 10.0.0.52
- 10.0.0.53
controller::horizon_cache_server_ip:
- 10.0.0.51:11211
- 10.0.0.52:11211
- 10.0.0.53:11211
I was wondering if there is functionality avaialble in hiera that is like Perl's map function.
If so then I could do something like:
controller::horizon_cache_server_ip: "%{hiera_map( {"$_:11211"}, %{hiera('controller_ips')})}"
Thanks
It depends on which puppet version you are using. I puppet 3.x, you can do the following:
common::test::var1: a
common::test::var2: b
common::test::variable:
- "%{hiera('common::test::var1')}"
- "%{hiera('common::test::var2')}"
common::test::variable2:
- "%{hiera('common::test::var1')}:1"
- "%{hiera('common::test::var2')}:2"
In puppet 4.0 you can try using a combination of zip, hash functions from stdlib, with built in function map.
Something like:
$array3 = zip($array1, $array2)
$my_hash = hash($array3)
$my_hash.map |$key,$val|{ "${key}:${val}" }
The mutation is a problem. It is simpler with identical data thanks to YAML's referencing capability.
controller_ips: &CONTROLLERS
- 10.0.0.51
- 10.0.0.52
- 10.0.0.53
controller::horizon_cache_server_ip: *CONTROLLERS
You will need more logic so that the port can be stored independently.
controller::horizon_cache_server_port: 11211
The manifest needs to be structured in a way that allows you to combine the IPs with the port.
I've seen someone doing a check on whether an agent's MAC address is on a specific regular expression before it runs the specified stuff below. The example is something like this:
if $is_virtual == "true" and $kernel == "Linux" and $macaddress =~ /^02:00:0A/ {
include nmonitor
include rootsh
include checkmk-agent
include backuppcacc
include onecontext
include sysstatpkg
include ensurekvmsudo
include cronntpdate
}
That's just it in that particular manifest file. Similarly another manifest example but via regular expression below:
node /^mi-cloud-(dev|stg|prd)-host/ {
if $is_virtual == 'false' {
include etchosts
include checkmk-agent
include nmonitor
include rootsh
include sysstatpkg
include cronntpdate
include fstab-ds-dev
}
}
I've been asked of whether can that similar concept be applied upon checking the agent's hostname with a master file of hostnames allowed to be run or otherwise.
I am not sure whether it can be done, but the rough idea goes around something like:
file { 'hostmasterfile.ini'
ensure => present,
source => puppet:///test/hostmaster.ini,
content => $hostname
}
$coname = content
#Usually the start / head of the manifest
if $hostname == $coname {
include <a>
include <b>
}
Note: $fqdn is out of the question.
To my knowledge, I have not seen any such sample manifest that matches the request. Whats more, it goes against a standard practice of keeping things easier to manage and not putting all eggs in a basket.
An ex-colleague of mine claims that idea above is about self-provisioning. However that concept is non-existent in Puppet (he posed that question at a workshop a few months back). I am not sure how true is that though.
If that thing above can be done, any suggestion of how can it be done? Or is it best to go back to the standard one manifest per node for easy maintenance?
Thanks very much.
M
Well, you can replace your node blocks with if constructs.
if $hostname == 'host1' {
# manifest for host1 here
}
You can combine this with some sort of inifile (e.g., using the generate) function. If the <a> and <b> for the include statements are then fetched from your ini file as well, you have constructed a crude ENC.
Note that this has security implications - any agent can claim to have any host name. It's even very simple to do:
FACTER_hostname=kerberos01 puppet agent --test
Any node can receive the catalog for kerberos01 this way. (node blocks rely on $certname instead, which cannot be forged.)
I could not decipher your precise intent from your question, but I suspect that you really want an ENC or a Hiera based approach.
Edit after feedback from your first comment:
To make the master read contents from local files, you should
get rid of the file { 'hostmasterfile.ini': } - it only allows you to set contents, not retrieve them
initialize the variable content using the file function (this will make all nodes fail if the file is not readable)
The code could look like this (assuming that there can be multiple host names in the ini file).
$ini_data = file('/etc/puppet/files/test/hostmaster.ini')
Next step would be a regex lookup like this:
if $ini_data =~ /name=$hostname/ {
Unfortunately, this does not work! Puppet will not expand variable values in regular expressions, apparently.
You can use this (kind of silly) workaround:
$ini_lookup = regsubst($ini_data, "name=$hostname", '__FOUND__')
if $ini_lookup =~ /__FOUND__/ {
...
}
Final remark about security: If your team is adamant about not using $certname for this lookup (although it should be easy to map host names to cert names), you should consider adding the host name to your trusted facts.