hiera.yaml
---
:hierarchy:
- node/%{host_fqdn}
- site_config/%{host_site_name}
- site_config/perf_%{host_performance_class}
- site_config/%{host_type}_v%{host_type_version}
- site/%{host_site_name}
- environments/%{site_environment}
- types/%{host_type}_v%{host_type_version}
- hosts
- sites
- users
- common
# options are native, deep, deeper
:merge_behavior: deeper
We currently have this hiera config. So the config gets merged in the following sequence common.yaml > users.yaml > sites.yaml > hosts.yaml > types/xxx_vxxx.yaml > etc. For the variable top hierarchies, it gets overwritten only if that file exists.
eg:
common.yaml
server:
instance_type: m3.medium
site_config/mysite.yaml
server:
instance_type: m4.large
So for all other sites, the instance type will be m3.medium, but only for mysite it will be m4.large.
How can I achieve the same in Ansible?
I think that #Xiong is right that you should go the variables way in Ansible.
You can set up flexible inventory with vars precedence from general to specific.
But you can try this snippet if it helps:
---
- hosts: loc-test
tasks:
- include_vars: hiera/{{ item }}
with_items:
- common.yml
- "node/{{ ansible_fqdn }}/users.yml"
- "node/{{ ansible_fqdn }}/sites.yml"
- "node/{{ ansible_fqdn }}/types/{{ host_type }}_v{{ host_type_version }}.yml"
failed_when: false
- debug: var=server
This will try to load variables from files with structure similar to your question.
Nonexistent files are ignored (because of failed_when: false).
Files are loaded in order of this list (from top to bottom), overwriting previous values.
Gotchas:
all variables that you use in the list must be defined (e.g. host_type in this example can't be defined in common.yml), because list of items to iterate is templated before the whole loop is executed (see update for workaround).
Ansible overwrite(replace) dicts by default, I guess your use case expects merging behavior. This can be achieved with hash_behavior setting – but this is unusual for Ansible playbooks.
P.S. You may alter top-to-bottom-merge behavior by changing with_items to with_first_found and reverse the list (from specific to general). In this case Ansible will load variables from first file found.
Update: use variables from previous includes in file path.
You can split the loop into multiple tasks, so Ansible will evaluate each task's result before templating next file's include path.
Make hiera_inc.yml:
- include_vars: hiera/common.yml
failed_when: false
- include_vars: hiera/node/{{ ansible_fqdn }}/users.yml
failed_when: false
- include_vars: hiera/node/{{ ansible_fqdn }}/sites.yml
failed_when: false
- include_vars: hiera/node/{{ ansible_fqdn }}/types/{{ host_type | default('none') }}_v{{ host_type_version | default('none') }}.yml
failed_when: false
And in your main playbook:
- include: hiera_inc.yml
This looks a bit clumsy, but this way you can define host_type in common.yaml and it will be honored in the path templating for next tasks.
With Ansible 2.2 it will be possible to include_vars into named variable (not global host space), so you can include_vars into hiera_facts and use combine filter to merge them without altering global hash behavior.
I'm not familiar with Puppet, so this may not be a direct mapping. But what I understand your question to be is "how do I use values in one shared location but override their definitions for different servers?". In Ansible, you do this with variables.
You can define variables directly in your inventory. You can define variables in host- and group-specific files. You can define variables at a playbook level. You can define variables at a role level. Heck, you can even define variables with command-line switches.
Between all of these places, you should be able to define overrides to suit your situation. You'll probably want to take a look at the documentation section on how to decide where to define a variable for more info.
It seems a little more basic than Hiera, but somebody has created a basic ansible lookup plugin with similar syntax
https://github.com/sailthru/ansible-oss/tree/master/tools/echelon
Related
How can one use !reference in variables block and then extend the list of entries?
For example, I want to be able to do something like
variables:
!reference [.common_variables, variables]
FOO: "bar"
At the moment I get an error in the GitLab linter:
This GitLab CI configuration is invalid: (): did not find expected key while parsing a block mapping at line 1 column 1
P.S. Note that this question is specifically about !reference.
Yaml anchors are out of scope of this question, as I want to place .common_variables into a separate file that I include in my pipeline.
extends: mechanism is out of scope of this question. It does work, but it has a weird behaviour that one also gets an extends variable in the environment, see GitLab: use `extends` in `variables`
variables: !reference [.common_variables, variables]
Is the correct way to reference, however gitlab yml structure will not support additional definitions after the !reference keyword.
My advice would be to have a more specific definition in your variables file that uses anchors to share the common varaibles.
e.g:
in 'variables.gitlab-ci.yml':
.common_variables: &common_variables
COMMON_VAR: value
.variables:Example-job
variables:
<<: *common_variables
FOO: "bar"
in '.gitlab.ci.yml':
Example-job:
variables: !reference [".variables:Example-job", "variables"]
script:
...
I achieve it by using multiple extends, one of them, .contract-env-vars-default, containing my shared variables.
# base template
.contract-env-vars-default:
variables:
GIT_CURRENT_BRANCH: $CI_COMMIT_BRANCH
PACT_BROKER_CLI: ./pact/bin/pact-broker
…
Then using two extends
# adding language specific variable
.contract-verify-java:
extends:
- .gradle-test-base
- .contract-env-vars-default # < < < < < < < <
variables:
ENVIRONMENT: contract-test # adding language specific variable
And finally
# adding project specific variable
gradle-contract-testing-verify:
extends: .contract-verify-java
variables:
PROJECT_SPECIFIC: foo_bar_biz # add project specific
Thus I got the expected result of:
gradle-contract-testing-verify:
variables:
CACHE_FALLBACK_KEY: main-cache-non_protected # platform?
GIT_CURRENT_BRANCH: "$CI_COMMIT_BRANCH" # base
PACT_BROKER_CLI: "./pact/bin/pact-broker" # base
…
ENVIRONMENT: contract-test # language specific
PROJECT_SPECIFIC: foo_bar_biz # project specific
variables:
!reference [.common_variables, variables]
FOO: "bar"
or if you want to assign the variables to a new variable, you can use this
variables:
MY_VAR: !reference [.common_variables, variables]
For additional information - https://docs.gitlab.com/ee/ci/yaml/yaml_optimization.html#reference-tags
is there anyway to do multiple AND variable expression?
let's say
.template1:
only:
variables:
- $flag1 == "true"
.template2:
only:
variables:
- $flag2 == "true"
job1:
extends:
- .template1
- .template2
script: echo "something"
How will this get evaluated?
Is this going to result in only:variables overwriting each other thus template2 is the final result?
or is this going to result in a combined variables such that it becomes an OR statement
only:
variables:
- $flag1 == "true"
- $flag2 == "true"
Is there anyway to make it as and AND statement instead? keeping the templating system, and without using rules: if since using rules if has its own quirk, triggering multiple pipeline during merge request
Problem
Any time two jobs get "merged", either using extends or anchors, GitLab will overwrite one section with another. The sections don't actually get merged. In your case, you're extending from two jobs, so GitLab will completely overwrite the first variables section with the second.
Solution
One way to achieve your desired result is by defining the variables in your template jobs. The problem you will have then is the two variables sections will overwrite each other. So..
You can use a before_script section to define the variable in the 2nd template. This approach works for your specific case of 2 templates. You can use script and after_script if you need a third template, buy you'd have to use a more advanced approach if you need more templates than that.
.template1:
# We can define a variables section here, no problem
variables:
flag1: "true"
.template2:
# You can't define a second variables section here, since it will overwrite the first
# Instead, define the environment variable directly in a before-script section
before_script:
- export flag2="true"
job1:
extends:
- .template1
- .template2
only:
variables:
- $flag1 == "true"
- $flag2 == "true"
script: echo "something"
I'm attempting to implement encrypted values in yaml in Hiera 5 to inject passwords securely into Puppet (enterprise) 5.3 via automatic lookup. There's excellent guidance from the Puppet blog and PUP-7284 on the necessary setup.
However, I can't seem to get lookup_options correct to ensure conversion to a Sensitive type (to match the class parameters).
Asserting with the puppet lookup command fails with:
[user#rhel7 ~]$ puppet lookup my_module::db_pass --environment test --type Sensitive[String]
Error: Could not run: Found value has wrong type, expects a Sensitive value, got String
It also appears the lookup_options are being found and they look sensible:
[user#rhel7 ~]$ puppet lookup my_module::db_pass --environment test --explain-options
Hierarchy entry "Passwords"
Path "/etc/puppetlabs/code/environments/test/modules/my_module/data/secrets.eyaml"
Original path: "secrets.eyaml"
Found key: "lookup_options" value: {
"^my_module::.*pass$" => {
"convert_to" => "Sensitive"
}
}
Decryption works just fine (unfortunately to cleartext -- not sure if that's expected?)
[user#rhel7 ~]$ puppet lookup my_module::db_pass --environment test
Found key: "my_module::db_pass" value: "password_is_taco"
The setup is as follows:
[user#rhel7 /etc/puppetlabs/puppet/environment/test/modules/my_module]$ cat hiera.eyaml
---
version: 5
defaults:
data_hash: yaml_data
datadir: data
hierarchy:
- name: "Passwords"
lookup_key: eyaml_lookup_key
paths:
- "secrets.eyaml"
options:
pkcs7_private_key: "/etc/puppetlabs/puppet/keys/private_key.pkcs7.pem"
pkcs7_public_key: "/etc/puppetlabs/puppet/keys/public_key.pkcs7.pem"
[user#rhel7 /etc/puppetlabs/puppet/environment/test/modules/my_module]$ cat ./data/secrets.eyaml
---
lookup_options:
'^my_module::.*pass$':
convert_to: "Sensitive"
my_module::db_pass: >
ENC[PKCS7,MIIBqQYJKoZ...snip]
I've also been unsuccessful with different regexes and/or just using keys directly:
lookup_options:
my_module::db_pass:
convert_to: "Sensitive"
Apologies in advance for any minor copy-paste issues with obfuscated code :)
I never quite figured out why the specific test setup above I tried never worked, but here's what I ultimately ended up implementing:
---
lookup_options:
"^my_module::.*(password|token)$":
convert_to: Sensitive
The pattern match will appropriately cast any of the following to Sensitive[String]:
my_module::password
my_module::service_password
my_module::api_token
my_module::any_number::of_subclasses::token_or_password
If you're considering going through this same process, you might consider:
instead of hard-coding your passwords, try leveraging new Puppet6+ features that avoid obfuscation altogether (by NOT putting credentials in catalogs):
https://puppet.com/blog/agent-side-functions-in-puppet-6
https://www.hashicorp.com/resources/agent-side-lookups-with-hashicorp-vault-puppet-6
If that's not feasible, follow the recommended process (and don't forget
to restart the Puppet Server)
make liberal use of the puppet lookup utility, especially with the --explain-options and type Sensitive[String]
I would like to create a default users for all servers, but in addition of this default uses, only for specific servers I want to create in addition of the default users a specifics ones.
My problem is that when I run puppet agent -t, puppet only create the users for the first match. If the server match in - node/%{::fqdn} create only the specific users but not the default ones.
in /etc/puppet/hiera.yaml I have the follow:
:backends:
- yaml
:yaml:
:datadir: "/etc/puppet/hieradata"
:hierarchy:
- node/%{::fqdn}
- common
How I can set up hiera in order to always run the common file?
Please use hiera hash merge. Define merge behaviour in hiera.yaml, possible values are native, deep, deeper e.g:
:merge_behavior: deeper
And than just use hiera. According to documentation:
In a deeper hash merge, Hiera recursively merges keys and values in each source hash.
Here you have merge behaviour in examples.
UPDATE:
I have setup the following simple example:
hiera.yaml:
:hierarchy:
- apps
- common
:merge_behavior: deeper
apps.yaml:
test_hash:
abc1:
value: apps
abc2:
value: apps
common.yaml:
test_hash:
abc1:
value: comm
abc3:
value: comm
test_hash.pp
class test_hash
{
$normal_hash = hiera('test_hash')
$hiera_hash = hiera_hash('test_hash')
notify{ " normal: ${normal_hash}":}
notify{ " hiera : ${hiera_hash}":}
}
include test_hash
Next call puppet script puppet apply test_hash.pp
In result:
Notice: normal: {"abc1"=>{"value"=>"apps"}, "abc2"=>{"value"=>"apps"}}
Notice: hiera : {"abc1"=>{"value"=>"apps"}, "abc3"=>{"value"=>"comm"}, "abc2"=>{"value"=>"apps"}}}
UPDATE2:
You can also consider using merge function from stdlib. But probably to use it you will have to change a bit your architecture e.g:
In common define common values, in node/%{::fqdn} define node specific values, and than use it as in example:
$common_hash = hiera('something_from_common')
$node_hash = hiera('something_from_fqdn')
$merged_hash = merge($node_hash, $common_hash)
(Yes it is a bit ugly :))
I have a hierachy like this:
- "nodes/%{::certname}"
- (what's here is my question)
- common
I'd like to assign a group to my nodes in their individual configuration in hiera, like this in nodes/hostname.yaml :
---
group: alpha
Now, I'd like to have a file alpha.yaml, where I state group-specific settings.
So my question is how do I write the hierachy to ask hiera for the filename of the group definition?
Is there another way to achieve this?
You can. Make sure you have the group defined in Facts.
- "nodes/%{::certname}"
- "%{::group}"
- common
So you can test with below command
FACTER_group=alpha puppet apply your.pp
For custom facts, you can go through this document: Custom Facts Walkthrough