I'm using hiera to assign classes like webserver or dbserver to my nodes. The webserver class just includes apache and sets some config on it (e.g. port). Obviously I don't want to replicate this config for every node, so I put it in common.yaml. However, my common.yaml is getting big, so I want to split it up. I'd like to have one file containing the config for the webserver role, another for the dbserver role etc. I'm imagining my hiera.yaml to look something like this:
:hierarchy:
- "fqdn/%{::fqdn}"
- "role/%{ROLE}"
- common
Where the role folder would contain files like webserver.yaml, appserver.yaml, dbserver.yaml. I've seen various blog posts saying that the solution is to create a custom 'role' fact, but most of them achieve this by loading that fact from a file on the agent node (e.g from /etc/role), which to me seems to defeat the point of puppet (I use puppet specifically so I don't have to log into each node and change some config every time I want it to have some new role).
To be clear, I don't want to have to edit files on the agent to get this to work, I want it all done using the config that's on the master.
I guess I could have something like the following and exhaustively list every role as an element in the hierarchy, but that doesn't seem that manageable.
:hierarchy:
- "fqdn/%{::fqdn}"
- "webserver"
- "appserver"
- "dbserver"
- common
Is there any way to solve this?
To be able to use $Role in your hiera config, it needs to be supplied as a fact/variable, however there is a way to do this on the master instead of on the node. This is one of the things that External Node Classifiers can be used for.
Basically, you need to write a script that takes the node name and prints out yaml that includes the Role parameter's value. For example, you could have one yaml file that is just a map of node names to roles, and then the script does a lookup and prints the result (as a parameter in the linked schema). Here is an example.
There are also more robust ENC's out there, if you are interested in new tooling. For example, Foreman gives you a web interface for grouping hosts together into similar roles, setting parameters to inject into puppet runs, etc.
I've come up with a solution for this. Only disadvantage is that the max number of roles is hardcoded. This will be better with hiera 3 until then try this:
/etc/puppet/hiera.yaml
---
:backends:
- yaml
:yaml:
:datadir: /etc/puppet/hieradata
:hierarchy:
- 'nodes/%{::clientcert}'
- 'roles/%{::role_4}'
- 'roles/%{::role_3}'
- 'roles/%{::role_2}'
- 'roles/%{::role_1}'
- common
/etc/puppet/manifests/site.pp
# Get roles
$roles = hiera_array('roles', [])
# Declare Roles in vars (not needed in puppet 4)
$role_1 = $roles[0]
$role_2 = $roles[1]
$role_3 = $roles[2]
$role_4 = $roles[3]
# Include Classes
hiera_include('classes')
/etc/puppet/hieradata/roles/webserver.yaml
---
classes:
- nginx
# put nginx config here
/etc/puppet/hieradata/nodes/your_node_name.yaml
---
roles:
- webserver
classes:
# put node specific stuff here
Related
We are using the roles pattern in puppet with hiera, meaning we have these lines in hiera.yaml:
- name: "Roles data"
path: "roles/%{::server_role}.yaml"
We have a custom fact that produces the role name when facter runs, but we would like to move this into hiera. Instead of the server_role variable being produced by facter, we want to specify the server_role inside of hiera, and let that variable be referenced elsewhere in hiera. Something like this:
hiera.yaml:
- name: "Per-node data"
path: "nodes/%{trusted.certname}.yaml"
- name: "Roles data"
path: "roles/%{lookup(server_role)}.yaml"
nodes/hostname.yaml:
server_role: foo_bar
I have seen this question, which says to use hiera() or lookup() but when I try to use those, I get this error message:
Interpolation using method syntax is not allowed in this context
So how can I use a hiera variable that's defined elsewhere in hiera?
Edit:
The prototypical code examples for defining roles could use any fact that's known to facter, often giving examples that are based on hostname. When you can't embed server config into hostname, a common(ish) workaround is to write a file such as /etc/server_role, but it seems to defeat the purpose of config management, when you need to ssh into a machine and edit a file. As the other comments & answer here so far mentioned, you could use an ENC, but again, the goal here is not to have config stored outside of version control. In fact, we have foreman as an ENC and we make a practice to never use it that way because then upgrades and other maintenance become unsustainable.
We could write a class which will pick up data from hiera, write it to /etc/server_role, and on the next puppet run, facter will pick that up and send it back to hiera, so then we'll have the server_role fact available to use in hiera.yaml. As gross as this sounds, so far, it's the best known solution. Still looking for better answers to this question.
Thanks.
As #MattSchuchard explained in comments, you cannot interpolate Hiera data into your Hiera config, because the config has to be known before the data can be looked up.
If you need a per-role level in your data hierarchy then an alternative would be to assign roles to machines via an external node classifier. You don't need it to assign any classes, just the server_role top-scope variable and probably also environment.
On the other hand, maybe you don't need a per-role level of your general hierarchy in the first place. Lots of people do roles & profiles without per-role data, but even if you don't want to do altogether without then it may be that module-specific data inside the module providing your role classes could be made to suffice.
I am using hiera5 and I want to read hierarchies from one of the groups based on their names.
For e.g. Currently I have:
version: 5
defaults:
datadir: "/"
data_hash: yaml_data
hierarchy:
- name: "ABC"
paths:
- "env/env-data1.yaml"
- "env/env-data2.yaml"
- "env/env-data3.yaml"
I want to add multiple hierarchies like this:
hierarchy:
- name: "ABC"
paths:
- "env/env-data1.yaml"
- "env/env-data2.yaml"
- "env/env-data3.yaml"
- name: "DEF"
paths:
- "env/env-data4.yaml"
- "env/env-data5.yaml"
- "env/env-data6.yaml"
- name: "GHI"
paths:
- "env/env-data7.yaml"
- "env/env-data8.yaml"
- "env/env-data9.yaml"
And read files from just one of the groups at a time lets say just 'ABC'.
Is there a way I can pass hierarchy name as parameter while calling puppet script ?
Additional Info:
I am running a shell script from jenkins which triggers the puppet script.
puppet apply --hiera_config=${WORKSPACE}/hiera.yaml ${WORKSPACE}/puppet-script.pp
Thanks in advance!
I want to add multiple hierarchies like this: [...]
The syntax is fine, but it does not describe multiple hierarchies. It describes one multi-level hierarchy (which is in fact the usual case). Having multiple levels is what makes it hierarchical -- it is the distinguishing characteristic of Hiera relative to other external-data mechanisms that Puppet has had.
And read files from just one of the groups at a time lets say just 'ABC'.
Nope, not happening. Hiera does not work that way.
Is there a way I can pass hierarchy name as parameter while calling puppet script ?
No, but you can specify a different hiera.yaml configuration file, as you already demonstrate. If you want different hierarchies for different runs then define each in its own (full) configuration file, and choose the appriate config file for each use.
I'm new to configuration management, just FYI.
I'm trying to puppetize elasticsearch, and want to have a master list of elasticsearch nodes in a file (which can be used for multiple things, not just this purpose).
I would like to add elasticsearch.yml via an ERB template and expand the list of FDQN's into the discovery.zen.ping.unicast.hosts: [] param.
For example I have an external file called es_hosts in module/files that contains:
host1.domain.com
host2.domain.com
host3.domain.com
host4.domain.com
Then when puppet builds the ERB template have this in the param:
discovery.zen.ping.unicast.hosts: ["host1.domain.com", "host2.domain.com", "host3.domain.com", "host4.domain.com"]
I've tried a few things, but I can't get my head wrapped around it.
I would be using this list for other things like building firewall rules, etc, so I'd like to have one master list for reference that can be updated by my team.
Thanks for any help!
Rather than have a list in a file, it would be better to have it in Hiera, since defining lists and other external data is specifically what Hiera is for.
(If you have not used Hiera yet, you definitely should read up on it.)
So in Hiera you would have:
---
es_hosts:
- host1.domain.com
- host2.domain.com
- host3.domain.com
- host4.domain.com
In your manifest, you would read that in from Hiera using the hiera function:
$es_hosts = hiera('es_hosts')
(Note that instead of the hiera function, we often use Puppet's Automatic Parameter Lookup feature instead to read data into our manifests from Hiera, but your requirement here - a list of ES hosts to be used in multiple contexts - suggests you will want this list not to be bound to a specific class input. If this does not make sense to you right now, you will need to learn about Parameterised Classes and Automatic Parameter Lookup, but it's otherwise not relevant to this answer.)
Finally, in your ERB template you would have:
discovery.zen.ping.unicast.hosts: ["<%= #es_hosts.join('", "') %>"]
Pay attention to the fact that the $es_hosts variable from your manifest is accessed via a Ruby instance variable #es_hosts in your ERB template.
Finally, note that there is an Elasticsearch Puppet module available on the Puppet Forget here. You may find that using that module is better than writing your own.
I have a running puppet master-agent setup and currently trying to figure out how to use hiera to provision php.
My Puppetfile:
forge "http://forge.puppetlabs.com"
mod "jfryman/nginx"
mod "puppetlabs/mysql"
mod "mayflower/php"
mod 'puppetlabs-vcsrepo'
mod 'puppetlabs/ntp', '4.1.0'
mod 'puppetlabs/stdlib'
My site.pp:
hiera_include('classes')
My environment.conf, where the modulepath is maintained:
manifest = site.pp
modulepath = modules:site
My hiera config on puppet master at /etc/puppetlabs/puppet/hiera.yml:
---
:backends:
- yaml
:hierarchy:
- "nodes/%{::trusted.certname}"
- "environment/%{server_facts.environment}"
- common
:yaml:
# datadir is empty here, so hiera uses its defaults:
# - /etc/puppetlabs/code/environments/%{environment}/hieradata on *nix
# - %CommonAppData%\PuppetLabs\code\environments\%{environment}\hieradata on Windows
# When specifying a datadir, make sure the directory exists.
:datadir:
From what I understand, general config that should be present on all servers goes into common.yaml. With this setup, I managed to install ntp on my node with this config at hieradata/common.yaml:
---
classes:
- 'profile::base'
ntp::servers:
- server 0.de.pool.ntp.org
- server 1.de.pool.ntp.org
- server 2.de.pool.ntp.org
- server 3.de.pool.ntp.org
Now, my hierarchy also states that all node specific config should go into hieradata/nodes/{fqdn-of-the-node}.yml.
Now, finally coming to my questions:
I have a file hieradata/nodes/myserver.example.com.yml which holds this:
classes:
- 'profile::php'
And a manifest under site/profile/manifests/php.pp:
class profile::php {
class { '::php': }
}
But this does not provision php. As you saw, I use mayflower/php from the forge.
Now, my two questions are:
Is my hiera file for php in the right location? What am I missing then to make it provision php to my agent?
You have multiple issues/possibilities here, so let us go through them iteratively.
First, you are using the default datadir of:
/etc/puppetlabs/code/environments/%{environment}/hieradata
However, you have a priority of:
"environment/%{server_facts.environment}"
This does not make sense, since you have a priority that distinguishes data for nodes based on their directory environment, but you also are placing hieradata directly in directory environments. If you want priority based on directory environment, then change your hieradata directory to be outside the direct environments at:
/etc/puppetlabs/code/hieradata
Otherwise, you should remove that level from your priority as it adds no value and will increase lookup times.
Second, you did not show your site.pp, but did you remember your hiera_include('classes')? That will lookup the array classes and then include them, which is what it seems you want. If you are not doing it, then the node provisioning issue you described would occur.
Third, is site in your modulepath? You need to append it in either your puppet.conf or your environment.conf.
Fourth, your node's fqdn may not match the certname. Check the certs directory on your Puppetmaster for the node's cert.
Side notes:
The first half of your question contains a lot of extraneous information and is missing a lot of helpful relevant information. Please consider editing the question to provide more helpful information and to be more concise.
Since ntp worked, I am assuming your module install with r10k into the environment directories succeeded. Also I am assuming that the modules are present for the directory environment of your node.
There is no real reason to specify the php class as global in your declaration with ::php.
So I have a directory of puppet manifests that I want to run.
Is it possible to do something like:
include /etc/puppet/users/server522/*.pp
and have puppet run them?
I've tried
include users::server522::*
and several other variations
I always get an error about puppet being unable to find it.
Is there anyway to do this?
So my final solution to this was write a script that would take the directory listing and for each .pp file add an include into the server522.pp file. Quite annoying that puppet won't include an entire directory.
What are you trying to do here, and are you sure you're doing it the correct way? To wit, if you have multiple manifests corresponding to multiple servers, you need to define the nodes for each server. If OTOH you're trying to apply multiple manifests to a single node it's not clear why you would be doing that, instead of just using your defined classes. A little more information would be helpful here.
I do not see the point of each user having its own manifest. I would rather create script that would automatically build one manifest file, basing on data from some source, for instance from HEAD of git repository containing CSV file with current list of users.
If you realy want to use separate manifest file for every user you may consider having seprate module for every user:
manifests
default.pp <-- here comes default manifest
module_for_user_foo/
manifests/
init.pp <-- here comes your 'foo' user
module_for_user_bar/
manifests/
init.pp <-- here comes your 'bar' user
Now you may copy modules containing manifests.