I'm new to configuration management, just FYI.
I'm trying to puppetize elasticsearch, and want to have a master list of elasticsearch nodes in a file (which can be used for multiple things, not just this purpose).
I would like to add elasticsearch.yml via an ERB template and expand the list of FDQN's into the discovery.zen.ping.unicast.hosts: [] param.
For example I have an external file called es_hosts in module/files that contains:
host1.domain.com
host2.domain.com
host3.domain.com
host4.domain.com
Then when puppet builds the ERB template have this in the param:
discovery.zen.ping.unicast.hosts: ["host1.domain.com", "host2.domain.com", "host3.domain.com", "host4.domain.com"]
I've tried a few things, but I can't get my head wrapped around it.
I would be using this list for other things like building firewall rules, etc, so I'd like to have one master list for reference that can be updated by my team.
Thanks for any help!
Rather than have a list in a file, it would be better to have it in Hiera, since defining lists and other external data is specifically what Hiera is for.
(If you have not used Hiera yet, you definitely should read up on it.)
So in Hiera you would have:
---
es_hosts:
- host1.domain.com
- host2.domain.com
- host3.domain.com
- host4.domain.com
In your manifest, you would read that in from Hiera using the hiera function:
$es_hosts = hiera('es_hosts')
(Note that instead of the hiera function, we often use Puppet's Automatic Parameter Lookup feature instead to read data into our manifests from Hiera, but your requirement here - a list of ES hosts to be used in multiple contexts - suggests you will want this list not to be bound to a specific class input. If this does not make sense to you right now, you will need to learn about Parameterised Classes and Automatic Parameter Lookup, but it's otherwise not relevant to this answer.)
Finally, in your ERB template you would have:
discovery.zen.ping.unicast.hosts: ["<%= #es_hosts.join('", "') %>"]
Pay attention to the fact that the $es_hosts variable from your manifest is accessed via a Ruby instance variable #es_hosts in your ERB template.
Finally, note that there is an Elasticsearch Puppet module available on the Puppet Forget here. You may find that using that module is better than writing your own.
Related
We are using the roles pattern in puppet with hiera, meaning we have these lines in hiera.yaml:
- name: "Roles data"
path: "roles/%{::server_role}.yaml"
We have a custom fact that produces the role name when facter runs, but we would like to move this into hiera. Instead of the server_role variable being produced by facter, we want to specify the server_role inside of hiera, and let that variable be referenced elsewhere in hiera. Something like this:
hiera.yaml:
- name: "Per-node data"
path: "nodes/%{trusted.certname}.yaml"
- name: "Roles data"
path: "roles/%{lookup(server_role)}.yaml"
nodes/hostname.yaml:
server_role: foo_bar
I have seen this question, which says to use hiera() or lookup() but when I try to use those, I get this error message:
Interpolation using method syntax is not allowed in this context
So how can I use a hiera variable that's defined elsewhere in hiera?
Edit:
The prototypical code examples for defining roles could use any fact that's known to facter, often giving examples that are based on hostname. When you can't embed server config into hostname, a common(ish) workaround is to write a file such as /etc/server_role, but it seems to defeat the purpose of config management, when you need to ssh into a machine and edit a file. As the other comments & answer here so far mentioned, you could use an ENC, but again, the goal here is not to have config stored outside of version control. In fact, we have foreman as an ENC and we make a practice to never use it that way because then upgrades and other maintenance become unsustainable.
We could write a class which will pick up data from hiera, write it to /etc/server_role, and on the next puppet run, facter will pick that up and send it back to hiera, so then we'll have the server_role fact available to use in hiera.yaml. As gross as this sounds, so far, it's the best known solution. Still looking for better answers to this question.
Thanks.
As #MattSchuchard explained in comments, you cannot interpolate Hiera data into your Hiera config, because the config has to be known before the data can be looked up.
If you need a per-role level in your data hierarchy then an alternative would be to assign roles to machines via an external node classifier. You don't need it to assign any classes, just the server_role top-scope variable and probably also environment.
On the other hand, maybe you don't need a per-role level of your general hierarchy in the first place. Lots of people do roles & profiles without per-role data, but even if you don't want to do altogether without then it may be that module-specific data inside the module providing your role classes could be made to suffice.
Following is a sentence from Bitbake user's manual:
"BitBake parses each recipe and append file located with BBFILES and stores the values of various variables into the datastore."
What data type is 'datastore' ? Is it list or Tuple or Dictionary ? Or what data type is it?
Bitbake's datastore is complex store of key+value pairs where keys also have flags (also key+value pairs). Its a custom structure written with a copy on write backend. It supports the idea of 'overrides' where one variable with special naming can override another. See https://git.openembedded.org/bitbake/tree/lib/bb/data_smart.py and https://git.openembedded.org/bitbake/tree/lib/bb/data.py within the codebase for the implementation, the Bitbake manual for information about how to use the data store and https://git.openembedded.org/bitbake/tree/lib/bb/tests/data.py for unittests of it.
You can work out the type of an object in python by executing type(foo) in the same environment. As for that specific type (datastore), a quick google indicates that it's neither a tuple or a dictionary, but a custom object with it's API documented here.
I am trying to set a module's source (this IS NOT a resource) based on a conditional trigger but it looks like the module is getting fired before the logic is applied:
module "my_module" {
source = "${var.my_field == "" ? var.standard_repo : var.custom_repo}"
stuff...
more stuff...
}
I have created the standard_repo and custom_repo vars as well and defined with URLs for respective repos (using git:: -- this all works w/o conditional)
All this being said, anyone know of a way to implement this conditional aspect? (again, this is a module and not a resource)
I tried using duplicate modules and calling based off the var value but this, too, does not work (condition is never met, even when it is):
repo = ["${var.my_field == "na" ? module.my_module_old : module.my_module_new}"]
One way to achieve this is described in this post
Basically, a common pattern is to have several folders for different environments such as dev/tst/prd. These environments often reuse large parts of the codebase. Some may be abstracted as modules, but there is still often a large common file which is either copy-pasted or symlinked.
The post offers a way that doesn't conditionally disable based on variables but it probably solves your issue of enabling a module based on different enviornments. It makes use of the override feature of terraform and adds a infra_override.tf file. Here, it defines a different source for the module which points to an empty directory. Voila, a disabled module.
Variables are not allowed to be used in the module source parameter. There also does not seem to be a plan for this to change. https://github.com/hashicorp/terraform/issues/1439 . Creating a wrapper script , or using something like mustache http://mustache.github.io/ seems to be the best way to solve the problem.
I'm using hiera to assign classes like webserver or dbserver to my nodes. The webserver class just includes apache and sets some config on it (e.g. port). Obviously I don't want to replicate this config for every node, so I put it in common.yaml. However, my common.yaml is getting big, so I want to split it up. I'd like to have one file containing the config for the webserver role, another for the dbserver role etc. I'm imagining my hiera.yaml to look something like this:
:hierarchy:
- "fqdn/%{::fqdn}"
- "role/%{ROLE}"
- common
Where the role folder would contain files like webserver.yaml, appserver.yaml, dbserver.yaml. I've seen various blog posts saying that the solution is to create a custom 'role' fact, but most of them achieve this by loading that fact from a file on the agent node (e.g from /etc/role), which to me seems to defeat the point of puppet (I use puppet specifically so I don't have to log into each node and change some config every time I want it to have some new role).
To be clear, I don't want to have to edit files on the agent to get this to work, I want it all done using the config that's on the master.
I guess I could have something like the following and exhaustively list every role as an element in the hierarchy, but that doesn't seem that manageable.
:hierarchy:
- "fqdn/%{::fqdn}"
- "webserver"
- "appserver"
- "dbserver"
- common
Is there any way to solve this?
To be able to use $Role in your hiera config, it needs to be supplied as a fact/variable, however there is a way to do this on the master instead of on the node. This is one of the things that External Node Classifiers can be used for.
Basically, you need to write a script that takes the node name and prints out yaml that includes the Role parameter's value. For example, you could have one yaml file that is just a map of node names to roles, and then the script does a lookup and prints the result (as a parameter in the linked schema). Here is an example.
There are also more robust ENC's out there, if you are interested in new tooling. For example, Foreman gives you a web interface for grouping hosts together into similar roles, setting parameters to inject into puppet runs, etc.
I've come up with a solution for this. Only disadvantage is that the max number of roles is hardcoded. This will be better with hiera 3 until then try this:
/etc/puppet/hiera.yaml
---
:backends:
- yaml
:yaml:
:datadir: /etc/puppet/hieradata
:hierarchy:
- 'nodes/%{::clientcert}'
- 'roles/%{::role_4}'
- 'roles/%{::role_3}'
- 'roles/%{::role_2}'
- 'roles/%{::role_1}'
- common
/etc/puppet/manifests/site.pp
# Get roles
$roles = hiera_array('roles', [])
# Declare Roles in vars (not needed in puppet 4)
$role_1 = $roles[0]
$role_2 = $roles[1]
$role_3 = $roles[2]
$role_4 = $roles[3]
# Include Classes
hiera_include('classes')
/etc/puppet/hieradata/roles/webserver.yaml
---
classes:
- nginx
# put nginx config here
/etc/puppet/hieradata/nodes/your_node_name.yaml
---
roles:
- webserver
classes:
# put node specific stuff here
So I have a directory of puppet manifests that I want to run.
Is it possible to do something like:
include /etc/puppet/users/server522/*.pp
and have puppet run them?
I've tried
include users::server522::*
and several other variations
I always get an error about puppet being unable to find it.
Is there anyway to do this?
So my final solution to this was write a script that would take the directory listing and for each .pp file add an include into the server522.pp file. Quite annoying that puppet won't include an entire directory.
What are you trying to do here, and are you sure you're doing it the correct way? To wit, if you have multiple manifests corresponding to multiple servers, you need to define the nodes for each server. If OTOH you're trying to apply multiple manifests to a single node it's not clear why you would be doing that, instead of just using your defined classes. A little more information would be helpful here.
I do not see the point of each user having its own manifest. I would rather create script that would automatically build one manifest file, basing on data from some source, for instance from HEAD of git repository containing CSV file with current list of users.
If you realy want to use separate manifest file for every user you may consider having seprate module for every user:
manifests
default.pp <-- here comes default manifest
module_for_user_foo/
manifests/
init.pp <-- here comes your 'foo' user
module_for_user_bar/
manifests/
init.pp <-- here comes your 'bar' user
Now you may copy modules containing manifests.