How to conveniently subscribe to multiple recently declared resources? - puppet

How do I conveniently trigger a refresh on some resource upon a change in one of many other resources (possibly of various resource types) that were declared nearby, in puppet 4?
E.g.
# these resources
resource_type1 {...}
resource_type2 {...}
resource_type2 {...}
resource_type2 {...}
resource_type3 {...}
# should refresh this when changed
service {...}
I know I could list each resource in subscribe, and use resource collectors in it; but I'd like something more fitting for the chronically lazy.
I considered using a with block, but this doesn't work:
with() || {
# these resources
resource_type1 {...}
resource_type2 {...}
resource_type2 {...}
resource_type2 {...}
resource_type3 {...}
}
~>
# will refresh this when any of them have changed
service {...}
Only the last resource within the with block triggers a refresh.
You can test this for yourself using:
with() || {
file { '/etc/test':
content => 'Change this and notice it does not trigger',
}
file { '/etc/test2':
content => 'foo',
}
}
~>
exec {'/bin/echo triggered':
refreshonly => true,
}

In Puppet 4, while still leading to a good deal of boiler plate, a robust way of doing it is to use a class to group the resources:
class config {
resource_type1 {...}
resource_type2 {...}
resource_type2 {...}
resource_type2 {...}
resource_type3 {...}
}
class { 'config': }
~>
service {...}
When part of a class, you'll want to add a contain as well (here's why), like so:
class foo {
class { 'foo::config': }
~>
service {...}
contain 'foo::config'
}

Related

Puppet - Set defined types in Nodes.pp

How to overwrite defined type in nodes.pp? I want to able to set custom domain using nodes.pp. Case Default isn't an option.
I'm using puppet 6.0..
The following method doesn't work. It says Could not find declared class resolv::resolv_config.
It looks like it used to work in 3.0 according to this answer.
nodes.pp
node "test001" {
class { 'resolv::resolv_config':
domain => "something.local",
}
}
modules/resolv/manifests/init.pp
class resolv {
case $hostname {
/^[Abc][Xyz]/: {
resolv:resolv_config { 'US':
domain => "mydomain.local",
}
}
}
}
define resolv::resolv_config($domain){
file { '/etc/resolv.conf':
content => template("resolv/resolv.conf.erb"),
}
}
resolv.conf.erb
domain <%= #domain %>
There are a couple of problems here, but the one causing the "could not find declared class" error is that you are using the wrong syntax for declaring a defined type. Your code should be something like this:
node "test001" {
resolv::resolv_config { 'something.local':
domain => "something.local",
}
}
There are examples of declaring defined types in the documentation, https://puppet.com/docs/puppet/latest/lang_defined_types.html.
Once you get that working, you'll find another problem, in that this definition
define resolv::resolv_config($domain){
file { '/etc/resolv.conf':
content => template("resolv/resolv.conf.erb"),
}
}
will cause a error if you try to declare more than one resolv::resolv_config, because they will both try to declare the /etc/resolv.conf file resource. You almost certainly wanted to use a file_line resource.

puppet 4.7 - how can I execute something at the end of a run?

I need to execute something as the very last thing of a puppet apply run. I tried to do that by defining a stage 'last', but the syntax restrictions on declaring a class in resource mode are a problem.
Is there a good way to use stages like this? Or is there some other way to make sure some class is executed last?
for example, this gives me an error for a duplicate declaration(sometimes, and I'm not sure why at this point):
class xyz::firstrun {
exec { 'exec_firstrun':
onlyif => '/usr/bin/test -e /tmp/firstrun.sh',
command => '/tmp/firstrun.sh',
path => ['/usr/bin/','/usr/sbin'],
creates => '/tmp/firstrun.done',
}
}
class { 'xyz::firstrun':
stage => last,
}
Sometimes, the firstrun class runs without error, but in the main stage.
I'm not a big fan of run stages, but they are the the right tool for this job. It's a bit unclear exactly what gives you the duplicate declaration error you describe, but if, for example, your class definition and class declaration both appear in the same file, then that might be a problem.
Here's how a solution using run stages might look:
environments/production/modules/site/manifests/stages.pp
class site::stages {
stage { 'last':
# Stage['main'] does not itself need to be declared
require => Stage['main'],
}
}
environments/production/modules/xyz/manifests/firstrun.pp
class xyz::firstrun {
exec { 'exec_firstrun':
onlyif => '/usr/bin/test -e /tmp/firstrun.sh',
command => '/tmp/firstrun.sh',
path => ['/usr/bin/','/usr/sbin'],
creates => '/tmp/firstrun.done',
}
}
environments/production/manifests/nodes.pp
node 'foobar.my.com' {
include 'site::stages'
include 'something::else'
# Must use a resource-like declaration to assign a class to a stage
class { 'xyz::firstrun':
stage => 'last'
}
}
Note that although include-like class declarations are generally to be preferred, you must use a resource-like declaration to assign a class to a non-default stage. You must therefore also be careful to avoid declaring such classes more than once.
You can use puppet relationship and ordering to do this.
(1) If you want to execute the entire class at the end, you can include your class in init.pp and user ordering arrow (->) to execute it after all other classes.
example:
file: /etc/puppet/modules/tomcat/init.pp
class tomcat {
include ::archive
include ::stdlib
class { '::tomcat::tomcatapiconf': }->
class { '::tomcat::serverconfig': }
}
(2) If you want a specific resource in a class to execute at the end, you can use the same arrow (->) within the class or use before or require in the resource
example:
file { '/etc/profile.d/settomcatparam.sh':
ensure => file,
before => File_line['/etc/profile.d/settomcatparam.sh'],
}
file_line { '/etc/profile.d/settomcatparam.sh':
path => '/etc/profile.d/settomcatparam.sh',
ine => 'export LD_LIBRARY_PATH=/usrdata/apps/sysapps/apr/lib:/usrdata/apps/sysapps/apr-util/lib:/usrdata/apps/sysapps/tomcat-native/lib:$LD_LIBRARY_PATH',
}
OR
exec { 'configure apr-util':
cwd => "/tmp/apr-util-${tomcat::aprutilversion}/",
command => "bash -c './configure --prefix=/usrdata/apps/sysapps/apr-util --with-apr=/usrdata/apps/sysapps/apr'",
} ->
exec { 'make apr-util':
cwd => "/tmp/apr-util-${tomcat::aprutilversion}/",
command => 'make',
}
You can also use combination of before, require and ->. Just make sure you don't create a dependency cycle.

Gradle plugin best practices for tasks that depend on extension objects

I would like feedback on the best practices for defining plugin tasks that depend on external state (i.e. defined in the build.gradle that referenced the plugin). I'm using extension objects and closures to defer accessing those settings until they're needed and available. I'm also interested in sharing state between tasks, e.g. configuring the outputs of one task to be the inputs of another.
The code uses "project.afterEvaluate" to define the tasks when the required settings have been configured through the extension object. This seems more complex than should be needed. If I move the code out of the "afterEvaluate", it gets compileFlag == null which isn't the external setting. If the code is changed again to use the << or doLast syntax, then it will get the external flag... but then it fails to work with type:Exec and other similarly helpful types.
I feel that I'm fighting Gradle in some ways, which means I don't understand better how to work well with it. The following is a simplified pseudo-code of what I'm using. This works but I'm looking to see if this can be simplified, or indeed what the best practices are. Also, the exception shouldn't be thrown unless the tasks are being executed.
apply plugin: MyPlugin
class MyPluginExtension {
String compileFlag = null
}
class MyPlugin implements Plugin<Project> {
void apply(Project project) {
project.extensions.create("myPluginConfig", MyPluginExtension)
project.afterEvaluate {
// Closure delays getting and checking flag until strictly needed
def compileFlag = {
if (project.myPluginConfig.compileFlag == null) {
throw new InvalidUserDataException(
"Must set compileFlag: myPluginConfig { compileFlag = '-flag' }")
}
return project.myPluginConfig.compileFlag
}
// Inputs for translateTask
def javaInputs = {
project.files(project.fileTree(
dir: project.projectDir, includes: ['**/*.java']))
}
// This is the output of the first task and input to the second
def translatedOutputs = {
project.files(javaInputs().collect { file ->
return file.path.replace('src/', 'build/dir/')
})
}
// Translates all java files into 'translatedOutputs'
project.tasks.create(name: 'translateTask', type:Exec) {
inputs.files javaInputs()
outputs.files translatedOutputs()
executable '/bin/echo'
inputs.files.each { file ->
args file.path
}
}
// Compiles 'translatedOutputs' to binary
project.tasks.create(name: 'compileTask', type:Exec, dependsOn: 'translateTask') {
inputs.files translatedOutputs()
outputs.file project.file(project.buildDir.path + '/compiledBinary')
executable '/bin/echo'
args compileFlag()
translatedOutputs().each { file ->
args file.path
}
}
}
}
}
I'd look at this problem another way. It seems like what you want to put in your extension is really owned by each of your tasks. If you had something that was a "global" plugin configuration option, would it be treated as an input necessarily?
Another way of doing this would have been to use your own SourceSets and wire those into your custom tasks. That's not quite easy enough yet, IMO. We're still pulling together the JVM and native representations of sources.
I'd recommend extracting your Exec tasks as custom tasks with a #TaskAction that does the heavy lifting (even if it just calls project.exec {}). You can then annotate your inputs with #Input, #InputFiles, etc and your outputs with #OutputFiles, #OutputDirectory, etc. Those annotations will help auto-wire your dependencies and inputs/outputs (I think that's where some of the fighting is coming from).
Another thing that you're missing is if the compileFlag effects the final output, you'd want to detect changes to it and force a rebuild (but not a re-translate).
I simplified the body of the plugin class by using the Groovy .with method.
I'm not completely happy with this (I think the translatedFiles could be done differently), but I hope it shows you some of the best practices. I made this a working example (as long as you have a src/something.java) by implementing the translate as a copy/rename and the compile as something that just creates an 'executable' file (contents is just the list of the inputs). I've also left your extension class in place to demonstrate the "global" plug-in config. Also take a look at what happens with compileFlag is not set (I wish the error was a little better).
The translateTask isn't going to be incremental (although, I think you could probably figure out a way to do that). So you'd probably need to delete the output directory each time. I wouldn't mix other output into that directory if you want to keep that simple.
HTH
apply plugin: 'base'
apply plugin: MyPlugin
class MyTranslateTask extends DefaultTask {
#InputFiles FileCollection srcFiles
#OutputDirectory File translatedDir
#TaskAction
public void translate() {
// println "toolhome is ${project.myPluginConfig.toolHome}"
// translate java files by renaming them
project.copy {
includeEmptyDirs = false
from(srcFiles)
into(translatedDir)
rename '(.+).java', '$1.m'
}
}
}
class MyCompileTask extends DefaultTask {
#Input String compileFlag
#InputFiles FileCollection translatedFiles
#OutputDirectory File outputDir
#TaskAction
public void compile() {
// write inputs to the executable file
project.file("$outputDir/executable") << "${project.myPluginConfig.toolHome} $compileFlag ${translatedFiles.collect { it.path }}"
}
}
class MyPluginExtension {
File toolHome = new File("/some/sane/default")
}
class MyPlugin implements Plugin<Project> {
void apply(Project project) {
project.with {
extensions.create("myPluginConfig", MyPluginExtension)
tasks.create(name: 'translateTask', type: MyTranslateTask) {
description = "Translates all java files into translatedDir"
srcFiles = fileTree(dir: projectDir, includes: [ '**/*.java' ])
translatedDir = file("${buildDir}/dir")
}
tasks.create(name: 'compileTask', type: MyCompileTask) {
description = "Compiles translated files into outputDir"
translatedFiles = fileTree(tasks.translateTask.outputs.files.singleFile) {
includes [ '**/*.m' ]
builtBy tasks.translateTask
}
outputDir = file("${buildDir}/compiledBinary")
}
}
}
}
myPluginConfig {
toolHome = file("/some/custom/path")
}
compileTask {
compileFlag = '-flag'
}

Puppet dependency resolution ordering

I have a couple hosts which have Docker containers on them, so I've defined a class called apps::docker which installs Docker on the hosts:
class apps::docker {
include apps::docker::repository, apps::docker::install,
apps::docker::service
Class["Apps::Docker::Repository"] ->
Class["Apps::Docker::Install"] ->
Class["Apps::Docker::Service"]
Package["Docker"] ~> Service["Docker"]
}
class apps::docker::repository {
apt::source { 'docker':
location => "http://get.docker.io/ubuntu",
key => "A88D21E9",
release => "docker",
repos => "main",
include_src => false
}
}
class apps::docker::install {
package { 'docker':
name => "lxc-docker",
ensure => present
}
class apps::docker::service {
service { 'docker':
provider => 'upstart',
enable => true,
ensure => running,
hasrestart => true,
hasstatus => true
}
}
Pretty simple stuff, actually.
The problem is that when I try to define a class which depends on this class, the execution happens out of order and commands fail. For example, my class profiles::shiningstar::containers depends on apps::docker as defined in profiles::shiningstar:
class profiles::shiningstar {
include apps::docker
include profiles::shiningstar::containers
Class["Apps::Docker"] -> Class["Profiles::Shiningstar::Containers"]
}
Unfortunately, this doesn't work as seen below:
Error: /Stage[main]/Profiles::Shiningstar::Containers::Puppetmaster::Pull/Docker::Image[rfkrocktk/puppetmaster:1.0.5]/Exec[docker pull rfkrocktk/puppetmaster:1.0.5]: Could not evaluate: Could not find command 'docker'
... (similar errors)
Notice: /Stage[main]/Apps::Docker::Repository/Apt::Source[docker]/Apt::Key[Add key: A88D21E9 from Apt::Source docker]/Apt_key[Add key: A88D21E9 from Apt::Source docker]/ensure: created
It's executing things completely out of order. What's wrong with my configuration and how can I specify that ALL of the dependencies of apps::docker must be satisfied before profiles::shiningstar::containers?
You probably want to contain the inner classes instead of just including them.
class apps::docker {
contain apps::docker::repository
contain apps::docker::install
contain apps::docker::service
Class['apps::docker::repository']
->
Class['apps::docker::install']
~>
Class['apps::docker::service']
}
Note that it makes sense (in your case at least) to make the ::install class as a whole notify all of the ::service class. The makes you more flexible in refactoring the respective implementation of those classes.
Edited after first comment - don't try to put chaining arrows between contain statements.
You should use an anchor, that will ensure that all dependencies are built
class apps::docker {
include apps::docker::repository, apps::docker::install,
apps::docker::service
Class["Apps::Docker::Repository"] ->
Class["Apps::Docker::Install"] ->
Class["Apps::Docker::Service"] ->
anchor{"apps::docker":}
Package["Docker"] ~> Service["Docker"]
}

In Puppet, how can I access a variable/attribute inside a defined type?

I would like to reference variables inside instances of defined types. For example, what can I do to reference $x and $y of foo a in bar b?
define foo($x, $y) {
}
define bar($foo) {
notify { "${::$foo::x}": } # <- how to make this reference work?
}
foo { 'a':
x => 'oh bar may you reference me',
y => 'please'
}
bar { 'b':
foo => Foo['a'],
require => Foo['a']
}
The reason why I would like this to work is that a foo instance may contain many values that I wouldn't like to repeat to each and every resource that might need them. Instead of passing those values again and again, thus repeating myself, I'd rather pass a reference to their container.
I've been looking all over and tried a bunch of things, but can't seem to find an answer to this question anywhere. I know it is possible to amend attributes, reference resources and read class attributes, but is it possible to read attributes of a resource/defined type? If it isn't what is then the best possible work around?
I've actually just found out that Puppetlab's stdlib module includes a getparam function that can be used to solve this problem.
So here is finally the solution to my own question:
define foo($x, $y) {
}
define bar($foo) {
notify { getparam(Foo[$foo], 'x'): }
notify { getparam(Foo[$foo], 'y'): }
}
foo { 'a':
x => 'oh bar may you reference me',
y => 'please'
}
bar { 'b':
foo => 'a'
}
Please note that the require => Foo['a'] in the definition of Bar['b'] does not appear to be needed.
It doesn't seem that you can access a defined type's attributes. Possible explanation here. What you can do however, is externalize it via hiera.
Hiera is a great way to separate your manifest logic from the data populating it, but it is not difficult to set up.
Installing
In my first try I was trying to access hiera variables by class reference; foo::a for example, but that doesn't work for defined types.
Using http://drewblessing.com/blog/-/blogs/puppet-hiera-implement-defined-resource-types-in-hiera as a guide, you can put declare all those attributes in hiera with a simple config:
Configuring:
hiera.yaml
:backends:
- yaml
:yaml:
:datadir: $hiera_dir
:hierarchy:
- common
$hiera_dir/common.yaml
foo:
a:
x: 'oh bar may you reference me'
y: 'please'
And then in your puppet manifest:
define foo ($x, $y) {
}
define bar($foo) {
require create_my_foos
$all_foos = hiera('foo')
# This is just for proof of concept, to show that the variable can be passed.
file { '/tmp/output.txt':
content => $all_foos[$foo]['x']
}
}
class create_my_foos {
$foo_instances = hiera('foo', [])
create_resources('foo', $foo_instances)
}
bar { 'b':
foo => 'a'
}
Now you can access foo's variables by calling the hiera('foo') function to get an array of foo's attributes, and do array lookups to get the exact parameter you need.
Note that hiera only looks up top-level keys, so you can't do hiera('foo'['a']['x]).

Resources