Limit chained collectors to current class - puppet

I have several classes, which follow a simple structure of ensuring a package, modifying some config files, then notifying the service.
Thus I came up with a way to apply dependencies for each class in a simple manner:
Package <| |> -> File <| |> ~> Service <| |>
However, once an error occured in one class, it would also break the dependency for all other classes. Apparently the chained collection does not apply to the current class scope, but to the global scope.
Is there a way to limit it to just the current class? I don't want to go back to specifying dependencies manually per resource!
As seen in the below graph, ALL services depend on ALL files, which in turn depend on ALL packages.
--EDIT--
I came up with the idea of using tags (a class' resources are conveniently tagged with the classname), but it didn't work either - in fact it functioned as though the class was not present (after verifying by looking at the puppet dependency graph)
Below is a sample test manifest I made:
class foo {
package { 'php-fpm':
ensure => present,
}
file { '/tmp/php-fpm.log':
ensure => file,
content => "test",
}
service { 'php-fpm':
ensure => running,
}
Package <| tag == "foo" |> -> File <| tag == "foo" |> ~> Service <| tag == "foo" |>
}
class bar {
package { 'nginx':
ensure => present,
}
file { '/tmp/nginx.log':
ensure => file,
content => "test",
}
service { 'nginx':
ensure => running,
}
Package <| tag == "bar" |> -> File <| tag == "bar" |> ~> Service <| tag == "bar" |>
}
include foo, bar
Even more curiously, in the below graph, the tags basically have no effect:

The reason the tags do not work is a defect in puppet where automatic tags are not applied until after resource collectors are evaluated. In order to get this to work with tags you actually need to explicitly declare the tag on each resource.
One other way that you could get something similar is to enable Manifest Ordered Resources. This basically executed resources in the order they are in the manifest file, and will be the default behavior in future versions of puppet.

Related

How to work around vcsrepo "duplicate declaration" evaluation error?

I am installing from github using puppet-vcsrepo. The code looks something like this:
class demo_class(
$my_repo = undef,
$my_tag = undef,
){
vcsrepo { "$my_repo",
path => "/home/user/$my_repo",
source => 'git#github.com:7yl4r/$my_repo.git',
ensure => latest,
provider => git,
}
# then declare resources specific to my_tag
}
This works just fine when called only once, but I am iterating over a list and installing dependencies so this resource sometimes gets declared twice. I think this is roughly equivalent to the code below.
class {"demo_class":
my_repo => test_repo,
my_tag => test_tag1,
}
class {"demo_class":
my_repo => test_repo,
my_tag => test_tag2,
}
Doing this yields a server-side "Duplicate declaration" error because vcsrepo is trying to map the the same path twice. However, this is exactly the behavior I want: for both resources declared by demo_class to require the same repo in the same location. This is so that I can declare one or more resources using demo_class and be sure the repo given by my_repo (which may be common to multiple my_tags) is there.
How can I modify this class so that I can call it twice without hitting an error?
I see the problem.
I reproduced the issue using this code:
define my_vcs_repo ($myRepo, $myTag) {
vcsrepo { "$myRepo-$myTag":
path => "/home/user/$myRepo",
source => "git#github.com:7yl4r/$myRepo.git",
revision => $myTag,
ensure => latest,
provider => git,
}
}
$data = [
{
myRepo => testRepo,
myTag => testTag1,
},
{
myRepo => testRepo,
myTag => testTag2,
},
]
$data.each |$i, $ref| {
$myRepo = $ref['myRepo']
$myTag = $ref['myTag']
my_vcs_repo { "$myRepo-$i":
myRepo => $myRepo,
myTag => $myTag,
}
}
That then results in:
Puppet::PreformattedError:
Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Resource Statement, Cannot alias Vcsrepo[testRepo-testTag2] to ["/home/user/testRepo"] at /
Users/alexharvey/git/modules/foo/spec/fixtures/modules/foo/manifests/init.pp:3; resource ["Vcsrepo", "/home/user/testRepo"] already declared at /Users/alexharvey/git/modules/foo/spec/fixtures/modules/foo/
manifests/init.pp:3 at /Users/alexharvey/git/modules/foo/spec/fixtures/modules/foo/manifests/init.pp:3:5 at /Users/alexharvey/git/modules/foo/spec/fixtures/modules/foo/manifests/init.pp:26 on node alexs-macbook-pro.local
The problem is that you are asking Puppet to clone the same Git module to a directory but with two different tags checked out. That does not make sense.
The fix is that you need to specify a unique path in the vcsrepo path attribute, e.g.:
vcsrepo { "$myRepo-$myTag":
path => "/home/user/$myRepo-$myTag",
source => "git#github.com:7yl4r/$myRepo.git",
revision => 'production',
ensure => latest,
provider => git,
}
By the way, I notice you are using camelCase for your variables. Don't do that. Aside from the fact that it is not idiomatic for Puppet, there are things that will break in some versions of Puppet/Rspec puppet that I have seen.
Use snake_case for your variable names and class parameter names.
Update
The question has been edited, and it is now a question about how to declare the same vcsrepo in more than one class.
In general, try to refactor so that you do not need to do this in the first place. In other words, just move it out of this class and put it somewhere that is only expected to be declared once.
If you cannot do this for some reason, then you can also use virtual resources, which will allow you to declare it in multiple classes that will be declared on the same node.
To do that, you just have to rewrite what you have there as:
#vcsrepo { $my_repo:
path => "/home/user/$my_repo",
source => "git#github.com:7yl4r/$my_repo.git",
ensure => latest,
provider => git,
}
realize Vcsrepo[$my_repo]
Keep in mind that you will not be able to declare the class demo_class twice on the same node either. You would need to turn it into a defined type, as I did above.
It is mentioned in the comments below that you can also use ensure_resource and ensure_resources; see docs in stdlib.

Puppet dependency loop when needing to notify a service from a class that first needs the service to be running

I am trying to build a module that uses the usual install->config->service pattern but with a twist. After the service starts there is more configuration done via an admin API. Unfortunately the vendor product also then requires some of the results of that API invocation to be edited into some config XML files which requires a service restart. So I have install->config->service->admin->service which is of course a dependency loop. Is there a clean way to do this? I've resorted to having to exec{'/sbin/service ... restart'} in the admin class. Not clean at all:
anchor { 'pingfederate::begin': } ->
class { '::pingfederate::install': } ->
class { '::pingfederate::config': } ~>
class { '::pingfederate::service': } ->
class { '::pingfederate::admin': } ->
anchor { 'pingfederate::end': }
and then exec {'/sbin/service pingfederate restart':}
See https://github.com/n2ygk/puppet-pingfederate

Puppet dependency resolution ordering

I have a couple hosts which have Docker containers on them, so I've defined a class called apps::docker which installs Docker on the hosts:
class apps::docker {
include apps::docker::repository, apps::docker::install,
apps::docker::service
Class["Apps::Docker::Repository"] ->
Class["Apps::Docker::Install"] ->
Class["Apps::Docker::Service"]
Package["Docker"] ~> Service["Docker"]
}
class apps::docker::repository {
apt::source { 'docker':
location => "http://get.docker.io/ubuntu",
key => "A88D21E9",
release => "docker",
repos => "main",
include_src => false
}
}
class apps::docker::install {
package { 'docker':
name => "lxc-docker",
ensure => present
}
class apps::docker::service {
service { 'docker':
provider => 'upstart',
enable => true,
ensure => running,
hasrestart => true,
hasstatus => true
}
}
Pretty simple stuff, actually.
The problem is that when I try to define a class which depends on this class, the execution happens out of order and commands fail. For example, my class profiles::shiningstar::containers depends on apps::docker as defined in profiles::shiningstar:
class profiles::shiningstar {
include apps::docker
include profiles::shiningstar::containers
Class["Apps::Docker"] -> Class["Profiles::Shiningstar::Containers"]
}
Unfortunately, this doesn't work as seen below:
Error: /Stage[main]/Profiles::Shiningstar::Containers::Puppetmaster::Pull/Docker::Image[rfkrocktk/puppetmaster:1.0.5]/Exec[docker pull rfkrocktk/puppetmaster:1.0.5]: Could not evaluate: Could not find command 'docker'
... (similar errors)
Notice: /Stage[main]/Apps::Docker::Repository/Apt::Source[docker]/Apt::Key[Add key: A88D21E9 from Apt::Source docker]/Apt_key[Add key: A88D21E9 from Apt::Source docker]/ensure: created
It's executing things completely out of order. What's wrong with my configuration and how can I specify that ALL of the dependencies of apps::docker must be satisfied before profiles::shiningstar::containers?
You probably want to contain the inner classes instead of just including them.
class apps::docker {
contain apps::docker::repository
contain apps::docker::install
contain apps::docker::service
Class['apps::docker::repository']
->
Class['apps::docker::install']
~>
Class['apps::docker::service']
}
Note that it makes sense (in your case at least) to make the ::install class as a whole notify all of the ::service class. The makes you more flexible in refactoring the respective implementation of those classes.
Edited after first comment - don't try to put chaining arrows between contain statements.
You should use an anchor, that will ensure that all dependencies are built
class apps::docker {
include apps::docker::repository, apps::docker::install,
apps::docker::service
Class["Apps::Docker::Repository"] ->
Class["Apps::Docker::Install"] ->
Class["Apps::Docker::Service"] ->
anchor{"apps::docker":}
Package["Docker"] ~> Service["Docker"]
}

difference between require and ordering with puppet classes

I want to achieve dependency between puppet classes, so that the classes (their content) get executed in a deterministic way. Reading the documentation I came up with the following two ways:
ORDERING
class Level1 {
Class['Level2']->Class[Level1']
package { "A":
ensure => "installed"
}
}
class Level2 {
include Level3
package { "B":
ensure => "installed"
}
}
class Level3 {
package { "C":
ensure => "installed"
}
}
according to the documentation
-> (ordering arrow) Causes the resource on the left to be applied before the resource on the right. Written with a hyphen and a
greater-than sign.
I expect the following to happen:
Level2 is called before Level1
package B or C get installed (order can be random because it was not specified)
package A gets installed.
REQUIRE
class Level1 {
require Level2
package { "A":
ensure => "installed"
}
}
class Level2 {
require Level3
package { "B":
ensure => "installed"
}
}
class Level3 {
package { "C":
ensure => "installed"
}
}
According to the documentation
cause every resource in the required classes to be applied
before any of the resources in the requiring class.
I expect the following to happen:
Level3 is called by Level2
package C gets installed
Level2 is called by Level1
package B gets installed
package A gets installed
alternative ORDERING
class Level1 {
Class['Level3']->Class['Level2']->Class[Level1']
package { "A":
ensure => "installed"
}
}
class Level2 {
package { "B":
ensure => "installed"
}
}
class Level3 {
package { "C":
ensure => "installed"
}
}
level 1 is needed by level 2
level 2 is needed by level 3
package C gets installed
package B gets installed
package A gets installed
Are my assumptions and conclusions correct?
I've been messing with these kind of dependencies for a while now and they don't seem to behave how I imagine them to behave. Maybe I'm missing something when it comes to the usage of require and -> when used with classes. Any feedback is appreciated!!
EDIT1
staging seems to be a good mechanism for fixing the dependency between two classes (see here), since you have to manually define the stage dependencies between each class pair. How would you use staging with three or four class levels?
EDIT2
Consider this more realistic case, where classes a used as wrappers to install multiple packages at the same time.
Every node is loading some pre-defined role:
node 'some-host' {
include role::processing_machine
}
the role is defined as follows:
class role::processing_machine {
include role::ubuntu_desktop_standard_software
include xorg::lts_12_04_quantal
include software::standard_packages
# define order in which classes should be executed
Class["role::ubuntu_desktop_standard_software"] -> Class["xorg::lts_12_04_quantal"] -> Class["software::standard_packages"] -> Class["role::processing_machine"]
}
and here's the role::ubuntu_desktop_standard_software definition:
class role::ubuntu_desktop_standard_software {
include role
include profile::ubuntu_desktop
include role::server::no_apache
# define order in which classes should be executed.
Class["role"] -> Class["profile::ubuntu_desktop"] -> Class["role::server::no_apache"] -> Class["role::ubuntu_desktop_standard_software"]
}
As you can see, I'm trying to chain multiple classes so that they get executed in a particular order (Class["a"] -> Class["b"]). Previously I've only used include in my classes but puppet would execute the include commands in arbitrary order, so that some commands (which have to run first!) wouldn't run first.
Despite these chaining efforts it still seems like puppet is executing the classes random fashion. What am I doing wrong? Is there a cleaner / better way of defining nodes so that I can be sure they are deployed in a particular, predefined way?
Here is an example :
class L {
package { "C" :
ensure => installed,
require => Package["A", "B",],
}
package { "B" :
ensure => installed,
require => Package["A"],
}
package { "A" :
ensure => installed,
}
}

Puppet Can't Find Variable for Template

Just getting started with Puppet, and I'm having trouble with my first template. It should be very easy, but I can't figure it out.
I have a module "base" at
/etc/puppet/modules/base/
./manifests
./manifests/service.pp
./manifests/init.pp
./manifests/params.pp
./manifests/config.pp
./manifests/install.pp
./templates
./templates/puppet.conf.erb
There's other stuff, but it's not necessary.
base/manifests/init.pp:
class base {
include base::install, base::service, base::config, base::params
}
base/manifests/config.pp
class base::config {
include base::params
File {
require => Class["base::install"],
ensure => present,
owner => root,
group => root,
}
file { "/etc/puppet/puppet.conf":
mode => 0644,
content => template("base/puppet.conf.erb"),
require => Class["base::install"],
nofity => Service["puppet"],
}
...
base/manifests/params.pp
class base::params {
$puppetserver = "pup01.sdirect.lab"
}
Finally the interesting part of the template at base/templates/puppet.conf.erb
...
server=<% puppetserver %>
The error message:
err: Failed to parse template base/puppet.conf.erb: Could not find
value for 'puppetserver' at
/etc/puppet/modules/base/manifests/config.pp:13 on node ...
I don't get what the problem is. I've copied this part straight out of the Pro Puppet book.
Could someone show me where $puppetserver should be defined and how?
The issue is that the name "puppetserver" needs to be fully qualified so Puppet can find the value, since it's defined in a different scope to the one the template is evaluated in.
The variable is defined in base::params so can only be referred to simply as "puppetserver" in that scope. When you're evaluating the template from within base::config, you're in a different scope and so you can't refer to the variable simply by its short name. The "include" adds the other class to the catalog, but doesn't change these rules.
This means to access it, you fully qualify it with the class name: base::params::puppetserver. If you were using it in the manifest itself, this would be $base::params::puppetserver. You'll see similar examples in Pro Puppet in the ssh::config and ssh::service classes where it refers to "ssh_service_name" in the params class (pages 43-45).
To access the variable in a template it's a bit different, use scope.lookupvar("base::params::puppetserver"). Taking your full example and adding a missing equals sign (to output the value) in the template:
...
server=<%= scope.lookupvar("base::params::puppetserver") %>
There's a bit more information about scoping on the Scope and Puppet as of 2.7 page.
(Edit: looks like it's listed on the confirmed errata page too with the same solution.)
Answer #1 is technically correct, but results in very verbose templates.
You can shorten them by bringing variable values from other classes into your own class scope:
class base::config {
include base::params
$puppetserver = $base::params::puppetserver
...
}
And then use them in your template as expected:
server=<% puppetserver %>
You could also use inherits:
class puppet::config inherits puppet::params {
....
In this way you don't have to define $puppetserver again in this class.

Resources