I'm trying to use Boxen to setup our dev environment. We have a number of repos that we want to pull down and run a script to get started. We landed on a convention: repos have a scripts/ directory with a bootstrap script that needs to be run.
It looks like this would be possible with the exec command. But in order to tell it what to run, I have to access the repo's directory. Other scripts use $repo_dir or ${boxen::config:srcdir}/${name}. I've tried each of these, and a number of different styles of exec, to no avail.
The Manifest
class projects::hero {
include ruby
boxen::project { 'hero':
ruby => '2.0.0',
source => 'myorg/hero'
}
->
Exec {
command => '$repo_dir/scripts/echo'
}
->
notify {'hero is running at $srcdir':}
}
This is simpler than the stated goal. The scripts need to be run within the directory they reside. So my first (and hopefully eventual) manifest would have something like this for the exec step:
->
exec { 'running bootstrap on hero':
command => '$repo_dir/scripts/bootstrap',
cwd => '$repo_dir/scripts'
}
The script
For right now, scripts/echo is super simple:
#!/bin/bash
echo "Echo File!"
touch `date`
Since the output isn't really going to be seen, we're making a file with the date so we can observe this side effect and know that the script actually ran.
Calling boxen
I just call this project directly from the manifests directory:
Chris:manifests chris$ boxen hero
The output
Warning: Scope(Class[Boxen::Environment]): Setting up 'hero'. This can be made permanent by having 'include projects::hero' in your personal manifest.
Error: Could not find resource 'command => $repo_dir/scripts/echo' for relationship from 'Boxen::Project[hero]' on node chris.local
Error: Could not find resource 'command => $repo_dir/scripts/echo' for relationship from 'Boxen::Project[hero]' on node chris.local
This is also true if I try ${boxen::config::srcdir} instead. Looking at other examples, these variables are used and seem to work. Am I calling it wrong? Is there a different variable I should be using?
I've noticed two mistakes in your manifest here:
->
Exec {
command => '$repo_dir/scripts/echo'
}
->
The first is that you've capitalized the first letter of exec. In puppet language this means you are specifying a default for all subsequent exec resource definitions (docs). This is not a resource definition itself, therefore resource ordering can not be applied, hence the error.
Another mistake is the use of single quotes in combination with variables. Single quoted strings are interpreted as literals. In other words, '$repo_dir' is interpreted literally as $repo_dir while "$repo_dir" is interpreted as the contents of the varialbe $repo_dir (docs).
Hope this helps,
Good luck
Related
I am trying to get a backup of a subdirectory before deleting the parent directory by copying the subdirectory into a different location.
This is how I have done this:
exec { "install_path_exists":
command => "/bin/true",
onlyif => "/usr/bin/test -d ${install_path}",
path => ['/usr/bin','/usr/sbin','/bin','/sbin'],
}
file { "server_backup_dir" :
ensure => 'directory',
path => "${distribution_path}/backup/server",
recurse => true,
source => "file:///${install_path}/repository/deployment/server",
require => Exec["install_path_exists"],
}
Exec checks if the directory exists, and returns true if so. The "server_backup_dir" file resource requires the "install_path_exists" exec to return true if the directory exists.
When the directory does not exist, and "install_path_exists" returns false, "server_backup_dir" executes anyway, and produces the following error.
Error: /Stage[main]/Is/File[server_backup_dir]: Could not evaluate:
Could not retrieve information from environment production source(s)
file:////usr/local/{project_location}/repository/deployment/server
What is wrong with my approach, and how can I fix this? Thanks in advance.
I'll break this up into two parts, what is wrong, and how to fix it.
What is wrong with my approach ...
You are misunderstanding the 'require' line and the nature of relationships in Puppet, and also how Puppet uses the return code of the command executed in an Exec.
When you use any of the four so-called metaparameters for relationships in Puppet - those being: require, before, subscribe & notify - you tell Puppet that you want the application of one resource to be ordered in time in relation to another. (Additionally, the 'subscribe' and 'notify' respond to refresh events but that's not relevant here.)
So, when Puppet applies a catalog built from your code, it will firstly apply the Exec resource, i.e. execute the /bin/true command, if and only if the install path exists; and then it will secondly manage the server_backup_dir File resource. Note also that it will apply the File resource irrespective of whether the Exec command actually was executed; the only guarantee being that /bin/true will never be run after the File resource.
Furthermore, the return code of the command in the Exec functions differently to what you're expecting. An exit status of 0 as the /bin/true command returns only tells Puppet to allow configuration to continue; compare that to an Exec command returning a non-zero exit status, which would cause Puppet to halt execution with an error.
Here's a simple demonstration of that:
▶ puppet apply -e "exec { '/usr/bin/false': }"
Notice: Compiled catalog for alexs-macbook-pro.local in environment production in 0.08 seconds
Error: '/usr/bin/false' returned 1 instead of one of [0]
Error: /Stage[main]/Main/Exec[/usr/bin/false]/returns: change from 'notrun' to ['0'] failed: '/usr/bin/false' returned 1 instead of one of [0]
Notice: Applied catalog in 0.02 seconds
For more info, read the page I linked above carefully. It generally takes a bit of time to get your head around relationships and ordering in Puppet.
how can I fix this?
You would normally use a custom fact like this:
# install_path.rb
Facter.add('install_path') do
setcode do
Facter::Core::Execution.execute('/usr/bin/test -d /my/install/path')
end
end
And then in your manifests:
if $facts['install_path'] {
file { "server_backup_dir" :
ensure => 'directory',
path => "${distribution_path}/backup/server",
recurse => true,
source => "file:///my/install/path/repository/deployment/server",
}
}
Consults docs for more info on writing and including custom facts in your code base.
Note:
I notice at the end that you reuse $install_path in the source parameter. If your requirement is to have a map of install paths to distribution paths, you can also build a structured fact. Without knowing exactly what you're trying to do, however, I can't be sure how you would write that piece.
I am trying to create some new env variables in the rhel machine using chef.
The block executes successfully but on trying to echo the value, i am getting black result.
Script-1:
execute 'JAVA_HOME' do
command 'export JAVA_HOME='+node['java']['home']
end
Script-2:
bash 'env_test' do
code <<-EOF
echo $chef
EOF
environment ({ 'chef' => 'chef' })
end
Also gave this a shot as it was mentioned in the documentation:
ENV['LIBRARY_PATH'] = node['my']['lib']
Please let me know where am i going wrong here..
So the thing you need to know about environment variables is they only work in one direction (parent process to children) so an export in a subcommand does nothing after that execute resource finishes. The second and third examples both work though, with the second setting it for just that bash resource and the third for both the Chef process and everything it spawns. Remember that you need to run with with -l debug to see the output from subcommands Chef runs.
Above explanation is pretty helpful. Updating the /etc/environments file using chef to make sure that env variables are present from the next session. Also using the 3rd approach to make the env variables available for the current session.
The following is a simplified manifest I am running:
package {'ruby2.4':
ensure => installed
}
exec { "gem2.4_install_bundler":
command => "/usr/bin/gem2.4 install bundler",
require => Package['ruby2.4']
}
Puppet apply runs this manifest correctly i.e
installs ruby2.4 package (which includes gem2.4)
Installs bundler using gem2.4
However, puppet apply --noop FAILS because puppet cannot find the executable '/usr/bin/gem2.4' because ruby2.4 is not installed with --noop.
My question is if there is a standard way to test a scenario like this with puppet apply --noop? To validate that my puppet manifest is executing correctly?
It occurs to me that I may have to parse the output and validate the order of the executions. If this is the case, is there a standard way/tool for this?
A last resort is a very basic check that the puppet at least runs, which can be determined with the --detailed-exitcodes option. (a code different to 1).
Thank you in advance
rspec-puppet is the standard tool for that level of verification. It can build a catalog from the manifest (e.g. for a class, defined type, or host) and then you can write tests to verify the contents.
In your case you could verify that the package resource exists, that the exec resource exists, and verify the ordering between them. This would be just as effective as running the agent with --noop mode and parsing the output - but easier and cheaper to run.
rspec-puppet works best with modules, so assuming you follow the setup for your module from the website (adding rspec-puppet to your Gemfile, running rspec-puppet-init), and let's say this is in a class called ruby24, a simple spec in spec/classes/ruby24_spec.rb would be:
require 'spec_helper'
describe 'ruby24' do
it { is_expected.to compile.with_all_deps }
it { is_expected.to contain_package('ruby2.4').with_ensure('installed') }
it { is_expected.to contain_exec('gem2.4_install_bundler').with_command('/usr/bin/gem2.4 install bundler') }
it { is_expected.to contain_exec('gem2.4_install_bundler').that_requires('Package[ruby2.4]') }
end
I wrote a simple module to install a package (BioPerl) on a Ubuntu VM. The whole init.pp file is here:
https://gist.github.com/anonymous/17b4c31bf7309aff14dfdcd378e44f40
The problem is it doesn't work, and it gives me no feedback to let me know why it doesn't work. There are 3 simple steps in the module. I checked and it didn't do any of them. Heres the first 2:
Step 1: Download an archive and save it to /usr/local/lib
exec { 'bioperl-download':
command => "sudo /usr/bin/wget --no-check-certificate -O ${archive_path} ${package_uri}",
require => Package['wget']
}
Step 2: Extract the archive
exec { 'bioperl-extract':
command => "sudo /usr/bin/tar zxvf ${archive_path} --directory ${install_path}; sudo rm ${archive_path}",
require => Exec['bioperl-download']
}
pretty simple. But I have no idea where the problem is because I can't see what its doing. The provisioner is set to verbose mode, and here are the output lines for my module:
==> default: Notice: /Stage[main]/Bioperl/Exec[bioperl-download]/returns: executed successfully
==> default: Notice: /Stage[main]/Bioperl/Exec[bioperl-extract]/returns: executed successfully
==> default: Notice: /Stage[main]/Bioperl/Exec[bioperl-path]/returns: executed successfully
So all I know is it executed these three steps successfully. It doesn't tell me anything about whether the steps did their job properly or not. I know that it didn't download the archive to /usr/local/lib that directory, and that it didn't add an environment variable file to /usr/profile.d. Maybe the issue is the variables containing the directories are wrong. Maybe the variable containing the archives download URI is wrong. How can I find these things out?
UPDATE:
It turns out the module does work. But to improve the module (since I want to upload it to forge.puppetlabs.com, I tried implementing the changes suggested by Matt. Heres the new code:
file { 'bioperl-download':
path => "${archive_path}",
source => "http://cpan.metacpan.org/authors/id/C/CJ/CJFIELDS/${archive_name}",
ensure => "present"
}
exec { 'bioperl-extract':
command => "sudo /bin/tar zxvf ${archive_name}",
cwd => "${bioperl_target_dir}",
require => File['bioperl-download']
}
A problem: It gives me an error telling me that the source cannot be http://. I see in the docs that they do indeed allow http:// files as the source for the file resource. Maybe I'm using an older version of puppet?
I want to try out the puppet-archive module, but I'm not sure how I can set it as a required dependency. By that, I mean how I can make sure its installed first. Do I need to get my module to download the module from github and save it to the modules directory? Or is there a way to let puppet install it automatically? I added it as a dependency to the metadata.json file, but that doesn't install it. I know I can just get my module to download the package, but I was wondering what best practice for this is.
The initial problem you describe is acceptance testing. Verifying that the Puppet resources and code you wrote actually resulted in the desired end state you wanted is normally accomplished with Serverspec: http://serverspec.org/. For example, you can write a Puppet module to deploy an application, but you only know that Puppet did what you told it to, and not that the application actually successfully deployed. Note Serverspec is also what people generally use to solve this problem for Ansible and Chef also.
You can write a Serverspec test similar to the following to help test your module's end state:
describe file('/usr/local/lib/bioperl.tar.gz') do
it { expect(subject).to be_file }
end
describe file('/usr/profile.d/env_file') do
it { expect_subject).to be_file }
its(:content) { is_expected.to match(/env stuff/) }
end
However, your problem also seems to deal with debugging why your acceptance tests failed. For that, you need unit testing. This is normally solved with RSpec-Puppet: http://rspec-puppet.com/. I would show you how to write some tests for your situation, but I don't think you should be writing your Puppet module the way that you did, so it would render the unit tests irrelevant.
Instead, consider using a file resource with the source attribute and a HTTP URI to grab the tarball instead of an exec with wget: https://docs.puppet.com/puppet/latest/type.html#file-attribute-source. Also, you might want to consider using the Puppet archive module to assist you: https://forge.puppet.com/puppet/archive.
If you have questions on how to use these tools to provide unit and acceptance testing, or have questions on how to refactor your module, then don't hesitate to write followup questions on StackOverflow and we can help you.
I'll need to add a apt repository key to a bunch of ubuntu hosts using puppet.
Would a statement like this work for that?
exec {"add apt key for elastic":
command => "/usr/bin/curl https://packages.elasticsearch.org/GPG-KEY-elasticsearch | /usr/bin/apt-key add -",
}
thanks
Yes, this should work but this will also apply the same configuration whenever you rerun Puppet.
From exec docs:
Any command in an exec resource must be able to run multiple times without causing harm — that is, it must be idempotent. There are three main ways for an exec to be idempotent:
The command itself is already idempotent. (For example, apt-get update.)
The exec has an onlyif, unless, or creates attribute, which prevents Puppet from running the command unless some condition is met.
The exec has refreshonly => true, which only allows Puppet to run the command when some other resource is changed. (See the notes on refreshing below.)
I try and only use execs if I really have to. One of the reasons being that you have to code the exec so that it only runs when you want to. Instead, you can use the apt module. This will only place the key on the host if it does not already exist. The resource is under puppet control so will not be added on subsequent runs:
include apt
apt::key { 'elasticsearch':
id => '46095ACC8548582C1A2699A9D27D666CD88E42B4',
options => 'https://packages.elasticsearch.org/GPG-KEY-elasticsearch',
}