I A have a Puppet module with acceptance tests based on Beaker. The module is working fine and when running locally all the acceptance tests run fine. But when I run the tests on Travis I got the following error in the module execution:
/Stage[main]/Alfred::Services/Service[alfred]: Could not evaluate: undefined method `[]' for nil:NilClass
Alfred is a system service based on upstart that is part of my module.
I am using Puppet 4.3.2. Here is the travis build: https://travis-ci.org/nicopaez/alfred-puppet
Any idea?
Looking at the code, there's a few issues.
One is that the environment variable you're using in Travis doesn't set the Puppet version. You need to add that code to your spec_helper_acceptance.rb:
hosts.each do |host|
install_puppet_on(host,
:puppet => ENV['PUPPET_VERSION'] || '4.3.2',
)
end
Right now it's still installing Puppet 3.8 (the default latest)
For further information on what is actually causing the issues in Travis, I forked your repo and did a build where I enabled the debug and trace options for beaker:
result = apply_manifest(pp, :trace => true, :debug => true)
From this, looking at the Travis build, theres an issue with the git clone exec:
Debug: Exec[clone-repo](provider=posix): Executing 'git clone https://github.com/fiuba/alfred.git /var/www/alfred'
Debug: Executing 'git clone https://github.com/fiuba/alfred.git /var/www/alfred'
Notice: /Stage[main]/Alfred::App/Exec[clone-repo]/returns: fatal: Could not change back to '/root': Permission denied
Error: git clone https://github.com/fiuba/alfred.git /var/www/alfred returned 128 instead of one of [0]
You could fix this by using the vcsrepo module, which performs git clones in an more idempotent way:
vcsrepo { '/var/www/alfred':
ensure => present,
source => 'https://github.com/fiuba/alfred.git',
provider => git,
user => 'deployer',
}
There's a few other fixes, I'm PRing some fixes to your module to fix them, and will add a summary here to the Stack Overflow answer after you've reviewed and merged, as some are significant refactors with a few different approaches.
Related
I'm using the pre-commit hooks configuration https://pre-commit.com/ to enable pre-commit hooks
repos:
- repo: local
hooks:
- id: pytest-check
name: pytest-check
entry: pytest
language: system
pass_filenames: false
always_run: false
- id: flake8
name: flake8
entry: flake8
language: python
types: [python]
args: ['src/']
If pytest-check fails it will also execute the flake8 hook.
Is it possible to terminate the execution if a previous hook failed, in this case flake8 would not run if the pytest-check failed.
I read through the docs but couldn't find any information on this...
you're looking for fail_fast: true
it can be specified both at the top level and at the hook level
an aside you have a few unrelated problems with your configuration:
~generally you don't want to run tests as part of pre-commit, they'll be slow which will frustrate your users and often lead to them turning the whole thing off
always_run: false is the default, no need to specify it
flake8 as you've configured it is both double-linting and fork bombing (you're passing src/ and pre-commit is additionally passing filenames, and you've misconfigured the multiprocessing mode) -- I'd recommend using the pycqa/flake8 repository directly which configures this correctly
for repo: local hooks there's no reason to use args -- just specify it directly in entry
disclaimer: I wrote pre-commit
I wrote a simple module to install a package (BioPerl) on a Ubuntu VM. The whole init.pp file is here:
https://gist.github.com/anonymous/17b4c31bf7309aff14dfdcd378e44f40
The problem is it doesn't work, and it gives me no feedback to let me know why it doesn't work. There are 3 simple steps in the module. I checked and it didn't do any of them. Heres the first 2:
Step 1: Download an archive and save it to /usr/local/lib
exec { 'bioperl-download':
command => "sudo /usr/bin/wget --no-check-certificate -O ${archive_path} ${package_uri}",
require => Package['wget']
}
Step 2: Extract the archive
exec { 'bioperl-extract':
command => "sudo /usr/bin/tar zxvf ${archive_path} --directory ${install_path}; sudo rm ${archive_path}",
require => Exec['bioperl-download']
}
pretty simple. But I have no idea where the problem is because I can't see what its doing. The provisioner is set to verbose mode, and here are the output lines for my module:
==> default: Notice: /Stage[main]/Bioperl/Exec[bioperl-download]/returns: executed successfully
==> default: Notice: /Stage[main]/Bioperl/Exec[bioperl-extract]/returns: executed successfully
==> default: Notice: /Stage[main]/Bioperl/Exec[bioperl-path]/returns: executed successfully
So all I know is it executed these three steps successfully. It doesn't tell me anything about whether the steps did their job properly or not. I know that it didn't download the archive to /usr/local/lib that directory, and that it didn't add an environment variable file to /usr/profile.d. Maybe the issue is the variables containing the directories are wrong. Maybe the variable containing the archives download URI is wrong. How can I find these things out?
UPDATE:
It turns out the module does work. But to improve the module (since I want to upload it to forge.puppetlabs.com, I tried implementing the changes suggested by Matt. Heres the new code:
file { 'bioperl-download':
path => "${archive_path}",
source => "http://cpan.metacpan.org/authors/id/C/CJ/CJFIELDS/${archive_name}",
ensure => "present"
}
exec { 'bioperl-extract':
command => "sudo /bin/tar zxvf ${archive_name}",
cwd => "${bioperl_target_dir}",
require => File['bioperl-download']
}
A problem: It gives me an error telling me that the source cannot be http://. I see in the docs that they do indeed allow http:// files as the source for the file resource. Maybe I'm using an older version of puppet?
I want to try out the puppet-archive module, but I'm not sure how I can set it as a required dependency. By that, I mean how I can make sure its installed first. Do I need to get my module to download the module from github and save it to the modules directory? Or is there a way to let puppet install it automatically? I added it as a dependency to the metadata.json file, but that doesn't install it. I know I can just get my module to download the package, but I was wondering what best practice for this is.
The initial problem you describe is acceptance testing. Verifying that the Puppet resources and code you wrote actually resulted in the desired end state you wanted is normally accomplished with Serverspec: http://serverspec.org/. For example, you can write a Puppet module to deploy an application, but you only know that Puppet did what you told it to, and not that the application actually successfully deployed. Note Serverspec is also what people generally use to solve this problem for Ansible and Chef also.
You can write a Serverspec test similar to the following to help test your module's end state:
describe file('/usr/local/lib/bioperl.tar.gz') do
it { expect(subject).to be_file }
end
describe file('/usr/profile.d/env_file') do
it { expect_subject).to be_file }
its(:content) { is_expected.to match(/env stuff/) }
end
However, your problem also seems to deal with debugging why your acceptance tests failed. For that, you need unit testing. This is normally solved with RSpec-Puppet: http://rspec-puppet.com/. I would show you how to write some tests for your situation, but I don't think you should be writing your Puppet module the way that you did, so it would render the unit tests irrelevant.
Instead, consider using a file resource with the source attribute and a HTTP URI to grab the tarball instead of an exec with wget: https://docs.puppet.com/puppet/latest/type.html#file-attribute-source. Also, you might want to consider using the Puppet archive module to assist you: https://forge.puppet.com/puppet/archive.
If you have questions on how to use these tools to provide unit and acceptance testing, or have questions on how to refactor your module, then don't hesitate to write followup questions on StackOverflow and we can help you.
I am following the pluralsight's tutorial for puppet fundamentals by Ben Piper.
When installing the "vcsrepo" module in agents and downloading the repository the puppet agent run hangs.
Below is the excerpt from my puppet master "init.pp" file
I changed my puppetmaster init.pp file to use the ssh instead.
`file { '/home/vagrant/.ssh/id_rsa':
ensure => 'present',
}
vcsrepo { '/var/www/html':
ensure => 'present',
provider => 'git',
source => 'git#github.com:wikimedia/mediawiki.git',
user => 'vagrant',
revision => 'REL1_23',
require => File['/home/vagrant/.ssh/id_rsa'],
}
file { '/var/www/html/index.html':
ensure => 'absent',
}
File['/var/www/html/index.html'] -> Vcsrepo['/var/www/html']`
It now throws the below error
Notice: /Stage[main]/Linux/File[/info.txt]/content: content changed '{md5}dd4735ab73567a89caba62c6607e44b5' to '{md5}e30fa7cc7448a09071a0e4d33efa5986'
Notice: /Stage[main]/Mediawiki/Vcsrepo[/var/www/html]/ensure: Creating repository from present
Error: Execution of 'git clone git#github.com:wikimedia/mediawiki.git /var/www/html' returned 1:
Error: /Stage[main]/Mediawiki/Vcsrepo[/var/www/html]/ensure: change from absent to present failed: Execution of 'git clone git#github.com:wikimedia/mediawiki.git /var/www/html' returned 1:
I have shared my system's public key in github account
On Linux
Execute the following in the command line before executing the Git command:
export GIT_TRACE_PACKET=1
export GIT_TRACE=1
export GIT_CURL_VERBOSE=1
On Windows
Execute the following in the command line before executing the Git command:
set GIT_TRACE_PACKET=1
set GIT_TRACE=1
set GIT_CURL_VERBOSE=1
Cause
Error code 56 indicates a curl receive error of CURLE_RECV_ERROR which means there was some issue that prevented the data from being received during the clone process. Typically this is caused by a network setting, firewall, VPN client, or anti-virus that is terminating the connection before all data has been transferred.
Workaround
Switch to using SSH to perform the clone.
The date is/was 12/17/2014, I'm trying to run gitlab-ce tests from within gitlab-development-kit. I'm hoping someone familiar with gitlab-ce development can help here. I want to have the tests pass before I begin development. I'm not sure if this warrants a bug report, it may be my environment (CentOS 6.5, rvm 1.26.3, ruby 2.1.3p242 )
I followed instructions on gitlab-development-kit to clone it, run make (to download latest gitlab + gitlab-shell).
I run bundle exec foreman start, redis and pgsql start.
Every thing is looking good, I ran gitlab and it worked fine in development env. I reset everything by recloning and following steps and then tested.
Within ./gitlab, I run "rake gitlab:test"; lots of passed, green tests.
Until I the end, I receive this:
...(many, and all, passing tests above here)...
Scenario: Navigate to project feed
✔ Given I sign in as a user # features/steps/shared/authentication.rb:7
✔ And I own a project # features/steps/shared/project.rb:5
✔ And I visit my project's home page # features/steps/shared/paths.rb:169
✔ Given I visit my project's files page # features/steps/shared/paths.rb:177
✔ Given I press "g" and "p" # features/steps/shared/shortcuts.rb:4
✔ Then the active main tab should be Home # features/steps/shared/project_tab.rb:7
/usr/local/rvm/gems/ruby-2.1.3/gems/actionview-4.1.1/lib/action_view/renderer/partial_renderer.rb:436:in `partial_path': 'nil' is not an ActiveModel-compatible object. It must implement :to_partial_path. (ActionView::Template::Error)
from /usr/local/rvm/gems/ruby-2.1.3/gems/actionview-4.1.1/lib/action_view/renderer/partial_renderer.rb:345:in `setup'
from /usr/local/rvm/gems/ruby-2.1.3/gems/actionview-4.1.1/lib/action_view/renderer/partial_renderer.rb:262:in `render'
from /usr/local/rvm/gems/ruby-2.1.3/gems/actionview-4.1.1/lib/action_view/renderer/renderer.rb:47:in `render_partial'
from /usr/local/rvm/gems/ruby-2.1.3/gems/actionview-4.1.1/lib/action_view/helpers/rendering_helper.rb:35:in `render'
from /usr/local/rvm/gems/ruby-2.1.3/gems/haml-4.0.5/lib/haml/helpers/action_view_mods.rb:10:in `block in render_with_haml'
from /usr/local/rvm/gems/ruby-2.1.3/gems/haml-4.0.5/lib/haml/helpers.rb:89:in `non_haml'
from /usr/local/rvm/gems/ruby-2.1.3/gems/haml-4.0.5/lib/haml/helpers/action_view_mods.rb:10:in `render_with_haml'
from /home/git/gitlab-development-kit/gitlab/app/views/projects/blob/_blob.html.haml:20:in `_app_views_projects_blob__blob_html_haml__1171767312904667641_107433960'
from /usr/local/rvm/gems/ruby-2.1.3/gems/actionview-4.1.1/lib/action_view/template.rb:145:in `block in render'
from /usr/local/rvm/gems/ruby-2.1.3/gems/activesupport-4.1.1/lib/active_support/notifications.rb:161:in `instrument'
from /usr/local/rvm/gems/ruby-2.1.3/gems/actionview-4.1.1/lib/action_view/template.rb:339:in `instrument'
from /usr/local/rvm/gems/ruby-2.1.3/gems/actionview-4.1.1/lib/action_view/template.rb:143:in `render'
from /usr/local/rvm/gems/ruby-2.1.3/gems/actionview-4.1.1/lib/action_view/renderer/partial_renderer.rb:306:in `render_partial'
...
When I inspect app/views/projects/blob/_blob.html.haml:20
I can see
%ul.blob-commit-info.bs-callout.bs-callout-info.hidden-xs
- blob_commit = #repository.last_commit_for_path(#commit.id, #blob.path)
= render blob_commit, project: #project
The error is complaining because blob_commit is nil, from the line
#repository.last_commit_for_path(#commit.id, #blob.path)
This is a pure clone of everything, I haven't began to make modifications yet. I waited a day to see if maybe the next update would fix things but it hasn't. I don't want to start a feature branch if I'm already having failing tests.
I found out the answer to my problem.
CentOS 6.5 hasn't upgraded their git rpm to anything beyond 1.7.1
I added some debugging to app/models/repository.rb, in def commit(id = HEAD), and def last_commit_for_path(sha, path)
It appears that in last_commit_for_path, on gitlab-test_bare, the following command is run:
git rev-list --max-count 1 5937ac0a7beb003549fc5fd26fc247adbce4a52e -- CHANGELOG
which results in an exception
fatal: bad revision '1'
which results in '#'repository.last_commit line above to assign nil to blob_commit.
It looks like --max-count 1, in my environment, must be --max-count=1.
git rev-list --max-count=1 5937ac0a7beb003549fc5fd26fc247adbce4a52e -- CHANGELOG
which results in a valid commit sha
913c66a37b4a45b9769037c55c2d238bd0942d2e
When I looked at my git version, i noticed it was extremely out of date (1.7 vs 2.2) so I updated using source and the tests pass since gitlab can execute the --max-count 1 as a parameter to git.
I am trying to create a custom provider for package but for some reasons I keep on getting
err: Could not run Puppet configuration client: Parameter provider
failed: Invalid package provider 'piprs' at
/usr/local/src/ops/services/puppet/modules/test/manifests/init.pp:5
I have added pluginsync=true in puppet.conf in both client and server. I have created the following rb file in module/test/lib/puppet/provider/package/piprs.rb. I am basically trying to create a custom provider for package resource type
#require 'puppet/provider/package'
Puppet::Type.type(:package).provide(:piprs,
:parent => ::Puppet::Provider::Package) do
commands : pip => "/usr/local/bin/pip"
desc "Python packages via `pip`."
def create
pip "freeze"
end
def destroy
end
def exists?
end
end
In the puppet.conf, there is the following source attribute
pluginsource = puppet://puppet/plugins
I am not sure what it is. If you need anymore details, please do post a comment.
First things first - you do realize there is already a Python pip provider in core?
https://github.com/puppetlabs/puppet/blob/master/lib/puppet/provider/package/pip.rb
If that isn't what you want - then lets move on ...
For starters - try your module without a Puppet master - this is going to be better for development anyway. You need to make sure Ruby can find the library path:
export RUBYLIB=<path_to_module>/lib
Then, try writing a small test in a .pp file:
package { "mypackage": provider => "piprs" }
And run it locally:
puppet apply mytest.pp
This will rule out a code bug in your provider versus a plugin sync issue.
I notice there is a space between the colon and the command - that isn't your problem is it?
commands : pip => "/usr/local/bin/pip"
If you can get this working without a puppetmaster, your problem is sync related.
There are a couple of things that can go wrong - make sure the file is sync'd properly on the client:
ls /var/lib/puppet/lib/puppet/provider/package
You should see the piprs.rb file there. If it is, you may need to make sure your libdir is set correctly:
puppet --configprint libdir
This should point to /var/lib/puppet/lib in most cases.