How to check if folder has any content in puppet - puppet

What I want to do is check whether a folder has any content before copying the content in the folder, since puppet is throwing an error if you are trying to copy content of a empty folder. This is what I have tried but it doesn't work :(
exec { "Copying_patches_$setupnode-$number":
path => '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/java/bin/',
command => "cp -r ${params::script_base_dir}/libs/patches/* ${params::deployment_target}/$setup/repository/patches/",
onlyif => "test -f ${params::script_base_dir}/libs/patches/*",
notify => Notify['${params::script_base_dir}/libs/patches/* found'],
require => File["${params::deployment_target}/$setupnode"],
}
params::script_base_dir will give the path up to the script location.

Try this approach:
package { 'rsync': ensure => 'installed' }
$from = "${params::script_base_dir}/libs/patches/"
$to = "${params::deployment_target}/$setup/repository/patches/"
file { "$from/.sync_marker": ensure => file }
exec { "Copying_patches_$setupnode-$number":
path => '/usr/bin:/bin',
command => "rsync -r $from $to",
require => [
File["${params::deployment_target}/$setupnode"],
Package['rsync'],
File["$from/.sync_marker"],
],
creates => "$to/.sync_marker",
}
Some remarks:
I shortened your path - no need for java or things in /sbin
Notifying a notify resource is usually no sensible - those always produce their message
The trailing slash on the target does not matter to rsync, but the one on the sources does!
The file in the source directory is created, just to make it possible to build a simple creates clause
The creates parameter makes sure that the command is run only once, and not during every run.
If you need Puppet to wait until the source directory is populated, you do have to use onlyif. Try this condition:
onlyif => "find $from | wc -l | grep -v '^2\$'",
The two lines of output would represent the directory itself and the marker file. The $ sign is escaped so that Puppet includes it in the command string literally.

spent a little bit of time to find it, This might help someone else as well
unless => "find /directory/path/ -mindepth 1 | read",
So wanted
exec { "Install wordpress in ${docroot}":
command => "git clone https://github.com/WordPress/WordPress.git ${docroot}",
unless => "find ${docroot} -mindepth 1 | read", # only cloen if directory is empty
cwd => $docroot,
path => ['/bin', '/usr/bin', '/usr/sbin'],
}

Related

rspec-puppet unit test for define type using resource

I have written a user define type which downloads a file using wget and stores in /root. I have used exec type to accomplish this. the code is working well with puppet apply, now when I am trying to write rspec test for the same, i am facing issues and getting failure messages. Please find the site.pp, download.pp, init_spec.rb file.
I am new to rspec and puppet and unable to find a solution for this, kindly help with suggestion on how to write the rspec test for this define type.
manifest/site.pp
$link='https://rpm.nodesource.com/pub_8.x/el/7/x86_64/'
$rpm = 'nodejs-8.0.0-1nodesource.el7.centos.x86_64.rpm'
node_js::download{'execute wget':
comm=>'/usr/bin/wget',
url=>"${link}${rpm}",
path=>'/root',
}
modules/node_js/manifests/download.pp
define node_js::download($comm=undef,
$url=undef,
$path=undef){
$validate= "nodejs-8.0.0-1nodesource.el7.centos.x86_64.rpm"
exec { 'execute wget':
command => "${comm} ${url}",
cwd => "${path}",
unless => "/usr/bin/ls ${path} | grep ${validate}",
}
}
rspec test file
modules/node_js/spec/defines/init_spec.rb
require 'spec_helper'
describe 'node_js::download', :type => 'define' do
let(:title) { 'node_js::download' }
it {
is_expected.to contain_exec('/usr/bin/wget h*tps://rpm.nodesource.com/pub_8.x/el/7/x86_64/nodejs-8.0.0-1nodesource.el7.centos.x86_64.rpm').with({
'cwd' => '/root/' ,
})
}
end
rspec execution error display message
[root#puppet node_js]# rspec spec/defines/init_spec.rb
F
Failures:
1) node_js::download should contain Exec[/usr/bin/wget https://rpm.nodesource.com/pub_8.x/el/7/x86_64/nodejs-8.0.0-1nodesource.el7.centos.x86_64.rpm] with cwd => "/root/"
should contain Exec[/usr/bin/wget h*tps://rpm.nodesource.com/pub_8.x/el/7/x86_64/nodejs-8.0.0-1nodesource.el7.centos.x86_64.rpm] with cwd => "/root/"
Failure/Error:
is_expected.to contain_exec('/usr/bin/wget h*tps://rpm.nodesource.com/pub_8.x/el/7/x86_64/nodejs-8.0.0-1nodesource.el7.centos.x86_64.rpm').with({
'cwd' => '/root/' ,
})
expected that the catalogue would contain Exec[/usr/bin/wget https://rpm.nodesource.com/pub_8.x/el/7/x86_64/nodejs-8.0.0-1nodesource.el7.centos.x86_64.rpm]
# ./spec/defines/init_spec.rb:8:in `block (2 levels) in <top (required)>'
Finished in 0.69964 seconds (files took 8.81 seconds to load)
1 example, 1 failure
Failed examples:
rspec ./spec/defines/init_spec.rb:7 # node_js::download should contain Exec[/usr/bin/wget ht*ps://rpm.nodesource.com/pub_8.x/el/7/x86_64/nodejs-8.0.0-1nodesource.el7.centos.x86_64.rpm] with cwd => "/root/"
[root#puppet node_js]#
It looks like you are on the right track.
Here is what you should have in init_spec.rb:
require 'spec_helper'
describe 'node_js::download', :type => 'define' do
let(:title) { 'node_js::download' }
let(:params) do
{
:comm => '/usr/bin/wget',
:path => '/root/',
:url => 'https://rpm.nodesource.com/pub_8.x/el/7/x86_64/nodejs-8.0.0-1nodesource.el7.centos.x86_64.rpm',
}
end
it {
is_expected.to contain_exec('execute wget').with({
'command' => '/usr/bin/wget https://rpm.nodesource.com/pub_8.x/el/7/x86_64/nodejs-8.0.0-1nodesource.el7.centos.x86_64.rpm',
'cwd' => '/root/',
})
}
end
You have correctly set the title of the defined-type-under-test using let(:title) { ... } but you have not set the input parameters. I added the input parameters that you apparently intended.
Second point is you expected the Exec resource to have a title containing the command, whereas the actual title is going to be execute wget, per what you have in download.pp.
Otherwise, it looks good.

logstash simple file input/output

I have trouble getting logstash to work. The Basic logstash Example works. But then I struggle with the Advanced Pipeline Example. Perhaps it could be as well a problem with elasticsearch.
Now I just want to check if a simple example work:
input: read textfile-a
output: generate new textfile-b with input of the textfile-a
But I am struggling with that. My config is the following:
# foo.conf
input {
file {
path => "C:/logstash-2.3.1/logstash-tutorial-dataset"
start_position => "beginning"
}
}
output {
stdout {}
file {
#message_format => "%{foo},%{bar},%{fii},%{bor},%{bing}"
#codec => { line { format => "custom format: %{message}"}}
path => "C:/output.txt"
}
}
When I run logstash, I get the following response and nothings happens.
bin/logstash -f foo.conf -v --debug --verbose
io/console not supported; tty will not be manipulated
{:timestamp=>"2016-04-22T13:41:15.514000+0200", :message=>"starting agent", :level=>:info}
{:timestamp=>"2016-04-22T13:41:15.518000+0200", :message=>"starting pipeline", :id=>"main", :level=>:info}
{:timestamp=>"2016-04-22T13:41:16.035000+0200", :message=>"Registering file input", :path=>["C:/logstash-2.3.1/logstash-tutorial-dataset"], :level=>:info}
{:timestamp=>"2016-04-22T13:41:16.039000+0200", :message=>"No sincedb_path set, generating one based on the file path", :sincedb_path=>"c:/Users/foobar/.sincedb_802dc9c88c8fad631bf3d3a5c96435e4", :path=>["C:/logstash-2.3.1/logstash-tutorial-dataset"], :level=>:info}
{:timestamp=>"2016-04-22T13:41:16.103000+0200", :message=>"Starting pipeline", :id=>"main", :pipeline_workers=>4, :batch_size=>125, :batch_delay=>5, :max_inflight=>500, :level=>:info}
{:timestamp=>"2016-04-22T13:41:16.106000+0200", :message=>"Pipeline main started"}
how do I get the simple example working?
ignore_older => 0 did the trick, see documentaion: ignore_older.
The working configuration is the following:
# foo.conf
input {
file {
path => "C:/logstash-2.3.1/logstash-tutorial-dataset"
start_position => "beginning"
ignore_older => 0
}
}
output {
stdout {}
file {
path => "C:/output.txt"
}
}
Now the .sincedb* file contains as well content.
Logstash remembers which files it has processed, and how much of them it has processed. In normal operations, this allows it to restart in case of failure and not reprocess logs.
In your case, I imagine that your log file has been processed once already, so logstash is ignoring it. The "start_position" parameter you've provided is documented to only apply to new files.
You would either need to reset your registry (perhaps files like /var/lib/logstash/.sincedb*), or set the "sincedb_path" parameter in your file{} into to /dev/null so that it doesn't maintain the history while you're testing.

logstash : file input is not working

I am trying to run sample eg. using logstash-1.4.2 in CDH 4.4. Whenever I use file input instead of stdin, the window freezes at the following message:
Using milestone 2 plugin 'file'. This plugin should be stable but if
you see strange behavior, please let us know! For more
information.....
My code looks like this:
input {
file {
path => "/tmp/access_log"
start_position => "beginning"
}
}
filter {
if [path] =~ "access" {
mutate { replace => { "type" => "apache_access" } }
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}
output {
file{
path =>"/logs/output_log"
}
}
Command- bin/logstash -f logstash-apache.conf
I have tried deleting all my previous sincedb files in the $HOME. directory and re-run logstash, but that doesn't seem to work either. Am I missing something?
if you have just one line in your input file,
you should add an empty line at the end!
that should work!
edited:
AND if you are on a windows machine, you need to write the absolute path like
"c:/dev/access-log.txt"
and take care of just using one / instead of // after the c:
I got stuck because logstash tracks which logs it has already read: https://stackoverflow.com/a/24034718/268907
Remember that this option only modifies “first contact” situations where a file is new and not seen before. If a file has already been seen before, this option has no effect. Otherwise you have to set your sincedb_path to /dev/null .
Set sincedb_path to /dev/null and you will prevent it from tracking the position in the file that it last read.
Are you running with root permissions? It looks like /logs/output_log needs root permission to be written to.
I tried your configuration locally with logstash 1.4.1 (and sudo) and it seems to be working fine.
Could you try the below one. It worked for me.
path => "/tmp/access_log/*"
instead of
path => "/tmp/access_log"

Puppet Syntax: how to include an array of objects into an ordering -> chain?

Let's say I have:
$files = ["file1", "file2"]
exec { "exec1" :
command => "mycommand";
}
file { $files :
ensure => present;
}
And I want to use the -> and ~> constructs to control the notify/require order of execution, like so:
Exec["exec1"] -> File[$files]
How do I do it?
If I do the above, I get Could not find resource 'File[file1]File[file2]' (for real file paths, of course). I've played with wrapping the $files variable in quotes and {}s, but to no avail.
What's the syntax for putting an array variable of resource names into an ordering chain?
You can use the 'What is in an array' hint from this great list of hints from R.I.Pienaar:
First define a function that handles the chaining, then pass in your array to the function.
The function will get called once for each item in the array.
Code sample time:
exec { "exec1":
command => "/bin/echo 'i am the very model of a modern major general'";
}
file {
"/var/tmp/file1":
ensure => present;
"/var/tmp/file2":
ensure => present;
}
define chaintest() {
notify{"Calling chaintest with ${name}": }
Exec["exec1"] -> File["${name}"]
}
$files = ["/var/tmp/file1","/var/tmp/file2"]
chaintest{$files: }
The output of 'puppet apply test.pp' on puppet 2.7.11 on Ubuntu 12.04 gives:
notice: Calling chaintest with /var/tmp/file1
notice: /Stage[main]//Chaintest[/var/tmp/file1]/Notify[Calling chaintest with /var/tmp/file1]/message: defined 'message' as 'Calling chaintest with /var/tmp/file1'
notice: /Stage[main]//Exec[exec1]/returns: executed successfully
notice: Calling chaintest with /var/tmp/file2
notice: /Stage[main]//Chaintest[/var/tmp/file2]/Notify[Calling chaintest with /var/tmp/file2]/message: defined 'message' as 'Calling chaintest with /var/tmp/file2'
notice: Finished catalog run in 0.11 seconds
Why not to use require instead?
$files = ["file1", "file2"]
exec { "exec1" :
command => "mycommand";
}
file { $files :
ensure => present;
require => Exec["exec1"]
}
or just do
Exec["exec1"] -> [File["file1"], File["file2"]]

puppet chown/chmod against files under a directory in batch

In puppet, you can chown/chmod a single file by doing:
file {
'/var/log/mylog/test.log':
ensure => 'present',
mode => '0644',
owner => 'me';
}
Two questions on this:
ensure=>'present' is gonna make sure '/var/log/mylog/test.log' exists, if it doesn't it creates it. Is there any way I can make it do actions if file exists, if file doesn't exist, don't bother to create/delete it, just ignore it and carry on.
Let's say I have 3 files under /var/log/mylog/, I want to chown/chmod against them all in a batch instead of having 3 file resource sections in my puppet code. Can I do something like below(of coz, the code below doesn't exist, it's in my dream now ^_^ ):
files {
'/var/log/mylog/*.log':
ensure => 'present',
mode => '0644',
owner => 'me';
}
If you want to specify to take a given action if file exists, if file doesn't exist etc. you have no choice (to my knownledge) currently than to use the exec resource with creates + onlyif or unless directives.
You could use for instance (see reference doc)
exec { "touch /var/log/mylog/test.log":
path => "/usr/bin:/usr/sbin:/bin",
user => "${yourmodule::params::user}",
group => "${yourmodule::params::group}",
creates => "/var/log/mylog/test.log",
unless => "test -f /var/log/mylog/test.log"
}
file { '/var/log/mylog/test.log':
ensure => 'present',
mode => "${${yourmodule::params::mode}",
owner => "${yourmodule::params::user}",
group => "${yourmodule::params::group}",
require => Exec["touch /var/log/mylog/test.log"]
}
No. Again, you'll have to use an execresource.

Resources