Do basic math inside Hiera - puppet

I'm trying to set a crontab's weekday in hiera based on a custom fact and a basic modulo but I can't even figure out if it's possible.
I'd like to do something like:
cron-job:
command: "do something"
user: myuser
hour: "%{::instance}"
minute: "%{::instance}"
weekday: "%{::instance}" % 7
Can that even be done?

No, this is not possible. Please keep in mind that the YAML is just data, not code.
Hiera does offer some transformations using Interpolation Tokens, but there are only two functions that you can use with these, no arithmetics.

I'm not sure I follow the use case, but you could probably get away with using inline_epp or inline_template to make it look like this feature exists.
For example:
# In real usage this would be the result of a hiera lookup
$simple_lookup_result = '9 % 7'
$simple_evaluated = inline_template("<%= ${simple_lookup_result} %>")
exec { 'simple':
command => "/bin/echo $simple_evaluated",
logoutput => true,
}
# Again, hiera...
$complex_lookup_result = 'sprintf("The value is %i", 9 % 7)'
$complex_evaluated = inline_template("<%= ${complex_lookup_result} %>")
exec { 'complex':
command => "/bin/echo $complex_evaluated",
logoutput => true,
}
And the results:
$ puppet apply eval.pp
Notice: Compiled catalog for box in environment production in 0.06 seconds
Notice: /Stage[main]/Main/Exec[simple]/returns: 2
Notice: /Stage[main]/Main/Exec[simple]/returns: executed successfully
Notice: /Stage[main]/Main/Exec[complex]/returns: The value is 2
Notice: /Stage[main]/Main/Exec[complex]/returns: executed successfully
Notice: Applied catalog in 0.05 seconds
Keep in mind that Hiera can interpolate variables or Hiera lookups, and lookups can also be done within the code that inline_epp or inline_template will ultimately evaluate.
N.B. that this is an example, and you shouldn't pass Hiera input into a shell command unless you trust your users and really like headaches.

Related

How `agenda.js` calculates timezone for `every()` operation

I am using agenda.js in my Node project, backed with a MongoDB database, to handle batch processes we need to run. This is working well. I do have a question about timezones, however. When I use the every() operation, it seems like it accepts the job name, and the schedule. So I have been seeding jobs to the database like so:
for (let job of dbJobs) {
await agenda.every(schedule, job.name);
}
Note that for the above, schedule is in cron format -- 00 05 * * 1-5.
This works. However, from what I can tell, every() doesn't accept an argument for repeatTimezone. So what does it do to calculate the timezone in those cases?
To clarify, when I look at the document in the database after the job has been added using every(), the repeatTimezone property exists, but its value is set to null.
Other agenda operations, like repeatEvery(), do accept an argument for timezone, like so:
job.repeatEvery('0 6 * * *', {
timezone: 'America/New_York'
});
Since I'm using every(), I have been managing this by first seeding the database using every(), and then running a Mongo updateMany() to add the timzeone explicitly to all jobs:
async function addTimezoneToJobs() {
try {
const db = await client.db(dbName);
await db.collection('batch_processes').updateMany({}, {
$set: {
repeatTimezone: 'America/New_York'
}
});
} catch (error) {
console.log(error);
}
}
But strangely enough, agenda seems to calculate the same time even when I don't explicitly add the repeatTimezone property value to the jobs as when I do.
What's happening here that I'm not understanding? How is the runtime calculated with every(), and is there a way to pass in timezone?
FYI: I am not in the same timezone as that which needs to be set in the db.
Your Question seems to be 2 part, I'm not exactly sure I'll be able to explain it very well but let me try
So, your first question
However, from what I can tell, every() doesn't accept an argument for Timezone
Well Technically you can add Timezone option to every() as well because what this method does is it calls job.repeatEvery internally and as you already know you can add timezone to that. To Support my answer, I found 2 evidence
From Documentation as every accepts 4 parameters
every(interval, name, [data], [options])
options is an optional argument that will be passed to job.repeatEvery. In order to use this argument, data must also be specified.
So you can technically pass timezone if you pass data as well
From SourceCode, here you can see they use job.repeatEvery(interval, options) internally.
Now, To your Second Question
what does it do to calculate the timezone in those cases?
Well They have a very unique yet required module named ComputeNextRunAt().
So I went through their Source Code and figured Out that this is to Compute when will be the next run for your job based on startingTime and Interval.
Your Code works because you have once (Initially) mentioned in your job to follow America/New_York timezone, so every next job interval is calculated based on that, that's the reason you don't need to specify it again.
So, If initially you haven't had specified the timezone attribute, you would have gotten your local Timezone but now you did so, it calculates the next interval based on that.

Are resources defined by an array of titles always evaluated in order?

In Puppet, it's possible to define a number of resources using an array of titles, like so:
file { ['/some/file1', '/some/file2']:
ensure => file,
}
In terms of order of application, is it the same as the following?
file { '/some/file1':
ensure => file,
}
-> file { '/some/file2':
ensure => file,
}
I'm using puppet version 3.8.7, though I am curious if this behavior changes between Puppet 3 / 4 / 5.
I've searched the docs extensively, and while they do talk about defining resources with an array of titles, they don't talk about what order those resources get applied in.
This is kind of a complicated question to answer, but I will split up the question into components to help explain.
Is this resource array ordering typical behavior in Puppet?
Yes. To help with this, we need to delve into the Puppet DSL. Since arrays are part of the asymmetric syntax tree portion of the Puppet DSL (and usually for most if not all DSL), we need to check the relevant AST code of the Puppet Parser. The code for the commit at this time of writing is here. You can change the link to point at your version to see how and if the code has changed to see what the behavior will be for the version of Puppet you are using at any given time.
The code we want to examine is:
# Second level of implicit iteration; build a resource for each
# title. This handles things like:
# file { ['/foo', '/bar']: owner => blah }
resource_titles.flatten.map do |resource_title| # <-- line of iterator
exceptwrap :type => Puppet::ParseError do
...
if resource.resource_type.is_a? Puppet::Resource::Type
resource.resource_type.instantiate_resource(scope, resource)
end
scope.compiler.add_resource(scope, resource)
scope.compiler.evaluate_classes([resource_title], scope, false) if fully_qualified_type == 'class'
resource
end
end
So we see that the array of resources is iterated via the Array.map() method in Ruby. This converts the specific question component into "Does the Array.map() method in Ruby preserve ordering?". The specific type being iterated upon in the array is the title as specified in the type code when an instance is instantiated. This will typically be a symbol or a string, but the main point of emphasis here is that the iteration is over an array of simple types (i.e. not a nested hash etc.).
The answer to this new question component is also yes as specified in the docs or a simple yes in this linked answer.
Is this resource array ordering a documented supported behavior in Puppet?
No. You can check the relevant documentation here or here and see an absence of any mention of this behavior. Also, a former colleague directly asked Puppet about this a few years ago and their high-level response was the same as what I have outlined here thus far.
The code for the resource arrays in the AST can change at any time without warning in the documentation since this is unsupported behavior.
Should I depend on this unsupported but seemingly always existent behavior?
Your call. If this is for something like creating nested directories, then the automatic require code in Puppet will catch any issues with resources being applied out of order anyway. If not, then you probably need to do a quick cost/benefit calculation on code cleanliness versus potential undefined behavior, mitigated by a code lookup every time you upgrade, plus the time cost of said code lookup.
In terms of order of application, is it the same as the following?
file { '/some/file1':
ensure => file,
}
-> file { '/some/file2':
ensure => file,
}
Contrary to the claims of the other answer, NO, the two are not equivalent. Nor does the other answer demonstrate differently. What it shows is that in Puppet's current implementation, your array-style declaration is equivalent to this:
file { '/some/file1':
ensure => file,
}
# no chain operator here
file { '/some/file2':
ensure => file,
}
Since you specifically ask about order of application, the absence of any explicit relationship between the declared resources is significant.
Now, by default, in the absence of an explicit relationship chain between two resources, their relative order of application will be the same as their relative order of evaluation in the manifest set. But there are two important caveats there:
That's only a default. That default order-of-application rule can be changed via Puppet's 'ordering' setting. If it is, then your array-based resource declaration cannot be relied upon to produce the same order of application as your alternative using the chain operator.
Implicit relationships created by the default rule can be overridden or circumvented by explicit ones. If a contrary chain of one or more explicit relationships were declared in your manifest set, then Puppet would accept and honor it with the array-based declaration, but would reject it as a circular dependency in the case with an explicit chain operator between the resource declarations.
Where relative order of application matters, it is essential to declare explicit relationships. Array-titled resource declarations do not have such semantics, but you can, in general, add the relationships separately and still obtain the code clarity and brevity advantages of array titles:
file { ['/some/file1', '/some/file2']:
ensure => file,
}
File['/some/file1'] -> File['/some/file2']
There are two interesting answers above which are in disagreement. I will add my 2 cents here, as well as providing some more info that might be interesting to the OP.
The OP has asked in general if in terms of order of application:
file { ['/some/file1', '/some/file2']:
ensure => file,
}
is equivalent to:
file { '/some/file1':
ensure => file,
}
-> file { '/some/file2':
ensure => file,
}
In general, the answer is "no", as John says.
Here's a simple demonstration to prove that:
Puppet version:
$ bundle exec puppet -V
5.3.3
My code:
# code.pp
include stdlib
$files = range(1, 10).map |$x| { "/tmp/file${x}" }
notice("Files array is: $files")
file { $files:
ensure => file,
}
Apply with --ordering=random:
$ bundle exec puppet apply --ordering=random code.pp
Notice: Scope(Class[main]): Files array is: [/tmp/file1, /tmp/file2, /tmp/file3, /tmp/file4, /tmp/file5, /tmp/file6, /tmp/file7, /tmp/file8, /tmp/file9, /tmp/file10]
Notice: Compiled catalog for alexs-macbook-pro.local in environment production in 0.05 seconds
Notice: /Stage[main]/Main/File[/tmp/file3]/ensure: created
Notice: /Stage[main]/Main/File[/tmp/file9]/ensure: created
Notice: /Stage[main]/Main/File[/tmp/file8]/ensure: created
Notice: /Stage[main]/Main/File[/tmp/file7]/ensure: created
Notice: /Stage[main]/Main/File[/tmp/file1]/ensure: created
Notice: /Stage[main]/Main/File[/tmp/file6]/ensure: created
Notice: /Stage[main]/Main/File[/tmp/file2]/ensure: created
Notice: /Stage[main]/Main/File[/tmp/file4]/ensure: created
Notice: /Stage[main]/Main/File[/tmp/file10]/ensure: created
Notice: /Stage[main]/Main/File[/tmp/file5]/ensure: created
Notice: Applied catalog in 0.06 seconds
Thus, without an explicit relationship declaration, Puppet's random ordering setting sees the resources ordered randomly. If we had explicit orderings declared in the manifest however:
# code.pp
file { '/tmp/file1':
ensure => file,
}
-> file { '/tmp/file2':
ensure => file,
}
The files are always ordered as we want them to be:
$ for i in {1..5} ; do rm -f /tmp/file* ; bundle exec puppet apply --ordering=random code.pp ; done
Notice: Compiled catalog for alexs-macbook-pro.local in environment production in 0.21 seconds
Notice: /Stage[main]/Main/File[/tmp/file1]/ensure: created
Notice: /Stage[main]/Main/File[/tmp/file2]/ensure: created
Notice: Finished catalog run in 0.02 seconds
Notice: Compiled catalog for alexs-macbook-pro.local in environment production in 0.20 seconds
Notice: /Stage[main]/Main/File[/tmp/file1]/ensure: created
Notice: /Stage[main]/Main/File[/tmp/file2]/ensure: created
Notice: Finished catalog run in 0.02 seconds
Notice: Compiled catalog for alexs-macbook-pro.local in environment production in 0.23 seconds
Notice: /Stage[main]/Main/File[/tmp/file1]/ensure: created
Notice: /Stage[main]/Main/File[/tmp/file2]/ensure: created
Notice: Finished catalog run in 0.02 seconds
Notice: Compiled catalog for alexs-macbook-pro.local in environment production in 0.22 seconds
Notice: /Stage[main]/Main/File[/tmp/file1]/ensure: created
Notice: /Stage[main]/Main/File[/tmp/file2]/ensure: created
Notice: Finished catalog run in 0.02 seconds
Notice: Compiled catalog for alexs-macbook-pro.local in environment production in 0.23 seconds
Notice: /Stage[main]/Main/File[/tmp/file1]/ensure: created
Notice: /Stage[main]/Main/File[/tmp/file2]/ensure: created
Notice: Finished catalog run in 0.02 seconds
We can also get pseudo-random ordering using Puppet's previous default title-hash ordering, where resources are ordered by a hash generated from the resource title:
$ bundle exec puppet apply --ordering=title-hash code.pp
Notice: Scope(Class[main]): Files array is: [/tmp/file1, /tmp/file2, /tmp/file3, /tmp/file4, /tmp/file5, /tmp/file6, /tmp/file7, /tmp/file8, /tmp/file9, /tmp/file10]
Notice: Compiled catalog for alexs-macbook-pro.local in environment production in 0.05 seconds
Notice: /Stage[main]/Main/File[/tmp/file3]/ensure: created
Notice: /Stage[main]/Main/File[/tmp/file6]/ensure: created
Notice: /Stage[main]/Main/File[/tmp/file8]/ensure: created
Notice: /Stage[main]/Main/File[/tmp/file1]/ensure: created
Notice: /Stage[main]/Main/File[/tmp/file4]/ensure: created
Notice: /Stage[main]/Main/File[/tmp/file2]/ensure: created
Notice: /Stage[main]/Main/File[/tmp/file7]/ensure: created
Notice: /Stage[main]/Main/File[/tmp/file10]/ensure: created
Notice: /Stage[main]/Main/File[/tmp/file5]/ensure: created
Notice: /Stage[main]/Main/File[/tmp/file9]/ensure: created
Notice: Applied catalog in 0.06 seconds
But if we switch to manifest ordering we get the files ordered numerically again:
$ bundle exec puppet apply --ordering=manifest code.pp
Notice: Scope(Class[main]): Files array is: [/tmp/file1, /tmp/file2, /tmp/file3, /tmp/file4, /tmp/file5, /tmp/file6, /tmp/file7, /tmp/file8, /tmp/file9, /tmp/file10]
Notice: Compiled catalog for alexs-macbook-pro.local in environment production in 0.05 seconds
Notice: /Stage[main]/Main/File[/tmp/file1]/ensure: created
Notice: /Stage[main]/Main/File[/tmp/file2]/ensure: created
Notice: /Stage[main]/Main/File[/tmp/file3]/ensure: created
Notice: /Stage[main]/Main/File[/tmp/file4]/ensure: created
Notice: /Stage[main]/Main/File[/tmp/file5]/ensure: created
Notice: /Stage[main]/Main/File[/tmp/file6]/ensure: created
Notice: /Stage[main]/Main/File[/tmp/file7]/ensure: created
Notice: /Stage[main]/Main/File[/tmp/file8]/ensure: created
Notice: /Stage[main]/Main/File[/tmp/file9]/ensure: created
Notice: /Stage[main]/Main/File[/tmp/file10]/ensure: created
Notice: Applied catalog in 0.07 seconds
I think as John said, Matt's answer is correct, but only assuming that the default ordering of "manifest" is used; and also assuming that none of the resources auto-require other resources.
See also Gary Larizza's post on ordering here.

Perl regex in Nagios check

I am unfamiliar with perl, and I have a need to modify a Nagios check. I'd appreciate any advice on how to proceed. The check I'm using is check_smart, found here:
https://www.claudiokuenzler.com/nagios-plugins/check_smart.php
This script lets you check SMART values from hard drives and present the results in a simple form for monitoring. As it stands, the script can take a regex in the form /dev/sd[a-c] for one of the options; I believe that this is the section which allows this:
# list of devices for a loop
my(#dev);
if ( $opt_d ){
# normal mode - push opt_d on the list of devices
push(#dev,$opt_d);
} else {
# glob all devices - try '?' first
#dev =glob($opt_g);
}
foreach my $opt_dl (#dev){
warn "Found $opt_dl\n" if $opt_debug;
if (-b $opt_dl || -c $opt_dl){
$device .= $opt_dl.":";
} else {
warn "$opt_dl is not a valid block/character special device!\n\n" if $opt_debug;
}
}
I don't quite understand why the variable is $opt_dl when earlier it seems to be $opt_d. The result, however, is that the script returns something like:
OK: [/dev/sda] - Device is clean --- [/dev/sdb] - Device is clean --- [/dev/sdc] - Device is clean
EDIT: Here's the code where $opt_d is set; on further thought it seems like $opt_dl is just $opt_d while it's in a loop or something?
use vars qw($opt_b $opt_d $opt_g $opt_debug $opt_h $opt_i $opt_v);
Getopt::Long::Configure('bundling');
GetOptions(
"debug" => \$opt_debug,
"b=i" => \$opt_b, "bad=i" => \$opt_b,
"d=s" => \$opt_d, "device=s" => \$opt_d,
"g=s" => \$opt_g, "global=s" => \$opt_g,
"h" => \$opt_h, "help" => \$opt_h,
"i=s" => \$opt_i, "interface=s" => \$opt_i,
"v" => \$opt_v, "version" => \$opt_v,
);
The part of the code I'd like to change in a similar fashion is:
# Allow all device types currently supported by smartctl
# See http://www.smartmontools.org/wiki/Supported_RAID-Controllers
if ($opt_i =~ m/(ata|scsi|3ware|areca|hpt|cciss|megaraid|sat)/) {
$interface = $opt_i;
} else {
print "invalid interface $opt_i for $opt_d!\n\n";
print_help();
exit $ERRORS{'UNKNOWN'};
}
Specifically, I'd like to be able to pass the script something like "megaraid,[5-8]" and let it run for each. In this case, I would not be passing the regex for the device, it would just be /dev/sda.
If anyone could give me advice on this I'd appreciate it!
$opt_dl is probably poorly named and has nothing to do with your $opt_d, those are two separate variables.
From the if statement, if $opt_d is not set (that is the script was not given any device name to act upon), then glob is called with the value of $opt_g and it is glob in fact that finds out all filenames based on the regex given inside $opt_g.
After this if statement, the #dev array is filed with the names of devices to handle.
And then you have a foreach statement which means a loop on each item inside the #dev array. And during the loop, each item is in the $opt_dl variable, due to its use on the foreach statement.
However I was not able to understand what you wanted to do in your last paragraph.
I'm the maintainer of check_smart and it's funny I accidentally stumbled on that question now.
I don't quite understand why the variable is $opt_dl when earlier it seems to be $opt_d. The result, however, is that the script returns something like: OK: [/dev/sda] - Device is clean --- [/dev/sdb] - Device is clean --- [/dev/sdc] - Device is clean
So basically when you use the -g parameter, you tell the check_smart plugin to use glob (https://perldoc.perl.org/functions/glob.html) - this is not the same as regular expression. The drives matching the glob expression (e.g. -d '/dev/sd[a-z]) will create a list ($opt_dl) and the plugin will run through each drive in a for loop.
Specifically, I'd like to be able to pass the script something like "megaraid,[5-8]" and let it run for each. In this case, I would not be passing the regex for the device, it would just be /dev/sda.
This is already possible since release 5.0 (which was released in April 2014, way before your question ;-) ). You just need to change the syntax. Instead of using the glob expression on -d, you use it on the interface parameter (-i). Practical example: -i 'megaraid,[5-8]'.
Since the newest release (6.6, released a couple of days ago), the output for multiple drive checks (using -g) and hardware storage/raid controllers has slightly changed and now indicates the interface's device id rather than the logical drive path:
# ./check_smart.pl -g /dev/sda -i 'megaraid,[1-3]'
OK: [megaraid,1] - Device is clean --- [megaraid,2] - Device is clean --- [megaraid,3] - Device is clean|
This is all described in the official documentation, too.
More info:
https://www.claudiokuenzler.com/monitoring-plugins/check_smart.php
https://www.claudiokuenzler.com/blog/914/check_smart-6.6-multiple-drives-check-megaraid-3ware-cciss-controllers
I hope this answers your question, although I am probably 2 years late.

How to use return value from a Puppet exec?

How can I make the below logic work? My aim is to compare the value of custom fact $environment and the content of the file /etc/facter/facts.d/oldvalue.
If the custom fact $environment is not equal to the content of file /etc/facter/facts.d/oldvalue, then execute the following code.
exec {'catenvchange' :
command => "/bin/cat /root/oldvalue"}
if $environment != exec['catenvchange'] {#code#}
Exec resources do not work that way. In fact, no resource works that way, or any way remotely like that. Moreover, the directory /etc/facter/facts.d/ serves a special purpose, and your expectation for how it might be appropriate to use a file within is not consistent with that purpose.
What you describe wanting to do looks vaguely like setting up an external fact and testing its value. If you drop an executable script named /etc/facter/facts.d/anything by some means (manually, plugin sync, File resource, ...) then that script will be executed before each Puppet run as part of the process of gathering node facts. The standard output generated by the script would be parsed for key=value pairs, each defining a fact name and its value. The facts so designated, such as one named "last_environment" will be available during catalog building. You could then use it like so:
if $::environment != $::last_environment {
# ...
}
Update:
One way to use this mechanism to memorialize the value that a given fact, say $::environment, has on one run so that it can be read back on the next run would be to declare a File resource managing an external fact script. For example,
file { '/etc/facter/facts.d/oldvalues':
ensure => 'file',
owner => 'root',
group => 'root',
mode => '0755',
content => "#!/bin/bash\necho 'last_environment=${::environment}'\n"
}

Subscribe to new file(s) in directory in Puppet

I know I can sync directory in Puppet:
file { 'sqls-store':
path => '/some/dir/',
ensure => directory,
source => "puppet:///modules/m1/db-updates",
recurse => true,
purge => true
}
So when the new files are added they are copied to '/some/dir/'. However what I need is to perform some action for every new file. If I "Subscribe" to such resource, I don't get an array of new files.
Currently I created external shell script which finds new files in that dir and executes action for each of them.
Naturally, I would prefer not to depend on external script. Is there a way to do that with Puppet?
Thanks!
The use case for that is applying changes to DB schema that are being made from time to time and should be applied to all clients managed by puppet. In the end it's mysql [args] < update.sql for every such file.
Not sure I would recommend to have puppet applying the db changes for me.
For small db, it may work but for real world db... you want to be aware of when and how these kind of changes got applied (ordering of the changes, sometime require temp disk space adjustement, db downtime, taking backup before/after, reorg,...), most of the times your app should be adapted at the same time. You want more orchestration (and puppet isn't good at orchestration)
Why not using a tool dedicated to this task like
liquid-base
rails db migrations and capistrano
...
A poor men solution would be to use vcs-repo module and an exec to list modified files since last "apply".
I agree with mestachs, puppet dealing with db updates it's not a great idea
You can try some kind of define:
define mydangerousdbupdate($name, $filename){
file { "/some/dir/$filename":
ensure => present,
source => "puppet:///modules/m1/db-updates/$filename",
}
exec{"apply $name":
command => "/usr/bin/mysql [args] < /some/dir/$filename > /some/dir/$filename.log",
creates => "/some/dir/$filename.log"
}
}
And then, you can instantiate with the different patches, in the preferred order
mydangerousdbupdate{"first_change":
name => "first",
filename => "first.sql",
}->mydangerousdbupdate{"second_change":
name => "second",
filename => "second.sql",
}

Resources