Can you think of a way to solve this problem in Puppet?
I have a custom fact with generates a string of IP addresses depending on the domain it is run on, it can resolve to have 1 to n addresses.
"10.1.29.1"
"10.1.29.1,10.1.29.5"
"10.1.29.1,10.1.29.5,10.1.29.7"
etc
I want to add these to the host file with a generated server names of servernameX for example;
10.1.29.1 myservername1
10.1.29.5 myservername2
10.1.29.7 myservername3
So how can you do this as puppet doesn't have an array iterator like "for each"?
Sadly, even if you go about and use a custom "define" to iterate over an array upon splitting your custom fact on a comma, the result will be rather not what you expect and not even close to a "for each" loop -- aside of causing you a headache, probably.
Said that, I am not sure if this is what you want to achieve, but have a look at this approach:
$fact = '1.1.1.1,2.2.2.2,3.3.3.3'
$servers = split($::fact, ',')
$count = size($servers)
$names = bracket_expansion("host[01-${count}].address")
file { '/tmp/test.txt':
content => inline_template('<%= #servers.each_with_index.map {|v,i| "#{v}\t\t#{#names[i]}\n" } %>'),
ensure => present
}
What we have there are two custom functions: size() and bracket_expansion(); which we then use values that they provide inside a hack that leverages the inline_template() function to render content of the file utilising parallel access to two arrays -- one with IP addresses from your fact and one with host names that should follow.
The result is a follows:
matti#acrux ~ $ cat | puppet apply
$fact = '1.1.1.1,2.2.2.2,3.3.3.3'
$servers = split($::fact, ',')
$count = size($servers)
$names = bracket_expansion("host[01-${count}].address")
file { '/tmp/test.txt':
content => inline_template('<%= #servers.each_with_index.map {|v,i| "#{v}\t\t#{#names[i]}\n" } %>'),
ensure => present
}
notice: /Stage[main]//File[/tmp/test.txt]/ensure: created
notice: Finished catalog run in 0.07 seconds
matti#acrux ~ $ cat /tmp/test.txt
1.1.1.1 host01.address
2.2.2.2 host02.address
3.3.3.3 host03.address
matti#acrux ~ $
Both size() and bracket_expansion() functions can be found here:
https://github.com/kwilczynski/puppet-functions/tree/master/lib/puppet/parser/functions/
I hope this helps a little :-)
Related
I'm using puppet 5 and writing a module which refers to some hiera which has some duplication in it (example below - see gpgkey):
profile::example1:
repo1:
descr: Centos repo
gpgkey: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
repo2:
descr: Centos repo
gpgkey: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
repo3:
descr: puppet repo
gpgkey: http://puppet.repo/GPG-KEY
I've successfully managed to retrieve the repo names (repo1, repo2 and repo3 in this example). What I'd like to do next is run an exec to import the gpgkey, however this is failing with a duplicate declaration error (I assume because the gpgkey is duplicated in the hiera). Any ideas or help on how to do this? Here's the lookup:
$repo_name = lookup('profile::example1', Hash, 'deep')
$repo_name.each | $name, Hash $config_hash | {
notify { "${name}": }
}
I've looked at embedding another loop to pull values from $config_hash but I usually get the same result regardless.
Puppet's built-in unique() function seems natural for this job. For example:
$repos = lookup('profile::example1', Hash, 'deep')
$unique_keys = $repos.map |$_name, $props| { $props['gpgkey'] } .unique
$unique_keys.each |$key| {
# ...
}
I have several SOAP::Lite clients running under mod_perl in Apache hhtpd.
Some of them use 1.1 soap-servers and some of them use 1.2 servers. So I have code like:
# In client 1:
my $soap1 = SOAP::Lite->soapversion("1.1");
$result1 = $soap1->method1();
# In client 2:
my $soap2 = SOAP::Lite->soapversion("1.2");
$result2 = $soap2->method2();
This works in stand-alone clients, but when I run the code under mod_perl, I seem to get stung by that the soapversion
method has side-effects:
# From SOAP::Lite.pm
sub soapversion {
my $self = shift;
my $version = shift or return $SOAP::Constants::SOAP_VERSION;
($version) = grep {
$SOAP::Constants::SOAP_VERSIONS{$_}->{NS_ENV} eq $version
} keys %SOAP::Constants::SOAP_VERSIONS
unless exists $SOAP::Constants::SOAP_VERSIONS{$version};
die qq!$SOAP::Constants::WRONG_VERSION Supported versions:\n#{[
join "\n", map {" $_ ($SOAP::Constants::SOAP_VERSIONS{$_}->{NS_ENV})"} keys %SOAP::Constants::SOAP_VERSIONS
]}\n!
unless defined($version) && defined(my $def = $SOAP::Constants::SOAP_VERSIONS{$version});
foreach (keys %$def) {
eval "\$SOAP::Constants::$_ = '$SOAP::Constants::SOAP_VERSIONS{$version}->{$_}'";
}
$SOAP::Constants::SOAP_VERSION = $version;
return $self;
}
This is what I believe happens:
Basically, the soapversion call rededefines essential constants in $SOAP::Constants. And since this is mod_perl, the $SOAP::Constants are global and shared between every server-thread (I believe. Please correct me if I'm wrong). This leads to a race-condition: Most of the times, the codelines gets executed more-or-less in the sequence seen above. But once in at while (actually about 2% of the calls) the execution sequence is:
Thread1: my $soap1 = SOAP::Lite->soapversion("1.1");
Thread2: my $soap2 = SOAP::Lite->soapversion("1.2");
Thread1: $result1 = $soap1->method1();
Thread2: $result2 = $soap2->method2();
And so, the $soap1->method1() gets called with $SOAP::Constants set as befitting version 1.2 - causing several namespace to be wrong, notably:
xmlns:soapenc="http://www.w3.org/2003/05/soap-encoding"
Which is wrong for 1.1 - who prefers:
xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/"
If I could somehow make $SOAP::Constants localized to a serverthread or something like that, I guess things would be fine. But any solution will be appreciated.
Run Apache with the prefork model instead of the threading model (mpm_prefork_module instead of mpm_event_module or mpm_worker_module), so that each Apache child will have its own Perl interpreter, hence its own set of constants.
Otherwise have a look on the modperl documentation regarding the PerlOptions directive, specifically the clone and/or parent value. But stop using threads seem simpler to me, threads and Perl were never friends.
My first question is:
Is this possible to do this, since now I have a perl script which reads Html file and extract data to display on another html file.
If the answer for the question above is Yes, my second question would be:
How to do this?
Sorry to ask frankly as this, but since I'm so new for perl, and I have to take this task, so I'm here for some useful advice or suggestion to guide me through this task. Appreciate your help in advance.
Here's a part of the code, since the whole chunk is quite long:
$date=localtime();
($TWDAY, $TMTH, $TD1D, $TSE, $TYY) = split(/\s+/, $date);
$TSE =~ s/\://g;
$STAMP=_."$TD1D$TMTH$TYY";
#ServerInfo=();
#--------------------------------------------------------------------------- -------------------------------
# Read Directory
#----------------------------------------------------------------------------------------------------------
$myDir=getcwd;
#----------------------------------------------------------------------------------------------------------
# INITIALIZE HTML FORMAT
#----------------------------------------------------------------------------------------------------------
&HTML_FORMAT;
#----------------------------------------------------------------------------------------------------------
# REPORT
#----------------------------------------------------------------------------------------------------------
if (! -d "$myDir/report") { mkdir("$myDir/report");};
$REPORTFILE="$myDir/report/checkpack".".htm";
open OUT,">$REPORTFILE" or die "\nCannot open out file $REPORTFILE\n\n";
print OUT "$Tag_Header";
#----------------------------------------------------------------------------------------------------------
sub numSort {
if ($b < $a) { return -1; }
elsif ($a == $b) { return 0;}
elsif ($b > $a) { return 1; }
}
#ArrayDir = sort numSort #DirArray;
#while (<#ArrayDir>) {
#OutputDir=grep { -f and -T } glob "$myDir/*.htm $myDir/*.html";
#}
#----------------------------------------------------------------------------------------------------------
#ReadLine3=();
$xyxycnt=0;
foreach $InputFile (#OutputDir) { #---- MAIN
$filename=(split /\//, $InputFile) [-1]; print "-"x80 ; print "\nFilename\t:$filename\n";
open IN, "<$InputFile" or die "Cannot open Input file $InputFile\n";
#MyData=();
$DataCnt=0;
#MyLine=();
$MyLineCnt=0;
while (<IN>) {
$LINE=$_;
chomp($LINE);
$LINE=~s/\<br\>/XYXY/ig;
$LINE=~s/\<\/td\>/ \nXYZXYZ\n/ig;
$LINE=~s/\<dirname\>/xxxdirnameyyy/ig;
$LINE=linetrim3($LINE);
$LINE=linetrim($LINE);
$LINE=~s/XYXY/\<br\>/ig;
$LINE=~s/xxxdirnameyyy/< dirname >/ig;
$LINE=~s/^\s+//ig;
print OUT2 "$LINE\n";
if (defined($LINE)) { $MyData[$DataCnt]="$LINE"; $DataCnt++ ; }
}
close IN;
foreach $ReadFile (#MyData) { #--- Mydata
$MyLineCnt++;
$MyLine[$MyLineCnt]="";
#### FILENAME
$ServerInfo[0]="$filename";
#### IP ADDRESS
if ($ReadFile =~ /Host\/Device Name\:/) {
#print "$ReadFile\n"
($Hostname)=(split /\:|\s+/, $ReadFile)[3]; print "$Hostname\n";
&myServerInfo("$Hostname","1");
}
if ($ReadFile =~ /IP Address\(es\)/) {#ListIP=(); $SwIP=1; $CntIP=0 ; };
#### OPERATING SYSTEM & VERSION
if ($ReadFile =~ /Operating System\:/) {
$SwIP=0;
$OS= (split /\:|\s+/, $ReadFile)[3]; &myServerInfo("$OS","3") ; print "$OS\n";
$OSVer= (split /\:|\s+/, $ReadFile)[-2]; &myServerInfo("$OSVer","4") ; print "$OSVer\n";
};
#### GET IP VALUE
if ($SwIP==1) {
$ReadFile=(split /\:/,$ReadFile) [2];
$ReadFile=~s/[a-z|A-Z]|\(|\)|\// /ig; print "$ReadFile\n";
if ($CntIP==0) {
#$ListIP[$CntIP]=(split /\s+/,$ReadFile) [1];
#ListIP="$ReadFile";
} elsif ($CntIP==1) { print "\n\t\t $ReadFile\n" ; $ListIP[$CntIP]="\n$ReadFile";
} else { print "\t\t $ReadFile\n" ; $ListIP[$CntIP]="\n$ReadFile"; };
$CntIP++;
}
I'm afraid if you don't understand what is going on in this program and you also don't understand how to approach a task like this at all, Stack Overflow might not be the right place to get help.
Let me try to show you the approach I would take with this. I'm assuming there is more code.
First, write down a list of everything you know:
What is the input format of the existing file
Where does the existing file come from now
What is the output format of the existing file
Where does the generated output file go afterwards
What does the new file look like
Where does the new file come from
Use perltidy to indent the inherited code so you can read it better. The default options should be enough.
Read the code, take notes about what pieces do what, add comments
Write a unit test for the desired output format. You can use Test::More. Another useful testing module here is Test::File.
Refactor the part that generated the output format to work with a certain data structure. Use your tests to make sure you don't break it.
Write code to parse the new file into the data structure from the point above. Now you can plug that in and get the expected output.
Refactor the part that takes the old input file from the existing file location to be a function, so you can later switch it for the new one.
Write code to get the new file from the new file location.
Document what you did so the next guy is not in the same situation. Remember that could be you in half a year.
Also add use strict and use warnings while you refactor to catch errors more easily. If stuff breaks because of that, make it work before you continue. Those pragmas tell you what's wrong. The most common one you will encounter is Global symbol "$foo" requires explicit package name. That means you need to put my in front of the first assignment, or declare the variable before.
If you have specific questions, ask them as a new question with a short example. Read how to ask to make sure you will get help on those.
Good luck!
After seing your comment I am thinking you want a different input and a different output. In that case, disregard this, throw away the old code and start from scratch. If you don't know enough Perl, get a book like Curtis Poe's Beginning Perl if you already know programming. If not, check out Learning Perl by Randal L. Schwartz.
I have an array as follows:
$servers = [
{name => 'server1', address => '10.10.10.11', port => 8081},
{name => 'server2', address => '10.10.10.12', port => 8082},
]
and I need to programmatically convert it into a hash based on the 'name' key:
$serversMap = {
server1 => {address => '10.10.10.11', port => 8081},
server2 => {address => '10.10.10.12', port => 8082},
}
this works in puppet 3:
$serversMap = $servers.reduce({}) |$cumulate, $server| {
$key = "${server['name']}"
$value = $server.filter |$entry| { $entry[0] != 'name' }
$tmp = merge($cumulate, {"$key" => $value })
$tmp
}
however, is there an easier way?
and why I need to create the temporal variable $tmp?
without that, I get at Error: Function 'merge' must be the value of a statement error.
ps: it's obvious but I will say it anyway: the $servers variable is given to me, I cannot change its structure. and I need that same data but in the form of $serversMap. and so, I am asking how to programmatically convert from $servers to $serversMap.
Except for the $tmp variable this code is optimal.
The $tmp variable needs to be there, because the merge function is an rvalue. From the docs:
There are actually two completely different types of functions available — rvalues (which return a value) and statements (which do not). If you are writing an rvalue function, you must pass :type => :rvalue when creating the function; see the examples below.
The puppet source that produces this error is pretty self explanatory when it comes to rvalue statements:
# It is harmless to produce an ignored rvalue, the alternative is to mark functions
# as appropriate for both rvalue and statements
# Keeping the old behavior when a pblock is not present. This since it is not known
# if the lambda contains a statement or not (at least not without a costly search).
# The purpose of the check is to protect a user for producing a meaningless rvalue where the
# operation has no side effects.
#
if !pblock && Puppet::Parser::Functions.rvalue?(#name)
raise Puppet::ParseError,
"Function '#{#name}' must be the value of a statement"
end
The real reason is that Puppet is just not that sophisticated as a language. If you want real power switch to Ruby.
Is there a way to parse thru a txt file that includes elements seperated by {}s
Here is a sample from the file:
virtual vs_devtnet_80 {
snat automap
pool pool_devtnet_80
destination 167.69.107.41:http
ip protocol tcp
profiles {
profile_http_health {}
tcp-lan-optimized {}
}
}
virtual vs_devdpp_4430 {
snat automap
pool pool_devdpp_5430
destination 167.69.107.31:https
ip protocol tcp
persist devdpp
profiles tcp-lan-optimized {}
}
virtual vs_devwww30_80 {
snat automap
pool pool_devwww30_80
destination 167.69.107.46:http
ip protocol tcp
profiles {
profile_http_health {}
tcp-lan-optimized {}
}
}
As you can see, the elements are separated, but with {}
Any help would be gladly appreciated. I was trying to use grep but it only returns one line...
I would like to be able to search by the top most element, for example searh.sh virtual vs_devtnet_80, and have it return the entire blob..furthermore perhaps be able to search for botht eh top layer and one of its sub layers, for example search.sh virtual vs_devtnet_80 pool which would return pool_devtnet_80
Something like:
cat .tmp | sed '/.*{/ ! { s/.*//g}'
this wont solve it completely, but i think it does something similar to what you want
Look at JSON pasers, they're written in all sort of languages, syntax looks similar enough to give you some ideas on how to tackle this.
Basically what you need to do is have a recursive function that calls itself whenever it encounters a '{' and returns the content whenever it encounters a '}'.
I wrote an article on a lisp-like parser that actually does just that. Check it out here for inspiration: http://www.codeproject.com/KB/linq/TinyLisp.aspx
Rgds Gert-Jan
This is Tcl syntax, so you can set up a mechanism to run it as a Tcl script that creates a data structure of itself.
#!/usr/bin/env tclsh
# create a safe slave interpreter
set slave [interp create -safe]
# set up the slave to parse the input file
$slave eval {
proc virtual {subkey body} {
eval $body
}
proc unknown {cmd args} {
upvar 1 subkey subkey ;# access the 'subkey' variable from the 'virtual' proc
if {[llength $args] == 1} {set args [lindex $args 0]}
dict set ::data $subkey $cmd $args
}
set data [dict create]
}
$slave expose source
# now do the parsing in the slave
$slave eval source [lindex $argv 0]
# fetch the data structure
set data [$slave eval set data]
# and do stuff with it.
dict for {subkey subdict} $data {
puts "$subkey destination = [dict get $subdict destination]"
}
And then, parser.tcl input_file outputs
vs_devaetnet_80 destination = 167.69.107.41:http
vs_devdpp_4430 destination = 167.69.107.31:https
vs_devwww30_80 destination = 167.69.107.46:http
This should give you the 2nd 'blob', for instance:
sed -n '/virtual vs_devdpp_4430.*/,/^}$/p' filename.txt
and pipe through grep -oP '(?<=pool ).*' to get what follows pool in that blob.
I ended up having to create a recursive function that first used strpos to find the search variable within the entire config file. Then using a recursive function pop'd on and pop'd off brackets to return the searched variable's entire body.