puppet not working with keys() and hiera_hash() - puppet

I have in hiera node variable solr_enabled = true. Also I have in this node list of fstab mount points like:
fstab_homes:
'/home1':
device: 'UUID=ac2ca97e-8bce-4774-92d7-051482253089'
'/home2':
device: 'UUID=d9daaeed-4e4e-40e9-aa6b-73632795e661'
'/home3':
device: 'UUID=21a358cf-2579-48cb-b89d-4ff43e4dd104'
'/home4':
device: 'UUID=c68041de-542a-4f72-9488-337048c41947'
'/home16':
device: 'UUID=d55eff53-3087-449b-9667-aeff49c556e7'
In solr.pp I want to get the first mounted home disk, create there folder and make symbolic link to /home/cpanelsolr.
For this I wrote the code /etc/puppet/environments/testing/modules/cpanel/manifests/solr.pp:
# Install SOLR - dovecot full text search plugin
class cpanel::solr(
$solr_enable = hiera('solr_enabled',false),
$homes = hiera_hash('fstab_homes', false),
$homesKeys = keys($homes),
)
{
if $solr_enable == true {
notify{"Starting Solr Installation ${homesKeys[0]}":}
if $homes != false and $homesKeys[0] != '/home' {
file { "Create Solr home symlink to ${homesKeys[0]}":
path => '/home/cpanelsolr',
ensure => 'link',
target => "${homesKeys[0]}/cpanelsolr",
}
}
exec { 'cpanel-dovecot-solr':
command => "/bin/bash -c
'/usr/local/cpanel/scripts/install_dovecot_fts'",
}
}
}
But when I run this in dev node I get error:
root#webcloud2 [/home1]# puppet agent -t --no-use_srv_records --server=puppet.development.internal --environment=testing --tags=cpanel::solr
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
2018-08-03 6:04:54 140004666824672 [Note] libgovernor.so found
2018-08-03 6:04:54 140004666824672 [Note] All governors functions found too
2018-08-03 6:04:54 140004666824672 [Note] Governor connected
2018-08-03 6:04:54 140004666824672 [Note] All governors lve functions found too
Error: Could not retrieve catalog from remote server: Error 400 on SERVER: keys(): Requires hash to work with at
/etc/puppet/environments/testing/modules/cpanel/manifests/solr.pp:6 on node webcloud2.development.internal
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
What's wrong?

You have at least two problems there.
First problem is $home won't be set at all in that context. You would need to rewrite as:
class cpanel::solr(
$solr_enable = hiera('solr_enabled',false),
$homes = hiera_hash('fstab_homes', false),
)
{
$homes_keys = keys($homes)
...
}
Second problem is that your YAML isn't correctly indented, so fstab_homes would not actually return a Hash. It should be:
fstab_homes:
'/home1':
device: 'UUID=ac2ca97e-8bce-4774-92d7-051482253089'
'/home2':
device: 'UUID=d9daaeed-4e4e-40e9-aa6b-73632795e661'
'/home3':
device: 'UUID=21a358cf-2579-48cb-b89d-4ff43e4dd104'
'/home4':
device: 'UUID=c68041de-542a-4f72-9488-337048c41947'
'/home16':
device: 'UUID=d55eff53-3087-449b-9667-aeff49c556e7'
Finally, be aware that use of camelCase in parameter names in Puppet can cause you issues in some contexts, so best to use snake_case.

Related

capture journald properties with rsyslog

I am struggling on how to capture systemd-journald properties into rsyslog files.
My setup
ubuntu inside docker on arm (raspberrypi): FROM arm64v8/ubuntu:20.04
docker command (all subsequent actions taken inside running docker container)
$ docker run --privileged -ti --cap-add SYS_ADMIN --security-opt seccomp=unconfined --cgroup-parent=docker.slice --cgroupns private --tmpfs /tmp --tmpfs /run --tmpfs /run/lock systemd:origin
rsyslog under $ sytemctl status rsyslog
● rsyslog.service - System Logging Service
Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor prese>
Active: active (running)
...
[origin software="rsyslogd" swVersion="8.2001.0" x-pid="39758" x-info="https://www.rsyslog.com"] start
...
My plan
Having a small c program to put some information into journal:
#include <systemd/sd-journal.h>
#include <stdio.h>
#include <unistd.h>
int main(int arcg, char** args) {
char buffer [50];
sprintf (buffer, "%lu", (unsigned long)getpid());
printf("writing to journal\n");
sd_journal_print(LOG_WARNING, "%s", "a little journal test message");
sd_journal_send("MESSAGE=%s", "there shoud be a text", "SYSLOG_PID=%s", buffer, "PRIORITY=%i", LOG_ERR, "DOCUMENTATION=%s", "any doc link", "MESSAGE_ID=%s", "e5e4132e441541f89bca0cc3e7be3381", "MEAS_VAL=%d", 1394, NULL);
return 0;
}
Compile it: $ gcc joutest.c -lsystemd -o jt
Execute it: $ ./jt
This results inside the journal as $ journalctl -r -o json-pretty:
{
"_GID" : "0",
"MESSAGE" : "there shoud be a text",
"_HOSTNAME" : "f1aad951c039",
"SYSLOG_IDENTIFIER" : "jt",
"_TRANSPORT" : "journal",
"CODE_FILE" : "joutest.c",
"DOCUMENTATION" : "any doc link",
"_BOOT_ID" : "06a36b314cee462591c65a2703c8b2ad",
"CODE_LINE" : "14",
"MESSAGE_ID" : "e5e4132e441541f89bca0cc3e7be3381",
"_CAP_EFFECTIVE" : "3fffffffff",
"__REALTIME_TIMESTAMP" : "1669373862349599",
"_SYSTEMD_UNIT" : "init.scope",
"CODE_FUNC" : "main",
"_MACHINE_ID" : "5aba31746bf244bba6081297fe061445",
"SYSLOG_PID" : "39740",
"PRIORITY" : "3",
"_COMM" : "jt",
"_SYSTEMD_SLICE" : "-.slice",
"MEAS_VAL" : "1394",
"__MONOTONIC_TIMESTAMP" : "390853282189",
"_PID" : "39740",
"_SOURCE_REALTIME_TIMESTAMP" : "1669373862336503",
"_UID" : "0",
"_SYSTEMD_CGROUP" : "/init.scope",
"__CURSOR" : "s=63a46a30bbbb4b8c9288a9b12c622b37;i=6cb;b=06a36b314cee46>
}
Now as a test, extracting all properties from that journal entry via rsyslog; property in the jargon of rsyslog in principle is the name of a key in the formatted json entry. But if a property (or key name) matches, the whole dictionary item (key and value) shall be captured
To start with this, I've configured rsyslog as:
module(load="imjournal")
module(load="mmjsonparse")
action(type="mmjsonparse")
if $programname == 'jt' and $syslogseverity == 3 then
action(type="omfile" file="/var/log/jt_err.log" template="RSYSLOG_DebugFormat")
This config is located in /etc/rsyslog.d/filter.conf and gets automatically included by /etc/rsyslog.conf:
# /etc/rsyslog.conf configuration file for rsyslog
#
# For more information install rsyslog-doc and see
# /usr/share/doc/rsyslog-doc/html/configuration/index.html
#
# Default logging rules can be found in /etc/rsyslog.d/50-default.conf
#################
#### MODULES ####
#################
#module(load="imuxsock") # provides support for local system logging
#module(load="immark") # provides --MARK-- message capability
# provides UDP syslog reception
#module(load="imudp")
#input(type="imudp" port="514")
# provides TCP syslog reception
#module(load="imtcp")
#input(type="imtcp" port="514")
# provides kernel logging support and enable non-kernel klog messages
module(load="imklog" permitnonkernelfacility="on")
###########################
#### GLOBAL DIRECTIVES ####
###########################
#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# Filter duplicated messages
$RepeatedMsgReduction on
#
# Set the default permissions for all log files.
#
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog
#
# Where to place spool and state files
#
$WorkDirectory /var/spool/rsyslog
#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf
Applied this config: $ systemctl restart rsyslog
Which results in the following: $ cat /var/log/jt_err.log
Debug line with all properties:
FROMHOST: 'f1aad951c039', fromhost-ip: '127.0.0.1', HOSTNAME:
'f1aad951c039', PRI: 11,
syslogtag 'jt[39765]:', programname: 'jt', APP-NAME: 'jt', PROCID:
'39765', MSGID: '-',
TIMESTAMP: 'Nov 25 11:47:50', STRUCTURED-DATA: '-',
msg: ' there shoud be a text'
escaped msg: ' there shoud be a text'
inputname: imuxsock rawmsg: '<11>Nov 25 11:47:50 jt[39765]: there
shoud be a text'
$!:{ "msg": "there shoud be a text" }
$.:
$/:
My problem
Looking on the resulting rsyslog, I miss a majority, if not all, of items originating from the journal entry.
There is really no property (key) matching. Shouldn't be there all properties matched as it is a debug output?
Specifically I am concentrating on my custom property, MEAS_VAL, it is not there.
The only property which occurs is "msg", which by the way is questionable whether it is a match of the journal, since the originating property name attached to the resulting content "there shoud be a text" is MESSAGE
So it feels that I don't hit the whole journal capturing mechanism at all, why?
Can we be sure that imjournal gets loaded properly?
I would say yes because of systemd's startup messages:
Nov 28 16:27:38 f1aad951c039 rsyslogd[144703]: imjournal: Journal indicates no msgs when positioned at head. [v8.2212.0.master try https://www.rsyslog.com/e/0 ]
Nov 28 16:27:38 f1aad951c039 rsyslogd[144703]: imjournal: journal files changed, reloading... [v8.2212.0.master try https://www.rsyslog.com/e/0 ]
Nov 28 16:27:38 f1aad951c039 rsyslogd[144703]: imjournal: Journal indicates no msgs when positioned at head. [v8.2212.0.master try https://www.rsyslog.com/e/0 ]
Edit 2022-11-29
Meanwhile I've compiled my own version 8.2212.0.master. But the phenomenon persists.
You're missing most items originating from the journal, because both templates RSYSLOG_DebugFormat and RSYSLOG_TraditionalFileFormat do not have the needed properties (See Reserved template names). RSYSLOG_DebugFormat, however, includes atleast some fields, e.g. procid, msgid and structured-data - which can be seen in the output you've provided.
This means, that if you want to include all the fields, you'll have to create your own template.
The journal fields are stored in key-value pairs. The imjournal module is able to parse these key-value pairs and generate the jsonf property,
which then can be used to access fields of the log message as if they were fields in a JSON object.
# load imjournal module
module(load="imjournal")
# specify journal as input source
input(type="imjournal")
template(name="journalTemplate" type="list") {
property(name="timestamp" dateFormat="rfc3339")
constant(value=" ")
property(name="hostname")
constant(value=" ")
property(name="syslogtag")
constant(value=": {")
property(name="jsonf")
constant(value="}")
}
if $programname == 'jt' and $syslogseverity == 3 then {
action(type="omfile" file="/var/log/jt_err.log" template="journalTemplate")
stop
}
The output of the provided log would then look something like the following:
YYYY-MM-DDTHH:mm:ss myHostname syslogtag: {"_GID" : "0", "MESSAGE" : "there shoud be a text", ... }
As seen in the log above, the output of the provided properties will be in JSON. By using the json property parser this can be prevented, as the output can be tailored as desired. If this is used, however, each property must be defined specifically.
template(name="journalTemplate" type="list") {
property(name="timestamp" dateFormat="rfc3339")
constant(value=" ")
property(name="hostname")
constant(value=" ")
property(name="syslogtag")
constant(value=": _GID=")
property(name="$._GID" format="json")
constant(value=" MESSAGE=")
property(name="$.MESSAGE" format="json")
constant(value=" _HOSTNAME=")
property(name="$._HOSTNAME" format="json")
...
}

Error: Could not retrieve catalog from remote server: Error 500 on SERVER:

Getting error 500
puppet agent -t
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Could not find node statement with name 'default' or 'client.domain.com' on node client.domain.com
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
Puppet.conf content below:
[master]
dns_alt_names = master.domain.com,server
certname = master.domain.com
[agent]
server = master.domain.com
ca_server = master.domain.com
certname = client.domain.com
use_cached_catalog = true
Your site.pp file under the manifests folder need to contain either a node 'client.domain.com' (as client.domain.com is your certname for the agent) or a node default.
For instance:
node default {
file { '/tmp/puppetdir':
ensure => 'directory',
owner => 'sysadmin',
group => 'wheel',
mode => '0755'
}
}
node 'client.domain.com' {
package { 'httpd':
ensure => "installed",
}
service { 'httpd':
ensure => running,
enable => true
}
file { '/tmp/puppetdir':
ensure => 'directory',
owner => 'sysadmin',
group => 'wheel',
mode => '0755'
}
}

Hiera value not able to receiving in puppet profile

I have created a proxy_match.yaml file as a hiera source file in
default hiera datalocation.
The proxy_match.yaml is added in hiera hierarchy
Looking up hiera data in profile
Where and what am I missing, I am not able to receive the hiera data
value and thus the error appears mentioned bellow.
Where,
proxy_match is the new environment created
hierafile 1
/etc/puppetlabs/code/environments/proxy_match/hiera.yaml
version: 5
defaults:
# The default value for "datadir" is "data" under the same directory as the hiera.yaml
# file (this file)
# When specifying a datadir, make sure the directory exists.
# See https://docs.puppet.com/puppet/latest/environments.html for further details on environments.
# datadir: data
# data_hash: yaml_data
hierarchy:
- name: "environment specific yaml"
path: "proxy_match.yaml"
- name: "Per-node data (yaml version)"
path: "nodes/%{::trusted.certname}.yaml"
- name: "Other YAML hierarchy levels"
paths:
- "common.yaml"
proxy_match.yaml hiera data source file
This is the yaml hiera source named as proxy_match.yaml as in herarchy
/etc/puppetlabs/code/environments/proxy_match/data/proxy-match.yaml
---
profiles::apache::servername: "taraserver.com"
profiles::apache::port: "80"
profiles::apache::docroot: "/var/www/tarahost"
hiera lookup in profile
$servername = hiera('profiles::apache::servername',{})
$port = hiera('profiles::apache::port',{})
$docroot = hiera('profiles::apache::docroot',{})
class profile::apache{
#configure apache
include apache
apache::vhost{$servername:
port => $port,
docroot => $docroot,
}
}
#ERROR:
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: {"message":"Server Error: Evaluation Error: Error while evaluating a Resource Statement, Apache::Vhost[7fba80ae621c.domain.name]: parameter 'docroot' expects a value of type Boolean or String, got Undef at /etc/puppetlabs/code/environments/proxy_match/modules/profile/manifests/apache.pp:29 on node 94707b03ff05.domain.name","issue_kind":"RUNTIME_ERROR"}
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
You are defining the variables outside the class definition. When Puppet loads that class, those lines you have before the class are ignored.
What you should have in your profile class is:
class profiles::apache (
String $servername,
Integer $port,
String $docroot,
) {
include apache
apache::vhost { $servername:
port => $port,
docroot => $docroot,
}
}
Note that I used the automatic parameter lookup feature to set your variables, instead of explicit calls to the hiera function.

Puppet can't find class firewall

I have a basic puppet install using this tutorial https://www.digitalocean.com/community/tutorials/how-to-install-puppet-4-on-ubuntu-16-04
When I run /opt/puppetlabs/bin/puppet agent --test on my node I get
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Error while evaluating a Resource Statement. Could not find declared class firewall at /etc/puppetlabs/code/environments/production/manifests/site.pp:7:1 on node mark-inspiron.
On my node:
/opt/puppetlabs/bin/puppet module list
returns
/etc/puppetlabs/code/environment/production/modules
----- puppetlabs-firewall (v1.9.0)
On my puppet master at /etc/puppetlabs/code/environments/production/manifests/site.pp:
file {'/tmp/it_works.txt': # resource type file and filename
ensure => present, # make sure it exists
mode => '0644', # file permissions
content => "It works on ${ipaddress_eth0}!\n", # Print the eth0 IP fact
}
class { 'firewall': }
resources { 'firewall':
purge => true,
}
firewall { "051 asterisk-set-rate-limit-register":
string => "REGISTER sip:",
string_algo => "bm",
dport => '5060',
proto => 'udp',
recent => 'set',
rname => 'VOIPREGISTER',
rsource => 'true';
}
firewall { "052 asterisk-drop-rate-limit-register":
string => "REGISTER sip:",
string_algo => "bm",
dport => '5060',
proto => 'udp',
action => 'drop',
recent => 'update',
rseconds => '600',
rhitcount => '5',
rname => 'VOIPREGISTER',
rsource => true,
rttl => true;
}
The file part works but not firewall.
You need to install the modules on your master in a master setup with Puppet. They need to be somewhere in your modulepath. You can either place it in the modules directory within your $codedir (normally /etc/puppetlabs/code/modules) or in your directory environment modules directory (likely /etc/puppetlabs/code/environments/production/modules in your case since your cited site.pp is there). If you defined additional module paths in your environment.conf, then you can also place the modules there.
You can install/deploy them with a variety of methods, such as librarian-puppet, r10k, or code-manager (in Enterprise). However, the easiest method for you would be puppet module install puppetlabs-firewall on the master. Your Puppet catalog will then find the firewall class during compilation.
On a side note, that:
resources { 'firewall':
purge => true,
}
will remove any changes to associated firewall configurations (as defined by Puppet's knowledge of the system firewall configuration according to the module's definition of what the resource manages) that are not managed by Puppet. This is nice for eliminating local changes that people make, but it can also have interesting side effects, so be careful.

Containing a defined resource type in Puppet

I'm trying to create an instance of a defined resource type (::apt::ppa) that comes before other resources. I am using the PuppetLabs Apt Module.
When adding a new repository via the module, the defined type contains an exec statement that notifies apt::update so that any packages that might be required can be installed correctly. However, when I run my below code, the notify gets scheduled after I attempt to install Java, thereby causing the Java install to fail. I've tried putting anchors around the apt::ppa declaration, but that doesn't help. What else can I do?
class rap::java(
$version = '7',
) {
$package = "oracle-java${version}-installer"
apt::ppa { 'ppa:webupd8team/java': } ->
exec { 'accept-java-license':
command => "/bin/echo ${package} shared/accepted-oracle-license-v1-1 select true | /usr/bin/sudo /usr/bin/debconf-set-selections",
unless => "/usr/bin/debconf-show ${package} | grep 'shared/accepted-oracle-license-v1-1: true'",
} ->
class { '::java':
package => $package,
distribution => 'oracle-jre',
}
file_line { 'java_environment':
path => '/etc/environment',
line => "JAVA_HOME=\"/usr/lib/jvm/java-${version}-oracle\"",
}
}
I believe the issue is that you need to include the apt class within the class you've made to get the ordering right.
This works for me on a new Precise box:
class rap::java(
$version = '7',
) {
$package = "oracle-java${version}-installer"
include apt
apt::ppa { 'ppa:webupd8team/java':
package_manage => true,
}
exec { 'accept-java-license':
command => "/bin/echo ${package} shared/accepted-oracle-license-v1-1 select true | /usr/bin/sudo /usr/bin/debconf-set-selections",
unless => "/usr/bin/debconf-show ${package} | grep 'shared/accepted-oracle-license-v1-1: true'",
}
class { '::java':
package => $package,
distribution => 'oracle-jre',
require => [
Apt::Ppa['ppa:webupd8team/java'],
Exec["accept-java-license"],
]
}
file_line { 'java_environment':
path => '/etc/environment',
line => "JAVA_HOME=\"/usr/lib/jvm/java-${version}-oracle\"",
}
}
Log of run:
Notice: Compiled catalog for precise64 in environment production in 0.78 seconds
Notice: /Stage[main]/Apt/File[preferences]/ensure: created
Notice: /Stage[main]/Rap::Java/Exec[accept-java-license]/returns: executed successfully
Notice: /Stage[main]/Rap::Java/File_line[java_environment]/ensure: created
Notice: /Stage[main]/Apt/Apt::Setting[conf-update-stamp]/File[/etc/apt/apt.conf.d/15update-stamp]/ensure: defined content as '{md5}0962d70c4ec78bbfa6f3544ae0c41974'
Notice: /Stage[main]/Rap::Java/Apt::Ppa[ppa:webupd8team/java]/Package[python-software-properties]/ensure: created
Notice: /Stage[main]/Rap::Java/Apt::Ppa[ppa:webupd8team/java]/Exec[add-apt-repository-ppa:webupd8team/java]/returns: executed successfully
Notice: /Stage[main]/Apt::Update/Exec[apt_update]: Triggered 'refresh' from 1 events
Notice: /Stage[main]/Java/Package[java-common]/ensure: created
Notice: /Stage[main]/Java/Package[java]/ensure: created
Notice: Applied catalog in 39.58 seconds
To extend the question further, generally things that are blockers for a standard setup to run are usually moved into a run stage (documented here).
So I would probably move all of the various repo setup puppet code into pre run stage with other prerequisites (normally you put in repo setup), the run stage will always be run first before the main stage, so you don't have to worry about explictly setting requirements that repos are setup on each package. This makes making changes to repos and prerequisites a lot easier

Resources