Puppet: Error 400 Could not find any files - puppet

I double checked all settings but did not find the issue and that's why I try to get help here.
Let me show the configuration:
puppet.conf:
[...]
[master]
environmentpath = $confdir/environments/
hiera_config = $confdir/environments/production/sites/xy/config/hiera.yaml
default_manifest = ./sites/xy/config/
environment_timeout = 0
fileserver.conf:
[...]
[sites]
path /etc/puppet/environments/production/sites/
allow *
auth.conf:
[...]
# extra mountpoint
path /sites
allow *
[...]
Now whenever I run Puppet and it tries to implement a specific file I get this:
Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find any files from puppet:///sites/xy/files/xy/xy.key.pub at /etc/puppet/environments/production/modules/xy/manifests/xy.pp:88 on node xy
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
Note that I had to replace sensitive information by xy but for debugging purposes I try to give every detail where possible.
So /sites points to /etc/puppet/environments/production/sites/ according to fileserver.conf and the directory exists like this (with correct permissions imho):
/etc/puppet % ls -ld /etc/puppet/environments/production/sites/
drwxr-xr-x 8 root puppet 4096 Oct 7 12:46 /etc/puppet/environments/production/sites/
The mentioned file puppet:///sites/xy/files/xy/xy.key.pub should therefore be located in /etc/puppet/environments/production/sites/xy/files/xy/xy.key.pub which looks like this:
/etc/puppet % ls -l /etc/puppet/environments/production/sites/*/files/*/*.key.pub
-rw-r--r-- 1 root puppet 725 Oct 7 12:46 /etc/puppet/environments/production/sites/xy/files/xy/xy.key.pub
And the in the error message mentioned line 88 of the module which loads the file it looks like this:
$sshpubkey = file("puppet:///${sitefiles}/xy/${s0_pubkey}")
where $s0_pubkey is xy.key.pub and ${sitefiles} is sites/$site/files which leads to the evaluated path of the requested file like this: puppet:///sites/xy/files/xy/xy.key.pub

The function file() in Puppet can not handle Puppet mountpoints with puppet://. The official docs at https://docs.puppet.com/puppet/latest/reference/function.html#file don't mention it specifically but obviously without mentioning it it means it can't handle extra mountpoints and wants to load files from a modules files directory.
My solution: I will use a variable which declares the absolute path to my "sites files" like this: $sitefiles_absolute = "/etc/puppet/environments/${environment}/sites/xy/files/" which will never change or at least not very often. And with keeping it in the site.pp file it can be used on every node and module.

Related

rsyslog , collect log from files outside /var/log

I have different logs that are written to our moutend nfs share that i need to send to our syslog-server (graylog) they are located outside /var/log folder.
So i add some extra conf in /etc/rsyslog.d/
For this example i have two files with following config:
atlassian-application-confluence-log.conf
module(load="imfile")
module(load="imklog")
$MaxMessageSize 50k
global(workDirectory="/atlassian/test/confluence/logs")
# This is the main application log file
input(type="imfile"
File="/atlassian/test/confluence/logs/atlassian-confluence.log"
Tag="atlassian"
PersistStateInterval="200"
)
# This file contains entries related to the search index.
input(type="imfile"
File="/atlassian/test/confluence/logs/atlassian-confluence-index.log"
Tag="atlassian"
PersistStateInterval="200"
)
# Send to Graylog
action(type="omfwd" target="log-server-company.com" port="5140")
# if you want to keep a local copy of the logs.
action(type="omfile" File="/var/log/rsyslog.log" template="RSYSLOG_TraditionalFileFormat")
atlassian-application-jira-log.conf
module(load="imfile")
module(load="imklog")
$MaxMessageSize 50k
global(workDirectory="/atlassian/test/jira/log")
# Contains logging for most of Jira, including logs that aren’t specifically written elsewhere
input(type="imfile"
File="/atlassian/test/jira/log/atlassian-jira.log"
Tag="atlassian"
PersistStateInterval="200"
)
# Send to Graylog
action(type="omfwd" target="log-server-company.com" port="5140")
# if you want to keep a local copy of the logs.
action(type="omfile" File="/var/log/rsyslog.log" template="RSYSLOG_TraditionalFileFormat")
So to my problem.
When i check the Rsyslogd configuration with following command:
rsyslogd -N1 -f /etc/rsyslog.d/atlassian-application-confluence-log.conf
It says it is valid.
When i restart the rsyslog service i get the following errors:
module 'imfile' already in this config, cannot be added [v8.2102.0-10.el8 try https://www.rsyslog.com/e/2221 ]
module 'imklog' already in this config, cannot be added [v8.2102.0-10.el8 try https://www.rsyslog.com/e/2221 ]
error during parsing file /etc/rsyslog.d/atlassian-tomcat-confluence-log.conf, on or before line 6: parameter 'workdirectory' specified more than once - one instance is ignored. Fix config [v8.2102.0-10.el8 try https://www.rsyslog.com/e/2207]>
error during parsing file /etc/rsyslog.d/atlassian-tomcat-confluence-log.conf, on or before line 6: parameter 'workDirectory' not known -- typo in config file? [v8.2102.0-10.el8 try https://www.rsyslog.com/e/2207]
module 'imfile' already in this config, cannot be added [v8.2102.0-10.el8 try https://www.rsyslog.com/e/2221 ]
module 'imklog' already in this config, cannot be added [v8.2102.0-10.el8 try https://www.rsyslog.com/e/2221 ]
[origin software="rsyslogd" swVersion="8.2102.0-10.el8" x-pid="379288" x-info="https://www.rsyslog.com"] start
imjournal: journal files changed, reloading... [v8.2102.0-10.el8 try https://www.rsyslog.com/e/0 ]
How can i get rid of the warnings?
I have already tried to put the two modules in /etc/rsyslog.conf
I get following errors from that config:
parameter 'PersistStateInterval' not known
parameter 'Tag' not known
parameter 'File' not known
If there are multiple configuration files, they are processed in ascending sort order of the file name (numerically/alphabetically), See: $IncludeConfig.
Therefore you don't have to include any configuration parameters (modules, work directories, rulesets etc.) multiple times. You can include them once in the config which is loaded first.

Referencing a YAML config file, from Python, when a softlink is defined

I have the following code;
#!/usr/bin/env python3
import yaml
with open('config.yml', 'r') as config_file:
config = yaml.load(config_file)
The file is called __init__.py which is in the directory ~/bin/myprogram/myprogram/ and in the same directory, I have a file called config.yml
My symlink is as follows;
user$ ls -la /usr/local/bin/
lrwxr-xr-x 1 user admin 55 27 Nov 13:25 myprogram -> /Users/user/bin/myprogram/myprogram/__init__.py
Every time I run myprogram, I get the error FileNotFoundError: [Errno 2] No such file or directory: 'config.yml'. I believe this is because the config.yml is not in /usr/local/bin/. What is the best way to work around this issue?
You can use __file__ to access the location of the __init__.py file when executing code in that file. It returns the full path, but care has to be taken as it may be the .pyc (or .pyo) version. Since you are using Python3 I would use the pathlib module:
import yaml
from pathlib import Path
my_path = Path(__file__).resolve() # resolve to get rid of any symlinks
config_path = my_path.parent / 'config.yaml'
with config_path.open() as config_file:
config = yaml.safe_load(config_file)
Please note:
If you have to use PyYAML, use safe_load(), even PyYAML's own documentation indicates .load() can be unsafe. It almost never necessary to use that. And in the unlikely event that safe_load() cannot load your config, e.g. if it has !!python/... tags, you should explicitly add register the classes that you actually need to the SafeLoader).
Since September 2006 the recommended extension for YAML files has been .yaml

MoveFileEx with MOVEFILE_REPLACE_EXISTING flag Fails with LastError = 183

We have an application that makes a backup of a configuration file by copying it to a sub-folder (see log file entry below) using the following command:-
BOOL b = MoveFileEx(ExistingFileSpec, NewFileSpec, MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH | MOVEFILE_COPY_ALLOWED);
This appears to work as expected on most systems but on two we get the following warning in our log file:-
11 Jun 2018 10:58:05.437 W. COMMON WARNING MoveFile failed on C:\Program Files\Configuration\Stores.xml to C:\Program Files\Configuration\Backup\Stores.xml LastError=183
Lasterror =183 means "ERROR_ALREADY_EXISTS"
This is odd since MOVEFILE_REPLACE_EXISTING is specifically meant to enable this command to work when the target file exists.
This is happening on Win7 x86 machines.
In a previous version of the code MoveFile was used but with a DeleteFile call immediately prior to ensure the target file did not exist. When the code was rolled back to this version it works.
Any suggestions?

java will intermittently not resolve symlinks on Linux

I'm trying to resolve canonical paths for all the files in a folder tree, but for some reason it will not resolve them (and intermittently the JVM security code will resolve the symlink properly within the FilePermission and cause a security error).
Env:
$ java -version
java version "1.6.0_23"
OpenJDK Runtime Environment (IcedTea6 1.11pre) (6b23~pre11-0ubuntu1.11.10.2)
OpenJDK 64-Bit Server VM (build 20.0-b11, mixed mode)
A known symlink in the system is /usr/share/java/gnome-java-bridge.jar:
$ ls -l /usr/lib/jvm/java-6-openjdk/jre/lib/ext/gnome-java-bridge.jar
lrwxrwxrwx 1 root root 50 2012-02-24 13:39 /usr/lib/jvm/java-6-openjdk/jre/lib/ext/gnome-java-bridge.jar -> ../../../../../../share/java/gnome-java-bridge.jar
The following code should resolve this known symlink:
String symlinkedFilePath =
"/usr/lib/jvm/java-6-openjdk/jre/lib/ext/gnome-java-bridge.jar";
File symlinkedFile = new File(symlinkedFilePath);
System.out.println(symlinkedFile.getAbsolutePath());
System.out.println(symlinkedFile.getCanonicalPath());
but produces:
/usr/lib/jvm/java-6-openjdk/jre/lib/ext/gnome-java-bridge.jar
/usr/lib/jvm/java-6-openjdk/jre/lib/ext/gnome-java-bridge.jar
A further test, using the following code, will sometimes return true for the permission check, but sometimes will return false:
String symlinkedFilePath =
"/usr/lib/jvm/java-6-openjdk/jre/lib/ext/gnome-java-bridge.jar";
File symlinkedFile = new File(symlinkedFilePath);
FilePermission recursivePermission = new FilePermission(
symlinkedFile.getParentFile().getParent() + "/-", "read");
FilePermission filePermission = new FilePermission(
symlinkedFile.getAbsolutePath(), "read");
System.out.println(recursivePermission);
System.out.println(filePermission);
System.out.println(
"Can read symlink: " + recursivePermission.implies(filePermission));
The typical result is:
(java.io.FilePermission /usr/lib/jvm/java-6-openjdk/jre/lib/- read)
(java.io.FilePermission /usr/lib/jvm/java-6-openjdk/jre/lib/ext/gnome-java-bridge.jar read)
Can read symlink: true
but when debugging, if I step through the creation of the FilePermission on the target file, internally the path is resolved to the symlink, and the output results in:
(java.io.FilePermission /usr/lib/jvm/java-6-openjdk/jre/lib/- read)
(java.io.FilePermission /usr/lib/jvm/java-6-openjdk/jre/lib/ext/gnome-java-bridge.jar read)
Can read symlink: false
The problem is that within the context of the app in which the permission checking actually takes place, the symlink is always resolved by the FilePermission object, but never by my own calls to file.getCanonicalPath() as demonstrated above.
Does this make sense to anyone?
A colleague of mine confirmed the issue on OpenJDK 6u23, but not on any prior or following versions. That being said, since the issue has
A) a work around in the form of the system property
-Dsun.io.useCanonCaches=false
OR
-Dsun.io.useCanonPrefixCache=false
B) appears to be resolved in the later build (u24)
there appears to be little motivation to dig any deeper.
In Unix a symlink is a "special" file, with its own permissions.
The fact that you have read permission on the symbolic link doesn't imply you'll have it for the file linked.
My guess here is that you are running your program as an user that can read the symlink but not the actual file.
When entering debug mode you trigger the call to some method that change the internal state of the FilePermission object making it resolve to the actual file and thus returning you "false".
When you get "true" it's just telling you that you can read the symbolic link.
In your place, I'll check permission on this file:
- /usr/share/java/gnome-java-bridge.jar
and to the two directories:
- /usr/share
- /usr/share/java

Sourcing Puppet files from outside of modules

I'm installing a package from a module (Nginx in this specific case) and would like to include a configuration file from outside of the module, i.e. from a top level files directory parallel to the top level manifests directory. I don't see any way to source the file though without including it in a module or in my current Vagrant environment referring to the absolute local path.
Does Puppet allow for sourcing files from outside of modules as described in the documentation?
if I understand your question correctly, you can.
In your module a simple code like this
file { '/path/to/file':
ensure => present,
source => [
"puppet:///files/${fqdn}/path/to/file",
"puppet:///files/${hostgroup}/path/to/file",
"puppet:///files/${domain}/path/to/file",
"puppet:///files/global/path/to/file",
],
}
will do the job. The /path/to/file will be sourced using a file located in the "files" Puppet share.
(in the example above, it search in 4 different locations).
update maybe you're talking about a directory to store files which is not shared by Puppet fileserver (look at http://docs.puppetlabs.com/guides/file_serving.html), and in this case you can't i think, Vagrant or not, but you can add it to your Puppet fileserver to do it. I thinks it's the best (and maybe only) way to do it.
If you have a number of Vagrant VMs you can simply store files within your Vagrant project directory (containing your VagrantFile).
This directory is usually available to all VMs as /vagrant within the VM on creation.
If you want other directories on your computer to be available to your VMs just add the following to your VagrantFile
# see http://docs.vagrantup.com/v1/docs/config/vm/share_folder.html
config.vm.share_folder "v-packages", "/vagrant_packages", "../../dpkg"
Then to use the files within puppet you can simply treat them as local files to the VM
# bad example, bub basically use 'source => 'file:///vagrant/foo/bar'
file { '/opt/cassandra':
ensure => directory,
replace => true,
purge => true,
recurse => true,
source => 'file:///vagrant/conf/dist/apache-cassandra-1.2.0',
}
This is probably only wise to do if you only using local puppet manifests/modules.
Probably too late to help bennylope, but for others who happen across this question, as I did before figuring it out for myself ...
Include stuff like this in your Vagrantfile ...
GUEST_PROVISIONER_CONFDIR = "/example/destination/path"
HOST_PROVISIONER_CONFDIR = "/example/source/path"
config.vm.synced_folder HOST_PROVISIONER_CONFIDIR, GUEST_PROVISIONER_CONFDIR
puppet.options = "--fileserverconfig='#{GUEST_PROVISIONER_CONFDIR}/fileserver.conf'"
Then make sure /example/source/path contains the referenced fileserver.conf file. It should look something like ...
[foo]
path /example/destination/path
allow *
Now, assuming example-file.txt exists in /example/source/path, the following will work in your manifests:
source => "puppet:///foo/example-file.txt",
See:
Puppet configuration reference entry for fileserverconfig
Serving Files From Custom Mount Points

Resources