Puppet eYAML to File - puppet

I have a file that I've encrypted using hiera-eyaml and included in my configuration as such:
mymodule::config_file: ENC[PKCS7,...]
Now I'm trying to write the contents of the decoded string into a file as such:
class mymodule (
String $config_file
) {
file { '/etc/agent_config.json':
content => $config_file,
mode => '0644',
}
}
I would expect the output file to be the decrypted string, but instead I'm getting the encrypted eYAML instead.
Is what I'm doing not possible to do in puppet?
Edit:
The file looks correct when running eyaml edit
ex:
itglue::fireeye::fireeye_config: DEC(1)::PKCS7[{
"serverlist": {
"servers": [
...
]
}
}]!
Edit 2:
Our hiera.yaml is as such:
---
:backends:
- yaml
- eyaml
:yaml:
:datadir: /etc/puppet/hiera/hieradata
:eyaml:
:datadir: /etc/puppet/hiera/hieradata
:pkcs7_private_key: /var/lib/puppet/keys/private_key.pkcs7.pem
:pkcs7_public_key: /var/lib/puppet/keys/public_key.pkcs7.pem
:hierarchy:
- secure
- "%{::clientcert}"
- "node_role/%{::node_role}"
- "env/%{::environment}"
- "node_types/%{nodetype}"
- global
:logger: console

I don't have a system with hiera version 3 to test this on so it's a complete stab in the dark, but try swapping the ordering of the backends so
:backends:
- yaml
- eyaml
becomes
:backends:
- eyaml
- yaml
If that doesn't work I'd suggest storing your secrets data in a different location so have
/etc/puppet/hiera/hieradata
for your yaml and
/etc/puppet/hiera/hierasecretdata
for the secrets, that way the yaml search will never be looking in the eyaml file where it can find a key match.

Related

Intake Cache specify filename/location

I'm trying to use intake and the intake-xarray to open and store remote files. I have a minimized catalog file here:
/isibhv/projects/paleo_pool/boundary_conditions/ice_sheet_reconstructions/ice_sheet_reconstructions.yaml
It looks like this:
metadata:
version: 1
sources:
glac1d:
description: The GLAC-1D Reconstruction
driver: netcdf
args:
urlpath: "https://sharebox.lsce.ipsl.fr/index.php/s/yfuUw91ruuJXroC/download?path=%2F&files=TOPicemsk.GLACD26kN9894GE90227A6005GGrBgic.nc"
cache_dir: "{{ CATALOG_DIR }}/glac1d"
cache:
- argkey: urlpath
type: file
I can open the files in Python:
import intake
cat = intake.open_catalog("ice_sheet_reconstructions.yaml")
ds = cat.glac1d.read()
This all works wonderfully; and I get the file as I would expect it. However, the cache doesn't show up where I would expect. I would have guessed a new folder is made under:
/isibhv/projects/paleo_pool/boundary_conditions/ice_sheet_reconstructions/glac1d
Instead, I get something in my home directory.
Did I specify the cache directory incorrectly?
As a second question: is it possible to directly specify how the cached files should be called when they are saved?
Thanks!
Paul
The location of the cache is specified by the config, which is a YAML file typically in ~/.intake/conf.yaml (key "cache_dir"), but can be elsewhere according to the INTAKE_CONF(_FILE) environment variable OR the metadata of the source, key "catalog_dir" (<- this may be incorrect?). The special value "catdir" means "in the directory where the catalog is".
However
With the appearance of caching in fsspec, the following will be possible:
sources:
glac1d:
description: The GLAC-1D Reconstruction
driver: netcdf
args:
urlpath: "filecache://sharebox.lsce.ipsl.fr/index.php/s/yfuUw91ruuJXroC/download?path=%2F&files=TOPicemsk.GLACD26kN9894GE90227A6005GGrBgic.nc"
storage_options:
target_protocol: https
cache_storage: "{{ CATALOG_DIR }}/glac1d"
unfortunately, the required change is not yet in intake-xarray.

eyaml is not decrypted on Puppet agent

We running a Puppet master server on CentOS 7.6 in combination with the repo from puppet (http://yum.puppetlabs.com/puppet6/el/7)
When I write am eyaml where I define a variable, I just see the encrypted value on the agent (Windows 2016).
On the Puppet master, I can edit the eyaml, everthing looks fine. The only difference to a standard installation is, that everything is in a different environment 'myenv'.
hiera.yaml:
---
version: 5
defaults:
datadir: data
data_hash: yaml_data
hierarchy:
### Encrypted eyaml files
- name: "Secret data: per-node, per-datacenter, common"
lookup_key: eyaml_lookup_key
path: "/etc/puppetlabs/code/environments/myenv/data/cmp/test/rdc/%{::trusted.certname}.eyaml"
options:
pkcs7_private_key: /etc/puppetlabs/puppet/eyaml/private_key.pkcs7.pem
pkcs7_public_key: /etc/puppetlabs/puppet/eyaml/public_key.pkcs7.pem
# Environments
- name: "env2"
glob: "env/test/*/%{::trusted.certname}.yaml"
- name: "env1"
glob: "env/test/%{::trusted.certname}.yaml"
# Components
- name: "cmp2"
glob: "cmp/test/*/%{::trusted.certname}.yaml"
- name: "cmp1"
glob: "cmp/test/%{::trusted.certname}.yaml"
# Others
- name: "Other YAML hierarchy levels"
paths:
- "common.yaml"
[ root #pup-mst-srv-10:/etc/puppetlabs/puppet]-$ puppetserver -v
puppetserver version: 6.3.0
[ root #pup-mst-srv-10:/etc/puppetlabs/puppet]-$ hiera -v
3.5.0
[ root #pup-mst-srv-10:/etc/puppetlabs/puppet]-$ eyaml version
[hiera-eyaml-core] hiera-eyaml (core): 3.0.0
[ root #pup-mst-srv-10:/etc/yum.repos.d]-$ puppetserver gem list -e hiera-eyaml
*** LOCAL GEMS ***
hiera-eyaml (3.0.0)
[ root #pup-mst-srv-10:/etc/eyaml]-$ cat config.yaml
pkcs7_private_key: '/etc/puppetlabs/puppet/eyaml/private_key.pkcs7.pem'
pkcs7_public_key: '/etc/puppetlabs/puppet/eyaml/public_key.pkcs7.pem'
[ root #pup-mst-srv-10:/etc/puppetlabs/code/environments/myenv/data/cmp/test/rdc]-$ ls -l /etc/puppetlabs/puppet/eyaml/
total 8
-r--r--r--. 1 puppet puppet 1679 Jul 11 15:39 private_key.pkcs7.pem
-r--r--r--. 1 puppet puppet 1050 Jul 11 15:39 public_key.pkcs7.pem
[ root #pup-mst-srv-10:/etc/puppetlabs/code/environments/myenv/data/cmp/test/rdc]-$ eyaml edit rdc.eyaml
---
classes:
- win_ad_abcd
win_ad_abcd::testpassword : DEC(1)::PKCS7[test12]!
[ root #pup-mst-srv-10:/etc/puppetlabs/code/environments/myenv/data/cmp/test/rdc]-$ cat rdc.eyaml
---
classes:
- win_ad_abcd
win_ad_abcd::testpassword : ENC[PKCS7,MIIBeQYJKoZIhvcNAQcDoIIBajCCAWYCAQAxggEhMIIBHQIBADAFMAACAQEwDQYJKoZIhvcNAQEBBQAEggEAS0E/Y3+QzFhRVZM+F+5kQ8ZQrvGddUno5sDeg3Np9P1/8I5Xetemrx5DTKQaD5C4DS3kgvxjrSqVk/GCCMtZUW5Ynlym1yvylHA7zXmn+g6pYbe5XW88y2Xv1IzdxHwPmgOlFAXJCRoieTrfph+Y4mQBWi2uyrTphHM/o31JcDREfzOeucTSGaHnq8SHeP7t5O7w5ZFG4++hasBLUTubG2ZOAgQRTlksmTK3oOJ0eLRDab4LpgBMaL/VaZgFiu3qmMb3IPtHlaSEAiTRQzdJW7WeHTJUqPSBNni1WmPXA3lFqmp8PFomxsLBTv7i9/gw7SQ2FHwpu5izH6iKwzmEcjA8BgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBBRpizv6doUY5DzpFaBg45lgBCJeK3Yi9qSUCulkHzBDzx6]
[ root #pup-mst-srv-10:/etc/puppetlabs/code/environments/myenv/modules/win_ad_abcd/manifests]-$ cat init.pp
class win_ad_abcd (
$testpassword = "Not Set",
)
{
notify{"eyaml --> ${testpassword} <--":}
file { 'C:\Windows\Temp\test.out':
content => $testpassword,
}
}
I expect an output in the test.out file on the agent with "test12", but I got always the encrypted output.
Would be great to have some hints.
Ivo
I spent a bit of time unsuccessfully trying to reproduce this. On the basis of this information, I also can't think of any reason why this would occur.
On attempting to create a minimal, complete example, I simplified the setup, and you could try using something similar to my setup to see if you can still reproduce the issue. Then, add in additional complexity similar to your original problem until you find the minimal, complete, verifiable example. We might be able to help further if you do this.
Simpler hiera.yaml:
---
version: 5
defaults:
datadir: data
data_hash: yaml_data
hierarchy:
- name: "Secret data: per-node, per-datacenter, common"
lookup_key: eyaml_lookup_key
path: "encrypted.yaml"
options:
pkcs7_private_key: ./keys/private_key.pkcs7.pem
pkcs7_public_key: ./keys/public_key.pkcs7.pem
spec/fixtures/hiera/data/encrypted.yaml:
---
test: >
ENC[PKCS7,MIIBeQYJKoZIhvcNAQcDoIIBajCCAWYCAQAxggEhMIIBHQIBADAFMAACAQEw
DQYJKoZIhvcNAQEBBQAEggEAedQX8FExcat6yk0zsUzNbzQ/07w8ghPOw4eY
ycrfz0H7Cr7KnuBMY0yloFmtWuhYcjXETfaU3U3zGr9IOl4Aiy7yD3ZIvH0Y
HoEWKiJeUzNGrpaH/QFk378cEbpd6LXG46nMzw6w21uhASmvVt3KmZBJwY29
sEk2MpZm32H4JxQQosns4SDMQ6tA5h1xSrgpBTKd1x5vKSTsNnLAahjW31aH
JbK7Se+hHJ4zi9P0/ZjT07OTq1X2rwnfNK8wgKJa/VEDSH+KoNub+4TDHfj/
CWyGQx3Y5U1J2R2/6P5Vp2zRaAf/0BT43Ud/M8H25BIjYosuGtDVCVbbxMNK
mXZITDA8BgkqhkiG9w0BBwEwHQYJYIZIAWUDBAEqBBDEPKno3R1K0XNat4a9
uCuDgBAdp579qk9MbgWLXyXSBD80]
Created keys:
▶ eyaml createkeys
▶ ls keys
private_key.pkcs7.pem public_key.pkcs7.pem
Create a secret:
▶ eyaml encrypt -l 'test' -s 'mySecretString' >> \
spec/fixtures/hiera/data/encrypted.yaml
Which I then cleaned up by further editing that file manually to get the content shown above.
I tested using lookup:
▶ puppet lookup \
--hiera_config=spec/fixtures/hiera/hiera.yaml test
--- mySecretString
Theories:
The information above doesn't truly represent your setup.
You could have actually encrypted the encrypted string?
Things I ruled out:
An error would be seen if there was some problem accessing your eyaml keys. Or if the eyaml library wasn't available. The fact that no error is seen suggests the keys are ok, the eyaml_lookup_key function is found, and eyaml is working basically.
Otherwise, see if you can boil this down to a truly minimal, verifiable, complete example.
Thanks Alex, I create a new VM with puppet and set them up like you described. This helped me to find a stupid mistake.
The eyaml with the encrypted password was rdc.eyaml and I had a symlink to this file like myHostname.yaml
So: myHostname.yaml -> rdc.eyaml
After moving myHostname.yaml to myHostname.eyaml everything works fine.

Not able to look up class parameter in hiera

I have look at other questions like Using hiera to set class parameters? and others which discusses hiera 3. I am using hiera 5.
Here is my hiera.yaml
[root#e64a2e5c7c79 fisherman]# cat /fisherman/fisherman/hiera/hiera.yaml
---
version: 5
defaults: # Used for any hierarchy level that omits these keys.
datadir: data # This path is relative to hiera.yaml's directory.
data_hash: yaml_data # Use the built-in YAML backend.
hierarchy:
- name: "Apps" # Uses custom facts.
path: "apps/%{facts.appname}.yaml"
I also have this hiera data file:
[root#e64a2e5c7c79 fisherman]# cat /fisherman/fisherman/hiera/apps/HelloWorld.yaml
---
fisherman::create_new_component::component_name: 'HelloWord'
But when I run my puppet agent like so ...
export FACTER_appname=HelloWorld
hiera_config=/fisherman/fisherman/hiera/hiera.yaml
modulepath=/fisherman/fisherman/modules
puppet apply --modulepath=$modulepath --hiera_config=$hiera_config -e 'include fisherman'
... I get this error ...
Error: Evaluation Error: Error while evaluating a Function Call, Class[Fisherman::Create_new_component]: expects a value for parameter $component_name (file: /fisherman/fisherman/modules/fish
erman/manifests/init.pp, line: 12, column: 9) on node e64a2e5c7c79
I tried debugging hiera with puppet lookup like so:
[root#e64a2e5c7c79 /]# export FACTER_appname=HelloWorld
[root#e64a2e5c7c79 /]# hiera_config=/fisherman/fisherman/hiera/hiera.yaml
[root#e64a2e5c7c79 /]# modulepath=/fisherman/fisherman/modules
[root#e64a2e5c7c79 /]# puppet lookup --modulepath=$modulepath --hiera_config=$hiera_config --node agent.local --explain fisherman::create_new_component::component_name
Searching for "lookup_options"
Global Data Provider (hiera configuration version 5)
Using configuration "/fisherman/fisherman/hiera/hiera.yaml"
Hierarchy entry "Apps"
Path "/fisherman/fisherman/hiera/data/apps/.yaml"
Original path: "apps/%{facts.appname}.yaml"
Path not found
Environment Data Provider (hiera configuration version 5)
Using configuration "/etc/puppetlabs/code/environments/production/hiera.yaml"
Merge strategy hash
Hierarchy entry "Per-node data (yaml version)"
Path "/etc/puppetlabs/code/environments/production/data/nodes/.yaml"
Original path: "nodes/%{::trusted.certname}.yaml"
Path not found
Hierarchy entry "Other YAML hierarchy levels"
Path "/etc/puppetlabs/code/environments/production/data/common.yaml"
Original path: "common.yaml"
Path not found
Module data provider for module "fisherman" not found
Searching for "fisherman::create_new_component::component_name"
Global Data Provider (hiera configuration version 5)
Using configuration "/fisherman/fisherman/hiera/hiera.yaml"
Hierarchy entry "Apps"
Path "/fisherman/fisherman/hiera/data/apps/.yaml"
Original path: "apps/%{facts.appname}.yaml"
Path not found
Environment Data Provider (hiera configuration version 5)
Using configuration "/etc/puppetlabs/code/environments/production/hiera.yaml"
Hierarchy entry "Per-node data (yaml version)"
Path "/etc/puppetlabs/code/environments/production/data/nodes/.yaml"
Original path: "nodes/%{::trusted.certname}.yaml"
Path not found
Hierarchy entry "Other YAML hierarchy levels"
Path "/etc/puppetlabs/code/environments/production/data/common.yaml"
Original path: "common.yaml"
Path not found
Module data provider for module "fisherman" not found
Function lookup() did not find a value for the name 'fisherman::create_new_component::component_name'
I noticed this in the above output:
Hierarchy entry "Apps"
Path "/fisherman/fisherman/hiera/data/apps/.yaml"
Original path: "apps/%{facts.appname}.yaml"
Path not found
It looks like facts.appname is empty and not HelloWorld as I had expected.
What am I doing wrong here?
Thanks
Based on the information in the question I can't reproduce this. Here is my setup if it helps:
# init.pp
class test (
String $component_name,
) {
notify { $facts['appname']:
message => "Component name: $component_name for fact appname of ${facts['appname']}"
}
}
# hiera.yaml
---
version: 5
defaults:
datadir: data
data_hash: yaml_data
hierarchy:
- name: "Apps" # Uses custom facts.
path: "apps/%{facts.appname}.yaml"
# data/apps/HelloWorld.yaml
---
test::component_name: 'MyComponentName'
# spec/classes/test_spec.rb
require 'spec_helper'
describe 'test' do
let(:hiera_config) { 'spec/fixtures/hiera/hiera.yaml' }
let(:facts) {{ 'appname' => 'HelloWorld' }}
it {
is_expected.to contain_notify("HelloWorld")
.with({
'message' => "Component name: MyComponentName for fact appname of HelloWorld"
})
}
end
Tested on Puppet version:
▶ bundle exec puppet -V
6.6.0
Output:
▶ bundle exec rake spec
I, [2019-07-07T16:42:51.219559 #22140] INFO -- : Creating symlink from spec/fixtures/modules/test to /Users/alexharvey/git/home/puppet-test
/Users/alexharvey/.rvm/rubies/ruby-2.4.1/bin/ruby -I/Users/alexharvey/.rvm/gems/ruby-2.4.1/gems/rspec-core-3.8.2/lib:/Users/alexharvey/.rvm/gems/ruby-2.4.1/gems/rspec-support-3.8.2/lib /Users/alexharvey/.rvm/gems/ruby-2.4.1/gems/rspec-core-3.8.2/exe/rspec --pattern spec/\{aliases,classes,defines,functions,hosts,integration,plans,tasks,type_aliases,types,unit\}/\*\*/\*_spec.rb
test
should contain Notify[HelloWorld] with message => "Component name: MyComponentName for fact appname of HelloWorld"
Finished in 0.1444 seconds (files took 0.9699 seconds to load)
1 example, 0 failures
You also can query the Hiera hierarchy directly using puppet lookup like this:
▶ FACTER_appname=HelloWorld bundle exec puppet lookup \
--hiera_config=spec/fixtures/hiera/hiera.yaml test::component_name
--- MyComponentName

puppet hiera fails to read yaml file

I am pretty new to puppet. I configured a hiera file, whose path is /etc/puppetlabs/puppet/hiera.yaml, as so
version: 5
hierarchy: []
backends:
- yaml
yaml:
- datadir: /etc/puppetlabs/puppet/some_dir
and I get this error
Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://docs.puppet.com/puppet/5.3/reference/deprecated_language.html
(file & line not available)
Error: Evaluation Error: Error while evaluating a Function Call, Lookup of key 'user_dir' failed: The Lookup Configuration at '/etc/puppetlabs/puppet/hiera.yaml' has wrong type, unrecognized key 'backends'
The Lookup Configuration at '/etc/puppetlabs/puppet/hiera.yaml' has wrong type, unrecognized key 'yaml' at /etc/puppetlabs/code/environments/production/manifests/site.pp:30:17 on node puppet,some_cluster_DNS.internal
Initially, I had those kinds of format for the keys :backends: :yaml: but it seemed it is not regular one for the 5 version, so that I deleted the : sign
Someone has an idea ?
First, in terms of that warning, you should definitely switch over to the Puppet lookup function from the Hiera hiera functions if you are using Hiera >= 4: https://puppet.com/docs/puppet/4.10/hiera_use_function.html
Second, in terms of that error, I would consult the documentation on how to setup a Hiera 5 config file: https://puppet.com/docs/puppet/4.10/hiera_config_yaml_5.html
Using the proper format, your config file would look like:
# /etc/puppetlabs/puppet/hiera.yaml
version: 5
defaults:
- data_hash: yaml_data
- datadir: /etc/puppetlabs/puppet/some_dir
hierarchy: []
What you are trying to do on the last line (specify a specific datadir for the yaml_data backend) is not allowed in Hiera 5. If you want to specify a datadir for a specific backend, then you need to specify a level of the hierarchy for just that backend (or just that backend's datadir; you can customize in several depths of matrices with it) and specify a datadir there. For example:
hierarchy:
- name: yaml data
data_hash: yaml_data
datadir: /etc/puppetlabs/puppet/some_dir
paths:
- "%{trusted.certname}.yaml"
- common.yaml

Why does hiera look in the "services" subdirectory?

I have been experimenting with hiera for configuration data. The hiera.yaml file has the following configuration:
---
:backends:
- yaml
:hierarchy:
- "servers/%{hostname}"
- common
but when I run hiera from the command line to test it,
hiera some::var hostname=foo
it does not pick up the configuration from servers/foo/yaml. Adding the -d (debug) option shows that it doesn't even look at that file. Instead, it says something like:
Looking for data source services/foo
Cannot find database /etc/puppet/hiera/services/foo.yaml, skipping
So, two questions:
Why is it not looking in the servers directory?
Why is it looking in a "services" directory? (Note that it looks for things in the services directory even if we remove the "servers/%{::hostname}" line from the hiera.yaml file!)
If by default hiera looks for things in special directories like "services", then where is this documented?
I think the reason may be that a) you are not using the default location for hiera datadir (which is /var/lib/hiera) and b) you have forgotten to specify that datadir within the hiera.yaml (or whatever filename you want your hiera config file to be). That is what a possible solution might be is just adding the following to your hiera config file:
:yaml:
:datadir: /tmp/var-lib-hiera
Check out the following example:
/tmp/var-lib-hiera $ hiera -c hiera.yaml some::var hostname=foo
100
/tmp/var-lib-hiera $ tree
.
├── hiera.yaml
└── servers
└── foo.yaml
/tmp/var-lib-hiera $ head hiera.yaml servers/foo.yaml
==> hiera.yaml <==
---
:backends:
- yaml
:hierarchy:
- "servers/%{hostname}"
- common
:yaml:
:datadir: /tmp/var-lib-hiera
==> servers/foo.yaml <==
some::var: 100
Apparently, the configuration will work if: hiera is run with the -c option to specify the exact location of the configuration file. (A symlink from /etc/hiera.yaml did not work, though that was a suggestion from another forum https://ask.puppetlabs.com/question/3149/where-does-hiera-search-for-data-sources/?answer=3152#post-id-3152)
If you are using puppet you can set the location of the hiera file in e master section of puppet.conf so you don't have to provide it on cmd line.
See: http://docs.puppetlabs.com/references/latest/configuration.html#hieraconfig

Resources