I am pretty new to puppet. I configured a hiera file, whose path is /etc/puppetlabs/puppet/hiera.yaml, as so
version: 5
hierarchy: []
backends:
- yaml
yaml:
- datadir: /etc/puppetlabs/puppet/some_dir
and I get this error
Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://docs.puppet.com/puppet/5.3/reference/deprecated_language.html
(file & line not available)
Error: Evaluation Error: Error while evaluating a Function Call, Lookup of key 'user_dir' failed: The Lookup Configuration at '/etc/puppetlabs/puppet/hiera.yaml' has wrong type, unrecognized key 'backends'
The Lookup Configuration at '/etc/puppetlabs/puppet/hiera.yaml' has wrong type, unrecognized key 'yaml' at /etc/puppetlabs/code/environments/production/manifests/site.pp:30:17 on node puppet,some_cluster_DNS.internal
Initially, I had those kinds of format for the keys :backends: :yaml: but it seemed it is not regular one for the 5 version, so that I deleted the : sign
Someone has an idea ?
First, in terms of that warning, you should definitely switch over to the Puppet lookup function from the Hiera hiera functions if you are using Hiera >= 4: https://puppet.com/docs/puppet/4.10/hiera_use_function.html
Second, in terms of that error, I would consult the documentation on how to setup a Hiera 5 config file: https://puppet.com/docs/puppet/4.10/hiera_config_yaml_5.html
Using the proper format, your config file would look like:
# /etc/puppetlabs/puppet/hiera.yaml
version: 5
defaults:
- data_hash: yaml_data
- datadir: /etc/puppetlabs/puppet/some_dir
hierarchy: []
What you are trying to do on the last line (specify a specific datadir for the yaml_data backend) is not allowed in Hiera 5. If you want to specify a datadir for a specific backend, then you need to specify a level of the hierarchy for just that backend (or just that backend's datadir; you can customize in several depths of matrices with it) and specify a datadir there. For example:
hierarchy:
- name: yaml data
data_hash: yaml_data
datadir: /etc/puppetlabs/puppet/some_dir
paths:
- "%{trusted.certname}.yaml"
- common.yaml
Related
I have a file that I've encrypted using hiera-eyaml and included in my configuration as such:
mymodule::config_file: ENC[PKCS7,...]
Now I'm trying to write the contents of the decoded string into a file as such:
class mymodule (
String $config_file
) {
file { '/etc/agent_config.json':
content => $config_file,
mode => '0644',
}
}
I would expect the output file to be the decrypted string, but instead I'm getting the encrypted eYAML instead.
Is what I'm doing not possible to do in puppet?
Edit:
The file looks correct when running eyaml edit
ex:
itglue::fireeye::fireeye_config: DEC(1)::PKCS7[{
"serverlist": {
"servers": [
...
]
}
}]!
Edit 2:
Our hiera.yaml is as such:
---
:backends:
- yaml
- eyaml
:yaml:
:datadir: /etc/puppet/hiera/hieradata
:eyaml:
:datadir: /etc/puppet/hiera/hieradata
:pkcs7_private_key: /var/lib/puppet/keys/private_key.pkcs7.pem
:pkcs7_public_key: /var/lib/puppet/keys/public_key.pkcs7.pem
:hierarchy:
- secure
- "%{::clientcert}"
- "node_role/%{::node_role}"
- "env/%{::environment}"
- "node_types/%{nodetype}"
- global
:logger: console
I don't have a system with hiera version 3 to test this on so it's a complete stab in the dark, but try swapping the ordering of the backends so
:backends:
- yaml
- eyaml
becomes
:backends:
- eyaml
- yaml
If that doesn't work I'd suggest storing your secrets data in a different location so have
/etc/puppet/hiera/hieradata
for your yaml and
/etc/puppet/hiera/hierasecretdata
for the secrets, that way the yaml search will never be looking in the eyaml file where it can find a key match.
I'm trying to use intake and the intake-xarray to open and store remote files. I have a minimized catalog file here:
/isibhv/projects/paleo_pool/boundary_conditions/ice_sheet_reconstructions/ice_sheet_reconstructions.yaml
It looks like this:
metadata:
version: 1
sources:
glac1d:
description: The GLAC-1D Reconstruction
driver: netcdf
args:
urlpath: "https://sharebox.lsce.ipsl.fr/index.php/s/yfuUw91ruuJXroC/download?path=%2F&files=TOPicemsk.GLACD26kN9894GE90227A6005GGrBgic.nc"
cache_dir: "{{ CATALOG_DIR }}/glac1d"
cache:
- argkey: urlpath
type: file
I can open the files in Python:
import intake
cat = intake.open_catalog("ice_sheet_reconstructions.yaml")
ds = cat.glac1d.read()
This all works wonderfully; and I get the file as I would expect it. However, the cache doesn't show up where I would expect. I would have guessed a new folder is made under:
/isibhv/projects/paleo_pool/boundary_conditions/ice_sheet_reconstructions/glac1d
Instead, I get something in my home directory.
Did I specify the cache directory incorrectly?
As a second question: is it possible to directly specify how the cached files should be called when they are saved?
Thanks!
Paul
The location of the cache is specified by the config, which is a YAML file typically in ~/.intake/conf.yaml (key "cache_dir"), but can be elsewhere according to the INTAKE_CONF(_FILE) environment variable OR the metadata of the source, key "catalog_dir" (<- this may be incorrect?). The special value "catdir" means "in the directory where the catalog is".
However
With the appearance of caching in fsspec, the following will be possible:
sources:
glac1d:
description: The GLAC-1D Reconstruction
driver: netcdf
args:
urlpath: "filecache://sharebox.lsce.ipsl.fr/index.php/s/yfuUw91ruuJXroC/download?path=%2F&files=TOPicemsk.GLACD26kN9894GE90227A6005GGrBgic.nc"
storage_options:
target_protocol: https
cache_storage: "{{ CATALOG_DIR }}/glac1d"
unfortunately, the required change is not yet in intake-xarray.
I have look at other questions like Using hiera to set class parameters? and others which discusses hiera 3. I am using hiera 5.
Here is my hiera.yaml
[root#e64a2e5c7c79 fisherman]# cat /fisherman/fisherman/hiera/hiera.yaml
---
version: 5
defaults: # Used for any hierarchy level that omits these keys.
datadir: data # This path is relative to hiera.yaml's directory.
data_hash: yaml_data # Use the built-in YAML backend.
hierarchy:
- name: "Apps" # Uses custom facts.
path: "apps/%{facts.appname}.yaml"
I also have this hiera data file:
[root#e64a2e5c7c79 fisherman]# cat /fisherman/fisherman/hiera/apps/HelloWorld.yaml
---
fisherman::create_new_component::component_name: 'HelloWord'
But when I run my puppet agent like so ...
export FACTER_appname=HelloWorld
hiera_config=/fisherman/fisherman/hiera/hiera.yaml
modulepath=/fisherman/fisherman/modules
puppet apply --modulepath=$modulepath --hiera_config=$hiera_config -e 'include fisherman'
... I get this error ...
Error: Evaluation Error: Error while evaluating a Function Call, Class[Fisherman::Create_new_component]: expects a value for parameter $component_name (file: /fisherman/fisherman/modules/fish
erman/manifests/init.pp, line: 12, column: 9) on node e64a2e5c7c79
I tried debugging hiera with puppet lookup like so:
[root#e64a2e5c7c79 /]# export FACTER_appname=HelloWorld
[root#e64a2e5c7c79 /]# hiera_config=/fisherman/fisherman/hiera/hiera.yaml
[root#e64a2e5c7c79 /]# modulepath=/fisherman/fisherman/modules
[root#e64a2e5c7c79 /]# puppet lookup --modulepath=$modulepath --hiera_config=$hiera_config --node agent.local --explain fisherman::create_new_component::component_name
Searching for "lookup_options"
Global Data Provider (hiera configuration version 5)
Using configuration "/fisherman/fisherman/hiera/hiera.yaml"
Hierarchy entry "Apps"
Path "/fisherman/fisherman/hiera/data/apps/.yaml"
Original path: "apps/%{facts.appname}.yaml"
Path not found
Environment Data Provider (hiera configuration version 5)
Using configuration "/etc/puppetlabs/code/environments/production/hiera.yaml"
Merge strategy hash
Hierarchy entry "Per-node data (yaml version)"
Path "/etc/puppetlabs/code/environments/production/data/nodes/.yaml"
Original path: "nodes/%{::trusted.certname}.yaml"
Path not found
Hierarchy entry "Other YAML hierarchy levels"
Path "/etc/puppetlabs/code/environments/production/data/common.yaml"
Original path: "common.yaml"
Path not found
Module data provider for module "fisherman" not found
Searching for "fisherman::create_new_component::component_name"
Global Data Provider (hiera configuration version 5)
Using configuration "/fisherman/fisherman/hiera/hiera.yaml"
Hierarchy entry "Apps"
Path "/fisherman/fisherman/hiera/data/apps/.yaml"
Original path: "apps/%{facts.appname}.yaml"
Path not found
Environment Data Provider (hiera configuration version 5)
Using configuration "/etc/puppetlabs/code/environments/production/hiera.yaml"
Hierarchy entry "Per-node data (yaml version)"
Path "/etc/puppetlabs/code/environments/production/data/nodes/.yaml"
Original path: "nodes/%{::trusted.certname}.yaml"
Path not found
Hierarchy entry "Other YAML hierarchy levels"
Path "/etc/puppetlabs/code/environments/production/data/common.yaml"
Original path: "common.yaml"
Path not found
Module data provider for module "fisherman" not found
Function lookup() did not find a value for the name 'fisherman::create_new_component::component_name'
I noticed this in the above output:
Hierarchy entry "Apps"
Path "/fisherman/fisherman/hiera/data/apps/.yaml"
Original path: "apps/%{facts.appname}.yaml"
Path not found
It looks like facts.appname is empty and not HelloWorld as I had expected.
What am I doing wrong here?
Thanks
Based on the information in the question I can't reproduce this. Here is my setup if it helps:
# init.pp
class test (
String $component_name,
) {
notify { $facts['appname']:
message => "Component name: $component_name for fact appname of ${facts['appname']}"
}
}
# hiera.yaml
---
version: 5
defaults:
datadir: data
data_hash: yaml_data
hierarchy:
- name: "Apps" # Uses custom facts.
path: "apps/%{facts.appname}.yaml"
# data/apps/HelloWorld.yaml
---
test::component_name: 'MyComponentName'
# spec/classes/test_spec.rb
require 'spec_helper'
describe 'test' do
let(:hiera_config) { 'spec/fixtures/hiera/hiera.yaml' }
let(:facts) {{ 'appname' => 'HelloWorld' }}
it {
is_expected.to contain_notify("HelloWorld")
.with({
'message' => "Component name: MyComponentName for fact appname of HelloWorld"
})
}
end
Tested on Puppet version:
▶ bundle exec puppet -V
6.6.0
Output:
▶ bundle exec rake spec
I, [2019-07-07T16:42:51.219559 #22140] INFO -- : Creating symlink from spec/fixtures/modules/test to /Users/alexharvey/git/home/puppet-test
/Users/alexharvey/.rvm/rubies/ruby-2.4.1/bin/ruby -I/Users/alexharvey/.rvm/gems/ruby-2.4.1/gems/rspec-core-3.8.2/lib:/Users/alexharvey/.rvm/gems/ruby-2.4.1/gems/rspec-support-3.8.2/lib /Users/alexharvey/.rvm/gems/ruby-2.4.1/gems/rspec-core-3.8.2/exe/rspec --pattern spec/\{aliases,classes,defines,functions,hosts,integration,plans,tasks,type_aliases,types,unit\}/\*\*/\*_spec.rb
test
should contain Notify[HelloWorld] with message => "Component name: MyComponentName for fact appname of HelloWorld"
Finished in 0.1444 seconds (files took 0.9699 seconds to load)
1 example, 0 failures
You also can query the Hiera hierarchy directly using puppet lookup like this:
▶ FACTER_appname=HelloWorld bundle exec puppet lookup \
--hiera_config=spec/fixtures/hiera/hiera.yaml test::component_name
--- MyComponentName
I'm trying (and struggling) to get a (multiple) role model implemented in Hiera.
I've worked in the last 2 years with exact the same model as a user and now want to rebuild the same structure on my own. For example, my node.yaml should contain only the roles I want to apply onto the host:
/etc/puppetlabs/code/environments/production/nodes/my.host.de.yaml
classes:
- ydixken_baseinstall
- additional_modules
[...]
For me it's way more intuitive, to place a yaml in the roles/ directory, with the name of the role, and avoid dealing with profiles:
/etc/puppetlabs/code/environments/production/roles/ydixken_baseinstall.yaml
classes:
- apt
- unattended_upgrades
- [...]
apt::update:
frequency: 'daily'
loglevel: 'debug'
[...]
Placing the role definitions as a node fact is not practicable for me. It's also nice-to-have to allow a customization of the already defined values inside of the node configuration, if needed.
Right now my directory, hiera.yaml & file-structure looks like this:
/etc/puppetlabs/puppet/hiera.yaml
version: 5
defaults:
datadir: /etc/puppetlabs/code/environments/production
data_hash: yaml_data
hierarchy:
- name: "Per-node data (yaml version)"
paths:
- "nodes/%{fqdn}.yaml"
- "roles/%{role}.yaml"
- common
/etc/puppetlabs/code/environments/production/hiera.yaml
version: 5
defaults:
hierarchy:
- name: "FQDN"
path: "nodes/%{fqdn}.yaml"
- name: "Roles"
path: "roles/%{role}.yaml"
- name: "Common Data"
path: "common.yaml"
/etc/puppetlabs/code/environments/production/manifests/site.pp
hiera_include('classes')
How can i achieve this?
My current error:
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Function Call, Could not find class ::ydixken_baseinstall for my.host.de (file: /etc/puppetlabs/code/environments/production/manifests/site.pp, line: 1, column: 1) on node my.host.de
I've found exactly, what i was looking for: r10k
I have been experimenting with hiera for configuration data. The hiera.yaml file has the following configuration:
---
:backends:
- yaml
:hierarchy:
- "servers/%{hostname}"
- common
but when I run hiera from the command line to test it,
hiera some::var hostname=foo
it does not pick up the configuration from servers/foo/yaml. Adding the -d (debug) option shows that it doesn't even look at that file. Instead, it says something like:
Looking for data source services/foo
Cannot find database /etc/puppet/hiera/services/foo.yaml, skipping
So, two questions:
Why is it not looking in the servers directory?
Why is it looking in a "services" directory? (Note that it looks for things in the services directory even if we remove the "servers/%{::hostname}" line from the hiera.yaml file!)
If by default hiera looks for things in special directories like "services", then where is this documented?
I think the reason may be that a) you are not using the default location for hiera datadir (which is /var/lib/hiera) and b) you have forgotten to specify that datadir within the hiera.yaml (or whatever filename you want your hiera config file to be). That is what a possible solution might be is just adding the following to your hiera config file:
:yaml:
:datadir: /tmp/var-lib-hiera
Check out the following example:
/tmp/var-lib-hiera $ hiera -c hiera.yaml some::var hostname=foo
100
/tmp/var-lib-hiera $ tree
.
├── hiera.yaml
└── servers
└── foo.yaml
/tmp/var-lib-hiera $ head hiera.yaml servers/foo.yaml
==> hiera.yaml <==
---
:backends:
- yaml
:hierarchy:
- "servers/%{hostname}"
- common
:yaml:
:datadir: /tmp/var-lib-hiera
==> servers/foo.yaml <==
some::var: 100
Apparently, the configuration will work if: hiera is run with the -c option to specify the exact location of the configuration file. (A symlink from /etc/hiera.yaml did not work, though that was a suggestion from another forum https://ask.puppetlabs.com/question/3149/where-does-hiera-search-for-data-sources/?answer=3152#post-id-3152)
If you are using puppet you can set the location of the hiera file in e master section of puppet.conf so you don't have to provide it on cmd line.
See: http://docs.puppetlabs.com/references/latest/configuration.html#hieraconfig