I need to be able to manage user accounts using pe_accounts and hiera. This page:
http://docs.puppetlabs.com/pe/latest/accounts_class.html#usage-example
says I can create a module like site_accounts and add this text
class {'pe_accounts':
data_store => yaml,
}
to site_accounts/manifests/init.pp
and then install this syntax
---
sysop:
locked: false
comment: System Operator
uid: '700'
gid: '700'
groups:
- admin
- sudonopw
into a file named /var/lib/hiera/pe_accounts_users_hash.yaml
and this will be enough to add the user sysop. But it's not, of course.
I did add the below syntax to hiera.conf
:hierarchy:
- "common"
- "pe_accounts_users_hash"
- "pe_accounts_group_hash"
The error I keep getting is:
Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find class pe_accounts for vmtest2.lala.kaka.org
Related
I'm trying to use intake and the intake-xarray to open and store remote files. I have a minimized catalog file here:
/isibhv/projects/paleo_pool/boundary_conditions/ice_sheet_reconstructions/ice_sheet_reconstructions.yaml
It looks like this:
metadata:
version: 1
sources:
glac1d:
description: The GLAC-1D Reconstruction
driver: netcdf
args:
urlpath: "https://sharebox.lsce.ipsl.fr/index.php/s/yfuUw91ruuJXroC/download?path=%2F&files=TOPicemsk.GLACD26kN9894GE90227A6005GGrBgic.nc"
cache_dir: "{{ CATALOG_DIR }}/glac1d"
cache:
- argkey: urlpath
type: file
I can open the files in Python:
import intake
cat = intake.open_catalog("ice_sheet_reconstructions.yaml")
ds = cat.glac1d.read()
This all works wonderfully; and I get the file as I would expect it. However, the cache doesn't show up where I would expect. I would have guessed a new folder is made under:
/isibhv/projects/paleo_pool/boundary_conditions/ice_sheet_reconstructions/glac1d
Instead, I get something in my home directory.
Did I specify the cache directory incorrectly?
As a second question: is it possible to directly specify how the cached files should be called when they are saved?
Thanks!
Paul
The location of the cache is specified by the config, which is a YAML file typically in ~/.intake/conf.yaml (key "cache_dir"), but can be elsewhere according to the INTAKE_CONF(_FILE) environment variable OR the metadata of the source, key "catalog_dir" (<- this may be incorrect?). The special value "catdir" means "in the directory where the catalog is".
However
With the appearance of caching in fsspec, the following will be possible:
sources:
glac1d:
description: The GLAC-1D Reconstruction
driver: netcdf
args:
urlpath: "filecache://sharebox.lsce.ipsl.fr/index.php/s/yfuUw91ruuJXroC/download?path=%2F&files=TOPicemsk.GLACD26kN9894GE90227A6005GGrBgic.nc"
storage_options:
target_protocol: https
cache_storage: "{{ CATALOG_DIR }}/glac1d"
unfortunately, the required change is not yet in intake-xarray.
I have look at other questions like Using hiera to set class parameters? and others which discusses hiera 3. I am using hiera 5.
Here is my hiera.yaml
[root#e64a2e5c7c79 fisherman]# cat /fisherman/fisherman/hiera/hiera.yaml
---
version: 5
defaults: # Used for any hierarchy level that omits these keys.
datadir: data # This path is relative to hiera.yaml's directory.
data_hash: yaml_data # Use the built-in YAML backend.
hierarchy:
- name: "Apps" # Uses custom facts.
path: "apps/%{facts.appname}.yaml"
I also have this hiera data file:
[root#e64a2e5c7c79 fisherman]# cat /fisherman/fisherman/hiera/apps/HelloWorld.yaml
---
fisherman::create_new_component::component_name: 'HelloWord'
But when I run my puppet agent like so ...
export FACTER_appname=HelloWorld
hiera_config=/fisherman/fisherman/hiera/hiera.yaml
modulepath=/fisherman/fisherman/modules
puppet apply --modulepath=$modulepath --hiera_config=$hiera_config -e 'include fisherman'
... I get this error ...
Error: Evaluation Error: Error while evaluating a Function Call, Class[Fisherman::Create_new_component]: expects a value for parameter $component_name (file: /fisherman/fisherman/modules/fish
erman/manifests/init.pp, line: 12, column: 9) on node e64a2e5c7c79
I tried debugging hiera with puppet lookup like so:
[root#e64a2e5c7c79 /]# export FACTER_appname=HelloWorld
[root#e64a2e5c7c79 /]# hiera_config=/fisherman/fisherman/hiera/hiera.yaml
[root#e64a2e5c7c79 /]# modulepath=/fisherman/fisherman/modules
[root#e64a2e5c7c79 /]# puppet lookup --modulepath=$modulepath --hiera_config=$hiera_config --node agent.local --explain fisherman::create_new_component::component_name
Searching for "lookup_options"
Global Data Provider (hiera configuration version 5)
Using configuration "/fisherman/fisherman/hiera/hiera.yaml"
Hierarchy entry "Apps"
Path "/fisherman/fisherman/hiera/data/apps/.yaml"
Original path: "apps/%{facts.appname}.yaml"
Path not found
Environment Data Provider (hiera configuration version 5)
Using configuration "/etc/puppetlabs/code/environments/production/hiera.yaml"
Merge strategy hash
Hierarchy entry "Per-node data (yaml version)"
Path "/etc/puppetlabs/code/environments/production/data/nodes/.yaml"
Original path: "nodes/%{::trusted.certname}.yaml"
Path not found
Hierarchy entry "Other YAML hierarchy levels"
Path "/etc/puppetlabs/code/environments/production/data/common.yaml"
Original path: "common.yaml"
Path not found
Module data provider for module "fisherman" not found
Searching for "fisherman::create_new_component::component_name"
Global Data Provider (hiera configuration version 5)
Using configuration "/fisherman/fisherman/hiera/hiera.yaml"
Hierarchy entry "Apps"
Path "/fisherman/fisherman/hiera/data/apps/.yaml"
Original path: "apps/%{facts.appname}.yaml"
Path not found
Environment Data Provider (hiera configuration version 5)
Using configuration "/etc/puppetlabs/code/environments/production/hiera.yaml"
Hierarchy entry "Per-node data (yaml version)"
Path "/etc/puppetlabs/code/environments/production/data/nodes/.yaml"
Original path: "nodes/%{::trusted.certname}.yaml"
Path not found
Hierarchy entry "Other YAML hierarchy levels"
Path "/etc/puppetlabs/code/environments/production/data/common.yaml"
Original path: "common.yaml"
Path not found
Module data provider for module "fisherman" not found
Function lookup() did not find a value for the name 'fisherman::create_new_component::component_name'
I noticed this in the above output:
Hierarchy entry "Apps"
Path "/fisherman/fisherman/hiera/data/apps/.yaml"
Original path: "apps/%{facts.appname}.yaml"
Path not found
It looks like facts.appname is empty and not HelloWorld as I had expected.
What am I doing wrong here?
Thanks
Based on the information in the question I can't reproduce this. Here is my setup if it helps:
# init.pp
class test (
String $component_name,
) {
notify { $facts['appname']:
message => "Component name: $component_name for fact appname of ${facts['appname']}"
}
}
# hiera.yaml
---
version: 5
defaults:
datadir: data
data_hash: yaml_data
hierarchy:
- name: "Apps" # Uses custom facts.
path: "apps/%{facts.appname}.yaml"
# data/apps/HelloWorld.yaml
---
test::component_name: 'MyComponentName'
# spec/classes/test_spec.rb
require 'spec_helper'
describe 'test' do
let(:hiera_config) { 'spec/fixtures/hiera/hiera.yaml' }
let(:facts) {{ 'appname' => 'HelloWorld' }}
it {
is_expected.to contain_notify("HelloWorld")
.with({
'message' => "Component name: MyComponentName for fact appname of HelloWorld"
})
}
end
Tested on Puppet version:
▶ bundle exec puppet -V
6.6.0
Output:
▶ bundle exec rake spec
I, [2019-07-07T16:42:51.219559 #22140] INFO -- : Creating symlink from spec/fixtures/modules/test to /Users/alexharvey/git/home/puppet-test
/Users/alexharvey/.rvm/rubies/ruby-2.4.1/bin/ruby -I/Users/alexharvey/.rvm/gems/ruby-2.4.1/gems/rspec-core-3.8.2/lib:/Users/alexharvey/.rvm/gems/ruby-2.4.1/gems/rspec-support-3.8.2/lib /Users/alexharvey/.rvm/gems/ruby-2.4.1/gems/rspec-core-3.8.2/exe/rspec --pattern spec/\{aliases,classes,defines,functions,hosts,integration,plans,tasks,type_aliases,types,unit\}/\*\*/\*_spec.rb
test
should contain Notify[HelloWorld] with message => "Component name: MyComponentName for fact appname of HelloWorld"
Finished in 0.1444 seconds (files took 0.9699 seconds to load)
1 example, 0 failures
You also can query the Hiera hierarchy directly using puppet lookup like this:
▶ FACTER_appname=HelloWorld bundle exec puppet lookup \
--hiera_config=spec/fixtures/hiera/hiera.yaml test::component_name
--- MyComponentName
I'm trying (and struggling) to get a (multiple) role model implemented in Hiera.
I've worked in the last 2 years with exact the same model as a user and now want to rebuild the same structure on my own. For example, my node.yaml should contain only the roles I want to apply onto the host:
/etc/puppetlabs/code/environments/production/nodes/my.host.de.yaml
classes:
- ydixken_baseinstall
- additional_modules
[...]
For me it's way more intuitive, to place a yaml in the roles/ directory, with the name of the role, and avoid dealing with profiles:
/etc/puppetlabs/code/environments/production/roles/ydixken_baseinstall.yaml
classes:
- apt
- unattended_upgrades
- [...]
apt::update:
frequency: 'daily'
loglevel: 'debug'
[...]
Placing the role definitions as a node fact is not practicable for me. It's also nice-to-have to allow a customization of the already defined values inside of the node configuration, if needed.
Right now my directory, hiera.yaml & file-structure looks like this:
/etc/puppetlabs/puppet/hiera.yaml
version: 5
defaults:
datadir: /etc/puppetlabs/code/environments/production
data_hash: yaml_data
hierarchy:
- name: "Per-node data (yaml version)"
paths:
- "nodes/%{fqdn}.yaml"
- "roles/%{role}.yaml"
- common
/etc/puppetlabs/code/environments/production/hiera.yaml
version: 5
defaults:
hierarchy:
- name: "FQDN"
path: "nodes/%{fqdn}.yaml"
- name: "Roles"
path: "roles/%{role}.yaml"
- name: "Common Data"
path: "common.yaml"
/etc/puppetlabs/code/environments/production/manifests/site.pp
hiera_include('classes')
How can i achieve this?
My current error:
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Function Call, Could not find class ::ydixken_baseinstall for my.host.de (file: /etc/puppetlabs/code/environments/production/manifests/site.pp, line: 1, column: 1) on node my.host.de
I've found exactly, what i was looking for: r10k
I am pretty new to puppet. I configured a hiera file, whose path is /etc/puppetlabs/puppet/hiera.yaml, as so
version: 5
hierarchy: []
backends:
- yaml
yaml:
- datadir: /etc/puppetlabs/puppet/some_dir
and I get this error
Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://docs.puppet.com/puppet/5.3/reference/deprecated_language.html
(file & line not available)
Error: Evaluation Error: Error while evaluating a Function Call, Lookup of key 'user_dir' failed: The Lookup Configuration at '/etc/puppetlabs/puppet/hiera.yaml' has wrong type, unrecognized key 'backends'
The Lookup Configuration at '/etc/puppetlabs/puppet/hiera.yaml' has wrong type, unrecognized key 'yaml' at /etc/puppetlabs/code/environments/production/manifests/site.pp:30:17 on node puppet,some_cluster_DNS.internal
Initially, I had those kinds of format for the keys :backends: :yaml: but it seemed it is not regular one for the 5 version, so that I deleted the : sign
Someone has an idea ?
First, in terms of that warning, you should definitely switch over to the Puppet lookup function from the Hiera hiera functions if you are using Hiera >= 4: https://puppet.com/docs/puppet/4.10/hiera_use_function.html
Second, in terms of that error, I would consult the documentation on how to setup a Hiera 5 config file: https://puppet.com/docs/puppet/4.10/hiera_config_yaml_5.html
Using the proper format, your config file would look like:
# /etc/puppetlabs/puppet/hiera.yaml
version: 5
defaults:
- data_hash: yaml_data
- datadir: /etc/puppetlabs/puppet/some_dir
hierarchy: []
What you are trying to do on the last line (specify a specific datadir for the yaml_data backend) is not allowed in Hiera 5. If you want to specify a datadir for a specific backend, then you need to specify a level of the hierarchy for just that backend (or just that backend's datadir; you can customize in several depths of matrices with it) and specify a datadir there. For example:
hierarchy:
- name: yaml data
data_hash: yaml_data
datadir: /etc/puppetlabs/puppet/some_dir
paths:
- "%{trusted.certname}.yaml"
- common.yaml
Puppet on the tst-01 works fine when using:
node "tst-01" inherits basenode {
But it breaks when I try to organize servers into groups with this configuration:
node "tst-01" inherits redhat6server {
The error with "inherits redhat6server" is:
err: Could not retrieve catalog; skipping run
[root#tst-01 ~]# puppet agent --test
err: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to parse template ldap/access.conf: Could not find value for 'netgroup' at 124:/etc/puppet/modules/ldap/templates/access.conf at /etc/puppet/modules/ldap/manifests/init.pp:82 on node tst-01.tst.it.test.com
warning: Not using cache on failed catalog
err: Could not retrieve catalog; skipping run
This is the access.conf file, that works fine if inherits is set to "inherits basenode".
[root#puppet]# grep -v "#" /etc/puppet/modules/ldap/templates/access.conf
+ : root : LOCAL
+ : #<%= netgroup %> : ALL
- : ALL : ALL
[root#puppet]#
This is the configuration in /etc/puppet/manifests/nodes.pp.
# Basenode configuration
node "basenode" {
include resolv_conf
include sshd
include ntpd
include motd
}
# Groups
node "redhat6server" inherits basenode {
include ldap_auth
}
# Testservers
node "tst-01" inherits redhat6server {
$netgroup = tst-01
}
I am planning to bring more organisation (read: avoid configuration repetition) in the nodes.pp by grouping machines, e.g. RH5 and RH6 machines instead of adding multiple lines of includes for all RH5 and RH6 servers.
Your running into a variable scoping problem. The official documentation discusses this issue.
In short, redhat6server doesn't have access to the netgroup variable.
The method I employ to work around this is to use hiera. With this, the ldap_auth module can be defined this way, and it will pull the value from a hiera configuration file (typically a yaml file in /etc/puppet/hiera).
You would defined ldap_auth like this:
ldap_auth/manifests/init.pp:
class ldap_auth($netgroup=hiera('netgroup')) {
...
}
Or if your on puppet 3.x, you can use automatic parameter lookup:
class ldap_auth($netgroup) {
...
}
And have a yaml file with:
ldap_auth::netgroup = 'netgroup'