Hiera value not able to receiving in puppet profile - puppet

I have created a proxy_match.yaml file as a hiera source file in
default hiera datalocation.
The proxy_match.yaml is added in hiera hierarchy
Looking up hiera data in profile
Where and what am I missing, I am not able to receive the hiera data
value and thus the error appears mentioned bellow.
Where,
proxy_match is the new environment created
hierafile 1
/etc/puppetlabs/code/environments/proxy_match/hiera.yaml
version: 5
defaults:
# The default value for "datadir" is "data" under the same directory as the hiera.yaml
# file (this file)
# When specifying a datadir, make sure the directory exists.
# See https://docs.puppet.com/puppet/latest/environments.html for further details on environments.
# datadir: data
# data_hash: yaml_data
hierarchy:
- name: "environment specific yaml"
path: "proxy_match.yaml"
- name: "Per-node data (yaml version)"
path: "nodes/%{::trusted.certname}.yaml"
- name: "Other YAML hierarchy levels"
paths:
- "common.yaml"
proxy_match.yaml hiera data source file
This is the yaml hiera source named as proxy_match.yaml as in herarchy
/etc/puppetlabs/code/environments/proxy_match/data/proxy-match.yaml
---
profiles::apache::servername: "taraserver.com"
profiles::apache::port: "80"
profiles::apache::docroot: "/var/www/tarahost"
hiera lookup in profile
$servername = hiera('profiles::apache::servername',{})
$port = hiera('profiles::apache::port',{})
$docroot = hiera('profiles::apache::docroot',{})
class profile::apache{
#configure apache
include apache
apache::vhost{$servername:
port => $port,
docroot => $docroot,
}
}
#ERROR:
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: {"message":"Server Error: Evaluation Error: Error while evaluating a Resource Statement, Apache::Vhost[7fba80ae621c.domain.name]: parameter 'docroot' expects a value of type Boolean or String, got Undef at /etc/puppetlabs/code/environments/proxy_match/modules/profile/manifests/apache.pp:29 on node 94707b03ff05.domain.name","issue_kind":"RUNTIME_ERROR"}
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

You are defining the variables outside the class definition. When Puppet loads that class, those lines you have before the class are ignored.
What you should have in your profile class is:
class profiles::apache (
String $servername,
Integer $port,
String $docroot,
) {
include apache
apache::vhost { $servername:
port => $port,
docroot => $docroot,
}
}
Note that I used the automatic parameter lookup feature to set your variables, instead of explicit calls to the hiera function.

Related

How to format the file path in an MLTable for Azure Machine Learning uploaded during a pipeline job?

How is the path to a (.csv) file to be expressed in a MLTable file
that is created in a local folder but then uploaded as part of a
pipline job?
I'm following the Jupyter notebook automl-forecasting-task-energy-demand-advance from the azuerml-examples repo (article and notebook). This example has a MLTable file as below referencing a .csv file with a relative path. Then in the pipeline the MLTable is uploaded to be accessible to a remote compute (a few things are omitted for brevity)
my_training_data_input = Input(
type=AssetTypes.MLTABLE, path="./data/training-mltable-folder"
)
compute = AmlCompute(
name=compute_name, size="STANDARD_D2_V2", min_instances=0, max_instances=4
)
forecasting_job = automl.forecasting(
compute=compute_name, # name of the compute target we created above
# name="dpv2-forecasting-job-02",
experiment_name=exp_name,
training_data=my_training_data_input,
# validation_data = my_validation_data_input,
target_column_name="demand",
primary_metric="NormalizedRootMeanSquaredError",
n_cross_validations="auto",
enable_model_explainability=True,
tags={"my_custom_tag": "My custom value"},
)
returned_job = ml_client.jobs.create_or_update(
forecasting_job
)
ml_client.jobs.stream(returned_job.name)
But running this gives the error
Error meassage:
Encountered user error while fetching data from Dataset. Error: UserErrorException:
Message: MLTable yaml schema is invalid:
Error Code: Validation
Validation Error Code: Invalid MLTable
Validation Target: MLTableToDataflow
Error Message: Failed to convert a MLTable to dataflow
uri path is not a valid datastore uri path
| session_id=857bd9a1-097b-4df6-aa1c-8871f89580d8
InnerException None
ErrorResponse
{
"error": {
"code": "UserError",
"message": "MLTable yaml schema is invalid: \nError Code: Validation\nValidation Error Code: Invalid MLTable\nValidation Target: MLTableToDataflow\nError Message: Failed to convert a MLTable to dataflow\nuri path is not a valid datastore uri path\n| session_id=857bd9a1-097b-4df6-aa1c-8871f89580d8"
}
}
paths:
- file: ./nyc_energy_training_clean.csv
transformations:
- read_delimited:
delimiter: ','
encoding: 'ascii'
- convert_column_types:
- columns: demand
column_type: float
- columns: precip
column_type: float
- columns: temp
column_type: float
How am I supposed to run this? Thanks in advance!
For Remote PATH you can use the below and here is the document for create data assets.
It's important to note that the path specified in the MLTable file must be a valid path in the cloud, not just a valid path on your local machine.

puppet not working with keys() and hiera_hash()

I have in hiera node variable solr_enabled = true. Also I have in this node list of fstab mount points like:
fstab_homes:
'/home1':
device: 'UUID=ac2ca97e-8bce-4774-92d7-051482253089'
'/home2':
device: 'UUID=d9daaeed-4e4e-40e9-aa6b-73632795e661'
'/home3':
device: 'UUID=21a358cf-2579-48cb-b89d-4ff43e4dd104'
'/home4':
device: 'UUID=c68041de-542a-4f72-9488-337048c41947'
'/home16':
device: 'UUID=d55eff53-3087-449b-9667-aeff49c556e7'
In solr.pp I want to get the first mounted home disk, create there folder and make symbolic link to /home/cpanelsolr.
For this I wrote the code /etc/puppet/environments/testing/modules/cpanel/manifests/solr.pp:
# Install SOLR - dovecot full text search plugin
class cpanel::solr(
$solr_enable = hiera('solr_enabled',false),
$homes = hiera_hash('fstab_homes', false),
$homesKeys = keys($homes),
)
{
if $solr_enable == true {
notify{"Starting Solr Installation ${homesKeys[0]}":}
if $homes != false and $homesKeys[0] != '/home' {
file { "Create Solr home symlink to ${homesKeys[0]}":
path => '/home/cpanelsolr',
ensure => 'link',
target => "${homesKeys[0]}/cpanelsolr",
}
}
exec { 'cpanel-dovecot-solr':
command => "/bin/bash -c
'/usr/local/cpanel/scripts/install_dovecot_fts'",
}
}
}
But when I run this in dev node I get error:
root#webcloud2 [/home1]# puppet agent -t --no-use_srv_records --server=puppet.development.internal --environment=testing --tags=cpanel::solr
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
2018-08-03 6:04:54 140004666824672 [Note] libgovernor.so found
2018-08-03 6:04:54 140004666824672 [Note] All governors functions found too
2018-08-03 6:04:54 140004666824672 [Note] Governor connected
2018-08-03 6:04:54 140004666824672 [Note] All governors lve functions found too
Error: Could not retrieve catalog from remote server: Error 400 on SERVER: keys(): Requires hash to work with at
/etc/puppet/environments/testing/modules/cpanel/manifests/solr.pp:6 on node webcloud2.development.internal
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
What's wrong?
You have at least two problems there.
First problem is $home won't be set at all in that context. You would need to rewrite as:
class cpanel::solr(
$solr_enable = hiera('solr_enabled',false),
$homes = hiera_hash('fstab_homes', false),
)
{
$homes_keys = keys($homes)
...
}
Second problem is that your YAML isn't correctly indented, so fstab_homes would not actually return a Hash. It should be:
fstab_homes:
'/home1':
device: 'UUID=ac2ca97e-8bce-4774-92d7-051482253089'
'/home2':
device: 'UUID=d9daaeed-4e4e-40e9-aa6b-73632795e661'
'/home3':
device: 'UUID=21a358cf-2579-48cb-b89d-4ff43e4dd104'
'/home4':
device: 'UUID=c68041de-542a-4f72-9488-337048c41947'
'/home16':
device: 'UUID=d55eff53-3087-449b-9667-aeff49c556e7'
Finally, be aware that use of camelCase in parameter names in Puppet can cause you issues in some contexts, so best to use snake_case.

How to create secured files in Puppet5 with Hiera?

I want to create SSL certificate and try to secure this operation.
I am using Puppet 5.5.2 and gem hiera-eyaml.
Created simple manifest
cat /etc/puppetlabs/code/environments/production/manifests/site.pp
package { 'tree':
ensure => installed,
}
package { 'httpd':
ensure => installed,
}
$filecrt = lookup('files')
create_resources( 'file', $filecrt )
Hiera config
---
version: 5
defaults:
# The default value for "datadir" is "data" under the same directory as the hiera.yaml
# file (this file)
# When specifying a datadir, make sure the directory exists.
# See https://puppet.com/docs/puppet/latest/environments_about.html for further details on environments.
datadir: data
data_hash: yaml_data
hierarchy:
- name: "Secret data: per-node, per-datacenter, common"
lookup_key: eyaml_lookup_key # eyaml backend
paths:
- "nodes/%{facts.fqdn}.eyaml"
- "nodes/%{trusted.certname}.eyaml" # Include explicit file extension
- "location/%{facts.whereami}.eyaml"
- "common.eyaml"
options:
pkcs7_private_key: /etc/puppetlabs/puppet/eyaml/keys/private_key.pkcs7.pem
pkcs7_public_key: /etc/puppetlabs/puppet/eyaml/keys/public_key.pkcs7.pem
- name: "YAML hierarchy levels"
paths:
- "common.yaml"
- "nodes/%{facts.fqdn}.yaml"
- "nodes/%{::trusted.certname}.yaml"
And common.yaml
---
files:
'/etc/httpd/conf/server.crt':
ensure: present
mode: '0600'
owner: 'root'
group: 'root'
content: 'ENC[PKCS7,{LOT_OF_STRING_SKIPPED}+uaCmcHgDAzsPD51soM+AIkIlv0ANpUXzBpwM3tqQ3ysFtz81S0xuVbKvslK]'
But have en error while applying manifest
Error: Evaluation Error: Error while evaluating a Function Call, create_resources(): second argument must be a hash (file: /etc/puppetlabs/code/environments/production/manifests/site.pp, line: 12, column: 1) on node test1.com
I really dont know what to do )
The problem appears to be that the indentation in common.yaml isn't right - currently, file will be null rather than a hash, which explains the error message. Also, the file should be called common.eyaml, otherwise the ENC string won't be decrypted. Try
---
files:
'/etc/httpd/conf/server.crt':
ensure: present
mode: '0600'
owner: 'root'
group: 'root'
content: 'ENC[PKCS7{LOTS_OF_STRING_SKIPPED}UXzBpwM3tqQ3ysFtz81S0xuVbKvslK]'
There is an online YAML parser at http://yaml-online-parser.appspot.com/ if you want to see the difference the indentation makes.
Found another solution.
Its was a problem with lookup and hashes. When I have multiply lines in hiera hash, I must specify them https://docs.puppet.com/puppet/4.5/function.html#lookup
So i decided use only 'content' variable to lookup
cat site.pp
$filecrt = lookup('files')
file { 'server.crt':
ensure => present,
path => '/etc/httpd/conf/server.crt',
content => $filecrt,
owner => 'root',
group => 'root',
mode => '0600',
}
and Hiera
---
files:'ENC[PKCS7{LOT_OF_STRING_SKIPPED}+uaCmcHgDAzsPD51soM+AIkIlv0ANpUXzBpwM3tqQ3ysFtz81S0xuVbKvslK]'

Passing elements in an array to hiera lookup as keys

I am getting an error when trying to execute my puppet module:
Error 400 on SERVER: certificatefqdn is not a hash or array when accessing it with certificatefile
This is what my hiera file looks like:
testmodule::install::certificates:
test1.domain.com:
certificatefile: 'certificate1.crt'
certificatepass: 'testpass1'
test2.domain.com:
certificatefile: 'certificate2.crt'
certificatepass: 'testpass2'
test3.domain.com:
certificatefile: 'certificate3.crt'
certificatepass: 'testpass3'
The init.pp in my module looks like this:
class testmodule (
$certificates = hiera('testmodule::install::certificates'),
)
{
$domains = [
test1.domain.com',
test2.domain.com',
test3.domain.com',
[
testmodule::install { $domains:
certificates => $certificates,
}
}
The install.pp in my module looks like this:
define testmodule::install ($certificates)
{
$domain = $name
$certificatefqdn = $certificates["$domain"]
$certificatefile = $certificatefqdn['certificatefile']
$certificatepass = $certificatefqdn['certificatepass']
notify{"This is the certificate file: $certificatefile" :}
}
I'm expecting to see an output like this for each of the elements in the domain array:
Notice: This is the certificate file: certificate2.crt
Notice: /Stage[main]/Testmodule/Testmodule::Install[certificate2.crt]/Notify[This is the certificate file: certificate2.crt]/message: defined 'message' as 'This is the certificate file: certificate2.crt'
Instead I see this:
Notice: This is the certificate file: test2.domain.com['certificatefile']
Notice: /Stage[main]/Testmodule/Testmodule::Install[test2.domain.com]/Notify[This is the certificate file: test2.domain.com['certificatefile']]/message: defined 'message' as 'This is the certificate file: test2.domain.com['certificatefile']'
How can I correctly access the keys in the nested hash in hiera using the elements in domains as the initial key?
The code appears to be correct already.
I have set this up according to your question:
1/
# manifests/init.pp
class test (
$certificates = hiera('test::install::certificates'),
) {
$domains = [
'test1.domain.com',
'test2.domain.com',
'test3.domain.com',
]
test::install { $domains:
certificates => $certificates,
}
}
2/
# manifests/install.pp
define test::install ($certificates) {
$domain = $name
$certificatefqdn = $certificates["$domain"]
$certificatefile = $certificatefqdn['certificatefile']
$certificatepass = $certificatefqdn['certificatepass']
notify{"This is the certificate file: $certificatefile" :}
}
3/
# spec/fixtures/hiera/data/common.yaml
---
test::install::certificates:
test1.domain.com:
certificatefile: 'certificate1.crt'
certificatepass: 'testpass1'
test2.domain.com:
certificatefile: 'certificate2.crt'
certificatepass: 'testpass2'
test3.domain.com:
certificatefile: 'certificate3.crt'
certificatepass: 'testpass3'
4/
# spec/fixtures/hiera/hiera.yaml
---
version: 5
defaults:
datadir: data
data_hash: yaml_data
hierarchy:
- name: "All levels"
paths:
- common.yaml
Compile and apply:
$ bundle exec puppet apply --modulepath spec/fixtures/modules --hiera_config spec/fixtures/hiera/hiera.yaml -e 'include test'
Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://docs.puppet.com/puppet/5.3/reference/deprecated_language.html
(file & line not available)
Notice: Compiled catalog for alexs-macbook-pro.local in environment production in 0.14 seconds
Notice: This is the certificate file: certificate1.crt
Notice: /Stage[main]/Test/Test::Install[test1.domain.com]/Notify[This is the certificate file: certificate1.crt]/message: defined 'message' as 'This is the certificate file: certificate1.crt'
Notice: This is the certificate file: certificate2.crt
Notice: /Stage[main]/Test/Test::Install[test2.domain.com]/Notify[This is the certificate file: certificate2.crt]/message: defined 'message' as 'This is the certificate file: certificate2.crt'
Notice: This is the certificate file: certificate3.crt
Notice: /Stage[main]/Test/Test::Install[test3.domain.com]/Notify[This is the certificate file: certificate3.crt]/message: defined 'message' as 'This is the certificate file: certificate3.crt'
Notice: Applied catalog in 0.04 seconds
So I can't reproduce this.

how to read ansible yml config file in groovy

I want to write a DSL job build script for jenkins in groovy which automatically make deploy job for our projects. There is a general yml file for ansible roles and hosts parameter in each project which I want to read it and use its contents to configure the job.
The problem is that so far I'm using the snakeyml for reading the yml file, but it returns an arraylist (more like a map) which I cannot use efficiently.
anyone knows a better solution?
my yml sample file:
---
- hosts: app.host
roles:
- role: app-db
db_name: myproje_db
db_port: "3306"
migrate_module: "my-proje-api"
- role: java-app
app_name: "myproje-api"
app_artifact_name: "my-proje-api"
app_links:
- myproje_db
I read the file from workspace in my main groovy script:
InputStream configFile = streamFileFromWorkspace('data/config.yml')
and process it in another function of another class:
public String configFileReader(def out, InputStream configFile){
def map
Yaml configFileYml = new Yaml()
map = configFileYml.load(configFile)
}
it returns map class type as arraylist.
It's an expected output, this configuration is starting with a "-" which represent a list. It's "a collection of hosts, and each host have a set of roles".
If you wants to iterate on each host, you can do :
Yaml configFileYml = new Yaml()
configFileYml.load(configFile).each { host -> ... }
When this configuration is read, it's equivalent to the following structure (in groovy format):
[ // collection of map (host)
[ // 1 map for each host
hosts:"app.host",
roles:[ // collection of map (role)
[ // 1 map for each role
role: 'app-db',
db_name: 'myproje_db',
db_port: "3306",
migrate_module: "my-proje-api"
],
[
role: 'java-app',
app_name: "myproje-api",
app_artifact_name: "my-proje-api",
app_links:['myproje_db']
]
]
]
]

Resources