How do I rewrite this YAML so it is more structured, then reference it in Puppet using hiera function?
Currently, I am working with a hieradata syntax that looks very flat and hard to read.
service::proxy::behind_reverse_proxy: true
service::proxy::proxy_timeout: 300
service::proxy::serverlist:
- host1.fqdn
- host2.fqdn
And grabbed these in a params.pp file, for example
$behind_reverse_proxy = hiera('service::proxy::behind_reverse_proxy', 'False')
$serverlist = hiera('service::proxy::serverlist')
I thought I could rewrite the YAML like so in an effort to make it more readable...
service::proxy:
behind_reverse_proxy: true
proxy_timeout: 300
serverlist:
- host1.fqdn
- host2.fqdn
And updated the params.pp file according to
Hiera Key.subkey syntax
interacting with structured data
$behind_reverse_proxy = hiera('service::proxy.behind_reverse_proxy', 'False')
$serverlist = hiera('service::proxy.serverlist')
However upon puppet agent -t that resulted in
Error 400 on SERVER: Could not find data item service::proxy.serverlist in any Hiera data file and no default supplied
I think these are relevant
[user#server ~]$ facter -y | grep 'version'
facterversion: 2.4.4
puppetversion: 3.8.2
Following up on my comment about how you can access your restructured data:
service::proxy:
behind_reverse_proxy: true
proxy_timeout: 300
serverlist:
- host1.fqdn
- host2.fqdn
In your manifest, instead of this ...
$behind_reverse_proxy = hiera('service::proxy.behind_reverse_proxy', 'False')
$serverlist = hiera('service::proxy.serverlist')
... you might do this:
$proxy_info = merge(
{ 'behind_reverse_proxy' => false, 'serverlist' => [] },
hiera('service::proxy', {})
)
$behind_reverse_proxy = $proxy_info{'behind_reverse_proxy'}
$serverlist = $proxy_info{'serverlist'}
The merge() function is not built-in, but rather comes from Puppet's (formerly PuppetLabs's) widely-used stdlib module. There's a good chance that you are already using that module elsewhere, but even if not, it may be well worth your while to introduce it to your stack.
I've never used Hiera, but I think the problem is that you have a sequence (array) when you wanted a mapping (hash).
In the below YAML, the value of the service::proxy key is a sequence with three elements, each of which is a mapping with one key:
service::proxy:
- behind_reverse_proxy: true
- proxy_timeout: 300
- serverlist:
- host1.fqdn
- host2.fqdn
What you probably wanted, though, was for service::proxy to be a mapping with three keys:
service::proxy:
behind_reverse_proxy: true
proxy_timeout: 300
serverlist:
- host1.fqdn
- host2.fqdn
The examples in the Hiera docs you linked to seem to support this.
Related
I have some hiera not unlike the following (I know this is invalid hiera with two keys... bare with me):
an::example::rule_files:
my_rules:
groups:
- name: my_rules
rules:
- alert: highCPU
expr: CPU > 90
for: 5m
annotations:
summary: "CPU is too high"
description: "CPU should be less than 90"
someone_elses_rules:
groups:
- name: someone_elses_rules
rules:
- alert: highCPU
expr: CPU > 70
for: 5m
annotations:
summary: "CPU is too high"
description: "CPU should be less than 70 on someone else's system"
I'm trying to turn this into a yaml file (the key is the filename). Now I know this is invalid hiera and I can remove the groups key to get this working (exactly what I've done), however when I try to reinsert it into the array, I can't get the formatting right. Here's the puppet code I'm using:
$alert_files = hiera('an::example::rule_files'),
$alert_files.each | String $alerts_file_name, Array $alert_config_pre | {
$prefix = [ "groups:" ]
$alert_config = $prefix + $alert_config_pre
file { "/etc/prometheus/${alerts_file_name}.rules":
ensure => file,
content => $alert_config.to_yaml,
}
}
Here's what I want:
cat /etc/prometheus/my_rules.rules
---
groups:
- name: my_rules
rules:
- alert: highCPU
expr: CPU > 90
for: 5m
annotations:
summary: CPU is too high
description: CPU should be less than 90
and here's what I get:
---
- 'groups:'
- name: my_rules
rules:
- alert: highCPU
expr: CPU > 90
for: 5m
annotations:
summary: CPU is too high
description: CPU should be less than 90
Any help would be massively appreciated. I feel like this should be simple but I've not really made any progress (I can't even remove the quotes from the word groups). If this is possible in either hiera or puppet (perhaps I've defined the hiera wrong) then great; any progress I can make in any way will be really appreciated.
This ...
$alert_files = hiera('an::example::rule_files'),
$alert_files.each | String $alerts_file_name, Array $alert_config_pre | {
... depends on the data associated with key an::example::rule_files to be a Hash with String keys and Array values. In the YAML presented at the top of the question, that item is instead a hash with String keys and Hash values. Inasmuch as the data seem to match the wanted file content, the problem seems to be not with the YAML (except for the inconsistent indentation), but rather with the Puppet code.
To work as you appear to want with the data you want, the Puppet code might look more like so:
$alert_files = lookup('an::example::rule_files'),
$alert_files.each |String $alerts_file_name, Hash $alert_config| {
file { "/etc/prometheus/${alerts_file_name}.rules":
ensure => 'file',
content => $alert_config.to_yaml,
}
}
Note that I have switched from the deprecated hiera() function to its replacement, lookup().
I am creating a MAW well and want to use it as an observation well to compare it later to field data, it should be screened over multiple layers. However, I am only getting the head value in the well of the very last timestep in my output file. Any ideas on how to get all timesteps in the output?
The FloPy manual says something about it needing to be in Output Control, but I can't figure out how to do that:
print_head (boolean) – print_head (boolean) keyword to indicate that the list of multi-aquifer well heads will be printed to the listing file for every stress period in which “HEAD PRINT” is specified in Output Control. If there is no Output Control option and PRINT_HEAD is specified, then heads are printed for the last time step of each stress period.
In the MODFLOW6 manual I see that it is possible to make a continuous output:
modflow6
My MAW definition looks like this:
maw = flopy.mf6.ModflowGwfmaw(gwf,
nmawwells=1,
packagedata=[0, Rwell, minbot, wellhead,'MEAN',OBS1welllayers],
connectiondata=OBS1connectiondata,
perioddata=[(0,'STATUS','ACTIVE')],
flowing_wells=False,
save_flows=True,
mover=True,
flow_correction=True,
budget_filerecord='OBS1wellbudget',
print_flows=True,
print_head=True,
head_filerecord='OBS1wellhead',
)
My output control looks like this:
oc = flopy.mf6.ModflowGwfoc(gwf,
budget_filerecord=budget_file,
head_filerecord=head_file,
saverecord=[('HEAD', 'ALL'), ('BUDGET', 'ALL'), ],
)
Hope this is all clear and someone can help me, thanks!
You need to initialise the MAW observations file... it's not done in the OC package.
You can find the scripts for the three MAW examples in the MF6 documentation here:
https://github.com/MODFLOW-USGS/modflow6-examples/tree/master/notebooks
It looks something like this:
obs_file = "{}.maw.obs".format(name)
csv_file = obs_file + ".csv"
obs_dict = {csv_file: [
("head", "head", (0,)),
("Q1", "maw", (0,), (0,)),
("Q2", "maw", (0,), (1,)),
("Q3", "maw", (0,), (2,)),
]}
maw.obs.initialize(filename=obs_file, digits=10, print_input=True, continuous=obs_dict)
This code was created by black:
def test_schema_org_script_from_list():
assert (
schema_org_script_from_list([1, 2])
== '<script type="application/ld+json">1</script>\n<script type="application/ld+json">2</script>'
)
But now flake8 complains:
tests/test_utils.py:59:9: W503 line break before binary operator
tests/test_utils.py:59:101: E501 line too long (105 > 100 characters)
How can I format above lines and make flake8 happy?
I use this .pre-commit-config.yaml
# See https://pre-commit.com for more information
# See https://pre-commit.com/hooks.html for more hooks
repos:
- repo: 'https://github.com/pre-commit/pre-commit-hooks'
rev: v3.2.0
hooks:
- id: trailing-whitespace
- id: end-of-file-fixer
- id: check-yaml
- id: check-added-large-files
- repo: 'https://gitlab.com/pycqa/flake8'
rev: 3.8.4
hooks:
- id: flake8
- repo: 'https://github.com/pre-commit/mirrors-isort'
rev: v5.7.0
hooks:
- id: isort
tox.ini:
[flake8]
max-line-length = 100
exclude = .git,*/migrations/*,node_modules,migrate
# W504 line break after binary operator
ignore = W504
(I think it is a bit strange that flake8 reads config from a file which belongs to a different tool).
from your configuration, you've set ignore = W504
ignore isn't the option you want as it resets the default ignore (bringing in a bunch of things, including W503).
If you remove ignore=, both W504 and W503 are in the default ignore so they won't be caught
as for your E501 (line too long), you can either extend-ignore = E501 or you can set max-line-length appropriately
for black, this is the suggested configuration:
[flake8]
max-line-length = 88
extend-ignore = E203
note that there are cases where black cannot make a line short enough (as you're seeing) -- both from long strings and from long variable names
disclaimer: I'm the current flake8 maintainer
We have custom test method libraries that support a couple different test method classes. For example:
test93k.common.Functional
test93kcustomext.common.Functional
Would I need to create different test method libraries in our test interface using the add_tml method or can they both exist in the same test method library? In the end, we need the correct class to show up in the generated flow as so:
testmethods
tm_jtag_regular:
testmethod_class = "test93k.common.Functional";
tm_jtag_extension:
testmethod_class = "test93kcustomext.common.Functional";
What controls the test that goes above?
regards
You can apply a class_name: option to both the library and the individual tests, so you could try:
add_tml :my_tml,
class_name: '', # Try setting this to nothing
functional: {
class_name: 'test93k.common.Functional',
},
functional_ext: {
class_name: 'test93kcustomext.common.Functional',
}
There's a chance that the final name might end up with a leading ., though it should be a simple patch to make it inhibit that if the TML class name is blank.
Defining them as two separate TMLs would definitely work too, and is probably how it should be handled:
add_tml :regular,
class_name: 'test93k.common',
functional: {
class_name: 'Functional', # May not even need this
}
add_tml :ext,
class_name: 'test93kcustomext.common',
functional: {
class_name: 'Functional', # May not even need this
}
See here for more - https://origen-sdk.org/origen/guides/program/v93k/#Custom_Test_Methods
So, a brief description of what I want, what my issue is, and what I have tried.
I want to declare and use a dictionary variable for my tests in pyrest, specifically for the [url, body] section so that I can conduct my POST tests targeting a specific endpoint and with a preformatted body.
Here is how mytest.yml file is structured:
- data:
- id: 63
- rate: 25
... a sizable set of field for reasons ...
- type: lab_test_authorization
- modified_at: ansible_date_time.datetime # Useful way to generate
- test:
- url: "some-valid-url/{the_url_question}" # data['special_key']
- method: 'POST'
- headers : {etc..etc}
- body: '{ "data": ${the_body_question} }' # data (the content)
Now the problem starts in my lack of understanding why (if true) does pyrest does not have support for dictionary mappings. I understand yaml supports these feature but am not sure if pyrest can parse through it. Knowing how to call and use dictionary variable in my url and body tags would be significantly helpful.
As of right now, if I try to convert my data Sequence into a data Dictionary, I will get an error stating:
yaml.parser.ParserError: while parsing a block mapping
in "<unicode string>", line 4, column 1:
data:
^
expected <block end>, but found '-'
in "<unicode string>", line 36, column 1:
- config:
I'm pretty sure there are gaps in my knowledge regarding how yaml and pyresttest interact with each other, so any insight would be greatly appreciated.