I'm struggling to understand the hiera way of working with data, it seems to me like plain yaml using frontmatter to include global data files would be simpler and more powerful.
In any case, I want to accomplish something like this:
# global.yaml
collection1: &collection1
foo: 1
collection2: &collection2
bar: 2
collection3: &collection3
baz: 3
# development_environment.yaml
collection:
<<: *collection1
<<: *collection2
# production_environment.yaml
collection:
<<: *collection2
<<: *collection3
Essentially, so that I can maintain a couple of lists of things in a single place and then combine them in different ways depending on the environment. Hiera has an option for merging top level keys vs deep merging, but I can't find anything about including data from higher up in the hierarchy (for my particular problem I could also get it working reasonably well if there were a way to overwrite the data in the global file rather than merge it in to the more specific file but that doesn't seem possible either).
How can I do this? Am I stuck manually duplicating the base data in all my different environments?
I realize that I could put an environment case statement in puppet code to choose which base collections to include, but that breaks the separation of concerns of keeping data in hiera and code in puppet. If I have to do that, I may as well skip hiera altogether and put my data in puppet modules.
You can do it by manually loading the list of collections and iterating over it :
# global.yaml
collection1:
foo: 1
collection2:
bar: 2
collection3:
baz: 3
# development_environment.yaml
collection:
- collection1
- collection2
# production_environment.yaml
collection:
- collection2
- collection3
Now you can write something like that :
# this variable will contain something like ['collection1','collection2']
$collections = hiera('collection')
# Now get all the corresponding values
$hashparts = $collections.map |$r| { $x = hiera($r); $x } # [{"baz"=>3}, {"bar"=>2}]
# Now we merge all the parts
$hash = $hashparts.reduce |$a,$b| { $x = merge($a,$b); $x } # {"baz"=>3, "bar"=>2}
This is ugly, but should do what you expect. The deal about $x = function(); $x is here because of the unfortunate decision that all the lambda functions can be used in any context (statement, or value), so we don't know at parsing time whether we expect the last "token" of the "block" to be a statement or an expression.
Related
i'm not sure if that's the right place to ask this kind of questions, but I feel like the way i'm doing things now is 'dumb way' and there's room for improvement in my code.
I'm trying to build stock data website as my side project, and im using rust for backend. One microservice i'm writing is responsible for scraping data from web and then saving it in database. The result of web scraping is 2d vector where each row is responsible for one attribute of struct i'll later construct. Then I save rows to variables.
Then i use izip! macro from itertools to make iterate over all those attributes and create struct.
izip!(
publication_dates,
quarter_dates,
income_revenue,
...
)
.for_each(
|(
publication_date,
quarter_date,
income_revenue,
...
)| {
Financials {
ticker: self.ticker.to_owned(),
publication_date,
quarter_date,
...
},
})
My issue is the fact, that one data table can have more than 40 attributes, to saving data from just one page can be over 250 lines of code so i'd have total of 2000 lines just to store webscraped data, most of it repetitive (parsing rows to correct data types). I'm pretty sure that's not correct approach since any changes i'd like to make would have to be done in many places.
One of my ideas to make it better was to create enum with desired types, then create vector of those enums like vec!([dataType::quarter_date, dataType::int32, dataType::int32 ...]) and iteratoe over both rows and new vector, and use match statement to use according function for data processing. That would get shorten rows allocation part a bit, but probably not by much.
Do you have any advice? Any hint would be great help, i just need a direction that i can later explore by myself :-)
If you want to only reduce the code duplication, I would recommend using a macro for that. A simple example is this (playground):
macro_rules! create_financials {
($rows:ident, $($fun:ident > $column:ident),+) => {{
$(
let $column = $rows
.next()
.ok_or("None")?
.into_iter()
.flat_map($fun);
)+
itertools::izip!($($column,)+).map(
|($($column,)+)| {
Financials {
$($column,)+
}
}
).collect::<Vec<_>>()
}}
}
Note that I removed the .collect::<Vec<_>>() part, it is not needed and allocates additional memory.
I also replaced the for_each with map to return a Vec from the macro which could be used outside of the macro.
The macro can be used simply like this:
let financials: Vec<Financials> = create_financials!(
rows,
quarter_string_date_to_naive_date > quarter_date,
publish_date_string_to_naive_date > publication_date,
income_revenue > income_revenue
);
To remove the code duplication of parsing to the different data types, look if the data types implement FromStr, From or TryFrom. Else you could define your own trait which does the conversion and which you can implement for each data type.
I have a list of valid values that I am storing in a data store. This list is about 20 items long now and will likely grow to around 100, maybe more.
I feel there are a variety of reasons it makes sense to store this in a data store rather than just storing in code. I want to be able to maintain the list and its metadata and make it accessible to other services, so it seems like a micro-service data store.
But in code, we want to make sure only values from the list are passed, and they can typically be hardcoded. So we would like to create an enum that can be used in code to ensure that valid values are passed.
I have created a simple node.js that can generate a JS file with the enum right from the data store. This could be regenerated anytime the file changes or maybe on a schedule. But sharing the enum file with any node.js applications that use it would not be trivial.
Has anyone done anything like this? Any reason why this would be a bad approach? Any feedback is welcome.
Piggy-backing off of this answer, which describes a way of creating an "enum" in JavaScript: you can grab the list of constants from your server (via an HTTP call) and then generate the enum in code, without the need for creating and loading a JavaScript source file.
Given that you have loaded your enumConstants from the back-end (here I hard-coded them):
const enumConstants = [
'FIRST',
'SECOND',
'THIRD'
];
const temp = {};
for (const constant of enumConstants) {
temp[constant] = constant;
}
const PlaceEnum = Object.freeze(temp);
console.log(PlaceEnum.FIRST);
// Or, in one line
const PlaceEnum2 = Object.freeze(enumConstants.reduce((o, c) => { o[c] = c; return o; }, {}));
console.log(PlaceEnum2.FIRST);
It is not ideal for code analysis or when using a smart editor, because the object is not explicitly defined and the editor will complain, but it will work.
Another approach is just to use an array and look for its members.
const members = ['first', 'second', 'third'...]
// then test for the members
members.indexOf('first') // 0
members.indexOf('third') // 2
members.indexOf('zero') // -1
members.indexOf('your_variable_to_test') // does it exist in the "enum"?
Any value that is >=0 will be a member of the list. -1 will not be a member. This doesn't "lock" the object like freeze (above) but I find it suffices for most of my similar scenarios.
I'm writing a puppet code and need to point to a variable with the information of others.... to make it clear here is the example:
These are the variables with the information:
Array $users_ap1_dev = ['userdev1,userdev2'],
Array $users_ap2_prd = ['userprd1,userprd2'],
but ap1 and ap2 values are store in a fact calles main_app and dev and prd values are store in a fact called env.
I want to retrieve the info and create the user based on the fact information, something like
$dmz_users.each | String $user |{
user { $user:
ensure => 'present',
}
So, how can i put the content of $users_ap1_dev into dmz_users, replacing ap1 and dev with the one store in the fact?
something like?:
Array $dmz_user = "${users_${main_app}_${env}}"
Thanks a lot in advance for the help!
I found the answer, the function: getvar()
It allows you to create a variable and put his content into another...
How to use it:
$dmz_users = getvar("users_${main_app}_${env}")
So let's says $main_app = web and '$env = prod', it will take this two values and will make '$dmz_users = $users_web_prod`
I try to create an attribute trait. The use case is to mark some attributes of a class as "crudable" in the context of an objects-to-documents-mapping while other are not.
role crud {
has Bool $.crud is default(True);
}
multi trait_mod:<is>(Attribute $a, crud, $arg) {
$a.container.VAR does crud($arg);
}
class Foo {
has $.bar is rw;
# Provide an extra nested information
has $.baz is rw is crud(True);
}
By reading and adapting some example code, I managed to get something that seems to do what I want. Here is a snippet with test case.
When I instantiate a new Foo object and set the $.bar attribute (that is not crud), it looks like that:
.Foo #0
├ $.bar is rw = 123456789
└ $.baz is rw = .Scalar+{crud} #1
└ $.crud +{crud} = True
What I understand from this is that the $.baz attribute got what I call a meta-attribute that is independent from its potential value.
It looks good to me (if I understood correctly what I did here and that my traits use is not a dirty hack). It is possible to reach $foo.baz.crud that is True. Though, I don't understand very well what .Scalar+{crud} means, and if I can set something there and how.
When I try to set the $.baz instance attribute, this error is returned:
Cannot modify an immutable Scalar+{crud} (Scalar+{crud}.new(crud => Bool::True))
in block <unit> at t/08-attribute-trait.t line 30
Note: This is the closest thing to a working solution I managed to get. I don't need different crud settings for different instances of instantiated Foo classes.
I never want to change the value of the boolean, in fact, once the object instantiated, just providing it to attributes with is crud. I am not even interested to pass a True or False value as an argument: if it would be possible to just set the boolean trait attribute to True by default, it would be enough. I didn't manage to do this though, like:
multi trait_mod:<is>(Attribute $a, :$crud!) {
# Something like this
$a.container.VAR does set-crud;
}
class Foo {
has $.bar is rw;
has $.baz is rw is crud;
}
Am I trying to do something impossible? How could I adapt this code to achieve this use case?
There are several things going on here. First of all, the signature of the trait_mod looks to be wrong. Secondly, there appears to be a bad interaction when the name of a trait is the same as an existing role. I believe this should be an NYI exception, but apparently it either goes wrong in parsing, or it goes wrong in trying to produce the error message.
Anyways, I think this is what you want:
role CRUD {}; # since CRUD is used as an acronym, I chose to use uppercase here
multi trait_mod:<is>(Attribute:D $a, :$crud!) { # note required named attribute!
$a.^mixin: CRUD if $crud; # mixin the CRUD role if a True value was given
}
class A {
has $.a is crud(False); # too bad "is !crud" is invalid syntax
has $.b is crud;
}
say "$_.name(): { $_ ~~ CRUD }" for A.^attributes; # $!a: False, $!b: True
Hope this helps.
My requirement is to do some repetitive file configuration stuff using a loop, Something like following,
$no_of_managers = 2
$array = ['One','two','Three']
define loop() {
notice("Configuring The Manager Nodes!!")
if ($name == $no_of_managers+1) {
notice("Loop Iteration Finished!!!")
}
else
{
notice("Iteration Number : $name \n")
# Doing All Stuff Here
resource {$array:}
$next = $name + 1
loop { $next: }
}
}
loop { "1":}
define resource () {
# Doing my other Stuff
notice ("The Parsed value Name : ${name}\n")
}
Now when The second iteration is running the following error occurs,
Error: Duplicate declaration: Resource[One] is already declared in file
How can I overcome this, What I'm doing is a cluster setup. Is there a workaround to do this, I'm a newbie for puppet so Your kind guidance highly appreciated.
The Use Case :
I'm trying to setup a cluster which have multiple Manager/Worker nodes, So using this script the user has the privilege to select how many manager nodes he needs. So the first loop is for that to copy necessary files and create required number of nodes.
The second loop is there to push all .erb templates. Because each Node has slightly different configs the .erb files have there own logic inside them.
So after each Iteration I want to push the .erb templates to the respective node.
In Puppet 3.x, you cannot build a loop in the fashion you are trying.
resource { $array: }
is a loop over the contents of $array if you will.
It is not really clear what you are trying to solve. If you can make your question a bit more concrete, we may be able to suggest an actual implementation.
Update
If you really want to go down this road, you need to generate unique names for your derived resources.
$local_names = regsubst($array, '$', "-$name")
resource { $local_names: }
In your defined type, you will have to retrieve the original meaning by removing the suffix.
define resource() {
$orig_name = regsubst($name, '-[0-9]+$', '')
# use $orig_name where you used $name before
}
Note that even exported resources must have unique names. So the transformation may have to happen on in the manifest of the receiving node.