According to the Puppet documentation:
Order does not matter in a declarative language.
If that is the case, why does this bit of code work:
class myserver {
$package_to_install = 'libcapture-tiny-perl'
package {
$package_to_install: ensure => present;
}
}
but this code does not work:
class myserver {
package {
$package_to_install: ensure => present;
}
$package_to_install = 'libcapture-tiny-perl'
}
If order matters, then I can see why one works and the other does not, but since order does not matter, why do they behave differently?
Disclaimer: I am one of the Puppet developers.
Because our language isn't, as our documentation claims, actually declarative. It is actually ordered. :(
Evaluation is more or less top to bottom inside the class or declaration. The product of that evaluation is a resource in the catalog, however, not evaluation of the catalog.
Think of the DSL as a not-entirely-declarative way to build the catalog, a graph of resources, that are entirely declarative in processing.
Related
I read a lot about ordering puppet classes with containment (iam using Puppet 6). But it still does not work for me in one case. Maybe my english is not good enough and i miss something. Maybe somebody know what iam doing wrong.
I have a profile to installing a puppetserver (profile::puppetserver). This profile has three sub-classes which I contain within the profile::puppetserver
class profile::puppetserver(
) {
contain profile::puppetserver::install
contain profile::puppetserver::config
contain profile::puppetserver::firewall
}
That works fine for me. Now I want to expand this profile and install PuppetDB. For this, i use the puppetdb module from puppet forge:
So what i do is add profile::puppetserver::puppetdb and the contain to the profile::puppetserver
class profile::puppetserver::puppetdb(
) {
# Configure puppetdb and its underlying database
class { 'puppetdb': }
# Configure the Puppet master to use puppetdb
class { 'puppetdb::master::config': }
}
When i provision my puppetserver first and add the profile::puppetserver::puppetdb after it, puppetdb installs and everything works fine.
If I add it directly with contain, and provisioning everything at once, it crashes. It's because the puppetdb module is installed randomly during my master server installs (and also the postgresql server and so on). That ends in my puppetserver is not running and my puppetdb generate no local ssl certificates and the service doesn't comes up.
What i try first:
I installed the puppetdb Package in my profile::puppetserver::puppetdb directly and use the required flag. It works when i provision all at once.
class profile::puppetserver::puppetdb (
) {
Package { 'puppetdb':
ensure => installed,
require => Class['profile::puppetserver::config']
}
}
So i think i could do the same in the code above:
class profile::puppetserver::puppetdb(
) {
# Configure puppetdb and its underlying database
class { 'puppetdb':
require => Class['profile::puppetserver::config']
}
# Configure the Puppet master to use puppetdb
class { 'puppetdb::master::config':
require => Class['profile::puppetserver::config']
}
}
But this does not work...
So i read about puppet class containment and ordering by chains. So i did this in my profile::puppetserver
class profile::puppetserver(
) {
contain profile::puppetserver::install
contain profile::puppetserver::config
contain profile::puppetserver::firewall
contain profile::puppetserver::puppetdb
Class['profile::puppetserver::install'] ->
Class['profile::puppetserver::config'] ->
Class['profile::puppetserver::firewall'] ->
Class['profile::puppetserver::puppetdb']
}
But it still does not have any effect... he still starts to install postgresql and the puppetdb package during my "puppetserver provisioning" in the install, config, firewall steps.
How i must write the ordering, that all things from the puppetdb module, which i call in profile::puppetserver::puppetdb, only starts when the rest of the provisioning steps are finished?
I really don't understand it. I think maybe it haves something to do with the fact, that i declare classes from the puppetdb module inside of profile::puppetserver::puppetdb and not the directly Resource Type. Because when i use the Package Resource Type with the Require Flag, it seems to work. But i really don't know how to handle this. I think there must be a way or?
I think maybe it haves something to do with the fact, that i declare
classes from the puppetdb module inside of
profile::puppetserver::puppetdb and not the directly Resource Type.
Because when i use the Package Resource Type with the Require Flag, it
seems to work.
Exactly so.
Resources are ordered with the class or defined-type instance that directly declares them, as well as according to ordering parameters and instructions applying to them directly.
Because classes can be declared multiple times, in different places, ordering is more complicated for them. Resource-like class declarations such as you demonstrate (and which you really ought to avoid as much as possible) do not imply any particular ordering of the declared class. Neither do declarations via the include function.
Class declarations via the require function place a single-ended ordering constraint on the declared class relative to the declaring class or defined type, and declarations via the contain function place a double-ended ordering constraint similar to that applying to all resource declarations. The chaining arrows and ordering metaparameters can place additional ordering constraints on classes.
But i really dont know how to handle this. I think there must be a way or?
Your last example shows a viable way to enforce ordering at the level of profile::puppetserver, but its effectiveness is contingent on each of its contained classes taking the same approach for any classes they themselves declare, at least where those third-level classes must be constrained by the order of the second-level classes. This appears to be where you are falling down.
Note also that although there is definitely a need to order some things relative to some others, it is not necessary or much useful to try to enforce an explicit total order over all resources. Work with the lightest hand possible, placing only those ordering constraints that serve a good purpose.
Looking at the Relationship Declarations section in the Puppet Style Guide about the '->' arrows, where 'x->y' means that y requires x to be installed before y, it says:
When possible, you should prefer metaparameters to relationship declarations.
that's a lot of syllables. Is that saying that you should prefer this
file { "/home/${user}/.ssh":
require => User[$user],
...
}
and not use the arrows like this?
User[$user] -> file { "/home/${user}/.ssh":
...
}
file { "/home/${user}/.ssh":
require => User[$user],
...
}
1.The above code even works with different manifests, when they are part of same catalog. So if you use metaparameters you have less work when you get a modification in future(unfortunately).
2.Chaining arrows is best used for Resource collectors, when you have to make relationship more than one resource, you can have this instead of putting require/before in all the resources.
I have encounter really weird behaviour which goes against what I have learned, tutorial says etc. So I would be glad if someone could explain why that is happening.
I have a role module which is made up of composition of profiles (role-profile pattern). My role consists:
class role::lab_prg_c2_dn inherits lab_prg_c2 {
class { 'profile::cluster_data_node':
namenode_fqdn => $role::lab_prg_c2::namenode_fqdn,
secondarynamenode_fqdn => $role::lab_prg_c2::secondarynamenode_fqdn,
}
->
class{'bigdatasolution':}
}
First class installs technology and second one installs our components and items which are build on top of technology. Hence the technology need to be installed first, thats the reason for "->" dependency. However this seems to me doesn't work correctly. As components from class 'bigdatasolution' are installed somewhere before the class profile::cluster_data_node finishes.
I tried to use require => Class['profile::cluster_data_node'] but that doesn't make any difference!
The content of class{'bigdatasolution':} :
class bigdatasolution {
$hdfs_default_conf = '/usr/local/hadoop.hdfs.conf'
$hbase_default_conf = '/usr/local/hadoop.hbase.conf'
include symlinks
include bdjar
}
Symlinks - create symlinks for the configuration installed in class profile::cluster_data_node and are not directly managed - it will be presented when actually specified package get installed.
bdjar - add our jar to a technology library so content is as follows:
class bigdatasolution::bdjar {
file { "/usr/lib/hadoop/lib/bigdata-properties.jar":
ensure => present,
mode => 0644,
group => 'root',
owner => 'root',
source => "puppet:///modules/bigdatasolution/bigdata-properties.jar"
}
}
I even tried to put require => "technologycalClass" here but that doesn't help either.
Can someone please help me understand what's wrong and how that should be solved properly?
I Using puppet 3 and ordering is specified explicetly - so no arbitrary ordering set by puppet should happen.
Thanks
If your 'profile::cluster_data_node' class 'includes' other classes/modules they will have no dependency ordering with the 'bigdatasolution' class.
I see you actually do include symlinks and bdjar. Basically every piece of ordering you want to have in puppet, you need to write explicitly.
Here you should replace the include statements with require, that way the class cluster_data_node will require the other two modules to complete before it says it has completed. Include is a pretty lose way of importing things in puppet and in my opinion is best to just avoid it and go with explicit require statements instead.
TL;DR: included modules have no transitive ordering; required modules do.
I would like to iterate over an array that is stored as a Facter fact, and for each element of the array create a new system user and a directory, and finally make API calls to AWS.
Example of the fact: my_env => [shared1,shared2,shared3]
How can I iterate over an array in Puppet?
This might work, depending on what you are doing
# Assuming fact my_env => [ shared1, shared2, shared3 ]
define my_resource {
file { "/var/tmp/$name":
ensure => directory,
mode => '0600',
}
user { $name:
ensure => present,
}
}
my_resource { $my_env: }
It will work if your requirements are simple, if not, Puppet makes this very hard to do. The Puppet developers have irrational prejudices against iteration based on a misunderstanding about how declarative languages work.
If this kind of resource doesn't work for you, perhaps you could give a better idea of which resource properties you are trying to set from your array?
EDIT:
With Puppet 4, this lamentable flaw was finally fixed. Current state of affairs documented here. As the documentation says, you'll find examples of the above solution in a lot of old code.
As of puppet 3.2 this is possible using the "future" parser like so:
$my_env = [ 'shared1', 'shared2', 'shared3', ]
each($my_env) |$value| {
file { "/var/tmp/$value":
ensure => directory,
mode => 0600,
}
user { $value:
ensure -> present,
}
}
See also: http://docs.puppetlabs.com/puppet/3/reference/lang_experimental_3_2.html#background-the-puppet-future-parser
Puppet 3.7 released earlier this month have the new DSL, which one feature is the iteration, check the following URL https://docs.puppetlabs.com/puppet/latest/reference/experiments_lambdas.html#enabling-lambdas-and-iteration
these new features can be enabled with the :
Setting parser = future in your puppet.conf file
or adding the command line switch --parser=future
hope that helps
As of latest Puppet (6.4.2), and since Puppet 4, iteration over arrays is supported in a few ways:
$my_arr = ['foo', 'bar', 'baz']
Each function:
$my_arr.each |$v| {
notice($v)
}
Each function alternative syntax:
each($my_arr) |$v| {
notice($v)
}
To get the index:
Pass a second argument to each:
$my_arr.each |$i, $v| {
notice("Index: $i, value: $v")
}
Comparison with Ruby:
Note that this grammar is inspired by Ruby but slightly different, and it's useful to show the two side by side to avoid confusion. Ruby would allow:
my_arr.each do |v|
notice(v)
end
Or:
my_arr.each { |v|
notice(v)
}
Other iteration functions:
Note that Puppet provides a number of other iteration functions:
each - Repeats a block of code a number of times, using a collection of values to provide different parameters each time.
slice - Repeats a block of code a number of times, using groups of values from a collection as parameters.
filter - Uses a block of code to transform a data structure by removing non-matching elements.
map - Uses a block of code to transform every value in a data structure.
reduce - Uses a block of code to create a new value, or data structure, by combining values from a provided data structure.
with - Evaluates a block of code once, isolating it in its own local scope. It doesn’t iterate, but has a family resemblance to the iteration functions.
Puppet 3 and earlier:
If you have inherited old code still using Puppet 3, the accepted answer is still correct:
define my_type {
notice($name)
}
my_type { $my_arr: }
Note however that this is usually considered bad style in modern Puppet.
itsbruce's answer is probably the best for now, but there is an iteration proposal going through puppetlabs' armatures process for possible implementation in future.
There is a "create_resources()" function in puppet. that will be very helpful while iterating over the list of itmes
I've taken a read to the Advantages of message based web services article and am wondering if there is there a recommended style/practice to versioning Restful resources in ServiceStack? The different versions could render different responses or have different input parameters in the Request DTO.
I'm leaning toward a URL type versioning (i.e /v1/movies/{Id}), but I have seen other practices that set the version in the HTTP headers (i.e Content-Type: application/vnd.company.myapp-v2).
I'm hoping a way that works with the metadata page but not so much a requirement as I've noticed simply using folder structure/ namespacing works fine when rendering routes.
For example (this doesn't render right in the metadata page but performs properly if you know the direct route/url)
/v1/movies/{id}
/v1.1/movies/{id}
Code
namespace Samples.Movies.Operations.v1_1
{
[Route("/v1.1/Movies", "GET")]
public class Movies
{
...
}
}
namespace Samples.Movies.Operations.v1
{
[Route("/v1/Movies", "GET")]
public class Movies
{
...
}
}
and corresponding services...
public class MovieService: ServiceBase<Samples.Movies.Operations.v1.Movies>
{
protected override object Run(Samples.Movies.Operations.v1.Movies request)
{
...
}
}
public class MovieService: ServiceBase<Samples.Movies.Operations.v1_1.Movies>
{
protected override object Run(Samples.Movies.Operations.v1_1.Movies request)
{
...
}
}
Try to evolve (not re-implement) existing services
For versioning, you are going to be in for a world of hurt if you try to maintain different static types for different version endpoints. We initially started down this route but as soon as you start to support your first version the development effort to maintain multiple versions of the same service explodes as you will need to either maintain manual mapping of different types which easily leaks out into having to maintain multiple parallel implementations, each coupled to a different versions type - a massive violation of DRY. This is less of an issue for dynamic languages where the same models can easily be re-used by different versions.
Take advantage of built-in versioning in serializers
My recommendation is not to explicitly version but take advantage of the versioning capabilities inside the serialization formats.
E.g: you generally don't need to worry about versioning with JSON clients as the versioning capabilities of the JSON and JSV Serializers are much more resilient.
Enhance your existing services defensively
With XML and DataContract's you can freely add and remove fields without making a breaking change. If you add IExtensibleDataObject to your response DTO's you also have a potential to access data that's not defined on the DTO. My approach to versioning is to program defensively so not to introduce a breaking change, you can verify this is the case with Integration tests using old DTOs. Here are some tips I follow:
Never change the type of an existing property - If you need it to be a different type add another property and use the old/existing one to determine the version
Program defensively realize what properties don't exist with older clients so don't make them mandatory.
Keep a single global namespace (only relevant for XML/SOAP endpoints)
I do this by using the [assembly] attribute in the AssemblyInfo.cs of each of your DTO projects:
[assembly: ContractNamespace("http://schemas.servicestack.net/types",
ClrNamespace = "MyServiceModel.DtoTypes")]
The assembly attribute saves you from manually specifying explicit namespaces on each DTO, i.e:
namespace MyServiceModel.DtoTypes {
[DataContract(Namespace="http://schemas.servicestack.net/types")]
public class Foo { .. }
}
If you want to use a different XML namespace than the default above you need to register it with:
SetConfig(new EndpointHostConfig {
WsdlServiceNamespace = "http://schemas.my.org/types"
});
Embedding Versioning in DTOs
Most of the time, if you program defensively and evolve your services gracefully you wont need to know exactly what version a specific client is using as you can infer it from the data that is populated. But in the rare cases your services needs to tweak the behavior based on the specific version of the client, you can embed version information in your DTOs.
With the first release of your DTOs you publish, you can happily create them without any thought of versioning.
class Foo {
string Name;
}
But maybe for some reason the Form/UI was changed and you no longer wanted the Client to use the ambiguous Name variable and you also wanted to track the specific version the client was using:
class Foo {
Foo() {
Version = 1;
}
int Version;
string Name;
string DisplayName;
int Age;
}
Later it was discussed in a Team meeting, DisplayName wasn't good enough and you should split them out into different fields:
class Foo {
Foo() {
Version = 2;
}
int Version;
string Name;
string DisplayName;
string FirstName;
string LastName;
DateTime? DateOfBirth;
}
So the current state is that you have 3 different client versions out, with existing calls that look like:
v1 Release:
client.Post(new Foo { Name = "Foo Bar" });
v2 Release:
client.Post(new Foo { Name="Bar", DisplayName="Foo Bar", Age=18 });
v3 Release:
client.Post(new Foo { FirstName = "Foo", LastName = "Bar",
DateOfBirth = new DateTime(1994, 01, 01) });
You can continue to handle these different versions in the same implementation (which will be using the latest v3 version of the DTOs) e.g:
class FooService : Service {
public object Post(Foo request) {
//v1:
request.Version == 0
request.Name == "Foo"
request.DisplayName == null
request.Age = 0
request.DateOfBirth = null
//v2:
request.Version == 2
request.Name == null
request.DisplayName == "Foo Bar"
request.Age = 18
request.DateOfBirth = null
//v3:
request.Version == 3
request.Name == null
request.DisplayName == null
request.FirstName == "Foo"
request.LastName == "Bar"
request.Age = 0
request.DateOfBirth = new DateTime(1994, 01, 01)
}
}
Framing the Problem
The API is the part of your system that exposes its expression. It defines the concepts and the semantics of communicating in your domain. The problem comes when you want to change what can be expressed or how it can be expressed.
There can be differences in both the method of expression and what is being expressed. The first problem tends to be differences in tokens (first and last name instead of name). The second problem is expressing different things (the ability to rename oneself).
A long-term versioning solution will need to solve both of these challenges.
Evolving an API
Evolving a service by changing the resource types is a type of implicit versioning. It uses the construction of the object to determine behavior. Its works best when there are only minor changes to the method of expression (like the names). It does not work well for more complex changes to the method of expression or changes to the change of expressiveness. Code tends to be scatter throughout.
Specific Versioning
When changes become more complex it is important to keep the logic for each version separate. Even in mythz example, he segregated the code for each version. However, the code is still mixed together in the same methods. It is very easy for code for the different versions to start collapsing on each other and it is likely to spread out. Getting rid of support for a previous version can be difficult.
Additionally, you will need to keep your old code in sync to any changes in its dependencies. If a database changes, the code supporting the old model will also need to change.
A Better Way
The best way I've found is to tackle the expression problem directly. Each time a new version of the API is released, it will be implemented on top of the new layer. This is generally easy because changes are small.
It really shines in two ways: first all the code to handle the mapping is in one spot so it is easy to understand or remove later and second it doesn't require maintenance as new APIs are developed (the Russian doll model).
The problem is when the new API is less expressive than the old API. This is a problem that will need to be solved no matter what the solution is for keeping the old version around. It just becomes clear that there is a problem and what the solution for that problem is.
The example from mythz's example in this style is:
namespace APIv3 {
class FooService : RestServiceBase<Foo> {
public object OnPost(Foo request) {
var data = repository.getData()
request.FirstName == data.firstName
request.LastName == data.lastName
request.DateOfBirth = data.dateOfBirth
}
}
}
namespace APIv2 {
class FooService : RestServiceBase<Foo> {
public object OnPost(Foo request) {
var v3Request = APIv3.FooService.OnPost(request)
request.DisplayName == v3Request.FirstName + " " + v3Request.LastName
request.Age = (new DateTime() - v3Request.DateOfBirth).years
}
}
}
namespace APIv1 {
class FooService : RestServiceBase<Foo> {
public object OnPost(Foo request) {
var v2Request = APIv2.FooService.OnPost(request)
request.Name == v2Request.DisplayName
}
}
}
Each exposed object is clear. The same mapping code still needs to be written in both styles, but in the separated style, only the mapping relevant to a type needs to be written. There is no need to explicitly map code that doesn't apply (which is just another potential source of error). The dependency of previous APIs is static when you add future APIs or change the dependency of the API layer. For example, if the data source changes then only the most recent API (version 3) needs to change in this style. In the combined style, you would need to code the changes for each of the APIs supported.
One concern in the comments was the addition of types to the code base. This is not a problem because these types are exposed externally. Providing the types explicitly in the code base makes them easy to discover and isolate in testing. It is much better for maintainability to be clear. Another benefit is that this method does not produce additional logic, but only adds additional types.
I am also trying to come with a solution for this and was thinking of doing something like the below. (Based on a lot of Googlling and StackOverflow querying so this is built on the shoulders of many others.)
First up, I don’t want to debate if the version should be in the URI or Request Header. There are pros/cons for both approaches so I think each of us need to use what meets our requirements best.
This is about how to design/architecture the Java Message Objects and the Resource Implementation classes.
So let’s get to it.
I would approach this in two steps. Minor Changes (e.g. 1.0 to 1.1) and Major Changes (e.g 1.1 to 2.0)
Approach for minor changes
So let’s say we go by the same example classes used by #mythz
Initially we have
class Foo { string Name; }
We provide access to this resource as /V1.0/fooresource/{id}
In my use case, I use JAX-RS,
#Path("/{versionid}/fooresource")
public class FooResource {
#GET
#Path( "/{id}" )
public Foo getFoo (#PathParam("versionid") String versionid, (#PathParam("id") String fooId)
{
Foo foo = new Foo();
//setters, load data from persistence, handle business logic etc
Return foo;
}
}
Now let’s say we add 2 additional properties to Foo.
class Foo {
string Name;
string DisplayName;
int Age;
}
What I do at this point is annotate the properties with a #Version annotation
class Foo {
#Version(“V1.0")string Name;
#Version(“V1.1")string DisplayName;
#Version(“V1.1")int Age;
}
Then I have a response filter that will based on the requested version, return back to the user only the properties that match that version. Note that for convenience, if there are properties that should be returned for all versions, then you just don’t annotate it and the filter will return it irrespective of the requested version
This is sort of like a mediation layer. What I have explained is a simplistic version and it can get very complicated but hope you get the idea.
Approach for Major Version
Now this can get quite complicated when there is a lot of changes been done from one version to another. That is when we need to move to 2nd option.
Option 2 is essentially to branch off the codebase and then do the changes on that code base and host both versions on different contexts. At this point we might have to refactor the code base a bit to remove version mediation complexity introduced in Approach one (i.e. make the code cleaner) This might mainly be in the filters.
Note that this is just want I am thinking and haven’t implemented it as yet and wonder if this is a good idea.
Also I was wondering if there are good mediation engines/ESB’s that could do this type of transformation without having to use filters but haven’t seen any that is as simple as using a filter. Maybe I haven’t searched enough.
Interested in knowing thoughts of others and if this solution will address the original question.