I have a cookbook "blah-deploy-nodejs-from-git" cookbook that installs a nodejs codebase from GIT and calls NPM install on the directory. It has the following attributes
git_repo
branch
destination
I have then written cookbooks that wrap that cookook for inidividual sites, that need to get installed. In this particar case "blah-pricing" and "blah-notifications" which have different overriding attributes:
me#me cat cookbooks/blah-svc-pricing/attributes/default.rb
node.override[:blah_deploy_nodejs_from_git][:destination] = "/var/blah/pricing"
node.override[:blah_deploy_nodejs_from_git][:branch] = "master"
node.override[:blah_deploy_nodejs_from_git][:git_repo] = "https://hqdevgit01.blah.lan/micro-services/blah-pricing.git"
me#me:~/chef-repo$ cat cookbooks/blah-svc-notifications/attributes/default.rb
node.override[:blah_deploy_nodejs_from_git][:destination] = "/var/blah/notifications"
node.override[:blah_deploy_nodejs_from_git][:branch] = "master"
node.override[:blah_deploy_nodejs_from_git][:git_repo] = "https://hqdevgit01.blah.lan/micro-services/blah-notifications.git"
And then the recipe is the same in both cases:
include_recipe 'blah-deploy-nodejs-from-git'
Unfortunately it is applying the inner recipe only once even though my node has both cookbooks applied to it. My understanding was that wrapper cookbooks are used to customize a cookbook and make it unique.
Can encapsulate the inner cookbook to two different cookbooks, with different attributes, and have the wrapper cookbooks both apply that inner recipe? OR Am I going to have to completely replicate the code that is in the inner cookbook?
This is due to a basic misunderstanding of how chef works. Recipes are not meant to be a procedure for how to do something, they are meant to be a declaration of what that something should look like. As such, you need to think of them as describing the end state, not the process for getting there.
Thus, chef will never run a recipe twice. And attributes really should not be changed mid run (unless they are updated to indicate something that happens mid run. Luckily, there are other chef capabilities that can solve your problem. You need either a definition or an LWRP (light weight resource provider)
Definitions are just groups of resources that are often repeated. So you can create a definition and then later call it multiple times in the same recipe with different attributes. Much like what you currently are doing with your recipe.
While definitions are sometimes appropriate, LWRPs are generally more powerful, and have become the prefered approach for most repetitive (library like) task in Chef. With LWRPs you will define a new chef primitive (much like file, service, etc), and then write the code for accomplishing the goal of that primitive. You can then use these resources anywhere in your recipes. In your case, you'd have an npm_deployer resource that took attributes for repo, branch, and destination. It would then do the work that is currently in your deployer recipe. Your "wrapper" recipes would then stop calling include_recipe and instead just declare a npm_deploy resource and pass in the attributes needed.
Related
Disclaimer: I can achieve the behavior I’m looking for with Active Choices plugin, BUT I really want this to work in a Jenkinsfile and controlled with scm because it’s tedious to configure the Active Choices on each job we may need them on. And with it being separate from the Jenkinsfile creation, it’s then one job defined in multiple places. :(
I am looking to verify if this is possible, because I can’t get the syntax right, if it is possible. And I haven’t been able to find any examples online:
pipeline {
environment {
ARTIFACTS = lib.myfunc() // this works well
}
parameters {
choice(name: "Artifacts", choices: ARTIFACTS) // I can’t get this to work
}
}
I cannot use the function inline in the declaration of the parameter. The errors were clear about that, but it seems as though I should be able to do what I’ve written out above.
I am not home, so I do not have the exceptions handy, but I will add them soon. They did not seem very helpful while I was working on this yesterday.
What have I tried?
I’ve tried having the the function return a List Because it requires a list according to the docs, and I’ve also tried (illogically) returning a String in the precise syntax of a list of strings. (It was hacky, like return "['" + artifacts.join("', '") + "']" to look like ['artifact1.zip', 'artifact2.zip']
I also tried things like "$ARTIFACTS" and ${ARTIFACTS} in desperation.
the list of choices has to be supplied as String containing new line characters (\n): choices: 'TESTING\nSTAGING\nPRODUCTION'
I was tipped off by this article:
https://st-g.de/2016/12/parametrized-jenkins-pipelines
Related to a bug:
https://issues.jenkins.io/plugins/servlet/mobile#issue/JENKINS-40358
:shrug:
First, we need to understand that Jenkins starts running your pipeline code by presenting you with Parameters page. Once you've set up the parameters, and pressed Build, then a node is allocated, variables are set, and your code starts to run.
But in your pipeline, as presented above, you want to run some code to prepare the parameters.
This is not how Jenkins usually works. It's definitely not doing the following: allocating a node, setting the variables, running some of your code until parameters clause is reached, stopping all that, presenting you with GUI, and then continuing where it left off. Again, it's not how Jenkins works.
This is why, when writing a new pipeline, your first option to build it is Build and not Build with Parameters. Jenkins hasn't run your code yet; it doesn't have any idea if there are any parameters. When running for the first time, it will remember the parameters (and any choices, if were) as were configured for this (first) run, so in the second run you will see the parameters as configured in the first run. (Generally, in run number n you will see the result of configuration in run number n-1.)
There are a number of ways to overcome this.
If having a "somewhat recent" (and not "current and absolutely up-to-date") situation fits you, your code may need minor changes to work — second time. (I don't know what exactly lib.myfunc() returns but if it's a choice of Development/Staging/Production this might be good enough.)
If having a "somewhat recent" situation is an absolute no-no (e.g. your lib.myfunc() returns the list of git branches, and "list of branches as of yesterday" is unacceptable), then your only solution is ActiveChoice. ActiveChoice allows you to run some code before showing you the Build with Parameters GUI (with script approval etc.).
We are using the roles pattern in puppet with hiera, meaning we have these lines in hiera.yaml:
- name: "Roles data"
path: "roles/%{::server_role}.yaml"
We have a custom fact that produces the role name when facter runs, but we would like to move this into hiera. Instead of the server_role variable being produced by facter, we want to specify the server_role inside of hiera, and let that variable be referenced elsewhere in hiera. Something like this:
hiera.yaml:
- name: "Per-node data"
path: "nodes/%{trusted.certname}.yaml"
- name: "Roles data"
path: "roles/%{lookup(server_role)}.yaml"
nodes/hostname.yaml:
server_role: foo_bar
I have seen this question, which says to use hiera() or lookup() but when I try to use those, I get this error message:
Interpolation using method syntax is not allowed in this context
So how can I use a hiera variable that's defined elsewhere in hiera?
Edit:
The prototypical code examples for defining roles could use any fact that's known to facter, often giving examples that are based on hostname. When you can't embed server config into hostname, a common(ish) workaround is to write a file such as /etc/server_role, but it seems to defeat the purpose of config management, when you need to ssh into a machine and edit a file. As the other comments & answer here so far mentioned, you could use an ENC, but again, the goal here is not to have config stored outside of version control. In fact, we have foreman as an ENC and we make a practice to never use it that way because then upgrades and other maintenance become unsustainable.
We could write a class which will pick up data from hiera, write it to /etc/server_role, and on the next puppet run, facter will pick that up and send it back to hiera, so then we'll have the server_role fact available to use in hiera.yaml. As gross as this sounds, so far, it's the best known solution. Still looking for better answers to this question.
Thanks.
As #MattSchuchard explained in comments, you cannot interpolate Hiera data into your Hiera config, because the config has to be known before the data can be looked up.
If you need a per-role level in your data hierarchy then an alternative would be to assign roles to machines via an external node classifier. You don't need it to assign any classes, just the server_role top-scope variable and probably also environment.
On the other hand, maybe you don't need a per-role level of your general hierarchy in the first place. Lots of people do roles & profiles without per-role data, but even if you don't want to do altogether without then it may be that module-specific data inside the module providing your role classes could be made to suffice.
I am trying to set a module's source (this IS NOT a resource) based on a conditional trigger but it looks like the module is getting fired before the logic is applied:
module "my_module" {
source = "${var.my_field == "" ? var.standard_repo : var.custom_repo}"
stuff...
more stuff...
}
I have created the standard_repo and custom_repo vars as well and defined with URLs for respective repos (using git:: -- this all works w/o conditional)
All this being said, anyone know of a way to implement this conditional aspect? (again, this is a module and not a resource)
I tried using duplicate modules and calling based off the var value but this, too, does not work (condition is never met, even when it is):
repo = ["${var.my_field == "na" ? module.my_module_old : module.my_module_new}"]
One way to achieve this is described in this post
Basically, a common pattern is to have several folders for different environments such as dev/tst/prd. These environments often reuse large parts of the codebase. Some may be abstracted as modules, but there is still often a large common file which is either copy-pasted or symlinked.
The post offers a way that doesn't conditionally disable based on variables but it probably solves your issue of enabling a module based on different enviornments. It makes use of the override feature of terraform and adds a infra_override.tf file. Here, it defines a different source for the module which points to an empty directory. Voila, a disabled module.
Variables are not allowed to be used in the module source parameter. There also does not seem to be a plan for this to change. https://github.com/hashicorp/terraform/issues/1439 . Creating a wrapper script , or using something like mustache http://mustache.github.io/ seems to be the best way to solve the problem.
I have a cucumber feature file 'A' that serves as setting up environment (data clean up and initialization). I want to have it executed before all other feature files can run.
It's it kind of like #before hook as in http://zsoltfabok.com/blog/2012/09/cucumber-jvm-hooks/. However, that does not work because my feature files 'A' contains hundreds of cucumber steps and it is not as simple as:
#Before
public void beforeScenario() {
tomcat.start();
tomcat.deploy("munger");
browser = new FirefoxDriver();
}
instead it's better to be able to run 'A' as a feature file as a whole.
I've searched around but did not find a answer. I am so surprised that no one has this type of requirement before.
The closest i found is 'background'. But that means i can have only one huge feature file with the content of 'A' as 'background' at the top, and rest of my test in the same file. I really do not want to do that.
Any suggestions?
By default, Cucumber features are run single thread in order by:
Alphabetically by feature file directory
Alphabetically by feature file name within directory
Scenario execution is then by order within the feature file.
So have your initialization feature in the first directory (alhpabetically) with a file name that sorts first (alphabetically) in that directory.
That being said it is generally a bad practice to require an execution order in your feature files. We run our feature files in parallel so order is meaningless. For Jenkins or TeamCity you could add a build step that executes the one feature file followed by a second build step that executes the rest of your feature files.
I have also a project, where we have a single feature file, that contains a very long scenario called Scenario: Test data with a lot of very long scenarios, like this:
Given the system knows about the following employees
|uuid|user-key|name|nickname|
|1|0101140000|Anna|annie|
... hundreds of lines like this follow ...
We see this long SystemKnows scenarios as quite valuable, so that our testers, Product Owner and developers have a baseline of what data are in the system. Our domain is quite complex, and we need this baseline of reference data for everyone to be able to understand the tests.
(These reference data become almost like well known personas, and are a shared team metaphore)
In the beginning, we were relying on the alphabetic naming convention, to have the AAA.feature to be run first.
Later, we discovered that this setup was brittle, and decided to use the following trick, inspired by the PageObject pattern:
Add a background with the single line Given(~'^I set test data for all feature files$')
In the step definition, have a factory to create the test data, and make sure inside the factore method, that it is only created once, like testFactory.createTestData()
In this way, you have both the convenience of expressing reference setup as a scenario, that enhances team communication, but you also have a stable test setup.
Hope this is helpful!
Agata
I am looking for specific instructions on how to override default values in a third party cookbook. For example, i am using apache_spark cookbook (https://github.com/clearstorydata-cookbooks/apache_spark)
And i want to override the attribute
default['apache_spark']['standalone']['master_host']
I tried it my making a main recipe, in which i add
node.default['apache_spark']['standalone']['master_host'] = 'foo.com'
And execute it using chef solo like:
run_list(
'recipe[main]',
'recipe[apache_spark::spark-standalone-worker]'
)
But that does not seem to work. Any suggestions on how it needs to be done?
My main recipe is here https://github.com/Vibhuti/chef-main
The correct fix would be to make a wrapper cookbook and set your values in the cookbook's attributes file (main/attributes/default.rb):
override['apache_spark']['standalone']['master_host'] = 'foo.com'
Also make sure you add a dependency in main's metadata.rb to force the load ordering to be correct:
depends 'apache_spark'