Best way to set reusable property in Jenkinsfile - groovy

My Jenkins pipeline shared library can send notification on demand. The user basically has to send the the notifcation channel details like slack channel name or email id for each stage that he wants notifcation.
I do not want user to repeat this property in every stage but rather define it once in Jenkinsfile and I can use that. What would be the best way to set this variable?
Sample:
// this properties needs to be accessed by my groovy files
def properties = "channelNm,token"
node('myNode'){
stage('checkout'){
slackNotify = "yes"
.....
}
stage('compile'){
slackNotify = "yes"
.....
}
}

When you use Jenkins shared library you can create a configuration class and you can expose a DSL script that allows you modifying your configuration object. Take a look at following example. Let's say you have a class called NotificationConfig located in src folder:
src/NotificationConfig.groovy
#Singleton
class NotificationConfig {
String slackChannelName
String email
String otherStuff
}
This class is a singleton which means that you can get instance (a single one) of this with NotificationConfig.instance. Now, let's say you have a DSL script called notificationConfig.groovy located in vars folder:
vars/notificationConfig.groovy
#!groovy
def call(Closure body) {
body.resolveStrategy = Closure.DELEGATE_FIRST
body.delegate = NotificationConfig.instance
body()
}
This is a very simple script that delegates closure body to be executed in context of NotificationConfig object. Now let's take a look at very simple scripted pipeline that uses notificationConfig DSL to set some configuration values:
Jenkinsfile
notificationConfig {
email = 'test#test.com'
slackChannelName = 'test'
}
node {
stage('Test') {
echo NotificationConfig.instance.email
}
}
When I run this simple pipeline I see:
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/test-pipeline
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Test)
[Pipeline] echo
test#test.com
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
As you can see you can expose with such DSL NotificationConfig object and let pipeline to define its own values and then access these values in the shared library with NotificationConfig.instance object.
ATTENTION: you can always set up some default values in NotificationConfig object so library users can override them in their pipelines or rely on defaults if needed.
This is pretty popular pattern in Jenkins Pipeline shared libraries. You can read more about it in this Jenkins blog post - https://jenkins.io/blog/2017/10/02/pipeline-templates-with-shared-libraries/

Related

Script to get projects urls using Gitlab API works in Jenkins Console but not in my scripted pipeline

I'm Jenkins a beginner and I'm trying to write a jenkins scripted pipeline to dynamically fetch git urls of repos under a given Gitlab group.
I'm using gitlab4j-api which unfortunately does not provide a straightforward way to get all projects in a group and it's sub-groups.
So I'm writing a function to recursively flatmap sub-groups when found into projects, inspired by this link which does basically the same but for files and folders :
https://nurkiewicz.com/2014/07/turning-recursive-file-system-traversal.html
The issue is the script I wrote works fine when run in jenkins console, but when embedded in my pipeline it gives below error. Here's my code :
import java.util.stream.Stream
import org.gitlab4j.api.GitLabApi
import org.gitlab4j.api.models.Group
import org.gitlab4j.api.models.Project
url = "https://gitlab.domain.com"
token = "xxxxxxx"
groupPath = 'MY/GROUP'
gitLabApi = new GitLabApi(url,token)
println("Connected to Gitlab, version: "+ gitLabApi.getApiVersion())
getProjectsRecursive(groupPath).each { project ->
println project.getSshUrlToRepo()
}
def getProjectsRecursive(path) {
//println 'getProjectsRecursive : ' + path
return getDirectProjectsAndSubGroups(path).flatMap { projectOrSubgroup ->
projectOrSubgroup instanceof Group ?
getProjectsRecursive(((Group)projectOrSubgroup).getFullPath()) :
Stream.of((Project) projectOrSubgroup)
}
}
def getDirectProjectsAndSubGroups(path2) {
//println 'getDirectProjectsAndSubGroups : ' + path2
def projectsAndGroups = []
projectsAndGroups.addAll(gitLabApi.getGroupApi().getProjectsStream(path2).toList())
projectsAndGroups.addAll(gitLabApi.getGroupApi().getSubGroupsStream(path2).toList())
return projectsAndGroups.stream()
}
and here's the error :
[Pipeline] Start of Pipeline
[Pipeline] echo
Connected to Gitlab, version: V4
[Pipeline] End of Pipeline
hudson.remoting.ProxyException: groovy.lang.MissingMethodException: No signature of method: java.util.stream.ReferencePipeline$Head.flatMap() is applicable for argument types: (org.jenkinsci.plugins.workflow.cps.CpsClosure2) values: [org.jenkinsci.plugins.workflow.cps.CpsClosure2#2fd9965c]
at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:158)
at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:178)
at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:182)
at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
at WorkflowScript.getProjectsRecursive(WorkflowScript:19)
at WorkflowScript.run(WorkflowScript:13)
at ___cps.transform___(Native Method)
at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:90)
Edit : As suggested by #MattSchuchard adding #NonCPS on both functions seems to be the right thing to do, but unfortunately this raises another error :
[Pipeline] Start of Pipeline
[Pipeline] echo
Connected to Gitlab, version: V4
[Pipeline] End of Pipeline
hudson.remoting.ProxyException: groovy.lang.MissingMethodException: No signature of method: java.util.stream.ReferencePipeline$Head.flatMap() is applicable for argument types: (WorkflowScript$_getProjectsRecursive_closure1) values: [WorkflowScript$_getProjectsRecursive_closure1#5795b4cf]

Calling a variadic function in a Jenkinsfile fails unpredictably

Context
I'm running Jenkins on Windows, writing declarative pipelines. I'm trying to put multiple commands in a single bat step, while still making the step fail if any of the included commands fail.
Purpose of this is twofold.
The best practices document suggests that creating a step for every little thing might not be the best idea (might also be solved by putting more stuff in batch files, but my builds aren't that big yet)
I want to execute some commands in a Visual Studio command prompt, which is achieved by first calling a .bat file that sets the environment, and then doing any necessary commands.
Code
I wrote the following Groovy code in my Jenkinsfile:
def ExecuteMultipleCmdSteps(String... steps)
{
bat ConcatenateMultipleCmdSteps(steps)
}
String ConcatenateMultipleCmdSteps(String... steps)
{
String[] commands = []
steps.each { commands +="echo ^> Now starting: ${it}"; commands += it; }
return commands.join(" && ")
}
The problem/question
I can't get this to work reliably. That is, in a single Jenkinsfile, I can have multiple calls to ExecuteMultipleCmdSteps(), and some will work as intended, while others will fail with java.lang.NoSuchMethodError: No such DSL method 'ExecuteMultipleCmdSteps' found among steps [addBadge, ...
I have not yet found any pattern in the failures. I thought it only failed when executing from within a warnError block, but now I also have a problem from within a dir() block, while in a different Jenkinsfile, that works fine.
This problem seems to be related to ExecuteMultipleCmdSteps() being a variadic function. If I provide an overload with the correct number of arguments, then that overload is used without problem.
I'm at a loss here. Your input would be most welcome.
Failed solution
At some point I thought it might be a scoping/importing thing, so I enclosed ExecuteMultipleCmdSteps() in a class (code below) as suggested by this answer. Now, the method is called as Helpers.ExecuteMultipleCmdSteps(), and that results in a org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: No such static method found: staticMethod Helpers ExecuteMultipleCmdSteps org.codehaus.groovy.runtime.GStringImpl org.codehaus.groovy.runtime.GStringImpl
public class Helpers {
public static environment
public static void ExecuteMultipleCmdSteps(String... steps)
{
environment.bat ConcatenateMultipleCmdSteps(steps)
}
public static String ConcatenateMultipleCmdSteps(String... steps)
{
String[] commands = []
steps.each { commands +="echo ^> Now starting: ${it}"; commands += it; }
return commands.join(" && ")
}
Minimal failing example
Consider the following:
hello = "Hello"
pipeline {
agent any
stages {
stage("Stage") {
steps {
SillyEcho("Hello")
SillyEcho("${hello}" as String)
SillyEcho("${hello}")
}
}
}
}
def SillyEcho(String... m)
{
echo m.join(" ")
}
I'd expect all calls to SillyEcho() to result in Hello being echoed. In reality, the first two succeed, and the last one results in java.lang.NoSuchMethodError: No such DSL method 'SillyEcho' found among steps [addBadge, addErrorBadge,...
Curiously succeeding example
Consider the following groovy script, pretty much equivalent to the failing example above:
hello = "Hello"
SillyEcho("Hello")
SillyEcho("${hello}" as String)
SillyEcho("${hello}")
def SillyEcho(String... m)
{
println m.join(" ")
}
When pasted into a Groovy Script console (for example the one provided by Jenkins), this succeeds (Hello is printed three times).
Even though I'd expect this example to succeed, I'd also expect it to behave consistently with the failing example, above, so I'm a bit torn on this one.
Thank you for adding the failing and succeeding examples.
I expect your issues are due to the incompatibility of String and GString.
With respect to the differences between running it as a pipeline job and running the script in the Jenkins Script Console, I assume based on this that the Jenkins Script Console is not as strict with type references or tries to cast parameters based upon the function signature. I base this assumption on this script, based upon your script:
hello = "Hello"
hello2 = "${hello}" as String
hello3 = "${hello}"
println hello.getClass()
println hello2.getClass()
println hello3.getClass()
SillyEcho(hello)
SillyEcho(hello2)
SillyEcho(hello3)
def SillyEcho(String... m)
{
println m.getClass()
}
This is the output I got in the Jenkins Script Console:
class java.lang.String
class java.lang.String
class org.codehaus.groovy.runtime.GStringImpl
class [Ljava.lang.String;
class [Ljava.lang.String;
class [Ljava.lang.String;
I expect the pipeline doesn't cast the GString to String but just fails as there is no function with the Gstring as parameter.
For debugging you could try to invoke .toString() an all elements you pass on to your function.
Update
This seems to be a known issue (or at least reported) with the pipeline interpreter: JENKINS-56758.
In the ticket an extra work-around has been described using collections instead of varargs. This would omit the need to type-cast everything.
Not sure if this will answer your question, if not, consider it as a bigger comment.
I like how you borrowed the 'variadic functions' from C++ :)
However, in groovy there is much elegant way how to deal with this.
Try this:
def ExecuteMultipleCmdSteps(steps)
{
sh steps
.collect { "echo \\> Now starting: $it && $it" }
.join(" && ")
}
pipeline {
agent any
stages {
stage ("test") {
steps {
ExecuteMultipleCmdSteps(["pwd", "pwd"])
}
}
}
}
which works just fine for me:
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/TestJob
[Pipeline] {
[Pipeline] stage
[Pipeline] { (test)
[Pipeline] sh
+ echo > Now starting: pwd
> Now starting: pwd
+ pwd
/var/lib/jenkins/workspace/TestJob
+ echo > Now starting: pwd
> Now starting: pwd
+ pwd
/var/lib/jenkins/workspace/TestJob
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
You may want to rewrite your function like this.
The 2 errors you mention may have different causes.
The fist one, "No such DSL method ..." is indeed a scoping one, you found yourself the solution, but I do not understand why the overload works in the same scope.
The second error, may be solved with this answer. However, for me your code from the second approach works also just fine.

How to get the name of locked resource in Jenkins Lockable Resource plugin

I am using Jenkins Lockable Resources plugin to decide which server to be used for various build operations in my declarative pipeline. I have set up my Lockable Resources as shown in the table below:
Resource Name Labels
Win_Res_1 Windows
Win_Res_2 Windows
Win_Res_3 Windows
Lx_Res_1 Linux
Lx_Res_2 Linux
Lx_Res_3 Linux
I want to lock a label and then get the name of the corresponding locked resource.
I'm writing following code but am not able to get the desired value of r
int num_resources = 1;
def label = "Windows"; /* I have hardcoded it here for simplicity. In actual code, I will get ${lable} from my code and its value can be either Windows or Linux. */
lock(label: "${label}", quantity: num_resources)
{
def r = org.jenkins.plugins.lockableresources.LockableResourcesManager; /* I know this is incomplete but unable to get correct function call */
println (" Locked resource r is : ${r} \n");
/* r should be name of resource for example Win_Res_1. */
}
The documentation for Lockable Resources Plugin is available here: https://jenkins.io/doc/pipeline/steps/lockable-resources/
and
https://plugins.jenkins.io/lockable-resources/
You can get the name of locked resource using variable parameter of the lock workflow step. This option defines the name of the environment variable that will store the name of the locked resource. Consider the following example.
pipeline {
agent any
stages {
stage("Lock resource") {
steps {
script {
int num = 1
String label = "Windows"
lock(label: label, quantity: num, variable: "resource_name") {
echo "Locked resource name is ${env.resource_name}"
}
}
}
}
}
}
In this example, the locked resource name can be accessed using env.resource_name variable. Here is the output of the pipeline run.
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Lock resource)
[Pipeline] script
[Pipeline] {
[Pipeline] lock
Trying to acquire lock on [Label: Windows, Quantity: 1]
Lock acquired on [Label: Windows, Quantity: 1]
[Pipeline] {
[Pipeline] echo
Locked resource name is Win_Res_1
[Pipeline] }
Lock released on resource [Label: Windows, Quantity: 1]
[Pipeline] // lock
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
You can see that the value Win_Res_1 was assigned to the env.resource_name variable.

Passing values from child declarative Jenkins pipeline to the parent

I have a declarative pipeline setup where the parent pipeline kicks off child pipeline via the "build job: workflow('workflow-name')" and i'm passing the parameters via the "parameters" directive
The question is, in the child pipeline in one of the stage i'm spawning a shell and writing few values to a file and via the readFile method the contents of the file is read and is set in a groovy variable defined at the top of the child pipeline.
This groovy variable (myVal) is visible in all the stages in the child pipeline, but I need to use the myVal in the parent pipeline,
Question 1 - Will the myVal be accessible in parent pipeline?
Question 2 - If it is not accessible then how can i access it, is it even viable?
As you can see, the child pipeline runs in a container but the parent pipeline will not, i.e. the agent on the parent pipeline is different at that of the child.
def myVal = ''
pipeline {
agent {
docker {
image 'myDockerImage'
label 'myRemoteVM'
args '-v /home/myuser:/home/myuser'
}
}
stages {
stage('step1') {
steps {
script {
sh '''
./myScript.sh
'''
myVal = readFile('myFileName.txt').trim()
}
}
}
}
}
}

String Interpolation in Groovy with Jenkins Pipeline file not working

So I have a Jenkins Pipeline which reads a text file (JSON) using the readFile method provided by Jenkins Pipeline. The text file app.JSON has multiple variables which are already defined in the Jenkins Pipeline.
While the readFile does read the file and convert into string it does not interpolate these variables. What are my options to interpolate these variables besides a simple string replace (Which I want to avoid)
I know I can use the readJSON or JSON parser but I want the output in string so it makes it easier for me to just read it as string and pass it along.
I have tried using Gstrings, ${-> variable} and .toString() method. Nothing worked for me.
Jenkins Pipeline Code
appServerName = 'gaga'
def appMachine = readFile file: 'util-silo-create-v2/app.json'
println appMachine
app.json
{
"name":"${appServerName}",
"fqdn":"${appServerName}"
}
There are more than one variable in both the pipeline and app.json that I want to substitute
The issue is with the readFile method provided by Jenkins Pipeline. Although it is very neat and easy to use it does not interpolate strings.
I expect below output
println appMachine
{
"name":"gaga",
"fqdn":"gaga"
}
Output I am getting
{
"name":"${appServerName}",
"fqdn":"${appServerName}"
}
Your assumption that readFile step (or any other method that reads content from a text file) should bind the variables from the current scope and interpolate variables placeholders in the raw text is wrong. However, you can use Groovy template engine to invoke something similar to GString variables interpolation. Consider the following example:
import groovy.text.SimpleTemplateEngine
def jsonText = '''{
"name":"${appServerName}",
"fqdn":"${appServerName}"
}'''
#NonCPS
def parseJsonWithVariables(String json, Map variables) {
def template = new SimpleTemplateEngine()
return template.createTemplate(json).make(variables.withDefault { it -> "\${$it}" }).toString()
}
node {
stage("Test") {
def parsed = parseJsonWithVariables(jsonText, [
appServerName: "gaga"
])
echo parsed
}
}
The method parseJsonWithVariables does what you expect to get. It is critical to make this method #NonCPS, because the SimpleTemplateEngine, as well as map created using withDefault() are not serializable. It takes a JSON read previously from a file (in this example I use a variable instead for simplicity) and a map of parameters. It converts this map to a map with a default value (the part variables.withDefault { ... } is responsible for that) so the template engine does not complain that there is no property with a given name. In this case the default method returns a variable "as is", but you can return an empty string or a null value instead. Whatever works for you better.
When you run it you will something like this:
[Pipeline] Start of Pipeline (hide)
[Pipeline] node
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Test)
[Pipeline] echo
{
"name":"gaga",
"fqdn":"gaga"
}
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

Resources