So I have a Jenkins Pipeline which reads a text file (JSON) using the readFile method provided by Jenkins Pipeline. The text file app.JSON has multiple variables which are already defined in the Jenkins Pipeline.
While the readFile does read the file and convert into string it does not interpolate these variables. What are my options to interpolate these variables besides a simple string replace (Which I want to avoid)
I know I can use the readJSON or JSON parser but I want the output in string so it makes it easier for me to just read it as string and pass it along.
I have tried using Gstrings, ${-> variable} and .toString() method. Nothing worked for me.
Jenkins Pipeline Code
appServerName = 'gaga'
def appMachine = readFile file: 'util-silo-create-v2/app.json'
println appMachine
app.json
{
"name":"${appServerName}",
"fqdn":"${appServerName}"
}
There are more than one variable in both the pipeline and app.json that I want to substitute
The issue is with the readFile method provided by Jenkins Pipeline. Although it is very neat and easy to use it does not interpolate strings.
I expect below output
println appMachine
{
"name":"gaga",
"fqdn":"gaga"
}
Output I am getting
{
"name":"${appServerName}",
"fqdn":"${appServerName}"
}
Your assumption that readFile step (or any other method that reads content from a text file) should bind the variables from the current scope and interpolate variables placeholders in the raw text is wrong. However, you can use Groovy template engine to invoke something similar to GString variables interpolation. Consider the following example:
import groovy.text.SimpleTemplateEngine
def jsonText = '''{
"name":"${appServerName}",
"fqdn":"${appServerName}"
}'''
#NonCPS
def parseJsonWithVariables(String json, Map variables) {
def template = new SimpleTemplateEngine()
return template.createTemplate(json).make(variables.withDefault { it -> "\${$it}" }).toString()
}
node {
stage("Test") {
def parsed = parseJsonWithVariables(jsonText, [
appServerName: "gaga"
])
echo parsed
}
}
The method parseJsonWithVariables does what you expect to get. It is critical to make this method #NonCPS, because the SimpleTemplateEngine, as well as map created using withDefault() are not serializable. It takes a JSON read previously from a file (in this example I use a variable instead for simplicity) and a map of parameters. It converts this map to a map with a default value (the part variables.withDefault { ... } is responsible for that) so the template engine does not complain that there is no property with a given name. In this case the default method returns a variable "as is", but you can return an empty string or a null value instead. Whatever works for you better.
When you run it you will something like this:
[Pipeline] Start of Pipeline (hide)
[Pipeline] node
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Test)
[Pipeline] echo
{
"name":"gaga",
"fqdn":"gaga"
}
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
Related
I'm Jenkins a beginner and I'm trying to write a jenkins scripted pipeline to dynamically fetch git urls of repos under a given Gitlab group.
I'm using gitlab4j-api which unfortunately does not provide a straightforward way to get all projects in a group and it's sub-groups.
So I'm writing a function to recursively flatmap sub-groups when found into projects, inspired by this link which does basically the same but for files and folders :
https://nurkiewicz.com/2014/07/turning-recursive-file-system-traversal.html
The issue is the script I wrote works fine when run in jenkins console, but when embedded in my pipeline it gives below error. Here's my code :
import java.util.stream.Stream
import org.gitlab4j.api.GitLabApi
import org.gitlab4j.api.models.Group
import org.gitlab4j.api.models.Project
url = "https://gitlab.domain.com"
token = "xxxxxxx"
groupPath = 'MY/GROUP'
gitLabApi = new GitLabApi(url,token)
println("Connected to Gitlab, version: "+ gitLabApi.getApiVersion())
getProjectsRecursive(groupPath).each { project ->
println project.getSshUrlToRepo()
}
def getProjectsRecursive(path) {
//println 'getProjectsRecursive : ' + path
return getDirectProjectsAndSubGroups(path).flatMap { projectOrSubgroup ->
projectOrSubgroup instanceof Group ?
getProjectsRecursive(((Group)projectOrSubgroup).getFullPath()) :
Stream.of((Project) projectOrSubgroup)
}
}
def getDirectProjectsAndSubGroups(path2) {
//println 'getDirectProjectsAndSubGroups : ' + path2
def projectsAndGroups = []
projectsAndGroups.addAll(gitLabApi.getGroupApi().getProjectsStream(path2).toList())
projectsAndGroups.addAll(gitLabApi.getGroupApi().getSubGroupsStream(path2).toList())
return projectsAndGroups.stream()
}
and here's the error :
[Pipeline] Start of Pipeline
[Pipeline] echo
Connected to Gitlab, version: V4
[Pipeline] End of Pipeline
hudson.remoting.ProxyException: groovy.lang.MissingMethodException: No signature of method: java.util.stream.ReferencePipeline$Head.flatMap() is applicable for argument types: (org.jenkinsci.plugins.workflow.cps.CpsClosure2) values: [org.jenkinsci.plugins.workflow.cps.CpsClosure2#2fd9965c]
at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:158)
at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:178)
at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:182)
at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
at WorkflowScript.getProjectsRecursive(WorkflowScript:19)
at WorkflowScript.run(WorkflowScript:13)
at ___cps.transform___(Native Method)
at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:90)
Edit : As suggested by #MattSchuchard adding #NonCPS on both functions seems to be the right thing to do, but unfortunately this raises another error :
[Pipeline] Start of Pipeline
[Pipeline] echo
Connected to Gitlab, version: V4
[Pipeline] End of Pipeline
hudson.remoting.ProxyException: groovy.lang.MissingMethodException: No signature of method: java.util.stream.ReferencePipeline$Head.flatMap() is applicable for argument types: (WorkflowScript$_getProjectsRecursive_closure1) values: [WorkflowScript$_getProjectsRecursive_closure1#5795b4cf]
Context
I'm running Jenkins on Windows, writing declarative pipelines. I'm trying to put multiple commands in a single bat step, while still making the step fail if any of the included commands fail.
Purpose of this is twofold.
The best practices document suggests that creating a step for every little thing might not be the best idea (might also be solved by putting more stuff in batch files, but my builds aren't that big yet)
I want to execute some commands in a Visual Studio command prompt, which is achieved by first calling a .bat file that sets the environment, and then doing any necessary commands.
Code
I wrote the following Groovy code in my Jenkinsfile:
def ExecuteMultipleCmdSteps(String... steps)
{
bat ConcatenateMultipleCmdSteps(steps)
}
String ConcatenateMultipleCmdSteps(String... steps)
{
String[] commands = []
steps.each { commands +="echo ^> Now starting: ${it}"; commands += it; }
return commands.join(" && ")
}
The problem/question
I can't get this to work reliably. That is, in a single Jenkinsfile, I can have multiple calls to ExecuteMultipleCmdSteps(), and some will work as intended, while others will fail with java.lang.NoSuchMethodError: No such DSL method 'ExecuteMultipleCmdSteps' found among steps [addBadge, ...
I have not yet found any pattern in the failures. I thought it only failed when executing from within a warnError block, but now I also have a problem from within a dir() block, while in a different Jenkinsfile, that works fine.
This problem seems to be related to ExecuteMultipleCmdSteps() being a variadic function. If I provide an overload with the correct number of arguments, then that overload is used without problem.
I'm at a loss here. Your input would be most welcome.
Failed solution
At some point I thought it might be a scoping/importing thing, so I enclosed ExecuteMultipleCmdSteps() in a class (code below) as suggested by this answer. Now, the method is called as Helpers.ExecuteMultipleCmdSteps(), and that results in a org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: No such static method found: staticMethod Helpers ExecuteMultipleCmdSteps org.codehaus.groovy.runtime.GStringImpl org.codehaus.groovy.runtime.GStringImpl
public class Helpers {
public static environment
public static void ExecuteMultipleCmdSteps(String... steps)
{
environment.bat ConcatenateMultipleCmdSteps(steps)
}
public static String ConcatenateMultipleCmdSteps(String... steps)
{
String[] commands = []
steps.each { commands +="echo ^> Now starting: ${it}"; commands += it; }
return commands.join(" && ")
}
Minimal failing example
Consider the following:
hello = "Hello"
pipeline {
agent any
stages {
stage("Stage") {
steps {
SillyEcho("Hello")
SillyEcho("${hello}" as String)
SillyEcho("${hello}")
}
}
}
}
def SillyEcho(String... m)
{
echo m.join(" ")
}
I'd expect all calls to SillyEcho() to result in Hello being echoed. In reality, the first two succeed, and the last one results in java.lang.NoSuchMethodError: No such DSL method 'SillyEcho' found among steps [addBadge, addErrorBadge,...
Curiously succeeding example
Consider the following groovy script, pretty much equivalent to the failing example above:
hello = "Hello"
SillyEcho("Hello")
SillyEcho("${hello}" as String)
SillyEcho("${hello}")
def SillyEcho(String... m)
{
println m.join(" ")
}
When pasted into a Groovy Script console (for example the one provided by Jenkins), this succeeds (Hello is printed three times).
Even though I'd expect this example to succeed, I'd also expect it to behave consistently with the failing example, above, so I'm a bit torn on this one.
Thank you for adding the failing and succeeding examples.
I expect your issues are due to the incompatibility of String and GString.
With respect to the differences between running it as a pipeline job and running the script in the Jenkins Script Console, I assume based on this that the Jenkins Script Console is not as strict with type references or tries to cast parameters based upon the function signature. I base this assumption on this script, based upon your script:
hello = "Hello"
hello2 = "${hello}" as String
hello3 = "${hello}"
println hello.getClass()
println hello2.getClass()
println hello3.getClass()
SillyEcho(hello)
SillyEcho(hello2)
SillyEcho(hello3)
def SillyEcho(String... m)
{
println m.getClass()
}
This is the output I got in the Jenkins Script Console:
class java.lang.String
class java.lang.String
class org.codehaus.groovy.runtime.GStringImpl
class [Ljava.lang.String;
class [Ljava.lang.String;
class [Ljava.lang.String;
I expect the pipeline doesn't cast the GString to String but just fails as there is no function with the Gstring as parameter.
For debugging you could try to invoke .toString() an all elements you pass on to your function.
Update
This seems to be a known issue (or at least reported) with the pipeline interpreter: JENKINS-56758.
In the ticket an extra work-around has been described using collections instead of varargs. This would omit the need to type-cast everything.
Not sure if this will answer your question, if not, consider it as a bigger comment.
I like how you borrowed the 'variadic functions' from C++ :)
However, in groovy there is much elegant way how to deal with this.
Try this:
def ExecuteMultipleCmdSteps(steps)
{
sh steps
.collect { "echo \\> Now starting: $it && $it" }
.join(" && ")
}
pipeline {
agent any
stages {
stage ("test") {
steps {
ExecuteMultipleCmdSteps(["pwd", "pwd"])
}
}
}
}
which works just fine for me:
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/TestJob
[Pipeline] {
[Pipeline] stage
[Pipeline] { (test)
[Pipeline] sh
+ echo > Now starting: pwd
> Now starting: pwd
+ pwd
/var/lib/jenkins/workspace/TestJob
+ echo > Now starting: pwd
> Now starting: pwd
+ pwd
/var/lib/jenkins/workspace/TestJob
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
You may want to rewrite your function like this.
The 2 errors you mention may have different causes.
The fist one, "No such DSL method ..." is indeed a scoping one, you found yourself the solution, but I do not understand why the overload works in the same scope.
The second error, may be solved with this answer. However, for me your code from the second approach works also just fine.
My Jenkins pipeline shared library can send notification on demand. The user basically has to send the the notifcation channel details like slack channel name or email id for each stage that he wants notifcation.
I do not want user to repeat this property in every stage but rather define it once in Jenkinsfile and I can use that. What would be the best way to set this variable?
Sample:
// this properties needs to be accessed by my groovy files
def properties = "channelNm,token"
node('myNode'){
stage('checkout'){
slackNotify = "yes"
.....
}
stage('compile'){
slackNotify = "yes"
.....
}
}
When you use Jenkins shared library you can create a configuration class and you can expose a DSL script that allows you modifying your configuration object. Take a look at following example. Let's say you have a class called NotificationConfig located in src folder:
src/NotificationConfig.groovy
#Singleton
class NotificationConfig {
String slackChannelName
String email
String otherStuff
}
This class is a singleton which means that you can get instance (a single one) of this with NotificationConfig.instance. Now, let's say you have a DSL script called notificationConfig.groovy located in vars folder:
vars/notificationConfig.groovy
#!groovy
def call(Closure body) {
body.resolveStrategy = Closure.DELEGATE_FIRST
body.delegate = NotificationConfig.instance
body()
}
This is a very simple script that delegates closure body to be executed in context of NotificationConfig object. Now let's take a look at very simple scripted pipeline that uses notificationConfig DSL to set some configuration values:
Jenkinsfile
notificationConfig {
email = 'test#test.com'
slackChannelName = 'test'
}
node {
stage('Test') {
echo NotificationConfig.instance.email
}
}
When I run this simple pipeline I see:
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/test-pipeline
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Test)
[Pipeline] echo
test#test.com
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
As you can see you can expose with such DSL NotificationConfig object and let pipeline to define its own values and then access these values in the shared library with NotificationConfig.instance object.
ATTENTION: you can always set up some default values in NotificationConfig object so library users can override them in their pipelines or rely on defaults if needed.
This is pretty popular pattern in Jenkins Pipeline shared libraries. You can read more about it in this Jenkins blog post - https://jenkins.io/blog/2017/10/02/pipeline-templates-with-shared-libraries/
I have defined a helper file for my logging purposes as such:
logger.groovy
import groovy.transform.Field
#Field color = [
reset : "\033[0m",
red : "\033[31m",
green : "\033[32m",
yellow : "\033[33m",
blue : "\033[34m",
magenta : "\033[35m",
cyan : "\033[36m"
]
def info(String str)
{
ansiColor()
{
println("${color.green}${str}")
}
}
I am then using it in my main build file as such:
logger.info("Logging at info level")
logger.info("another log message")
Doing so correctly shows my text as green (can't mimic here), but it's also printing a lot of junk:
Output:
[Pipeline] ansiColor
[Pipeline] {
[Pipeline] echo
Logging at info level
[Pipeline] }
[Pipeline] // ansiColor
[Pipeline] ansiColor
[Pipeline] {
[Pipeline] echo
another log message
[Pipeline] }
[Pipeline] // ansiColor
How can I get it to disable all the extraneous source lines? I would like to keep that feature enabled in my main script, I just really don't need it every time I want to log something.
I have tried putting in the shebangs at the top of the logger.groovy file, but to no avail. I also have my jenkins instance running on Windows, which might help explain why. I have also tried putting set +x in the info() method, but that only results in the following exception, which is likely due to the sandbox environment
groovy.lang.MissingPropertyException: No such property: set for class: logger
I'm not sure why I'm having such a hard time with this. Do other people just deal with it or am I doing something out of the ordinary?
https://github.com/jenkinsci/ansicolor-plugin#using-in-pipeline-workflows
TERM is a variable that's set to the color mapping when the ansiColor block is in scope. If TERM is not null, we're already inside an ansiColor block so we don't need to re-add it.
So here is my work around for the ansicolor logging:
def info(String str) {
if (env.TERM) {
println("${color.green}${str}")
} else {
ansiColor {
println("${color.green}${str}")
}
}
}
I don't know if it's possible to get rid of the logging of "echo"
The output from println from within a class function is lost.
An example script (outputclass.groovy):
class OutputClass
{
OutputClass()
{
println("Inside class") // This will not show in the console
}
}
println("Outside class") // Only this is shown in the console
output = new OutputClass()
I use Jenkins CLI to execute the groovy script
java -jar ..\jenkins-cli.jar -s JENKINS_SERVER_URL groovy outputclass.groovy
It only outputs this:
Outside class
It seems like the class inmplicitly uses println from System.out.println, and System.out is directed to the log files, but the println outside the class is using something else, which is outputted in the script console. The following code shows the behavior.
System.out.println("First")
println("Second")
Output:
Second
How do I explicitly set the output device to output to the Jenkins script console?
I found the solution myself here http://mriet.wordpress.com.
When the Groovy plugin starts is passes two bindings to the script. From the bindings we can get the out variable. Get it and use out.println to output to the script console, not the plain println.
The script below shows full solution.
import hudson.model.*
// Get the out variable
def out = getBinding().out;
class OutputClass
{
OutputClass(out) // Have to pass the out variable to the class
{
out.println ("Inside class")
}
}
out.println("Outside class")
output = new OutputClass(out)
If you use the skript as a post build step (I'm not shure whether it works with the mentioned CLI) you can use the build in logger:
manager.listener.logger.println("some output")
So in your case something like this may be helpful:
class OutputClass
{
OutputClass(logger) // Have to pass the out variable to the class
{
logger.println ("Inside class")
}
}
output = new OutputClass(manager.listener.logger)
See also Example 10 in Groovy Plugin Doc
Does this mailing list post help?
the output is sent to standard output, so if you check your log file, you
will probably see something like this: INFO [STDOUT] Hello World
if you insist on using system script, you have to pass out variable to
your class, as the binding is not visible inside the class (so it's
passed to standard output). You should use something like this
public class Hello {
static void say(out) {
out << "Hello World "
}
}
println "Started ..."
Hello.say(out)
A simple solution that worked well for me was to add this line on top of each script. This enables usage of traditional println commands all over the code (inside and outside of classes) leaving the code intuitive.
import hudson.model.*
System.out = getBinding().out;
This enables to create log entries like this:
println("Outside class");
class OutputClass {
OutputClass() {
println ("Inside class")
}
}
new OutputClass();
It replaces the default print stream in System.out with the one handed over from Jenkins via bindings.