Context
I'm running Jenkins on Windows, writing declarative pipelines. I'm trying to put multiple commands in a single bat step, while still making the step fail if any of the included commands fail.
Purpose of this is twofold.
The best practices document suggests that creating a step for every little thing might not be the best idea (might also be solved by putting more stuff in batch files, but my builds aren't that big yet)
I want to execute some commands in a Visual Studio command prompt, which is achieved by first calling a .bat file that sets the environment, and then doing any necessary commands.
Code
I wrote the following Groovy code in my Jenkinsfile:
def ExecuteMultipleCmdSteps(String... steps)
{
bat ConcatenateMultipleCmdSteps(steps)
}
String ConcatenateMultipleCmdSteps(String... steps)
{
String[] commands = []
steps.each { commands +="echo ^> Now starting: ${it}"; commands += it; }
return commands.join(" && ")
}
The problem/question
I can't get this to work reliably. That is, in a single Jenkinsfile, I can have multiple calls to ExecuteMultipleCmdSteps(), and some will work as intended, while others will fail with java.lang.NoSuchMethodError: No such DSL method 'ExecuteMultipleCmdSteps' found among steps [addBadge, ...
I have not yet found any pattern in the failures. I thought it only failed when executing from within a warnError block, but now I also have a problem from within a dir() block, while in a different Jenkinsfile, that works fine.
This problem seems to be related to ExecuteMultipleCmdSteps() being a variadic function. If I provide an overload with the correct number of arguments, then that overload is used without problem.
I'm at a loss here. Your input would be most welcome.
Failed solution
At some point I thought it might be a scoping/importing thing, so I enclosed ExecuteMultipleCmdSteps() in a class (code below) as suggested by this answer. Now, the method is called as Helpers.ExecuteMultipleCmdSteps(), and that results in a org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: No such static method found: staticMethod Helpers ExecuteMultipleCmdSteps org.codehaus.groovy.runtime.GStringImpl org.codehaus.groovy.runtime.GStringImpl
public class Helpers {
public static environment
public static void ExecuteMultipleCmdSteps(String... steps)
{
environment.bat ConcatenateMultipleCmdSteps(steps)
}
public static String ConcatenateMultipleCmdSteps(String... steps)
{
String[] commands = []
steps.each { commands +="echo ^> Now starting: ${it}"; commands += it; }
return commands.join(" && ")
}
Minimal failing example
Consider the following:
hello = "Hello"
pipeline {
agent any
stages {
stage("Stage") {
steps {
SillyEcho("Hello")
SillyEcho("${hello}" as String)
SillyEcho("${hello}")
}
}
}
}
def SillyEcho(String... m)
{
echo m.join(" ")
}
I'd expect all calls to SillyEcho() to result in Hello being echoed. In reality, the first two succeed, and the last one results in java.lang.NoSuchMethodError: No such DSL method 'SillyEcho' found among steps [addBadge, addErrorBadge,...
Curiously succeeding example
Consider the following groovy script, pretty much equivalent to the failing example above:
hello = "Hello"
SillyEcho("Hello")
SillyEcho("${hello}" as String)
SillyEcho("${hello}")
def SillyEcho(String... m)
{
println m.join(" ")
}
When pasted into a Groovy Script console (for example the one provided by Jenkins), this succeeds (Hello is printed three times).
Even though I'd expect this example to succeed, I'd also expect it to behave consistently with the failing example, above, so I'm a bit torn on this one.
Thank you for adding the failing and succeeding examples.
I expect your issues are due to the incompatibility of String and GString.
With respect to the differences between running it as a pipeline job and running the script in the Jenkins Script Console, I assume based on this that the Jenkins Script Console is not as strict with type references or tries to cast parameters based upon the function signature. I base this assumption on this script, based upon your script:
hello = "Hello"
hello2 = "${hello}" as String
hello3 = "${hello}"
println hello.getClass()
println hello2.getClass()
println hello3.getClass()
SillyEcho(hello)
SillyEcho(hello2)
SillyEcho(hello3)
def SillyEcho(String... m)
{
println m.getClass()
}
This is the output I got in the Jenkins Script Console:
class java.lang.String
class java.lang.String
class org.codehaus.groovy.runtime.GStringImpl
class [Ljava.lang.String;
class [Ljava.lang.String;
class [Ljava.lang.String;
I expect the pipeline doesn't cast the GString to String but just fails as there is no function with the Gstring as parameter.
For debugging you could try to invoke .toString() an all elements you pass on to your function.
Update
This seems to be a known issue (or at least reported) with the pipeline interpreter: JENKINS-56758.
In the ticket an extra work-around has been described using collections instead of varargs. This would omit the need to type-cast everything.
Not sure if this will answer your question, if not, consider it as a bigger comment.
I like how you borrowed the 'variadic functions' from C++ :)
However, in groovy there is much elegant way how to deal with this.
Try this:
def ExecuteMultipleCmdSteps(steps)
{
sh steps
.collect { "echo \\> Now starting: $it && $it" }
.join(" && ")
}
pipeline {
agent any
stages {
stage ("test") {
steps {
ExecuteMultipleCmdSteps(["pwd", "pwd"])
}
}
}
}
which works just fine for me:
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/TestJob
[Pipeline] {
[Pipeline] stage
[Pipeline] { (test)
[Pipeline] sh
+ echo > Now starting: pwd
> Now starting: pwd
+ pwd
/var/lib/jenkins/workspace/TestJob
+ echo > Now starting: pwd
> Now starting: pwd
+ pwd
/var/lib/jenkins/workspace/TestJob
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
You may want to rewrite your function like this.
The 2 errors you mention may have different causes.
The fist one, "No such DSL method ..." is indeed a scoping one, you found yourself the solution, but I do not understand why the overload works in the same scope.
The second error, may be solved with this answer. However, for me your code from the second approach works also just fine.
Related
So I have a Jenkins Pipeline which reads a text file (JSON) using the readFile method provided by Jenkins Pipeline. The text file app.JSON has multiple variables which are already defined in the Jenkins Pipeline.
While the readFile does read the file and convert into string it does not interpolate these variables. What are my options to interpolate these variables besides a simple string replace (Which I want to avoid)
I know I can use the readJSON or JSON parser but I want the output in string so it makes it easier for me to just read it as string and pass it along.
I have tried using Gstrings, ${-> variable} and .toString() method. Nothing worked for me.
Jenkins Pipeline Code
appServerName = 'gaga'
def appMachine = readFile file: 'util-silo-create-v2/app.json'
println appMachine
app.json
{
"name":"${appServerName}",
"fqdn":"${appServerName}"
}
There are more than one variable in both the pipeline and app.json that I want to substitute
The issue is with the readFile method provided by Jenkins Pipeline. Although it is very neat and easy to use it does not interpolate strings.
I expect below output
println appMachine
{
"name":"gaga",
"fqdn":"gaga"
}
Output I am getting
{
"name":"${appServerName}",
"fqdn":"${appServerName}"
}
Your assumption that readFile step (or any other method that reads content from a text file) should bind the variables from the current scope and interpolate variables placeholders in the raw text is wrong. However, you can use Groovy template engine to invoke something similar to GString variables interpolation. Consider the following example:
import groovy.text.SimpleTemplateEngine
def jsonText = '''{
"name":"${appServerName}",
"fqdn":"${appServerName}"
}'''
#NonCPS
def parseJsonWithVariables(String json, Map variables) {
def template = new SimpleTemplateEngine()
return template.createTemplate(json).make(variables.withDefault { it -> "\${$it}" }).toString()
}
node {
stage("Test") {
def parsed = parseJsonWithVariables(jsonText, [
appServerName: "gaga"
])
echo parsed
}
}
The method parseJsonWithVariables does what you expect to get. It is critical to make this method #NonCPS, because the SimpleTemplateEngine, as well as map created using withDefault() are not serializable. It takes a JSON read previously from a file (in this example I use a variable instead for simplicity) and a map of parameters. It converts this map to a map with a default value (the part variables.withDefault { ... } is responsible for that) so the template engine does not complain that there is no property with a given name. In this case the default method returns a variable "as is", but you can return an empty string or a null value instead. Whatever works for you better.
When you run it you will something like this:
[Pipeline] Start of Pipeline (hide)
[Pipeline] node
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Test)
[Pipeline] echo
{
"name":"gaga",
"fqdn":"gaga"
}
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
I have a Groovy application that can be custimized by a small Groovy DSL I wrote. On startup, the application loads several Groovy scripts, applies some AST transformations and finally executes whatever was specified in the scripts.
One of the AST transformations inserts a couple of lines of code into certain methods. That works fine and I can see the different behavior during runtime. However, sometimes the generated code is not correct. Although I load the scripts with the TypeChecked customizer in place, my generated code is never checked for soundness.
To show my problem, I constructed an extreme example. I have the following script:
int test = 10
println test // prints 10 when executed without AST
I load this script and insert a new line of code between the declaration of test and println:
public void visitBlockStatement(BlockStatement block) {
def assignment = (new AstBuilder().buildFromSpec {
binary {
variable "test"
token "="
constant 15
}
}).first()
def newStmt = new ExpressionStatement(assignment)
newStmt.setSourcePosition(block.statements[1])
block.statements.add(2, newStmt)
super.visitBlockStatement(block)
}
After applying this AST, the script prints 15. When I use AstNodeToScriptVisitor to print the Groovy code of the resulting script, I can see the new assignment added to the code.
However, if I change the value of the assignment to a String value:
// ...
def assignment = (new AstBuilder().buildFromSpec {
binary {
variable "test"
token "="
constant "some value"
}
}).first()
// ...
I get a GroovyCastExcpetion at runtime. Although the resulting script looks like this:
int test = 10
test = "some value" // no compile error but a GroovyCastException at runtime here. WHY?
println test
no error is raised by TypeChecked. I read in this mailing list, that you need to set the source position for generated code to be checked, but I'm doing that an it still doesn't work. Can anyone provide some feedback of what I am doing wrong? Thank you very much!
Update
I call the AST by attaching it to the GroovyShell like this:
def config = new CompilerConfiguration()
config.addCompilationCustomizers(
new ASTTransformationCustomizer(TypeChecked)
)
config.addCompilationCustomizers(
new ASTTransformationCustomizer(new AddAssignmentAST())
)
def shell = new GroovyShell(config)
shell.evaluate(new File("./path/to/file.groovy"))
The class for the AST itself looks like this:
#GroovyASTTransformation(phase = CompilePhase.CANONICALIZATION)
class AddAssignmentAST implements ASTTransformation {
#Override
public void visit(ASTNode[] nodes, SourceUnit source) {
def transformer = new AddAssignmentTransformer()
source.getAST().getStatementBlock().visit(transformer)
}
private class AddAssignmentTransformer extends CodeVisitorSupport {
#Override
public void visitBlockStatement(BlockStatement block) {
// already posted above
}
}
}
Since my Groovy script only consists of one block (for this small example) the visitBlockStatement method is called exactly once, adds the assignment (which I can verify since the output changes) but does not ever throw a compile-time error.
c:\>cat invalid.groovy
com.test.build(binding)
c:\junk>groovyc invalid.groovy
c:\junk>ls invalid.class
invalid.class
Why does this not result in an compilation error? There is no such method com.test.build!
In what scenario will this compilation be successful?
As Groovy is dynamic it cannot know that there will be no com.test.build at runtime
com may be an object added to the binding at a later date in the code's execution
An example:
Say we have this bit of groovy inside Script.groovy
com.test.build( binding )
Compile it with groovyc:
groovyc Script.groovy
Then, keep the Script.class file, and throw away the groovy script to make sure we are using the class file:
rm Script.groovy
Now, say we have the following in Runner.groovy
// Create a class which you can call test.build( a ) on
class Example {
Map test = [
build : { it ->
println "build called with parameter $it"
}
]
}
// Create one of our Com objects
def variable = new Example()
// Load the Script.class, and make an instance of it
def otherscript = new GroovyClassLoader().loadClass( 'Script' ).newInstance()
// Then, set `com` in the scripts binding to be our variable from above
otherscript.binding.com = variable
// Then, run the script
otherscript.run()
That prints:
build called with parameter groovy.lang.Binding#7bf35647
So as you can see, it gets the test parameter of the com object (the map above) and calls the build closure with the binding as a parameter...
Hope this makes sense...
The output from println from within a class function is lost.
An example script (outputclass.groovy):
class OutputClass
{
OutputClass()
{
println("Inside class") // This will not show in the console
}
}
println("Outside class") // Only this is shown in the console
output = new OutputClass()
I use Jenkins CLI to execute the groovy script
java -jar ..\jenkins-cli.jar -s JENKINS_SERVER_URL groovy outputclass.groovy
It only outputs this:
Outside class
It seems like the class inmplicitly uses println from System.out.println, and System.out is directed to the log files, but the println outside the class is using something else, which is outputted in the script console. The following code shows the behavior.
System.out.println("First")
println("Second")
Output:
Second
How do I explicitly set the output device to output to the Jenkins script console?
I found the solution myself here http://mriet.wordpress.com.
When the Groovy plugin starts is passes two bindings to the script. From the bindings we can get the out variable. Get it and use out.println to output to the script console, not the plain println.
The script below shows full solution.
import hudson.model.*
// Get the out variable
def out = getBinding().out;
class OutputClass
{
OutputClass(out) // Have to pass the out variable to the class
{
out.println ("Inside class")
}
}
out.println("Outside class")
output = new OutputClass(out)
If you use the skript as a post build step (I'm not shure whether it works with the mentioned CLI) you can use the build in logger:
manager.listener.logger.println("some output")
So in your case something like this may be helpful:
class OutputClass
{
OutputClass(logger) // Have to pass the out variable to the class
{
logger.println ("Inside class")
}
}
output = new OutputClass(manager.listener.logger)
See also Example 10 in Groovy Plugin Doc
Does this mailing list post help?
the output is sent to standard output, so if you check your log file, you
will probably see something like this: INFO [STDOUT] Hello World
if you insist on using system script, you have to pass out variable to
your class, as the binding is not visible inside the class (so it's
passed to standard output). You should use something like this
public class Hello {
static void say(out) {
out << "Hello World "
}
}
println "Started ..."
Hello.say(out)
A simple solution that worked well for me was to add this line on top of each script. This enables usage of traditional println commands all over the code (inside and outside of classes) leaving the code intuitive.
import hudson.model.*
System.out = getBinding().out;
This enables to create log entries like this:
println("Outside class");
class OutputClass {
OutputClass() {
println ("Inside class")
}
}
new OutputClass();
It replaces the default print stream in System.out with the one handed over from Jenkins via bindings.
How do I print a Groovy stack trace? The Java method, Thread.currentThread().getStackTrace() produces a huge stack trace, including a lot of the Groovy internals. I'm seeing a function called twice from a StreamingMarkupBuilder that looks like it should only be called once and I would like to see why Groovy thinks it should be calling it twice.
Solution:
org.codehaus.groovy.runtime.StackTraceUtils.sanitize(new Exception()).printStackTrace()
Original answer:
A Google search returns the following information:
Apparently, there is a method in org.codehaus.groovy.runtime.StackTraceUtils called printSanitizedStackTrace. There isn't much documentation for the method, though there is a method called sanitize which is described as
remove all apparently groovy-internal
trace entries from the exception
instance This modifies the original
instance and returns it, it does not
clone
So I would try org.codehaus.groovy.runtime.StackTraceUtils.printSanitizedStackTrace(Throwable t) (it is static)
and see if that works for you.
I found this questions when searching for "spock print full stack trace".
My unit tests are written in Groovy, using the Spock testing framework and they're run in the context of a Gradle build.
The fix for me was as simple as adding exceptionFormat = 'full' to my Gradle test task specification:
test {
testLogging {
exceptionFormat = 'full'
}
}
I have designed this simple code for stack trace printing, based on artificial simulation of a NullPointerException.
This code produces the same output in both modes: from a Jenkinsfile (Pipeline) and from a normal .groovy script in a command line.
def getStackTrace() {
try {
null.class.toString() // simulate NPE
} catch (NullPointerException e) {
return e.getStackTrace()
}
return null
}
def printStackTrace() {
def stackTraceStr = ""
def callingFuncFound = false
for (StackTraceElement ste : getStackTrace()) {
if (callingFuncFound) {
stackTraceStr += ste.toString() + '\n'
}
if (!callingFuncFound && ste.toString().startsWith(this.class.name + '.printStackTrace(')) {
callingFuncFound = true
}
}
println(stackTraceStr)
}
Some explanations:
The output is concatenated into a single string to avoid being mixed with "[Pipeline] echo" message prefix of Jenkins Pipeline's println()).
The number of "unnecessary" upper stack trace elements related to the NPE is different in Jenkinsfile and in a normal command line. This is why I calculate callingFuncFound and don't use just something like e.getStackTrace()[2..-1] to skip them.