I have a Jenkins job that has some weird behavior and I'm not sure what's going on.
node(){
myFuncs = [:]
String[] myFuncsToCall = ["one", "two"]
for (int i = 0; i < myFuncsToCall.size(); i++) {
func = myFuncsToCall[$i]
myFuncs[func] = {
"${func}"()
}
}
parallel myFuncs
}
def one(){
echo "one"
}
def two(){
echo "two"
}
It's interesting, because the job output shows parallel branching for both 'one' and 'two', but only calls method 'two' for both parallel branches.
I imagine there's something groovy related that I'm missing here, but am hoping it's possible to dynamically call method names like this so I can have large sets of dynamic inputs to jobs that need to run with variable inputs.
func is not a local variable but a dynamically defined property on the main script. If no type or the keyworkd def specified to the left from the var name then groovy will try to resolve it and Jenkins configures the main script in such a way that func will be assinged to the main script as a new property. Basically, it creates a global var since there is only one instance of the main script (Jenkinsfile itself).
Therefore, this global var gets changed in the loop and stores the last result only ("two"). When it comes to running the closures every closure the value from the global var. Hence, "two".
To change this behaviour simple add def or explicit type to make func a local variable.
node(){
def myFuncs = [:]
String[] myFuncsToCall = ["one", "two"]
for (int i = 0; i < myFuncsToCall.size(); i++) {
def func = myFuncsToCall[i]
myFuncs[func] = {
"${func}"()
}
}
parallel myFuncs
}
def one(){
echo "one"
}
def two(){
echo "two"
}
Related
I have these 3 groovy files:
A.groovy:
// ...
stage("Test")
{
throw new Exception("This script fails")
}
B.groovy:
// ...
stage("Test")
{
// Nothing, want this to pass
}
main.groovy:
// ...
m = [:]
status = [:]
scripts = ["A", "B"]
for script in scripts
{
m["${script}"] =
{
stage("${script}")
{
try
{
load "${script}.groovy"
status["${script}"] = true
}
catch (Exception e)
{
status["${script}"] = false
}
}
}
}
parallel m
for result_iterator in status:
print "${result_iterator.key} resulted in ${result_iterator.value}"
The code above is a sketch of the real code =)
When I run main.groovy to see the results in the status dictionary, I only get to see B. A threw an exception and thus it didn't add itself to the dictionary. Is there a way I can catch A's exception somehow?
Problem 1
You got bitten by the way Groovy closures capture variables outside of their scope. The references to script inside the closure will receive the value of script at the time when the closure is run, which will be the last value of script. So both closures actually load "B.groovy" and the exception is never thrown!
The fix is simple, just create a local variable inside of the loop that captures the current value of script:
for( script in scripts ) {
def theScript = script
// Now only use theScript inside of the closure!
}
Problem 2
You are not serializing write access to the status variable from parallel threads. This could result in very nasty, hard-to-reproduce bugs.
The fix is to use Map.asSynchronized() to create a synchronized map:
status = [:].asSynchronized()
Complete 'main.groovy' fix
m = [:]
status = [:].asSynchronized() // because multiple threads write to this variable
scripts = ["A", "B"]
for( script in scripts ) {
def theScript = script // Important to pass the current value into the closure!
m[ theScript ] = {
stage( theScript ) {
try {
load "${theScript}.groovy"
status[ theScript ] = true
}
catch (Exception e) {
status[ theScript ] = false
}
}
}
}
parallel m
for( result_iterator in status ) {
print "${result_iterator.key} resulted in ${result_iterator.value}"
}
Note: I have also removed the unnecessary string interpolations.
I am using geb spock. I am trying to read the system variable from GebConfig file, however it returns the null value.
Below is my GebConfig.groovy file.
def final DEFAULT_BROWSER = "chrome"
def final DEFAULT_LANGUAGE = "nl" //"en" or "nl"
def browser = System.getProperty("geb.env")
//Default browser
if (!correctBrowser(browser)) {
browser = DEFAULT_BROWSER
}
def envLang = System.getProperty("geb.env.lang")
//Default language
if (!correctLanguage(envLang)) {
envLang = DEFAULT_LANGUAGE
}
System.setProperty("geb.env.lang", envLang)
System.setProperty("geb.env", browser)
environments {
driver = { getDriver(browser, envLang) }
}
Below is my spec file where I am trying to get the language value in a variable.
#Stepwise
class TC001_SMO_Scenario_Spec extends GebReportingSpec {
#Shared
def lang = System.getProperty("geb.env.lang")
def "Step 1: Perform Login"() {
when: "Load File"
to WUPage
then: " File loaded successfully"
println " Getting data from Geb Config File: " + lang
}
}
Can you please help me how to do this, as this is very important for me to access and store it in a variable. Thanks
The problem is your #Shared variable. The Geb manual says:
(...) declare a #Shared field. Again it’s best to initialize the field right at the point of declaration. (Semantically, this is equivalent to initializing the field at the very beginning of the setupSpec() method.)
The thing is, setupSpec() runs before GebConfig is evaluated. You can see it if you add this to the end of your GebConfig:
println "Finished evaluating GebConfig"
Then run this Geb specification (I have wrapped the variable assignments into closures and added print statements, then I am evaluating the closures so as to make the assignments work):
package de.scrum_master.stackoverflow
import geb.spock.GebReportingSpec
import spock.lang.Shared
class GebConfigIT extends GebReportingSpec {
#Shared
def sharedLang = {
println "Initialising sharedLang"
System.getProperty("geb.env.lang")
}()
def normalLang = {
println "Initialising normalLang"
System.getProperty("geb.env.lang")
}()
def setup() {
println "sharedLang = $sharedLang"
println "normalLang = $normalLang"
}
def foo() {
expect:
!sharedLang
normalLang
}
def bar() {
expect:
!sharedLang
normalLang
}
}
Initialising sharedLang
Finished evaluating GebConfig
Initialising normalLang
sharedLang = null
normalLang = nl
Initialising normalLang
Finished evaluating GebConfig
sharedLang = null
normalLang = nl
Can you see how sharedLang is initialised only once at the very beginning, before GebConfig even gets a chance to kick in?
Bottom line: Just remove #Shared from your code. It is over-used by most people anyway, they think they save time this way for cheap resources and tamper with their clean fixture setups. #Shared really is only for very expensive resources and very exceptional cases.
I am doing some integration tests with Spock with 3rd party apps. Now I am struggling with a problem that I am not sure wether I am approaching the issue properly or not.
In one of the tests I am connecting to a 3rd party service to get some information in an array. Then each of these items are passed to another method to process them individually.
def get3rdPartyItems = {
[item1, item2, item3]
}
def processItem = { item ->
//do something with item
}
get3rdPartyItems.each {
processItem(it)
}
Then I have a test that connects to real 3rd party service using the method get3rdPartyItems() in which I am testing that processItem is called as many times as items has returned the method get3rdPartyItems().
What I am trying to do is to save one of the items as #Shared variable to write another test to know that the item is processed properly as I don't want to mock the content retrieved from the 3rd party service as I want real data.
Basically, this is what I am doing:
#Shared def globalItem
MyClass.metaClass.processItem = { i ->
if (!globalItem)
globalItem = i
//And now I would need to call the original method processItem
}
Any clue how to achieve this? I am probably overheading too much so I am open to change the solution.
Not sure if this is what you want, as it's hard to see your existing structure from the code and the code isn't runnable as-is, but given this class:
class MyClass {
def get3rdPartyItems = {
['item1', 'item2', 'item3']
}
def processItem( item ) {
println item
//do something with item
}
def run() {
get3rdPartyItems().each {
processItem( it )
}
}
}
You can do this:
def globalItem
def oldProcessItem = MyClass.metaClass.getMetaMethod("processItem", Object)
MyClass.metaClass.processItem = { item ->
if (!globalItem) {
println "Setting global item to $item"
globalItem = item
}
oldProcessItem.invoke( delegate, item )
}
def mc = new MyClass()
new MyClass().run()
Just as a matter of concision, that should be the way of passing the parameters to the metamethod in case you pass multiple parameters:
def globalItem
def oldProcessItem = MyClass.metaClass.getMetaMethod("processItem", ["",[:]] as Object[])
MyClass.metaClass.processItem = { String p1, Map p2 ->
if (!globalItem) {
println "Setting global item to $item"
globalItem = p2
}
oldProcessItem.invoke( delegate, [p1,p2] as Object[] )
}
def mc = new MyClass()
new MyClass().run()
I want to store objects in a map (called result). The objects are created or updated from SQL rows.
For each row I read I access the map as follows:
def result = [:]
sql.eachRow('SELECT something') { row->
{
// check if the Entry is already existing
def theEntry = result[row.KEY]
if (theEntry == null) {
// create the entry
theEntry = new Entry(row.VALUE1, row.VALUE2)
// put the entry in the result map
result[row.KEY] = theEntry
}
// use the Entry (create or update the next hierarchie elements)
}
I want to minimize the code for checking and updating the map. How can this be done?
I know the function map.get(key, defaultValue), but I will not use it, because it is to expensive to create an instance on each iteration even if I don't need it.
What I would like to have is a get function with a closure for providing the default value. In this case I would have lazy evaluation.
Update
The solution dmahapatro provided is exactly what I want. Following an example of the usage.
// simulate the result from the select
def select = [[a:1, b:2, c:3], [a:1, b:5, c:6], [a:2, b:2, c:4], [a:2, b:3, c:5]]
// a sample class for building an object hierarchie
class Master {
int a
List<Detail> subs = []
String toString() { "Master(a:$a, subs:$subs)" }
}
// a sample class for building an object hierarchie
class Detail {
int b
int c
String toString() { "Detail(b:$b, c:$c)" }
}
// the goal is to build a tree from the SQL result with Master and Detail entries
// and store it in this map
def result = [:]
// iterate over the select, row is visible inside the closure
select.each { row ->
// provide a wrapper with a default value in a closure and get the key
// if it is not available then the closure is executed to create the object
// and put it in the result map -> much compacter than in my question
def theResult = result.withDefault {
new Master(a: row.a)
}.get(row.a)
// process the further columns
theResult.subs.add new Detail(b: row.b, c: row.c )
}
// result should be [
// 1:Master(a:1, subs:[Detail(b:2, c:3), Detail(b:5, c:6)]),
// 2:Master(a:2, subs:[Detail(b:2, c:4), Detail(b:3, c:5)])]
println result
What I learned from this sample:
withDefault returns a wrapper, so for manipulating the map use the wrapper and not the original map
row variable is visible in the closure!
create the wrapper for the map in each iteration again, since row var changed
You asked for it, Groovy has it for you. :)
def map = [:]
def decoratedMap = map.withDefault{
new Entry()
}
It works the same way you would expect it to work lazily. Have a look at withDefault API for a detailed explanation.
I ran into a rather odd closure issue related to spock unit testing and wondered if anyone could explain this.
If we imagine a dao, model, and service as follows:
interface CustomDao {
List<Integer> getIds();
Model getModelById(int id);
}
class CustomModel {
int id;
}
class CustomService {
CustomDao customDao
public List<Object> createOutputSet() {
List<Model> models = new ArrayList<Model>();
List<Integer> ids = customDao.getIds();
for (Integer id in ids) {
models.add(customDao.getModelById(id));
}
return models;
}
}
I would like to unit test the CustomService.createOutputSet. I have created the following specification:
class TestSpec extends Specification {
def 'crazy closures'() {
def mockDao = Mock(CustomDao)
def idSet = [9,10]
given: 'An initialized object'
def customService = new CustomService
customService.customDao = mockDao
when: 'createOutput is called'
def outputSet = customService.createOutputSet()
then: 'the following methods should be called'
1*mockDao.getIds() >> {
return idSet
}
for (int i=0; i<idSet.size(); i++) {
int id = idSet.get(i)
1*mockDao.getModelById(idSet.get(i)) >> {
def tmp = new Model()
int tmpId = id // idSet.get(i)
return tmp
}
}
and: 'each compute package is accurate'
2 == outputSet.size()
9 == outputSet.get(0).getId()
10 == outputSet.get(1).getId()
}
}
Notice that in here I test two things. First, I initialize the dao with my mock, verify that the daos are correctly called and return the proper data, and then I verify that I get the proper output (i.e. "and:").
The tricky part is the for loop, in which I wanted to return models from the mock dao that are related to the method parameter. In the above example, if I use a simple for (__ in idSet), the models only return with id 10: outputSet.get(0).getId() == outputSet.get(1).getId() == 10. If I use the traditional for loop, and set the model with idSet.get(i), I get an IndexOutOfBoundsException . The only way to make this work is by retrieving the value in a local variable (id) and setting with variable, as above.
I know this is related to groovy closures and I suspect that spock captures the mock calls into a set of closures before executing them, which means that the model creation depends on the outer state of the closure. I understand why I would get the IndexOutOfBoundsException, but I don't understand why int id = idSet.get(i) is captured by the closure whereas i is not.
What is the difference?
Note: this is not the live code but rather simplified to demonstrate the crux of my challenge. I would not and do not make two subsequent dao calls on getIds() and getModelById().
While stubbing getModelById by a closure, the arguments to the closure has to match with that of the method. If you try something like below, you would not need the local variable id inside for anymore.
for (int i=0; i<idSet.size(); i++) {
//int id = idSet.get(i)
mockDao.getModelById(idSet.get(i)) >> {int id ->
def tmp = new Model()
tmp.id = id // id is closure param which represents idSet.get(i)
return tmp
}
}
Simplified version would be to use each
idSet.each {
mockDao.getModelById(it) >> {int id ->
def tmp = new Model()
tmp.id = id // id is closure param which represents idSet.get(i)
tmp
}
}
Do we need to worry about how many times method is called if it is being stubbed?
Accessing mutable local variables from a closure whose execution is deferred is a common source of errors not specific to Spock.
I don't understand why int id = idSet.get(i) is captured by the closure whereas i is not.
The former gives rise to a separate hoisted variable per iteration whose value is constant. The latter gives rise to a single hoisted variable whose value changes over time (and before the result generator executes).
Instead of solving the problem by introducing a temporary variable, a better solution (already given by #dmahapatro) is to declare an int id -> closure parameter. If it's deemed good enough to stub the calls without enforcing them, the loop can be omitted altogether. Yet another potential solution is to construct the return values eagerly:
idSet.each { id ->
def model = new Model()
model.id = id
1 * mockDao.getModelById(id) >> model
}