I want to use the Sauce Labs Java REST API to send Pass/Fail status back to the Sauce Labs dashboard. I am using Geb+Spock, and my Gradle build creates a test results directory where results are output in XML. My problem is that the results XML file doesn't seem to be generated until after the Spock specification's cleanupSpec() exits. This causes my code to report the results of the previous test run, rather than the current one. Clearly not what I want!
Is there some way to get to the results from within cleanupSpec() without relying on the XML? Or a way to get the results to file earlier? Or some alternative that will be much better than either of those?
Some code:
In build.gradle, I specify the testResultsDir. This is where the XML file is written after the Spock specifications exit:
drivers.each { driver ->
task "${driver}Test"(type: Test) {
cleanTest
systemProperty "geb.env", driver
testResultsDir = file("$buildDir/test-results/${driver}")
systemProperty "proj.test.resultsDir", testResultsDir
}
}
Here is the setupSpec() and cleanupSpec() in my LoginSpec class:
class LoginSpec extends GebSpec {
#Shared def SauceREST client = new SauceREST("redactedName", "redactedKey")
#Shared def sauceJobID
#Shared def allSpecsPass = true
def setupSpec() {
sauceJobID = driver.getSessionId().toString()
}
def cleanupSpec() {
def String specResultsDir = System.getProperty("proj.test.resultsDir") ?: "./build/test-results"
def String specResultsFile = this.getClass().getName()
def String specResultsXML = "${specResultsDir}/TEST-${specResultsFile}.xml"
def testsuiteResults = new XmlSlurper().parse( new File( specResultsXML ))
// read error and failure counts from the XML
def errors = testsuiteResults.#errors.text()?.toInteger()
def failures = testsuiteResults.#failures.text()?.toInteger()
if ( (errors + failures) > 0 ) { allSpecsPass = false }
if ( allSpecsPass ) {
client.jobPassed(sauceJobID)
} else {
client.jobFailed(sauceJobID)
}
}
}
The rest of this class contains login specifications that do not interact with SauceLabs. When I read the XML, it turns out that it was written at the end of the previous LoginSpec run. I need a way to get to the values of the current run.
Thanks!
Test reports are generated after a Specification has finished execution and the generation is performed by the build system, so in your case by Gradle. Spock has no knowledge of that so you are unable to get that information from within the test.
You can on the other hand quite easily get that information from Gradle. Test task has two methods that might be of interest to you here: addTestListener() and afterSuite(). It seems that the cleaner solution here would be to use the first method, implement a test listener and put your logic in afterSuite() of the listener (and not the task configuration). You would probably need to put that listener implementation in buildSrc as it looks like you have a dependency on SauceREST and you would need to build and compile your listener class before being able to use it as an argument to addTestListener() in build.gradle of your project.
Following on from erdi's suggestion, I've created a Sauce Gradle helper library, which provides a Test Listener that parses the test XML output and invokes the Sauce REST API to set the pass/fail status.
The library can be included by adding the following to your build.gradle file:
import com.saucelabs.gradle.SauceListener
buildscript {
repositories {
mavenCentral()
maven {
url "https://repository-saucelabs.forge.cloudbees.com/release"
}
}
dependencies {
classpath group: 'com.saucelabs', name: 'saucerest', version: '1.0.2'
classpath group: 'com.saucelabs', name: 'sauce_java_common', version: '1.0.14'
classpath group: 'com.saucelabs.gradle', name: 'sauce-gradle-plugin', version: '0.0.1'
}
}
gradle.addListener(new SauceListener("YOUR_SAUCE_USERNAME", "YOUR_SAUCE_ACCESS_KEY"))
You will also need to output the Selenium session id for each test, so that the SauceListener can associate the Sauce Job with the pass/fail status. To do this, include the following output in the stdout:
SauceOnDemandSessionID=SELENIUM_SESSION_ID
Related
I'm trying to use Test NG with Groovy. I implemented a simple test class, but whenever I try to run it I get the following error:
No tests found for given includes: [tests.OTP.get]
Here is what the tests.OTP class looks like:
class OTP{
public static Logger log = LogManager.getLogger(HomePage.class.getName())
String environment = 'qatesting'
String number = '0769878787'
#Test
def get(){
// SoapuiOTP s = new SoapuiOTP(environment,number)
// s.getOTP()
log.info("hello")
Assert.assertEquals(0,System.getProperty("SOAPUI_OTP"))
log.info System.getProperty('SOAPUI_OTP')
}
}
I use the play button (IntelliJ) next to the def get() to run the test but i get the same error on class level. Please assist, I've tried invaliding my cache and restarting. I have looked at TestNG No tests found. Nothing was run and
TestNG - no tests were found, but it didn't help.
You need to replace def with void in the method annotated with #Test. As the documentation says:
Test methods are annotated with #Test. Methods annotated with #Test that happen to return a value will be ignored, unless you set allow-return-values to true in your testng.xml:
<test allow-return-values="true">
Source: https://testng.org/doc/documentation-main.html#test-methods
The def return type compiles to Object, not void, so TestNG ignores this method by default.
This misconception is not TestNG specific and can be done in JUnit as well: Running Groovy test cases with JUnit 5
I have an issue where tags from my gauge gradle task is not being passed to beforespec whereas the the tags are passed to spec files
Any idea what is the issue?
I'm in Gauge version: 0.9.1
Plugins
html-report (3.1.0)
java (0.6.2)
gradle gauge task :
task runTestsInQA(type: GaugeTask) {
doFirst {
println 'Running tests for the V1 in QA environment...'
gauge {
specsDir = 'specs'
tags = 'V1'
env = 'qa'
additionalFlags = '--verbose'
}
}
}
_
My beforespec code:
#beforespec(tags = "V1")
public void beforeSpec(ExecutionContext context)
{
System.out.println("Tags in scenario "+context.getAllTags());
}
Here, print statement is throwing null array [ ]
This gist contains an example project with gauge + java + gradle.
In your case, note that the BeforeSpec hook is a tagged execution hook, so it will get executed only when the respective tags are passed.
Also note that if you wish to get tags for a scenario, you are better off using the BeforeScenario hook, since you can then get all the tags (scenarios inherit spec tags).
First off, this is my first foray into Gradle/Groovy (using Gradle 1.10). I'm setting up a multi-project environment where I'm creating a jar artifact in one project and then want to define an Exec task, in another project, which depends on the created jar. I'm setting it up something like this:
// This is from the jar building project
jar {
...
}
configurations {
loaderJar
}
dependencies {
loaderJar files(jar.archivePath)
...
}
// From the project which consumes the built jar
configurations {
loaderJar
}
dependencies {
loaderJar project(path: ":gfxd-demo-loader", configuration: "loaderJar")
}
// This is my test task
task foo << {
configurations.loaderJar.each { println it }
println configurations.loaderJar.collect { it }[0]
// The following line breaks!!!
println configurations.loaderJar[0]
}
When executing the foo task it fails with:
> Could not find method getAt() for arguments [0] on configuration ':loaderJar'.
In my foo task I'm just testing to see how to access the jar. So the question is, why does the very last println fail? if a Configuration object is a Collection/Iterable then surely I should be able to index into it?
Configuration is-a java.util.Iterable, but not a java.util.Collection. As can be seen in the Groovy GDK docs, the getAt method (which corresponds to the [] operator) is defined on Collection, but not on Iterable. Hence, you can't index into a Configuration.
I'm using Cucumber for my tests. How do I rerun only the failed tests?
Run Cucumber with rerun formatter:
cucumber -f rerun --out rerun.txt
It will output locations of all failed scenarios to this file.
Then you can rerun them by using
cucumber #rerun.txt
Here is my simple and neat solution.
Step 1: Write your cucumber java file as mentioned below with rerun:target/rerun.txt. Cucumber writes the failed scenarios line numbers in rerun.txt as shown below.
features/MyScenaios.feature:25
features/MyScenaios.feature:45
Later we can use this file in Step 2. Name this file as MyScenarioTests.java. This is your main file to run your tagged scenarios. If your scenarios has failed test cases, MyScenarioTests.java will write/mark them rerun.txt under target directory.
#RunWith(Cucumber.class)
#CucumberOptions(
monochrome = true,
features = "classpath:features",
plugin = {"pretty", "html:target/cucumber-reports",
"json:target/cucumber.json",
"rerun:target/rerun.txt"} //Creates a text file with failed scenarios
,tags = "#mytag"
)
public class MyScenarioTests {
}
Step 2: Create another scenario file as shown below. Let's say this as FailedScenarios.java. Whenever you notice any failed scenario run this file. This file will use target/rerun.txt as an input for running the scenarios.
#RunWith(Cucumber.class)
#CucumberOptions(
monochrome = true,
features = "#target/rerun.txt", //Cucumber picks the failed scenarios from this file
format = {"pretty", "html:target/site/cucumber-pretty",
"json:target/cucumber.json"}
)
public class FailedScenarios {
}
Everytime if you notice any failed scenarios run the file in Step 2
task cucumber() {
dependsOn assemble, compileTestJava
doLast {
javaexec {
main = "io.cucumber.core.cli.Main"
classpath = configurations.cucumberRuntime + sourceSets.main.output + sourceSets.test.output
args = ['--plugin', 'json:target/cucumber-reports/json/cucumber.json',
'--plugin', "rerun:target/rerun.txt",
'--glue', 'steps',
'src/test/resources']
}
}
}
task cucumberRerunFailed() {
doLast {
javaexec {
main = "io.cucumber.core.cli.Main"
classpath = configurations.cucumberRuntime + sourceSets.main.output + sourceSets.test.output
args = ['--plugin', 'json:target/cucumber-reports/json/cucumber.json',
'#target/rerun.txt']
}
}
}
I know this is old, but I found my way here first and then later found a much more up to date answer (not the accepted, Cyril Duchon-Doris' answer):
https://stackoverflow.com/a/41174252/215789
Since cucumber 3.0 you can use --retry to specify how many times to retry a scenario that failed.
https://cucumber.io/blog/open-source/announcing-cucumber-ruby-3-0-0/
Just tack it on to your cucumber command:
cucumber ... --retry 2
You need at least version 1.2.0 in order to use the #target/rerun.txt new feature. After that just create a runner that runs at the end and uses this file. Also, if you are using Jenkins, you can put a tag on the random failures features so the build doesn't fails unless after being ran twice.
I have a parameterized job that uses the Perforce plugin and would like to retrieve the build parameters/properties as well as the p4.change property that's set by the Perforce plugin.
How do I retrieve these properties with the Jenkins Groovy API?
Update: Jenkins 2.x solution:
With Jenkins 2 pipeline dsl, you can directly access any parameter with the trivial syntax based on the params (Map) built-in:
echo " FOOBAR value: ${params.'FOOBAR'}"
The returned value will be a String or a boolean depending on the Parameter type itself. The syntax is the same for scripted or declarative syntax. More info at: https://jenkins.io/doc/book/pipeline/jenkinsfile/#handling-parameters
If your parameter name is itself in a variable:
def paramName = "FOOBAR"
def paramValue = params.get(paramName) // or: params."${paramName}"
echo """ FOOBAR value: ${paramValue}"
Original Answer for Jenkins 1.x:
For Jenkins 1.x, the syntax is based on the build.buildVariableResolver built-ins:
// ... or if you want the parameter by name ...
def hardcoded_param = "FOOBAR"
def resolver = build.buildVariableResolver
def hardcoded_param_value = resolver.resolve(hardcoded_param)
Please note the official Jenkins Wiki page covers this in more details as well, especially how to iterate upon the build parameters:
https://wiki.jenkins-ci.org/display/JENKINS/Parameterized+System+Groovy+script
The salient part is reproduced below:
// get parameters
def parameters = build?.actions.find{ it instanceof ParametersAction }?.parameters
parameters.each {
println "parameter ${it.name}:"
println it.dump()
}
For resolving a single parameter (I guess what's most commonly needed), this is the simplest I found:
build.buildVariableResolver.resolve("myparameter")
in your Groovy System script build step.
Regarding parameters:
See this answer first. To get a list of all the builds for a project (obtained as per that answer):
project.builds
When you find your particular build, you need to get all actions of type ParametersAction with build.getActions(hudson.model.ParametersAction). You then query the returned object for your specific parameters.
Regarding p4.change: I suspect that it is also stored as an action. In Jenkins Groovy console get all actions for a build that contains p4.change and examine them - it will give you an idea what to look for in your code.
I've just got this working, so specifically, using the Groovy Postbuild plugin, you can do the following:
def paramText
def actionList = manager.build.getActions(hudson.model.ParametersAction)
if (actionList.size() != 0)
{
def pA = actionList.get(0)
paramText = pA.createVariableResolver(manager.build).resolve("MY_PARAM_NAME")
}
In cases when a parameter name cannot be hardcoded I found this would be the simplest and best way to access parameters:
def myParam = env.getProperty(dynamicParamName)
In cases, when a parameter name is known and can be hardcoded the following 3 lines are equivalent:
def myParam = env.getProperty("myParamName")
def myParam = env.myParamName
def myParam = myParamName
To get the parameterized build params from the current build from your GroovyScript (using Pipeline), all you need to do is:
Say you had a variable called VARNAME.
def myVariable = env.VARNAME
Get all of the parameters:
System.getenv().each{
println it
}
Or more sophisticated:
def myvariables = getBinding().getVariables()
for (v in myvariables) {
echo "${v} " + myvariables.get(v)
}
You will need to disable "Use Groovy Sandbox" for both.
If you are trying to get all parameters passed to Jenkins job you can use the global variable params in your groovy pipeline to fetch it.
http://jenkins_host:8080/pipeline-syntax/globals
params
Use something like below.
def dumpParameter()
{
params.each {
println it.key + " = " + it.value
}
}
thanks patrice-n! this code worked to get both queued and running jobs and their parameters:
import hudson.model.Job
import hudson.model.ParametersAction
import hudson.model.Queue
import jenkins.model.Jenkins
println("================================================")
for (Job job : Jenkins.instanceOrNull.getAllItems(Job.class)) {
if (job.isInQueue()) {
println("------------------------------------------------")
println("InQueue " + job.name)
Queue.Item queue = job.getQueueItem()
if (queue != null) {
println(queue.params)
}
}
if (job.isBuilding()) {
println("------------------------------------------------")
println("Building " + job.name)
def build = job.getBuilds().getLastBuild()
def parameters = build?.getAllActions().find{ it instanceof ParametersAction }?.parameters
parameters.each {
def dump = it.dump()
println "parameter ${it.name}: ${dump}"
}
}
}
println("================================================")
The following can be used to retreive an environment parameter:
println System.getenv("MY_PARAM")
The following snippet worked for me to get a parameter value in a parameterized project:
String myParameter = this.getProperty('binding').getVariable('MY_PARAMETER')
The goal was to dynamically lock a resource based on the selected project parameter.
In "[✓] This build requires lockable resources" I have the following "[✓] Groovy Expression":
if (resourceName == 'resource_lock_name') {
Binding binding = this.getProperty('binding')
String profile = binding.getVariable('BUILD_PROFILE')
return profile == '-Poradb' // acquire lock if "oradb" profile is selected
}
return false
In "[✓] This project is parameterized" section I have a "Choice Parameter" named e.g. BUILD_PROFILE
Example of Choices are:
-Poradb
-Ph2db
-DskipTests -T4
The lock on "resource_lock_name" will be acquired only if "-Poradb" is selected when building project with parameters
[-] Use Groovy Sandbox shall be unchecked for this syntax to work