Geb Reporter / Extension to retrieve Test Status - groovy

Question

I am attempting to replace some custom java selenium extensions by utilizing geb. I have hit a bit of a brick wall when I attempt to utilize a grid in the cloud (i.e. SauceLabs). When my tests complete, it'd be nice to send an update back to indicate whether or not the test has failed or succeeded. To utilize this, I need the sessionId from the RemoteWebDriver instance. This can be obtained in a custom Reporter, however I can't determine the success with this interface. Since I am extending the GebReportingSpec, I attempted to create my own custom version, which had a custom Junit rule to track success or failure:
public class TestSuccess extends TestWatcher {
boolean success;
String message;
#Override
protected void starting(Description d) {
message = d.getMethodName();
}
#Override
protected void succeeded(final Description description) {
System.out.println("Test Success [succeeded] " + description);
this.success = true;
}
#Override
protected void failed(final Throwable e, final Description description) {
System.out.println("Test Success [failed] " + description);
this.success = false;
}
public boolean isSuccess() {
return success;
}
#Override
public String toString() {
return message + " success: <" + success + ">.";
}
}
I then added that to my CustomReportingSpec:
class CustomReportingSpec extends GebReportingSpec {
/* I also tried creating this as a RuleChain with:
* #Rule TestRule chain = RuleChain.outerRule(
super._gebReportingSpecTestName).around(new TestSuccess());
* however, this results in a NPE. Placing the super rule in the around
* still results in a NPE.
*/
#Rule public TestSuccess _gebTestSuccesswatcher = new TestSuccess();
// I never see this called
void report() {
System.out.println("Custom Reporting Spec: " + _gebTestSuccesswatcher + "\t")
super.report()
}
}
I have also attempted to set this up in a custom reporter:
public CustomReporter extends ScreenshotAndPageSourceReporter implements Reporter {
#Rule
public TestSuccess _gebTestSuccesswatcher = new TestSuccess();
#Override
public void writeReport(Browser browser, String label, File outputDir) {
System.out.println("Custom Reporter: " + _gebTestSuccesswatcher);
super.writeReport(browser, label, outputDir)
}
}
However, regardless of whether or not my test fails, the success method on the watcher seems to be called. Here is my sample test:
class OneOff extends CustomReportingSpec {
def "Check One off"() {
when:
go "http://www.google.com"
then:
1 == 2
}
}
And the output:
Custom Reporter: null success: <false>.
Test Success [succeeded] Check One off(OneOff)
As you can see the success method is called upon completion of this failing test. If I modify the test to pass (i.e. 1 == 1), here is my output:
Custom Reporter: null success: <false>.
Test Success [succeeded] Check One off(OneOff)
Is there any way for me to get this Rule to work properly in the Custom Reporter? Or is there a way to get the browser instance in an extension? I've followed this guide to create a custom annotation and listener, but I can't access the Browser object. I have attempted adding an #Shared onto the declaration of the browser, but it isn't pulling the one in the Geb Configuration.

Answer 1

Your TestSuccess class doesn't work correctly due to a known limitation in Spock's TestRule support. Due to subtle differences between Spock's and JUnit's test execution model, calling base.evaluate() from a TestRule will not throw an exception in Spock, even if the test has failed. In many cases this won't make a difference, but for TestWatcher it will.
This is the only known limitation in Spock's rule support, and hopefully we'll find a way to overcome it at some point. There is no such semantic mismatch when using MethodRule.
If you want to implement your requirement with the help of a JUnit rule (which I think is fine), MethodRule is probably the better choice anyway. In contrary to TestRule, MethodRule provides access to the test instance, which will allow you to grab the session ID with browser.driver.sessionId.