for example:
testing.xml
<suite name="name" parallel="methods" thread-count="5">
<test>
...
</test>
</suite>
case #1: when I run the following feature file with #smoke tag per say, it will execute parallel testing normally:
something1.feature
#smoke
Feature: <feature message>
Scenario: <scenario1 message>
...
Scenario: <scenario2 message>
case#2: when I execute the same scenario using 'Scenario Outlines' is when parallel testing goes concurrently even though the parallel testing configuration is setup in the TestNG xml file.
something2.feature
Feature: <feature message>
Scenario Outline: <scenario outline messsage>
...
Examples:
<data table>
does anybody knows why is this happening with QMETRY framework?
When you are using TestNG, Scenario Outline is converted to TestNG data-driven tests. In TestNG there is separate mechanism for data driven test case parallel execution. QAF provides property global.datadriven.parallel to set parallel behavior for data driven tests. Default value is false. Make sure that you have set it to true in order to run data-driven tests in parallel.
global.datadriven.parallel=true
Related
I have a Cucumber project running thanks to JUnit with the following class:
#RunWith(Cucumber.class)
#CucumberOptions(
features = "src/test/resources/features",
glue = "com.steps",
plugin = {"pretty", "json:target/cucumber-report/cucumber.json"})
public class RunCucumber {
}
I want to use QAF Gherkin Scenario Factory in order to put some scenario examples in external files in order to reuse them (ex : Cucumber: Scenario Outline reusing examples table or CustomDataProvider for feature file (QAF)).
I read the last section called "For Existing cucumber implementation" and did what it says, but I currently get an exception when running Cucumber with QAF (see edit section below).
Java: 1.8
Cucumber : 4.8.0
Junit: 4.12
src/test/resources/features/test.feature
Feature: Test for QAF
Scenario: success
Given my scenario works
src/test/java/com/steps/TestSteps.java
#QAFTestStepProvider
public class TestSteps {
#Given("my scenario works")
public void myScenarioWorks() {
assertTrue(true);
}
}
Edit: Base on user861594 help, here is where I am:
I downloaded the library QAF 2.1.15 through maven configuration.
<dependency>
<groupId>com.qmetry</groupId>
<artifactId>qaf</artifactId>
<version>2.1.15</version>
</dependency>
I annotated my step class with #QAFTestStepProvider
I created a xml configuration file for testNG in order to configure cucumber and run it with the QAF class called GherkinScenarioFactory:
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="TestNG - Cucumber" verbose="0">
<test name="Gherkin-QAF-Test">
<parameter name="step.provider.pkg" value="com.steps" />
<parameter name="scenario.file.loc" value="src/test/resources/features/test.feature" />
<classes>
<class name="com.qmetry.qaf.automation.step.client.gherkin.GherkinScenarioFactory" />
</classes>
</test>
</suite>
I ran the previous configuration with the testNG plugin for IntelliJ IDEA (test kind = suite. I also can choose Class, Method, Group, Pattern or All in package)
When running, I get the following exception (before the test, so it is ignored):
"C:\Program Files\Java\jdk1.8.0_212\bin\java.exe" -ea -Didea.test.cyclic.buffer.size=1048576 "-javaagent:C:\Program Files\JetBrains\IntelliJ IDEA Community Edition 2019.1\lib\idea_rt.jar=51974:C:\Program Files\JetBrains\IntelliJ IDEA Community Edition 2019.1\bin" -Dfile.encoding=UTF-8 -classpath "C:\Program Files\JetBrains\IntelliJ IDEA Community Edition 2019.1\lib\idea_rt.jar;C:\Program Files\JetBrains\IntelliJ IDEA Community Edition 2019.1\plugins\testng\lib\testng-plugin.jar;C:\Program Files\Java\jdk1.8.0_212\jre\lib\charsets.jar;C:\Program Files\Java\jdk1.8.0_212\jre\lib\deploy.jar;C:\Program Files\Java\jdk1.8.0_212\jre\lib\ext\access-bridge-64.jar;C:\Program Files\Java\jdk1.8.0_212\jre\lib\ext\cldrdata.jar;C:\Program Files\Java\jdk1.8.0_212\jre\lib\ext\dnsns.jar;C:\Program Files\Java\jdk1.8.0_212\jre\lib\ext\jaccess.jar;C:\Program Files\Java\jdk1.8.0_212\jre\lib\ext\jfxrt.jar;C:\Program Files\Java\jdk1.8.0_212\jre\lib\ext\localedata.jar;C:\Program Files\Java\jdk1.8.0_212\jre\lib\ext\nashorn.jar;C:\Program Files\Java\jdk1.8.0_212\jre\lib\ext\sunec.jar;C:\Program Files\Java\jdk1.8.0_212\jre\lib\ext\sunjce_provider.jar;C:\Program Files\Java\jdk1.8.0_212\jre\lib\ext\sunmscapi.jar;C:\Program Files\Java\jdk1.8.0_212\jre\lib\ext\sunpkcs11.jar;C:\Program Files\Java\jdk1.8.0_212\jre\lib\ext\zipfs.jar;C:\Program Files\Java\jdk1.8.0_212\jre\lib\javaws.jar;C:\Program Files\Java\jdk1.8.0_212\jre\lib\jce.jar;C:\Program Files\Java\jdk1.8.0_212\jre\lib\jfr.jar;C:\Program Files\Java\jdk1.8.0_212\jre\lib\jfxswt.jar;C:\Program Files\Java\jdk1.8.0_212\jre\lib\jsse.jar;C:\Program Files\Java\jdk1.8.0_212\jre\lib\management-agent.jar;C:\Program Files\Java\jdk1.8.0_212\jre\lib\plugin.jar;C:\Program Files\Java\jdk1.8.0_212\jre\lib\resources.jar;C:\Program Files\Java\jdk1.8.0_212\jre\lib\rt.jar;C:\Utilisateurs\a754206\IdeaProjects\perso\cucumber\target\test-classes;C:\Utilisateurs\a754206\IdeaProjects\perso\cucumber\target\classes;C:\Utilisateurs\a754206\.m2\repository\junit\junit\4.12\junit-4.12.jar;C:\Utilisateurs\a754206\.m2\repository\org\hamcrest\hamcrest-core\1.3\hamcrest-core-1.3.jar;C:\Utilisateurs\a754206\.m2\repository\io\cucumber\cucumber-java\4.8.0\cucumber-java-4.8.0.jar;C:\Utilisateurs\a754206\.m2\repository\io\cucumber\cucumber-core\4.8.0\cucumber-core-4.8.0.jar;C:\Utilisateurs\a754206\.m2\repository\io\cucumber\gherkin\5.1.0\gherkin-5.1.0.jar;C:\Utilisateurs\a754206\.m2\repository\io\cucumber\tag-expressions\1.1.1\tag-expressions-1.1.1.jar;C:\Utilisateurs\a754206\.m2\repository\io\cucumber\cucumber-expressions\7.0.2\cucumber-expressions-7.0.2.jar;C:\Utilisateurs\a754206\.m2\repository\io\cucumber\datatable\1.1.14\datatable-1.1.14.jar;C:\Utilisateurs\a754206\.m2\repository\io\cucumber\datatable-dependencies\1.1.14\datatable-dependencies-1.1.14.jar;C:\Utilisateurs\a754206\.m2\repository\org\apiguardian\apiguardian-api\1.1.0\apiguardian-api-1.1.0.jar;C:\Utilisateurs\a754206\.m2\repository\io\cucumber\cucumber-junit\4.8.0\cucumber-junit-4.8.0.jar;C:\Utilisateurs\a754206\.m2\repository\io\cucumber\cucumber-spring\4.8.0\cucumber-spring-4.8.0.jar;C:\Utilisateurs\a754206\.m2\repository\org\springframework\spring-context\5.2.0.RELEASE\spring-context-5.2.0.RELEASE.jar;C:\Utilisateurs\a754206\.m2\repository\org\springframework\spring-aop\5.2.0.RELEASE\spring-aop-5.2.0.RELEASE.jar;C:\Utilisateurs\a754206\.m2\repository\org\springframework\spring-beans\5.2.0.RELEASE\spring-beans-5.2.0.RELEASE.jar;C:\Utilisateurs\a754206\.m2\repository\org\springframework\spring-core\5.2.0.RELEASE\spring-core-5.2.0.RELEASE.jar;C:\Utilisateurs\a754206\.m2\repository\org\springframework\spring-jcl\5.2.0.RELEASE\spring-jcl-5.2.0.RELEASE.jar;C:\Utilisateurs\a754206\.m2\repository\org\springframework\spring-expression\5.2.0.RELEASE\spring-expression-5.2.0.RELEASE.jar;C:\Utilisateurs\a754206\.m2\repository\net\masterthought\cucumber-reporting\4.10.0\cucumber-reporting-4.10.0.jar;C:\Utilisateurs\a754206\.m2\repository\org\apache\velocity\velocity\1.7\velocity-1.7.jar;C:\Utilisateurs\a754206\.m2\repository\commons-collections\commons-collections\3.2.2\commons-collections-3.2.2.jar;C:\Utilisateurs\a754206\.m2\repository\velocity-tools\velocity-tools\1.4\velocity-tools-1.4.jar;C:\Utilisateurs\a754206\.m2\repository\joda-time\joda-time\2.10.2\joda-time-2.10.2.jar;C:\Utilisateurs\a754206\.m2\repository\org\apache\commons\commons-lang3\3.9\commons-lang3-3.9.jar;C:\Utilisateurs\a754206\.m2\repository\commons-io\commons-io\2.6\commons-io-2.6.jar;C:\Utilisateurs\a754206\.m2\repository\org\codehaus\plexus\plexus-utils\3.2.0\plexus-utils-3.2.0.jar;C:\Utilisateurs\a754206\.m2\repository\org\jsoup\jsoup\1.12.1\jsoup-1.12.1.jar;C:\Utilisateurs\a754206\.m2\repository\com\googlecode\owasp-java-html-sanitizer\owasp-java-html-sanitizer\20190610.1\owasp-java-html-sanitizer-20190610.1.jar;C:\Utilisateurs\a754206\.m2\repository\com\google\guava\guava\27.1-jre\guava-27.1-jre.jar;C:\Utilisateurs\a754206\.m2\repository\com\google\guava\failureaccess\1.0.1\failureaccess-1.0.1.jar;C:\Utilisateurs\a754206\.m2\repository\com\google\guava\listenablefuture\9999.0-empty-to-avoid-conflict-with-guava\listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar;C:\Utilisateurs\a754206\.m2\repository\com\google\code\findbugs\jsr305\3.0.2\jsr305-3.0.2.jar;C:\Utilisateurs\a754206\.m2\repository\org\checkerframework\checker-qual\2.5.2\checker-qual-2.5.2.jar;C:\Utilisateurs\a754206\.m2\repository\com\google\errorprone\error_prone_annotations\2.2.0\error_prone_annotations-2.2.0.jar;C:\Utilisateurs\a754206\.m2\repository\com\google\j2objc\j2objc-annotations\1.1\j2objc-annotations-1.1.jar;C:\Utilisateurs\a754206\.m2\repository\org\codehaus\mojo\animal-sniffer-annotations\1.17\animal-sniffer-annotations-1.17.jar;C:\Utilisateurs\a754206\.m2\repository\commons-configuration\commons-configuration\1.10\commons-configuration-1.10.jar;C:\Utilisateurs\a754206\.m2\repository\ch\qos\logback\logback-classic\1.2.3\logback-classic-1.2.3.jar;C:\Utilisateurs\a754206\.m2\repository\ch\qos\logback\logback-core\1.2.3\logback-core-1.2.3.jar;C:\Utilisateurs\a754206\.m2\repository\org\slf4j\slf4j-api\1.7.25\slf4j-api-1.7.25.jar;C:\Utilisateurs\a754206\.m2\repository\com\jayway\jsonpath\json-path\2.2.0\json-path-2.2.0.jar;C:\Utilisateurs\a754206\.m2\repository\net\minidev\json-smart\2.2.1\json-smart-2.2.1.jar;C:\Utilisateurs\a754206\.m2\repository\net\minidev\accessors-smart\1.1\accessors-smart-1.1.jar;C:\Utilisateurs\a754206\.m2\repository\org\ow2\asm\asm\5.0.3\asm-5.0.3.jar;C:\Utilisateurs\a754206\.m2\repository\org\apache\httpcomponents\httpclient\4.5.3\httpclient-4.5.3.jar;C:\Utilisateurs\a754206\.m2\repository\org\apache\httpcomponents\httpcore\4.4.6\httpcore-4.4.6.jar;C:\Utilisateurs\a754206\.m2\repository\commons-logging\commons-logging\1.2\commons-logging-1.2.jar;C:\Utilisateurs\a754206\.m2\repository\commons-codec\commons-codec\1.9\commons-codec-1.9.jar;C:\Utilisateurs\a754206\.m2\repository\com\fasterxml\jackson\core\jackson-databind\2.9.8\jackson-databind-2.9.8.jar;C:\Utilisateurs\a754206\.m2\repository\com\fasterxml\jackson\core\jackson-annotations\2.9.0\jackson-annotations-2.9.0.jar;C:\Utilisateurs\a754206\.m2\repository\com\fasterxml\jackson\core\jackson-core\2.9.8\jackson-core-2.9.8.jar;C:\Utilisateurs\a754206\.m2\repository\com\fasterxml\jackson\dataformat\jackson-dataformat-xml\2.9.8\jackson-dataformat-xml-2.9.8.jar;C:\Utilisateurs\a754206\.m2\repository\com\fasterxml\jackson\module\jackson-module-jaxb-annotations\2.9.8\jackson-module-jaxb-annotations-2.9.8.jar;C:\Utilisateurs\a754206\.m2\repository\org\codehaus\woodstox\stax2-api\3.1.4\stax2-api-3.1.4.jar;C:\Utilisateurs\a754206\.m2\repository\com\fasterxml\woodstox\woodstox-core\5.0.3\woodstox-core-5.0.3.jar;C:\Utilisateurs\a754206\.m2\repository\com\qmetry\qaf\2.1.15\qaf-2.1.15.jar;C:\Utilisateurs\a754206\.m2\repository\org\hamcrest\hamcrest-library\1.3\hamcrest-library-1.3.jar;C:\Utilisateurs\a754206\.m2\repository\com\sun\jersey\jersey-client\1.19\jersey-client-1.19.jar;C:\Utilisateurs\a754206\.m2\repository\com\sun\jersey\jersey-core\1.19\jersey-core-1.19.jar;C:\Utilisateurs\a754206\.m2\repository\javax\ws\rs\jsr311-api\1.1.1\jsr311-api-1.1.1.jar;C:\Utilisateurs\a754206\.m2\repository\com\sun\jersey\contribs\jersey-multipart\1.19\jersey-multipart-1.19.jar;C:\Utilisateurs\a754206\.m2\repository\org\jvnet\mimepull\mimepull\1.9.3\mimepull-1.9.3.jar;C:\Utilisateurs\a754206\.m2\repository\com\sun\jersey\contribs\jersey-apache-client\1.17.1\jersey-apache-client-1.17.1.jar;C:\Utilisateurs\a754206\.m2\repository\commons-httpclient\commons-httpclient\3.1\commons-httpclient-3.1.jar;C:\Utilisateurs\a754206\.m2\repository\org\aspectj\aspectjrt\1.8.9\aspectjrt-1.8.9.jar;C:\Utilisateurs\a754206\.m2\repository\org\aspectj\aspectjweaver\1.8.9\aspectjweaver-1.8.9.jar;C:\Utilisateurs\a754206\.m2\repository\org\aspectj\aspectjtools\1.8.9\aspectjtools-1.8.9.jar;C:\Utilisateurs\a754206\.m2\repository\net\sourceforge\jexcelapi\jxl\2.6.12\jxl-2.6.12.jar;C:\Utilisateurs\a754206\.m2\repository\log4j\log4j\1.2.14\log4j-1.2.14.jar;C:\Utilisateurs\a754206\.m2\repository\commons-lang\commons-lang\2.6\commons-lang-2.6.jar;C:\Utilisateurs\a754206\.m2\repository\com\google\code\gson\gson\2.8.5\gson-2.8.5.jar;C:\Utilisateurs\a754206\.m2\repository\org\testng\testng\6.10\testng-6.10.jar;C:\Utilisateurs\a754206\.m2\repository\com\beust\jcommander\1.48\jcommander-1.48.jar;C:\Utilisateurs\a754206\.m2\repository\com\google\inject\guice\3.0\guice-3.0.jar;C:\Utilisateurs\a754206\.m2\repository\javax\inject\javax.inject\1\javax.inject-1.jar;C:\Utilisateurs\a754206\.m2\repository\aopalliance\aopalliance\1.0\aopalliance-1.0.jar;C:\Utilisateurs\a754206\.m2\repository\velocity\velocity-dep\1.4\velocity-dep-1.4.jar;C:\Utilisateurs\a754206\.m2\repository\org\seleniumhq\selenium\selenium-remote-driver\2.53.0\selenium-remote-driver-2.53.0.jar;C:\Utilisateurs\a754206\.m2\repository\cglib\cglib-nodep\2.1_3\cglib-nodep-2.1_3.jar;C:\Utilisateurs\a754206\.m2\repository\org\seleniumhq\selenium\selenium-api\2.53.0\selenium-api-2.53.0.jar;C:\Utilisateurs\a754206\.m2\repository\org\apache\commons\commons-exec\1.3\commons-exec-1.3.jar;C:\Utilisateurs\a754206\.m2\repository\net\java\dev\jna\jna\4.1.0\jna-4.1.0.jar;C:\Utilisateurs\a754206\.m2\repository\net\java\dev\jna\jna-platform\4.1.0\jna-platform-4.1.0.jar;C:\Utilisateurs\a754206\.m2\repository\org\seleniumhq\selenium\selenium-java\2.53.0\selenium-java-2.53.0.jar;C:\Utilisateurs\a754206\.m2\repository\org\seleniumhq\selenium\selenium-chrome-driver\2.53.0\selenium-chrome-driver-2.53.0.jar;C:\Utilisateurs\a754206\.m2\repository\org\seleniumhq\selenium\selenium-edge-driver\2.53.0\selenium-edge-driver-2.53.0.jar;C:\Utilisateurs\a754206\.m2\repository\org\seleniumhq\selenium\selenium-firefox-driver\2.53.0\selenium-firefox-driver-2.53.0.jar;C:\Utilisateurs\a754206\.m2\repository\org\seleniumhq\selenium\selenium-ie-driver\2.53.0\selenium-ie-driver-2.53.0.jar;C:\Utilisateurs\a754206\.m2\repository\org\seleniumhq\selenium\selenium-safari-driver\2.53.0\selenium-safari-driver-2.53.0.jar;C:\Utilisateurs\a754206\.m2\repository\io\netty\netty\3.5.7.Final\netty-3.5.7.Final.jar;C:\Utilisateurs\a754206\.m2\repository\org\seleniumhq\selenium\selenium-support\2.53.0\selenium-support-2.53.0.jar;C:\Utilisateurs\a754206\.m2\repository\org\seleniumhq\selenium\selenium-leg-rc\2.53.0\selenium-leg-rc-2.53.0.jar;C:\Utilisateurs\a754206\.m2\repository\org\json\json\20180813\json-20180813.jar;C:\Program Files\JetBrains\IntelliJ IDEA Community Edition 2019.1\plugins\testng\lib\jcommander-1.27.jar" org.testng.RemoteTestNGStarter -usedefaultlisteners false -socket51973 #w#C:\Utilisateurs\a754206\AppData\Local\Temp\idea_working_dirs_testng.tmp -temp C:\Utilisateurs\a754206\AppData\Local\Temp\idea_testng.tmp
10:00:20.385 [main] INFO c.q.q.a.core.ConfigurationManager - ISFW build info: {qaf-Type=core, qaf-Revision=15, qaf-Version=2.1, qaf-Build-Time=20-Nov-2019 22:28:06}
10:00:20.409 [main] INFO c.q.q.a.core.ConfigurationManager - Resource dir: C:\Utilisateurs\a754206\IdeaProjects\perso\cucumber\resources. Found property files to load: 0
10:00:20.417 [main] INFO c.q.q.a.core.ConfigurationManager - Resource dir: C:\Utilisateurs\a754206\IdeaProjects\perso\cucumber\resources. Found property files to load: 0
include groups []
exclude groups: [] Scanarios location: src/test/resources/features/test.feature
10:00:20.430 [main] INFO c.q.q.a.s.c.g.GherkinScenarioFactory - scenario.file.loc[Ljava.lang.String;#2a40cd94
10:00:20.511 [main] INFO c.q.q.a.s.c.text.BehaviorScanner - loading feature file: src/test/resources/features/test.feature
10:00:20.542 [main] INFO c.q.q.a.s.c.g.GherkinScenarioFactory - total test found: 1
Test ignored.
Preparing For Shut Down...
java.lang.NullPointerException
at java.util.ArrayList.<init>(ArrayList.java:178)
at com.qmetry.qaf.automation.testng.pro.QAFTestNGListener.report(QAFTestNGListener.java:234)
at com.qmetry.qaf.automation.testng.pro.QAFTestNGListener2.report(QAFTestNGListener2.java:174)
at com.qmetry.qaf.automation.testng.pro.QAFTestNGListener.onConfigurationFailure(QAFTestNGListener.java:126)
at org.testng.internal.Invoker.runConfigurationListeners(Invoker.java:1686)
at org.testng.internal.Invoker.handleConfigurationFailure(Invoker.java:305)
at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:243)
at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:144)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:326)
at org.testng.SuiteRunner.run(SuiteRunner.java:289)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1293)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1218)
at org.testng.TestNG.runSuites(TestNG.java:1133)
at org.testng.TestNG.run(TestNG.java:1104)
at org.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:73)
at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:123)
I debbuged, and in fact there is a double error: a NullPointerException happens during the run (see below), so then we arrive in onConfigurationFailure, but then again there is a NullPointerException (the one above)
java.lang.NullPointerException
at com.qmetry.qaf.automation.testng.pro.QAFTestNGListener.beforeInvocation(QAFTestNGListener.java:174)
at org.testng.internal.invokers.InvokedMethodListenerInvoker$InvokeBeforeInvocationWithContextStrategy.callMethod(InvokedMethodListenerInvoker.java:92)
at org.testng.internal.invokers.InvokedMethodListenerInvoker$InvokeBeforeInvocationWithContextStrategy.callMethod(InvokedMethodListenerInvoker.java:88)
at org.testng.internal.invokers.InvokedMethodListenerInvoker.invokeListener(InvokedMethodListenerInvoker.java:62)
at org.testng.internal.Invoker.runInvokedMethodListeners(Invoker.java:557)
at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:494)
at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:217)
at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:144)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:326)
at org.testng.SuiteRunner.run(SuiteRunner.java:289)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1293)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1218)
at org.testng.TestNG.runSuites(TestNG.java:1133)
at org.testng.TestNG.run(TestNG.java:1104)
at org.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:73)
at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:123)
Recap of when and where the exceptions happen:
// TestNG SuiteRunner#run begins
// Eventually we arrive in QAFTestBase where the first NullPointerException happens
public List<LoggingBean> getLog() {
return (List<LoggingBean>) getContext().getObject(COMMAND_LOG); // NullPointerException. getContext() is OK but getContext().getObject(CHECKPOINTS) returns null
}
// QAFTestNGListener#onConfigurationFailure is called
// Eventually we arrive in QAFTestBase again, where the second NullPointerException happens
public List<CheckpointResultBean> getCheckPointResults() {
return (List<CheckpointResultBean>) getContext().getObject(CHECKPOINTS); // NullPointerException. getContext() is OK but getContext().getObject(CHECKPOINTS) returns null
}
Create XML configuration file to run your features:
<test name="Gherkin-QAF-Test">
<parameter name="step.provider.pkg" value="pkg.to.look.for.steps" />
<parameter name="scenario.file.loc" value="resources/features" />
<classes>
<class name="com.qmetry.qaf.automation.step.client.gherkin.GherkinScenarioFactory" />
</classes>
</test>
Refer documentation for BDD configuration. Run it as TestNG test.
Which version of cucumber step implementation you have?
If it is cucumber 5+ add qaf-cucumber dependency. You can find latest dependency in maven central. Also use latest version (as of today QAF 2.1.15)
EDIT:
Cucumber 4:
Cucumber 4 users can use qaf-cucmber4 and keep using cucumber as it is with additional capabilities. It will allow you to use external test data, and parameterize step arguments as well as meta-data rule and meta-data format. Below are steps:
add qaf-cucumber4 dependency. Make suer to put dependency before
other cucumner dependencies.
add QAFCucumberPlugin (see junit example below).
You can then use BDD2 syntax features which is super set of gherkin.
#RunWith(Cucumber.class)
#CucumberOptions(plugin = {"com.qmetry.qaf.automation.cucumber.QAFCucumberPlugin"}, <other options>)//
public class RunCucumberTest {
}
Cucumber 5:
add qaf-cucumber dependency. Make suer to put dependency befor
other cucumner dependencies.
To run using cucumner runner add QAFCucumberPlugin (see junit example above).
To Run using TestNG use QAF BDD2 factory with TestNG
Here is example TestNG configuration file.
<test name="Gherkin-QAF-Test">
<parameter name="step.provider.pkg" value="pkg.to.look.for.steps" />
<parameter name="scenario.file.loc" value="resources/features" />
<classes>
<class name="com.qmetry.qaf.automation.step.client.text.BDDTestFactory2" />
</classes>
</test>
Note: qaf-cucumber4 is not supporting running with testng using bdd factory.
Turns out the exceptions were due to the following dependency, which has nothing to do with QAF as it simply is a plugin to generate cucumber reports in html format:
<dependency>
<groupId>net.masterthought</groupId>
<artifactId>cucumber-reporting</artifactId>
<version>4.10.0</version>
</dependency
Without it, everything is fine. I did not bother to dig why because I don't care and have no time for this.
I am now able to run tests with QAF, but it is important to note that:
#QAFTestStep and #QAFTestStepProvider do not work with cucumber annotations of io.cucumber.java.en.*, but only with their deprecated version of cucumber.api.java.en.*. Otherwise there will be a com.qmetry.qaf.automation.step.StepNotFoundException
Cucumber parameters like {string} must be escaped like \\{string\\} every single time. Otherwise there will be a java.util.regex.PatternSyntaxException: Illegal repetition
Spring dependency injection is more complex with TestNG than it is with Cucumber alone: steps classes must extends AbstractTestNGSpringContextTests (which imply to import the spring-test libray) and be annotated with #ContextConfiguration. Also I could not make this work, so may be it is even more complex when using QAF.
I discovered that while just scraching the surface of TestNG+QAF, so my opinion is that it is not worth the pain if like me you only want to use QAF for its feature of externalizing data.
If you use qaf to run testng with cucumber, in your log there is "include groups []" shows that you didn't include your test case.
So you should add you test case in testng xml like below:
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="TestNG - Cucumber" verbose="0">
<test name="Gherkin-QAF-Test">
<parameter name="step.provider.pkg" value="com.steps" />
<parameter name="scenario.file.loc" value="src/test/resources/features/test.feature" />
<groups>
<run>
<include name="#test"/>
</run>
</groups>
<classes>
<class name="com.qmetry.qaf.automation.step.client.gherkin.GherkinScenarioFactory" />
</classes>
</test>
</suite>
And add #test tag in cucumber feature file:
Feature: Test for QAF
#test
Scenario: success
Given my scenario works
This should not be posted in cucumberjs. From what I can see this is Java?
I want to change the scenario status for known issues into the After hook.
Something like:
#After
public void afterScenario(Scenario scenario) {
if(scenario.isFailed() && scenario.getSourceTagNames().contains("knownIssue")){
//scenario.add(Result.SKIPPED)
}
}
The idea is tests, which fail because of known bug to be skipped into the test report.
Thanks,
Nayden
You can annotate the scenario with #KnownIssue and then run cucumber with --tags "not #KnownIssue" or its #CucumberOptions equivalent.
it is your test execution engine that is responsible for this aspect, aka TestNG, JUnit.
So the question is: How to change Test Execution Status Programmatically? Here is one explained. You can hook in to the Test Execution Engine either by the way explained in the artcicle or from your method - what is important - you have to work with the Test Execution Engine and not Cucumber.
Gherkin with qaf supports what you are looking for. With qaf
you can achieve it by test ng after method invocation listener.
In addition you also can add meta-data at step level and handle it in onFailure method of step listener to modify exception to Skip exception depending on step meta data. Which automatically skips tests where that step is called and failed.
I am using Serenity with BDD and need to perform a teardown step that must get executed after completion of each Scenario. Also, this teardown step should not be visible to report as it is technical thing and nothing to do with behavior to be exposed as part of cucumber such as releasing few expensive resource that got
I used cucumber's #After annotation which is working as expected, but the problem is now this step is also shown in my Report which I don't want to be visible.
Could someone please suggest me a solution that allows me to perform teardown step that gets executed per scenario but should not be added as step in my Serenity Report.
Current Solution I have is which does not satisfy my need:
Step Definition Class has following method:
#After
public void tearDown() {
systemAction.deleteCostlyResource(id);
}
but #After annotation makes it a candidate for Reporting Step.
If you are using Dependency Injection, you could have your DI framework teardown the resources at the end of the scenarios?
For instance, if you are using Spring:
If the "costly resource" is a class that you yourself have created, mark it with:
#Component
#Scope("cucumber-glue")
If the "costly resource" is not a class you created, but provided by a framework or whatever, you can register it as a bean in your spring (test)configuration and mark it with a "destroy method".
For example, to register Selenium WebDriver using annotation based configuration and making sure to quit after each Scenario, mark it with:
#Bean(destroyMethod = "quit")
In this example, quit() is WebDriver's method to quit(). In your situation, call "costly resource's" quit method, or equivalent thereof.
We use threadcount attribute in #Test annotation of TestNg but there is no such attribute for #DataProvider annotation so is it possible to specify the thread count for Dataprovider, if yes then how? I specified Parallel=true in #DataProvider annotation and tried to read the data from Excel, there were 19 sets of test data and when executed then by default I found that 10 threads got created. I used Thread.currentThread() along with each set of Test Data to display the thread being used but I failed to understand the process and reason for 10 threads getting created.
That is because 10 is used as the default value by testng when parallel is set to true on a DataProvider. This can be overridden by data-provider-thread-count attribute on the suite tag. Example:
<suite name="Sample Suite" data-provider-thread-count="20">
<test name="Component Tests">
<packages>
<package name="com.something.tests" />
</packages>
</test>
</suite>
The testng documentation on DataProvider has this and much more: http://testng.org/doc/documentation-main.html#parameters-dataproviders
How can you get a jenkins groovy script to produce a junit xml results file? I'm doing this purely for the purpose of generating junit results with a specific number of passed/failed and skipped test cases. I need this so that I have a set of test data to test against for another application. This other app goes out to various jenkins jobs and analyzes the junit results from the job's json output. I want to point my functional tests at this jenkins job for testing. (I can't use my real continuous integration jobs because that wouldn't be deterministic).
I've got a basic groovy test case like what's below. It runs but doesn't produce junit output. I didn't expect it to, but I'm also not sure how to get it to generate one.
class BunchOfTests extends GroovyTestCase {
void testOne(){}
void testTwo(){fail()}
}
I also played around with writing code that prints the junit results xml but it's getting lengthy and quite ugly. I've seen the threads on here about what the junit results xsd looks like but I'm thinking there's got to be an easier route to generating some results without needing a pre-made results file. 10 results or so ought to be enough for what I need.
Generally unit testcases in groovy has to be written as below
import groovy.util.GroovyTestCase
class sampleTest extends GroovyTestCase {
assertEquals(true, val);
}
Incase you want only Junit reporting in groovy.
How would I produce JUnit test report for groovy tests, suitable for consumption by Jenkins/Hudson?