I am trying to create a test step in Jenkins using a python script. This python script will be used to automate regression tests. These tests are in a GitHub repository and what my python script will do is manipulate the requests after reading one test case at a time and send it across to the server which will return a pass or fail for that test case.
I wanted to visualize the test results of this python script and I stumbled upon Cucumber reports for Jenkins. I have installed the plugin and I wanted to know how to generate the HTML report of the tests I am going to execute using my python script.
Is there a way to create the json report in Python and feed that to the Cucumber UI plugin to generate the HTML report? Thanks! Please let me know what do I need to read/do to accomplish this task.
Based on my experience, but with Java, we were doing the following:
In your Runner class, you set a path to your json cucumber report, smth like (I took this example from here: Cucumber Report Path:
#CucumberOptions(
features = "src/test/resources/functionalTests",
glue= {"stepDefinitions"},
plugin = { "pretty", "json:target/cucumber-reports/Cucumber.json" },
monochrome = true
)
Jenkins has Cucumber Reports on Jenkins plugin for Cucumber reports
This plugin can 'eat' your json file report and will build a beautiful graphs for each build run.
Hope it will help you.
Related
We have few soapui projects each sending testrequests at different web services. The Groovy script that executes the tests is however for the most part identical for each project. Therefore we decided it would be good with regard to easy versioning and maintenance to keep the common script in separate "dummy" project ("TestWSScript-soapui-project.xml") with one testsuite/case (Autotest/Test) with only one testStep (Groovy script named "Run").
The idea is to have one project for each WebService (say WS1-soapui-project.xml) which has testSuite with one TestCase. Within this TestCase will be
Groovy test step to set WS specific properties and call the universal script from TestWSScript-soapui-project.xml
Request test step to call the webservice and perform assertions
Ending Groovy test step.
This works from within SoapUI, but I want to run the tests from Windows command line (batch file for automatization purposes). Here I ran into a problem: when invoking testrunner from command line with
set "SOAPUI_FOR_TEST_DIR=..\..\..\programs\SoapUI-5.6.0"
"%SOAPUI_FOR_TEST_DIR%\bin\testrunner.bat" -sAutoTest -r -a -j -I "..\resources\WS1-soapui-project.xml"
it does not load whole workspace with all SoapUi projects. Therefore the following script (in WS1-soapui-project.xml/AutoTest suite/Test TestCase) that should run testStep from project TestWSScript-soapui-project.xml/AutoTest suite/Test TestCase returns Null (more specifically "Cannot invoke method getProjectByName() on null object")
import com.eviware.soapui.model.project.ProjectFactoryRegistry
import com.eviware.soapui.impl.wsdl.WsdlProjectFactory
def workspace = testRunner.testCase.testSuite.project.workspace
def testProject = (workspace==null) ?
ProjectFactoryRegistry.getProjectFactory(WsdlProjectFactory.WSDL_TYPE).createNew("TestWSScript.xml") :
workspace.getProjectByName("TestWSScript")
if(!testProject.open && workspace!=null) workspace.openProject(testProject)
// Connect to the test step in another project.
def prj = testRunner.testCase.testSuite.project.workspace.getProjectByName('TestWSScript')
tCase = prj.testSuites['AutoTest'].testCases['Test']
tStep = tCase.getTestStepByName("Run")
// Call the test runner and check if it can run the specified step.
def runner = tStep.run(testRunner, context)
The called script just loops through parameters read from csv file and calls request step. It is irrelevant for the problem I need to solve as the issue happens before the script is called.
Is there even a way to achieve what we want? We are using the free version of SoapUI-5.6.0.
Thanks in advance.
Here is what I suggest in your case.
Instead of a separate project for just one groovy script, create a library out of it.
Create a class and methods as needed. Use method arguments or class members in case if you are passing data from the callers.
Use your existing script, convert them into class, methods.
Create the class(es) based on the need.
It can be programmed either java or groovy
Compile the class(es) and create library
Copy this library under %SOAPUI_HOME%\bin\ext directory
Now you can just call those methods in any project. No more dummy project is required.
Good thing is all the projects are independent.
Here is blog content created by Rupert Anderson, one of the SoapUI export and Author.
Steps
1.Create the following directory structure
soapuilib/src/main/groovy/custom
2.Get some Groovy code
For this example, I have knocked together a simple script to generate sequential ids. It may be of little practical use, but I wanted something with a simple public static method to call. Since the method is static, there will be no need to instantiate the class before calling it in step #8.
[groovy title=”SequentialIdGenerator.groovy”]
package custom
import java.util.concurrent.atomic.AtomicLong
public class SequentialIdGenerator {
public static final long counterSeed = 1000
public static final String prefix = "id"
private static AtomicLong counter = new AtomicLong(counterSeed)
public static String nextId() {
return prefix + counter.incrementAndGet()
}
}
[/groovy]
create the above script as a text file called SequentialIdGenerator.groovy
copy it to soapuilib/src/main/groovy/custom
3.Create Gradle build script
For this part, there are plenty of options to build the code and package it, such as Maven, Ant or just running the right shell commands! The following minimal Gradle script allows us to compile and package the code as a jar in one easy statement.
[code language=”groovy”]
apply plugin: ‘groovy’
version = ‘1.0’
jar {
classifier = ‘library’
manifest {
attributes ‘Implementation-Title’: ‘SoapUI Sample Groovy Library’, ‘Implementation-Version’: version
}
}
repositories {
mavenCentral()
}
dependencies {
compile ‘org.codehaus.groovy:groovy:2.1.7’ //Matches Groovy in SoapUI 5.2.1
}
[/code]
Create the above Gradle script as soapuilib/build.gradle
INFO: Groovy Version – (At time of writing) The current version of Groovy is v2.4.6, but SoapUI 5.2.1 ships with Groovy 2.1.7. If you try to compile with a Groovy version 2.3+ and use it with SoapUI, you will see an error popup and log message in like ‘org/codehaus/groovy/runtime/typehandling/ShortTypeHandling‘ – see http://glaforge.appspot.com/article/groovy-2-3-5-out-with-upward-compatibility for more details and options. Basically, you can still use the latest Groovy version, but will need to include an additional groovy-backports-compat23 dependency!
5.Compile it & Create jar file
Now we’re ready to use the Gradle script to compile the sample script from step #2 and package it as a jar file.
Open a shell/command prompt at soapuilib/
gradle clean build jar
You should then see output like:
[bash]
tests-MacBook-Pro:soapuilib test$ gradle clean build jar
:clean
:compileJava UP-TO-DATE
:compileGroovy
:processResources UP-TO-DATE
:classes
:jar
:assemble
:compileTestJava UP-TO-DATE
:compileTestGroovy UP-TO-DATE
:processTestResources UP-TO-DATE
:testClasses UP-TO-DATE
:test UP-TO-DATE
:check UP-TO-DATE
:build
BUILD SUCCESSFUL
Total time: 5.499 secs
This build could be faster, please consider using the Gradle Daemon: https://docs.gradle.org/2.12/userguide/gradle_daemon.html
[/bash]
and our new library jar file created under the directory:
soapuilib/build/soapuilib-1.0-sample.jar
6.Add jar file to SoapUI
To make our new Groovy library jar available for use in SoapUI, it should be added in SoapUI Home under the following external library directory:
SoapUI ext Directory
Or the Windows equivalent e.g. C:\Program Files\SmartBear\SoapUI-5.2.1\bin\ext
7.Verify jar file is imported
When SoapUI is restarted, you should see the following log entry indicating that the jar file has been successfully added to the SoapUI classpath:
SoapUI ext Lib Loaded
8.Call the code
Our SequentialIdGenerator has a public static method nextId() that we can call, so to do this we can either import the class (Example 1) or just prefix the class with its package (Example 2). See below:
Example 1 – Call from Groovy TestStep:
[code]
import custom.*
log.info SequentialIdGenerator.nextId()
[/code]
Gives output like:
[code]
Thu May 12 16:49:20 BST 2016:INFO:id1001
[/code]
Example 2 – Call from Property Expansion:
[code]
${= custom.SequentialIdGenerator.nextId()}
[/code]
EDIT:
Here is the sample code with context, log variable access.
<script src="https://gist.github.com/nmrao/c489a485bbe3418cf49d8442f9fb92eb.js"></script>
I have recently upgraded to version 1.0.0 from 0.9.6 and noticed that the generated karate-summary.html file, it doesn't display all the tested feature files in the JUnit 5 Runner unlike in 0.9.6.
What it displays instead was the last tested feature file only.
The below screenshots are from the provided SampleTest.java sample code (excluding other Tests for simplicity).
package karate;
import com.intuit.karate.junit5.Karate;
class SampleTest {
#Karate.Test
Karate testSample() {
return Karate.run("sample").relativeTo(getClass());
}
#Karate.Test
Karate testTags() {
return Karate.run("tags").relativeTo(getClass());
}
}
This is from Version 0.9.6.
And this one is from Version 1.0.0
However, when running the test below in 1.0.0, all the features are displayed in the summary correctly.
#Karate.Test
Karate testAll() {
return Karate.run().relativeTo(getClass());
}
Would anyone be kind to confirm if they are getting the similar result? It would be very much appreciated.
What it displays instead was the last tested feature file only.
This is because for each time you run a JUnit method, the reports directory is backed up by default. Look for other directories called target/karate-reports-<timestamp> and you may find your reports there. So maybe what is happening is that you have multiple JUnit tests that are all running, so you see this behavior. You may be able to over-ride this behavior by calling the method: .backupReportDir(false) on the builder. But I think it may not still work - because the JUnit runner has changed a little bit. It is designed to run one method at a time, when you are in local / dev-mode.
So the JUnit runner is just a convenience. You should use the Runner class / builder for CI execution, and when you want to run multiple tests and see them in one report: https://stackoverflow.com/a/65578167/143475
Here is an example: ExamplesTest.java
But in case there is a bug in the JUnit runner (which is quite possible) please follow the process and help the project developers replicate and then fix the issue to release as soon as possible.
Using soapUI free version(5.4.0). I have simple project with test-case which has test-steps and load-tests like:
TestCase
Test-steps
one
two
LoadTests
LoadOne
LoadTwo
I want to run LoadOne and LoadTwo load tests, using Groovy scripting, in separate script. How to do that?
Assuming you're executing a Groovy test step from the same project, the following script will execute a load test called LoadOne:
import com.eviware.soapui.impl.wsdl.loadtest.*;
// Get the load test
def loadTest = testRunner.testCase.getLoadTestByName("LoadOne");
// Run the load test
WsdlLoadTestRunner loadTestRunner = new WsdlLoadTestRunner(loadTest);
loadTestRunner.start(true);
loadTestRunner.waitUntilFinished();
Then, of course, do the same for your other load test.
We have introduced the PDK lately into our developments chain and are now trying to make everybody happy with the test outputs it generates.
We need an output as JUnit test report for our jenkins jobs. That we have solved.
And we need the output still on the console because some of the developers find it very annoying having to open the JUnit report file before they can see failed tests.
pdk test unit --format=junit:report.xml
Is how we configured the output for JUnit.
Unfortunately as soon as you configure the JUnit report no output gets printed on the console/stdout anymore. Even if you add another format like --format=text without target file.
Is there a way to achieve both without running the PDK twice?
It doesn't appear to be in the docs, but this should work.
pdk test unit --format=junit:report.xml --format=text:stdout
See https://github.com/puppetlabs/pdk/blob/7b2950bc5fb2e88ead7321c82414459540949eb1/lib/pdk/cli/util/option_normalizer.rb#L10-L24
I've filed a ticket to ensure that gets promoted to the docs at https://puppet.com/docs/pdk/1.x/pdk_reference.html#pdk-test-unit-command
From PDK documentation
--format=[:]
Specifies the format of the output. Optionally, you can specify a target file
for the given output format,
such as --format=junit:report.xml . Multiple --format options can be
specified as long as they all have distinct output targets
So I believe ,you can try as below
pdk test unit --tests=testcase_name --format=junit:report.xml --format=text:log.txt
Hope it helps.
I'm attempting to run tagged features in the order that they are submitted.
example:
I have tests that i'd like to run in a specific order (#test1, #test2, #test3). After looking at the cucumber documentation is looks like i'm only able to run them in an and/or option like
cucumber features/.feature --t #test1; cucumber features/.feature --t #test2; cucumber features/*.feature --t #test3;
but this prevents me from having a single report which contains all of the results.
Is there anyway which I can run these tests in their respective order and have all of the results contained in the same report?
If you put the tests that have to run in a specific order in a feature file together cucumber will run them in the order they are given. As this will be in your normal test run it should all show up in the same report.
But it might be worth looking into why your tests are dependant on each other and if there is a way to remove this dependancy as it is generally bad practice to have it.