We use Hudson (well, Jenkins now) for CI. I have just started a project based on Node.js, and am investigating Expresso and Gently (testing and mocking). I really like the fact that Expresso works with node-jscoverage to generate code coverage reports.
Has anybody started a project to display Expresso and node-jscoverage reports (or JScoveage) in Hudson? Failing that, is there some documantation on what kind of output Hudson is expecting, short of inventing an entire new plugin?
In summary, I'm looking for two types of output here. Test results (like junit) and coverage reports (like Cobertura).
Do Expresso and node-jscoverage produce xml output?
If so,
Jenkins and cFix unit testing (C++)
Following the above, you could convert your xml output to formats that junit understand (for tests, using an XSLT), and convert your coverage xml output into cobertura format (again, using XSLT).
See this also:
http://www.van-porten.de/2009/05/cunit-tests-in-hudson/
for a sample XSLT.
You could try the xUnit plugin. On their wiki page it says that it can handle txt and csv files using custom style sheets. In theory this should work for your test reports t least. I have never tried that though.
As for coverage I am not aware of any plugins that can deal with arbitrary coverage tools.
If the HTML reports the tools produce are usable you could use the HTML Publisher plugin to link those reports in your job, and make them accessible from Jenkins. Not as nice an integration as a test tool plugin can provide, but depending on your expectations it might be enough.
Otherwise you will probably be forced to write a custom plugins. You could also try a request on the Jenkins Mailing List, maybe someone is working on such a plugin already.
Related
I want to have an option on the cucumber report to mute/hide scenarios with a given tag from the results and numbers.
We have a bamboo build that runs our karate repository of features and scenarios. At the end it produces nice cucumber html reports. On the "overview-features.html" I would like to have an option added to the top right, which includes "Features", "Tags", "Steps" and "Failures", that says "Excluded Fails" or something like that. That when clicked provides the same exact information that the overview-features.html does, except that any scenario that's tagged with a special tag, for example #bug=abc-12345, is removed from the report and excluded from the numbers.
Why I need this. We have some existing scenarios that fail. They fail due to defects in our own software, that might not get fixed for 6 months to a year. We've tagged them with a specified tag, "#bug=abc-12345". I want them muted/excluded from the cucumber report that's produced at the end of the bamboo build for karate so I can quickly look at the number of passed features/scenarios and see if it's 100% or not. If it is, great that build is good. If not, I need to look into it further as we appear to have some regression. Without these scenarios that are expected to fail, and continue to fail until they're resolved, it is very tedious and time consuming to go through all the individual feature file reports and look at the failing scenarios and then look into why. I don't want them removed completely as when they start to pass I need to know so I can go back and remove the tag from the scenario.
Any ideas on how to accomplish this?
Karate 1.0 has overhauled the reporting system with the following key changes.
after the Runner completes you can massage the results and even re-try some tests
you can inject a custom HTML report renderer
This will require you to get into the details (some of this is not documented yet) and write some Java code. If that is not an option, you have to consider that what you are asking for is not supported by Karate.
If you are willing to go down that path, here are the links you need to get started.
a) Example of how to "post process" result-data before rendering a report: RetryTest.java and also see https://stackoverflow.com/a/67971681/143475
b) The code responsible for "pluggable" reports, where you can implement a new SuiteReports in theory. And in the Runner, there is a suiteReports() method you can call to provide your implementation.
Also note that there is an experimental "doc" keyword, by which you can inject custom HTML into a test-report: https://twitter.com/getkarate/status/1338892932691070976
Also see: https://twitter.com/KarateDSL/status/1427638609578967047
I want to have an option on the cucumber report to mute/hide scenarios with a given tag from the results and numbers.
We have a bamboo build that runs our karate repository of features and scenarios. At the end it produces nice cucumber html reports. On the "overview-features.html" I would like to have an option added to the top right, which includes "Features", "Tags", "Steps" and "Failures", that says "Excluded Fails" or something like that. That when clicked provides the same exact information that the overview-features.html does, except that any scenario that's tagged with a special tag, for example #bug=abc-12345, is removed from the report and excluded from the numbers.
Why I need this. We have some existing scenarios that fail. They fail due to defects in our own software, that might not get fixed for 6 months to a year. We've tagged them with a specified tag, "#bug=abc-12345". I want them muted/excluded from the cucumber report that's produced at the end of the bamboo build for karate so I can quickly look at the number of passed features/scenarios and see if it's 100% or not. If it is, great that build is good. If not, I need to look into it further as we appear to have some regression. Without these scenarios that are expected to fail, and continue to fail until they're resolved, it is very tedious and time consuming to go through all the individual feature file reports and look at the failing scenarios and then look into why. I don't want them removed completely as when they start to pass I need to know so I can go back and remove the tag from the scenario.
Any ideas on how to accomplish this?
Karate 1.0 has overhauled the reporting system with the following key changes.
after the Runner completes you can massage the results and even re-try some tests
you can inject a custom HTML report renderer
This will require you to get into the details (some of this is not documented yet) and write some Java code. If that is not an option, you have to consider that what you are asking for is not supported by Karate.
If you are willing to go down that path, here are the links you need to get started.
a) Example of how to "post process" result-data before rendering a report: RetryTest.java and also see https://stackoverflow.com/a/67971681/143475
b) The code responsible for "pluggable" reports, where you can implement a new SuiteReports in theory. And in the Runner, there is a suiteReports() method you can call to provide your implementation.
Also note that there is an experimental "doc" keyword, by which you can inject custom HTML into a test-report: https://twitter.com/getkarate/status/1338892932691070976
Also see: https://twitter.com/KarateDSL/status/1427638609578967047
I've been using Cypress for a few weeks now and I want to know it is possible to access to the data on the left of the report video, so that I could make more specific report. I couldn't get the data from mochaawesome reports or some other reports that's been suggested on elsewhere.
Cypress-terminal-report is a nice plugin that allows you to get reports on the cypress console activity and has nice filtering functions. It appears on the report that are like a long console log.
I am re factoring large sets of tests in SoapUi.
Is there a way to automate creation and renaming of test cases/test steps through Groovy?
Thanks.
Possibly not what you're looking for but I've had some success manually editing the test suite xml using find and replace in a text editor. You need to be careful and make sure to back up a copy.
You can run Groovy code within SoapUI using SoapUI Groovy Console plugin. This way, you can change any property you want programmatically (within API constraints of course).
As for technical details on how to solve your actual problem, I can only refer you to this blog post and SoapUI's javadoc. Based on the blog post, you need to figure out what's given to you, and based on the API, you need to figure out how to achieve what you need.
In my case, I started with my project being bound as a project variable, and moved on from there.
There is really not so much information in the question about what you really need to achieve, but given the little you provided, one way to go would be to directly modify the soapUI project XML file. I have done this with some success in the past. The last I used soapUI, it UI did not come with massive refactoring functionality.
I want to use The GATE Predicate-Argument EXtractor Component (PAX), but I can't figure out how to load the plugin on GATE developer (ver. 7)
Please help me.
How do I load it?
The MultiPaX plugin is a bit of a complex beast. You need to download the package from the homepage, unpack it (use something like 7-zip if you're on Windows) and then build it using Ant, which you will need to install separately - GATE used to bundle a copy of Ant but that is no longer the case since version 7. Once you have the plugin compiled and packaged you should be able to load it through the plugin manager the same as any other plugin.
However, to get anything useful out of the PR you need to build quite a complex pipeline including at least one of SUPPLE, MiniPar or the Stanford Parser PR. Getting these to work is itself a non-trivial task...
The predicate argument extractor is not developed by the core GATE team, so specific questions are probably better posted in the semanticsoftware.info forum (linked from the bottom of http://www.semanticsoftware.info/pax) rather than the usual GATE user list.