I want to have an option on the cucumber report to mute/hide scenarios with a given tag from the results and numbers.
We have a bamboo build that runs our karate repository of features and scenarios. At the end it produces nice cucumber html reports. On the "overview-features.html" I would like to have an option added to the top right, which includes "Features", "Tags", "Steps" and "Failures", that says "Excluded Fails" or something like that. That when clicked provides the same exact information that the overview-features.html does, except that any scenario that's tagged with a special tag, for example #bug=abc-12345, is removed from the report and excluded from the numbers.
Why I need this. We have some existing scenarios that fail. They fail due to defects in our own software, that might not get fixed for 6 months to a year. We've tagged them with a specified tag, "#bug=abc-12345". I want them muted/excluded from the cucumber report that's produced at the end of the bamboo build for karate so I can quickly look at the number of passed features/scenarios and see if it's 100% or not. If it is, great that build is good. If not, I need to look into it further as we appear to have some regression. Without these scenarios that are expected to fail, and continue to fail until they're resolved, it is very tedious and time consuming to go through all the individual feature file reports and look at the failing scenarios and then look into why. I don't want them removed completely as when they start to pass I need to know so I can go back and remove the tag from the scenario.
Any ideas on how to accomplish this?
Karate 1.0 has overhauled the reporting system with the following key changes.
after the Runner completes you can massage the results and even re-try some tests
you can inject a custom HTML report renderer
This will require you to get into the details (some of this is not documented yet) and write some Java code. If that is not an option, you have to consider that what you are asking for is not supported by Karate.
If you are willing to go down that path, here are the links you need to get started.
a) Example of how to "post process" result-data before rendering a report: RetryTest.java and also see https://stackoverflow.com/a/67971681/143475
b) The code responsible for "pluggable" reports, where you can implement a new SuiteReports in theory. And in the Runner, there is a suiteReports() method you can call to provide your implementation.
Also note that there is an experimental "doc" keyword, by which you can inject custom HTML into a test-report: https://twitter.com/getkarate/status/1338892932691070976
Also see: https://twitter.com/KarateDSL/status/1427638609578967047
Related
I want to have an option on the cucumber report to mute/hide scenarios with a given tag from the results and numbers.
We have a bamboo build that runs our karate repository of features and scenarios. At the end it produces nice cucumber html reports. On the "overview-features.html" I would like to have an option added to the top right, which includes "Features", "Tags", "Steps" and "Failures", that says "Excluded Fails" or something like that. That when clicked provides the same exact information that the overview-features.html does, except that any scenario that's tagged with a special tag, for example #bug=abc-12345, is removed from the report and excluded from the numbers.
Why I need this. We have some existing scenarios that fail. They fail due to defects in our own software, that might not get fixed for 6 months to a year. We've tagged them with a specified tag, "#bug=abc-12345". I want them muted/excluded from the cucumber report that's produced at the end of the bamboo build for karate so I can quickly look at the number of passed features/scenarios and see if it's 100% or not. If it is, great that build is good. If not, I need to look into it further as we appear to have some regression. Without these scenarios that are expected to fail, and continue to fail until they're resolved, it is very tedious and time consuming to go through all the individual feature file reports and look at the failing scenarios and then look into why. I don't want them removed completely as when they start to pass I need to know so I can go back and remove the tag from the scenario.
Any ideas on how to accomplish this?
Karate 1.0 has overhauled the reporting system with the following key changes.
after the Runner completes you can massage the results and even re-try some tests
you can inject a custom HTML report renderer
This will require you to get into the details (some of this is not documented yet) and write some Java code. If that is not an option, you have to consider that what you are asking for is not supported by Karate.
If you are willing to go down that path, here are the links you need to get started.
a) Example of how to "post process" result-data before rendering a report: RetryTest.java and also see https://stackoverflow.com/a/67971681/143475
b) The code responsible for "pluggable" reports, where you can implement a new SuiteReports in theory. And in the Runner, there is a suiteReports() method you can call to provide your implementation.
Also note that there is an experimental "doc" keyword, by which you can inject custom HTML into a test-report: https://twitter.com/getkarate/status/1338892932691070976
Also see: https://twitter.com/KarateDSL/status/1427638609578967047
i use the API to read my projects and tasks with C#. This works like a charm.
I read my tasks with the following endpoints.
https://quire.io/api/task/list/{project.oid}
and
https://quire.io/api/task/list/{task.oid}
However, the tasks have no references to the assigned tags. In the JSON-response the tags-property is always an empty array. But most of my tasks have at least one tag assigned.
Is this a known bug or did i miss something in the documentation?
Thanks for your Help.
I verified the api endpoints you describe. And in my case they result in the expected tags being listed per task. On both root tasks for a {project.oid} and subtasks for another {task.oid}.
Here a screenshot of what I get from command line (formatted the output using jq)
compared to the view inside quire:
So the answers to your questions...
No it's not a known bug on our side, and it doesn't look you missed anything based on your description. Maybe the URLs and results from my screenshots give you something to compare with.
If the problem persists on your side you can also contact support#quire.io with additional information you can't share in public.
We are trying to validate the HTML reports generated by our application. We have planned the below approach to do this
Capture that data related to the report from application
Generate the report
Identify the report’s elements and compare the data captured from application against the data/elements in report.
We started with identification of elements of the report and found that using 'Developers tool' we are able get some of the object properties where the Object ID is missing from these properties.
Can anyone please let us know the possibility of capturing the report elements and comparing them with the application data.
Frankly if you are just trying to parse HTML, and not trying to drive a browser, you'd probably be better off using something like Nokogiri, or another library aimed specifically at parsing HTML See https://www.ruby-toolbox.com/categories/html_parsing for a selection of such tools
Watir is 'Web Application Testing In Ruby" it is designed to drive web-browsers, in order to test websites and webapps. Validation that portions of the HTML are as expected is a part of that, but not the core functionality of Watir.
For what you are trying to do, if I understand you right, you could use watir to do that, in somewhat the same way you can use excel for word processing and page layout, which is to say it can be done, but may not be the most preferred tool or easiest way to go about it.
We use Hudson (well, Jenkins now) for CI. I have just started a project based on Node.js, and am investigating Expresso and Gently (testing and mocking). I really like the fact that Expresso works with node-jscoverage to generate code coverage reports.
Has anybody started a project to display Expresso and node-jscoverage reports (or JScoveage) in Hudson? Failing that, is there some documantation on what kind of output Hudson is expecting, short of inventing an entire new plugin?
In summary, I'm looking for two types of output here. Test results (like junit) and coverage reports (like Cobertura).
Do Expresso and node-jscoverage produce xml output?
If so,
Jenkins and cFix unit testing (C++)
Following the above, you could convert your xml output to formats that junit understand (for tests, using an XSLT), and convert your coverage xml output into cobertura format (again, using XSLT).
See this also:
http://www.van-porten.de/2009/05/cunit-tests-in-hudson/
for a sample XSLT.
You could try the xUnit plugin. On their wiki page it says that it can handle txt and csv files using custom style sheets. In theory this should work for your test reports t least. I have never tried that though.
As for coverage I am not aware of any plugins that can deal with arbitrary coverage tools.
If the HTML reports the tools produce are usable you could use the HTML Publisher plugin to link those reports in your job, and make them accessible from Jenkins. Not as nice an integration as a test tool plugin can provide, but depending on your expectations it might be enough.
Otherwise you will probably be forced to write a custom plugins. You could also try a request on the Jenkins Mailing List, maybe someone is working on such a plugin already.
Is there a way to get some help from RubyMine's code completion when using Capybara in Cucumber's step definitions? I'm new to Capybara, so not having to check the reference site all the time would be really helpful.
The best I can get at the moment is by explicitly calling Session.new, something like:
session = Capybara::Session.new(:rack_test, my_app)
This way Ctrl+Space after session. shows me methods from Capybara::Session (only) so at least I know it's somehow reachable. But that's not how I really use Capybara in my step definitions. I thought that helping the type inference engine by manually annotating page could do the trick, but I suppose all this DSL magic is too much to handle.
So basically, is it somehow possible to have
page.<Ctrl+Space>
pop up with all the exposed DSL methods? RubyMine API maybe? Or, as an alternative, some other way to bring the reference docs closer (I don't think RubyMine supports external docs in the IDE yet)?
As of RubyMine 8.0.3 the answer is no, RubyMine does not complete Capybara methods following page. in Cucumber step definitions, at least not when Capybara is included in Cucumber via the cucumber-rails gem. I don't see a feature request in the RubyMine issue tracker; someone could add one if they like.
Note that cucumber-rails, at least, includes the Capybara DSL in the Cucumber world, so you don't need to type page. in front of Capybara methods. You can just call visit, fill_in, etc. as self methods. I wouldn't want unnecessary page. in my step definitions just for the sake of RubyMine completion.
Unfortunately, RubyMine also doesn't include Capybara methods in the list of names it completes when you invoke completion in a step definition before you type anything. It does include Capybara methods in the list of names it completes when you invoke completion twice (all names in all available code), but since that list is so long it's only helpful if you already know the method you want already, or at least a correct prefix.
Finally, found a solution.
I am using Cucumber with Capybara and I included all the matchers I wanted to code complete in /features/support/spec/spec_helper.rb. Cucumber auto-loads everything in this file. I bet there are other places you can include these statements if you aren't using cucumber.
# Needed for RubyMine code completion
include Capybara::Node::DocumentMatchers
include Capybara::Node::Matchers
include Capybara::SessionMatchers
include Capybara::RSpecMatchers
include Capybara::RSpecMatcherProxies
For your specific case:
include Capybara::DSL
Then