I have a coded ui test that is written in C#.
When the test case is opened in MTM, no test steps are observed.
I have associated the automated, and this seems to be ok.
I tried to add some methods in my test method, and commented these methods with a summary, but this didn't help.
I will need these steps and I would like them tied back into the code, how is this done?
You must create an action recording for your test case/steps. A test method is created per step. You can then edit those test methods manually.
If you've handcoded a codedUi test then you can only associated the codedUI test method to the overall test case. Manual test steps should indicate what the codedUI test is doing, but there it's no connection between the two.
See Generating a Coded UI Test from an Existing Action Recording
Test steps are added to test cases in MTM. Commonly they provide instructions for manual testing. Such a manual test can be recorded when executed through MTM and that is referred to as an "action recording". The recorded test can be executed again via MTM, that avoids the tester having to perform the text entry and mouse-clicking needed. However, the action recording will not perform any validations of the expected results; that must be done manually.
The next facility is that a Coded UI test can be created from an action recording. The new Coded UI test does not included any validations of the expected results but the facilities of Coded UI can be used to assertions that can make the test fully automated. Having created a Coded UI test it can be linked back to the test case and will then be seen in the "associated automation" part of the test case. The linkage is created via the "Team Explorer" window in Visual Studio.
The order of events stated in the question suggests that the Coded UI test was created without using an "action recording"; which is a perfectly valid approach. The Coded UI test was then linked to an MTM test case. MTM has no mechanisms to decode the Coded UI test to create the test steps.
It would be possible to create a test case in MTM and specify its test steps but have an associated Coded UI test that does something different.
Related
I want to have an option on the cucumber report to mute/hide scenarios with a given tag from the results and numbers.
We have a bamboo build that runs our karate repository of features and scenarios. At the end it produces nice cucumber html reports. On the "overview-features.html" I would like to have an option added to the top right, which includes "Features", "Tags", "Steps" and "Failures", that says "Excluded Fails" or something like that. That when clicked provides the same exact information that the overview-features.html does, except that any scenario that's tagged with a special tag, for example #bug=abc-12345, is removed from the report and excluded from the numbers.
Why I need this. We have some existing scenarios that fail. They fail due to defects in our own software, that might not get fixed for 6 months to a year. We've tagged them with a specified tag, "#bug=abc-12345". I want them muted/excluded from the cucumber report that's produced at the end of the bamboo build for karate so I can quickly look at the number of passed features/scenarios and see if it's 100% or not. If it is, great that build is good. If not, I need to look into it further as we appear to have some regression. Without these scenarios that are expected to fail, and continue to fail until they're resolved, it is very tedious and time consuming to go through all the individual feature file reports and look at the failing scenarios and then look into why. I don't want them removed completely as when they start to pass I need to know so I can go back and remove the tag from the scenario.
Any ideas on how to accomplish this?
Karate 1.0 has overhauled the reporting system with the following key changes.
after the Runner completes you can massage the results and even re-try some tests
you can inject a custom HTML report renderer
This will require you to get into the details (some of this is not documented yet) and write some Java code. If that is not an option, you have to consider that what you are asking for is not supported by Karate.
If you are willing to go down that path, here are the links you need to get started.
a) Example of how to "post process" result-data before rendering a report: RetryTest.java and also see https://stackoverflow.com/a/67971681/143475
b) The code responsible for "pluggable" reports, where you can implement a new SuiteReports in theory. And in the Runner, there is a suiteReports() method you can call to provide your implementation.
Also note that there is an experimental "doc" keyword, by which you can inject custom HTML into a test-report: https://twitter.com/getkarate/status/1338892932691070976
Also see: https://twitter.com/KarateDSL/status/1427638609578967047
I want to have an option on the cucumber report to mute/hide scenarios with a given tag from the results and numbers.
We have a bamboo build that runs our karate repository of features and scenarios. At the end it produces nice cucumber html reports. On the "overview-features.html" I would like to have an option added to the top right, which includes "Features", "Tags", "Steps" and "Failures", that says "Excluded Fails" or something like that. That when clicked provides the same exact information that the overview-features.html does, except that any scenario that's tagged with a special tag, for example #bug=abc-12345, is removed from the report and excluded from the numbers.
Why I need this. We have some existing scenarios that fail. They fail due to defects in our own software, that might not get fixed for 6 months to a year. We've tagged them with a specified tag, "#bug=abc-12345". I want them muted/excluded from the cucumber report that's produced at the end of the bamboo build for karate so I can quickly look at the number of passed features/scenarios and see if it's 100% or not. If it is, great that build is good. If not, I need to look into it further as we appear to have some regression. Without these scenarios that are expected to fail, and continue to fail until they're resolved, it is very tedious and time consuming to go through all the individual feature file reports and look at the failing scenarios and then look into why. I don't want them removed completely as when they start to pass I need to know so I can go back and remove the tag from the scenario.
Any ideas on how to accomplish this?
Karate 1.0 has overhauled the reporting system with the following key changes.
after the Runner completes you can massage the results and even re-try some tests
you can inject a custom HTML report renderer
This will require you to get into the details (some of this is not documented yet) and write some Java code. If that is not an option, you have to consider that what you are asking for is not supported by Karate.
If you are willing to go down that path, here are the links you need to get started.
a) Example of how to "post process" result-data before rendering a report: RetryTest.java and also see https://stackoverflow.com/a/67971681/143475
b) The code responsible for "pluggable" reports, where you can implement a new SuiteReports in theory. And in the Runner, there is a suiteReports() method you can call to provide your implementation.
Also note that there is an experimental "doc" keyword, by which you can inject custom HTML into a test-report: https://twitter.com/getkarate/status/1338892932691070976
Also see: https://twitter.com/KarateDSL/status/1427638609578967047
I am doing BDD test on an app with cucumber, and I want to have clear instruction as it is recommanded in cucumber doc. The thing is that we have to do reusable step definitions so the maintenance cost is acceptable.
Example of scenario we have
Given I am on project page
When I click on 'buttonAddProject' //not easily readable
And I click on 'switchProjectPrivate'
And I click on 'buttonDeleteProject'
etc..
I don't want to have a function for each step like that: I change projet visibily or I delete project,
because this is basically just a click on a button, and we are going to have hundred of function like this. I also can't change the param in key to something more suitable, because every button key should be unique to avoid ambiguity.
So is there a way to do this with cucumber ?:
Given I am on project page
When I click on 'Add' //easily readable
And I click on 'Private'
And I click on 'Delete'
Bindings: //this keyword doesn't exist
'Add' : 'buttonAddProject'
'Private': 'switchProjectPrivate'
'Delete':'buttonDeleteProject'
I have tried that:
Scenario Outline:
Given I am on project page
When I click on <Add> //easily readable
And I click on <Private>
And I click on <Delete>
Examples:
|Add |Private |Delete |
|'buttonAddProject'|switchProjectPrivate'|'buttonDeleteProject'|
it works... but I need to do this for every scenario in the file, and if I really want to use scenario outline to iterate several times, I would have to copy paste this for every line, not really what I want.
How to organize this tests to make them more readable without making things to complex ?
First of all Cucumber scenarios that show HOW each thing is done are not maintainable or particularly useful.
What are cucumber scenario should describe and document is WHAT you are doing. To do this you need to determine WHY you are clicking on these buttons and what is achieved by these actions.
Now I have no idea from your scenarios about WHAT you are adding, WHY it is private or WHY you are then deleting it. But I can speculate from your post. The scenarios you should be writing should be something like.
Scenario: Delete a project
Given there is an existing project
And I am viewing the project
When I delete the project
Then ...
Scenario: Create a project
When I create a project
Then a project should be created
When you write your scenarios in this manner you push the details of how you interact with your UI down into your step definitions. So you might have something like
When 'I create a project' do
visit project_page
click "Create Project"
end
or better just
When 'I create a project' do
# must be on project page
click "Create Project"
When you work this way step definition re-use becomes less relevant and valuable. Each step does more and does something more specific.
You can continue this pattern of pushing the HOW down by having step definitions make calls to helper methods. This is particularly useful when dealing with Given's which get alot of re-use. Lets explore this with Given there is an existing project
Given 'there is an existing project' do
#project = create_project
end
Here we are pushing how we create an existing project down into the helper method create_project. The crude way to this would be to go through your UI visiting the project page and adding a new project. However this is really slow. You can optimise this process by bypassing your UI.
The most important point, whatever you decide to do, is that you are taking HOW you do something out of Cucumber and into some underlying code so now Cucumber is only interested in WHAT you are doing and WHY its important.
Making this change is probably the single most important thing you can do when Cuking. If you keep the HOW in your cucumber scenarios and step definitions you will end with a large number of brittle step definitions and very large scenarios that break all the time because everything is coupled together. You will get lots of bugs where making a change to get one step definition working causes lots of other scenarios to break. You will get lots of bugs where small changes to how you do a particular thing cause lots of unrelated scenarios to break.
Finally you are not doing BDD if you are writing the test after the code has been written. You can only do BDD if you write your scenarios collaboratively before the code is written.
Each step must be tied to a step definition. If you like to reuse an existing step def, you can just pass the command as argument (" Add", "Private","Delete"). You will have to use both the scenario name and the corresponding command to perform the required action.It will be something like this,
Scenario: scenario1_deleteproject
Given I am on project page
When I click on 'Add'
And I click on 'Private'
And I click on 'Delete'
Scenario: scenario2_createproject
Given I am on project page
When I click on 'Add'
And I click on 'Private'
And I click on 'Delete'
The step definition:
#When("When I click on {string}")
public void I_Click_On_Something(String command)
{
Switch(Command)
{
case Add:
//perform steps here
case delete:
//perform steps here
default:
}
If you want to differentiate the commands between the scenario, you will have to use scenario name ( need a class with definitions of scenario & command). You can grab the scenario name #Before hook.
I created data-driven coded ui test with more iterations. When I run the test and when first iteration is done, I want do put some clean up code before every other iteration to clean all changes that are made in the previous iteration.
Any idea how to do that?
Coded UI tests allow method with the [TestCleanup] attribute. Such methods are run after each test. If you are creating Coded UI test file you should find an example of a [TestCleanup] in comments in the CodedUItestN.cs file.
Methods with [ClassCleanup] and [AssemblyCleanup] attributes are also supported.
This SO question has more information Test Method that runs once at the Start of the Test?
In VS2012 RC, I noticed that the test list editor, tests view and test lists are absent.
I have an ordered test list - how do I run this within VS2012?
Normally, I would go to the test view, select the ordered test list, and run it.
But without the test view, what do I do?
From the new Test Explorer window, all the tests appear in the 'Not Run' section. When I attempt to run the test associated with the ordered test list, I get this error:
An exception occurred while invoking executor 'executor://orderedtestadapter/v1': Cannot find the test 'PhaseOneAdminLogonTest' with storage '..\bin\debug\codeduisanity.dll'.
Is there any way around this?
I was hoping to be able to start automating my tests using VS2012.
Looks like when projects are upgraded from VS2010, the ordered test lists aren't upgraded too. I created a new ordered test list, and added the same items, then the tests worked.