I have two e2e tests that run with browser automation that test the following user stories:
C: As a manager, I can add a member to my team.
D: As a manager, I can pay a member of my team.
The thing is, in order to do D I have to first do C. So writing one e2e test that goes ABC and once that goes ABCD is redundant!
My question is: what are some best practices for avoiding redundancy in e2e testing? One thing I have thought about is just writing ABCD, and including an assert after the C step. Though that means that if we have a failure at C, we have no information if D would have worked or not.
The answer is there in your question. Think of it from user flow. You know that ABCD is one logical chunk and step D cannot be performed without performing step C.So from e2e automation perspective, I would approach it in a way where I can mimic the end user action in one flow. having expectation post step C makes sense and if there is a failure it is absolutely correct to not perform step D and test result should tell the same.
Also, Since e2e tests come at the top in the test pyramid, you can very well figure out the cost, Time, efforts of not having redundant tests.
Related
We're using cucumberJS on a fairly large codebase with hundreds of cucumber scenarios and we've been running into issues with steps reuse.
Since all the steps in Cucumber are global, it's quite difficult to write steps like "and I select the first item in the list" or similar that would be similarly high-level. We end up having to append "on homepage" (so: "I select the first item in the list of folders on homepage") which just feels wrong and reads wrong.
Also, I find it very hard to figure out what the dependencies between steps are. For example we use a "and I see " pattern for storing a page object reference on the world cucumber instance to be used in some later steps. I find that very awkward since those dependencies are all but invisible when reading the .feature files.
What's your tips on how to use cucumber within a large team? (Including "ditch cucumber and use instead" :) )
Write scenarios/steps that are about what you are doing and why you are doing it rather than about how you do things. Cucumber is a tool for doing BDD. The key word here is Behaviour, and its interpretation. The fundamental idea behind Cucumber and steps is that each piece of behaviour (the what) has a unique name and place in the application, and in the application context you can talk about that behaviour using that name without ambiguity.
So your examples should never be in steps because they are about HOW your do something. Good steps never talk about clicking or selecting. Instead they talk about the reason Why you are clicking or selecting.
When you follow this pattern you end up with fewer steps at a higher level of abstraction that are each focused on a particular topic.
This pattern is easy to implement, and moderately easy to maintain. The difficulty is that to write the scenarios you have to have a profound understanding of what you are doing and why its important so you can discover/uncover the language you need to express yourself distinctly, clearly and simply.
I'll give my standard example about login. I use this because we share an understanding of What login is and Why its important. Realise before you can login that you have to be registered and that is complex.
Scenario: Login
Given I am registered
When I login
Then I should be logged in
The implementation of this is interesting in that I delegate all work to helper methods
Given I am registered
#i = create_registered_user
end
When I login
login_as(user: #i)
end
Then I should be logged in
should_be_logged_in
end
Now your problem becomes one of managing helper methods. What you have is a global namespace with a large number of helper methods. This is now a code and naming problem and All you have to do is
keep the number of helper methods as small as possible
keep each helper method simple
ensure there is no ambiguity between method names
ensure there is no duplication
This is still a hard problem, but
- its not as hard as what you are dealing with
- getting to this point has a large number of additional benefits
- its now a code problem, lots of people have experience of managing code.
You can do all these things with
- naming discipline (all my methods above have login in their name)
- clever but controlled use of arguments
- frequent refactoring and code cleaning
The code of your helper methods will have
- the highest churn of all your application code
- the greatest need to be simple and clear
So currently your problem is not about Cucumber its about debt you have with your existing scenarios and their implementation. You have to pay of your debt if you want things to improve, good luck
In Behavior Driven Development style of writing automated functional tests, it is generally understood that Givens should be the pre-condition that the system must be in, in order to begin the test, When should be the user action performed and Then should be asserting whether the observed matches the expected and fail or pass the test accordingly.
Off late my team has started performing assertions in Givens and Whens too which lead me to wonder if this is a correct practice.
For e.g -
Given a user with xyz privileges is logged in
When I click on the abc tab
Then records should be displayed
Should the Given in this test actually assert that the logged in user indeed has xyz privileges or assume the user has required privileges and just perform login
Also should the When assert that tab is visible before clicking?
If "logging in" is a behaviour that's interesting* and you need examples to illustrate it** then it should be a "When", with the context in which it happens being the "Given", and the outcome that results being the "Then".
This is the case for any behaviour you need to illustrate with an example.
However, sometimes it can be useful to make assertions in a Given, just to check that the context really is set up properly. Sometimes when people start adopting BDD the environments can be a bit flaky, and it's handy to know if it's your scenario finding a bug that made it fail, or something earlier in the process. So for that reason, you might find assertions there.
The Given doesn't concern itself with how the system got into that state, though. If it has assertions, it should merely be to check that it is.
Another form I've seen is a check that the system is in the correct state for the context, with corrective action if it isn't.
Note that these are largely interim patterns. They can be helpful while teams are adopting BDD and getting their pipelines and automated deployments into shape.
Assertions which check the results of the "When" are part of the outcome, so part of the "Then". I can't imagine a case where you'd need to check the results of a "When" without it being an outcome. If you've got one, please give me an example.
We discourage using clicking and UI details in scenarios. Work out what you're trying to achieve, and do that. Hide the clicking under the covers.
Most of the time automated scenarios aren't actually there to catch bugs; they're living documentation that helps people think about what they're trying to achieve and what the system already does, thus encouraging good design and preventing bugs in the first place.
I'd say something like "navigate to the ABC tab" and just do it; you'll get a relevant exception if it isn't there, and that won't happen as often as people reading the scenario will.
* It's logging in. It probably isn't.
** It's logging in. You probably don't.
Assertions in Givens and Whens are generally an indicator of immaturity of step definitions. So I might put one in a step that I am working on, but I wouldn't keep it there for very long.
I'd implement your step
Given a user with xyz privileges is logged in
as something like
'Given a user with xyz privileges is logged in' do
user = create_user(privileges: xyz)
login_as user: user
end
I would expect that the create_user method would be trusted pretty quickly and would not need an assertion. Same for the login_as method. (if methods like this aren't working properly you'd expect many scenarios to be broken)
Notice how this code makes it clear that there are two things going on and provides/uses an api that you would expect many other stepdefs to use. And how any assertion you might want to keep really belongs in the helper methods not in the step def.
There are no strict rules however in my experience I find it handy to define that there are no assertions there so it is clear what is actually tested and the test itself will run faster.
In the case if your abc tab is not displayed it shouldn't be clickable and then the test will fail either when identifying the object to click or when performing the next step(depends on the tools and method you use).
However you should make sure that the actual implementation of the test is not cheating and working with internals that are able to trigger a click even if the actual component isn't.
There is another point about the Given, there normally it is recommended to set up the environment for the test. This means you make sure that in your system there is a user and that user has been logged in. It makes no meaning to very that as you set it up, however if any failure occur you should fail fast to know what is wrong.
Generally I would be careful about using assertions in the Given or When steps in the automation code.
However, I use login steps like this all over the place in my code as - when used correctly - can turn the step into a piece of context rather than an assertion itself.
For instance:
Is this user logged in?
Yes => Do Nothing.
No => Log out, then Log in as this user
In the example you gave, we know that we need the specific user to be logged in for this scenario to work, however we do not know if there is a different user logged in (other scenarios may have been run before it). If you use this step as a check to make sure that the correct user is logged in (as in my example), it is part of the context and it will speed up run time of the automation as you won't have to log in before and log out after every scenario.
Recently, while looking for a job change on manual QA, I had interviewed on regular testing concepts questions. But, in a few companies,they gave some scenario and asked to write test case enumerations for it. Is it like test steps I need to write.? As per my knowledge, enumeration means complete, ordered list of all the items in a collection, so, is it writing all the test steps with description.?
Listing all possible test case names which could be extracted out of the scenario provided and classifying them in terms of priority and positive/negative/types is test case enumeration.
Kindly comment if you need anything, here is an example for better understanding.
Enumerate test cases for Login:(Classifying priorities into P1>P2>P3)
Positive cases include:
P1-Verify the login dialog box
P1-Verify the login id
P1-Verify the password
P1-Verify the submit button
Negative cases include:
1. P3-Verify logging in with empty id and password fields
Note: Haven't covered all the test cases.
Test Enumeration orders those scripts one by one- like 1,2,3... etc present in the test suite.It is just like defining the priority with which you want to run a specific script in a test suite.
For me enumeration means give for each test case identifier which is no 1, 2, 3 etc but which can tell you something, for example in very simple project you have three modules Users, Orders, Reports you can enumerate your use cases User.Accounts.1, User.Accounts.2..., User.Roles.1, User.Roles.2, Orders.Add.1, Orders.Edit.1, Orders.Edit.2, etc.
I gave long identifiers but you can short it or even replace names by numbers.
Other way (which is even much clear) you can gave names to use cases:
User.Accounts.Add account
User.Accounts.Edit account
User.Accounts.Remove account
User.Accounts.Remove account - negative (cannot remove)
User.Roles.Add role
etc...
This helps you (and others) to see if list of test cases you planned is full or you should add some new.
Can we achieve regression tesing coverage with BDD using JBehave/Cucumber?
Please share your inputs that the complete regression testing is achievable with Behavior Driver Development. (Jbehave/Cucmber).
In all but the most trivial products, it's impossible to perform complete regression testing.
Consider these acceptance criteria:
Items can be replaced or refunded.
This leads to two scenarios; one where we refund the item, and one where we return it. Now let's add a bit more to that:
Items are put in stock when returned or refunded, unless faulty.
Now we have four scenarios:
The one where we replace the item and it's faulty
The one where we refund the item and it's faulty
The one where we replace the item and put it back in stock
The one where we refund the item and put it back in stock.
Now let's add the criteria that a receipt must be in date. We need to check that refunds and replacements are both refused, but also that the item doesn't accidentally go back into stock, nor that any fault label is printed. So now we have eight scenarios.
Now let's think about the scenarios where we have a discount, and the ones where we can't scan the barcode, so we manually input the number, and the ones where the customer lost the receipt so we have to look it up using his loyalty card, and the ones where he paid by gift certificate...
Every scenario could, if the code was poorly designed, affect every other scenario. The number of potential combinations becomes exponential, very quickly.
We hope that the code is well-designed, and that the different aspects of behaviour are well-encapsulated. We hope that all the scenarios had been considered. However, if that was the case, we wouldn't be accidentally changing behaviour we didn't mean to, and we wouldn't need regression testing at all. So we know that at least some of the time, in most teams, changes to one scenario do affect changes in another.
Thinking about the responsibility of each piece of code can help to reduce this, which is why most teams practice both BDD and TDD (or BDD at a class level).
Additionally, it's impossible to ensure that every scenario has been thought of up-front, especially since every software project involves something new (or you wouldn't be doing it).
The only thing we can do is get confidence that the code works.
BDD is pretty good at giving us confidence. Not only does it help people to understand what the code does - so they are less likely to make mistakes and write bugs - but it also helps with automating the scenarios, so that there's less work for the testers, and they can focus more on looking for scenarios nobody's thought of yet (exploratory testing).
So, BDD can definitely help with regression testing... but nothing, not even BDD, can perform complete regression test coverage.
If Automation is excluded and from manual testing point of view, what is diffrerence between Test Strategy, Test Scenario, Test case and Test Script
**
Test Strategy
A Test Strategy document is a high level document and normally developed by project manager. This document defines “Software Testing Approach” to achieve testing objectives. The Test Strategy is normally derived from the Business Requirement Specification document.
Some companies include the “Test Approach” or “Strategy” inside the Test Plan, which is fine and it is usually the case for small projects. However, for larger projects, there is one Test Strategy document and different number of Test Plans for each phase or level of testing.
Components of the Test Strategy document
1)Scope and Objectives
2)Business issues
3)Roles and responsibilities
4)Communication and status reporting
5)Test deliverability
6)Industry standards to follow
7)Test automation and tools
8)Testing measurements and metrices
9)Risks and mitigation
10)Defect reporting and tracking
11)Change and configuration management
12)Training plan
**
Test Scenario
A scenario is a story that describes a hypothetical situation. In testing, you check how the program copes with this hypothetical situation.
The ideal scenario test is credible, motivating, easy to evaluate, and complex.
Scenarios are usually different from test cases in that test cases are single steps and scenarios cover a number of steps. Test suites and scenarios can be used in concert for complete system tests.
A Scenario is any functionality that can be tested. It is also called Test Condition ,or Test Possibility.
**
Test Cases
In software engineering, a test case is a set of conditions or variables under which a tester will determine if a requirement upon an application is partially or fully satisfied. It may take many test cases to determine that a requirement is fully satisfied. In order to fully test that all the requirements of an application are met, there must be at least one test case for each requirement unless a requirement has sub requirements. In that situation, each sub requirement must have at least one test case .
A test case is also defined as a sequence of steps to test the correct behavior of a functionality/feature of an application.
A sequence of steps consisting of actions to be performed on the system under test. (These steps are sometimes called the test procedure or test script). These actions are often associated with some set of data (preloaded or input during the test). The combination of actions taken and data provided to the system under test leads to the test condition. This condition tends to produce results that the test can compare with the expected results; I.e assess quality under the given test condition. The actions can be performed serially, in parallel, or in some other combination of consecution.
**
Test Script
Test Script is a set of instructions (written using a scripting/programming language) that is performed on a system under test to verify that the system performs as expected. Test scripts are used in automated testing.
Sometimes, a set of instructions (written in a human language), used in manual testing, is also called a Test Script but a better term for that would be a Test Case.
Test Scenario means " What to be tested" and test case means " How to be tested".
Test case: It consist of test case name, Precondition, steps / input condition, expected result.
Test Scenario: Test scenario consists of a detailed test procedure. We can also say that a test scenario has many test cases associated with it. Before executing the test scenario we need to think of test cases for each scenario.
Test Script: A Test Script is a set of instructions (written using a programming language) that is performed on a system under test to verify that the system performs as expected.
Test scripts is the term used when referring to automated testing. When you're creating a test script, you are using an automation tool to create your script.
Test strategy
outlines the testing approach and everything else that surrounds it. It is different from the test plan, in the sense that a Test strategy is only a sub set of the test plan. It is a hard core test document that is to an extent generic and static. There is also an argument about at what levels test strategy or plan is used- but I really do not see any discerning difference.
Example: Test plan gives the information of who is going to test at what time. For example: Module 1 is going to be tested by “X tester”. If tester Y replaces X for some reason, the test plan has to be updated.
On the contrary, test strategy is going to have details like – “Individual modules are to be tested by test team members. “ In this case, it does not matter who is testing it- so it’s generic and the change in the team member does not have to be updated, keeping it static.
Test scenario
This is a one line pointer that testers create as an initial, transitional step into the test design phase. This is mostly a one line definition of “What” we are going to test with respect to a certain feature. Usually, test scenarios are an input for the creation of test cases. In agile projects, Test scenarios are the only test design outputs and no test cases are written following these. A test scenario might result in multiple tests.
Examples test scenarios:
Validate if a new country can be added by the Admin
Validate if an existing country can be deleted by the admin
Validate if an existing country can be updated
Test Case:
Test Case is a commonly used term for a specific test. This is usually the smallest unit of testing. A Test Case will consist of information such as requirements testing, test steps, verification steps, prerequisites, outputs, test environment, etc.
A set of inputs, execution preconditions, and expected outcomes developed for a particular objective, such as to exercise a particular program path or to verify compliance with a specific requirement.
Test Script:
Commonly used to refer to the instructions for a particular test that will be carried out by an automated test tool
Test Scenarios: A high-level/simple/individual test panorama of actual system capability. We no need to define a clear step-by-step way of validation at this stage as we define test scenarios at very early stages of software life cycle. This will not be considered for test plan as this is a non-defined item in terms resource allocation.
Test Case: Is a document which consists of system specific prerequisites, but no step-by-step validation. In test case traceability we use a test case document against requirements. This is how we will define the test coverage matrix against requirements. In most of the cases, a test case will cover multiple test scenarios. A test case will carry complexity. Test cases are used for calculation of testing efforts for a particular release with respect to code version.
Test Script(without Automation/programming language context): Every one aware of the fact that a test script is an automation program which is uniquely mapped to a test case. But without automation as well we can use this term especially when you are using Rational Quality Manager(RQM) as your test repo.
1.When a test case has multiple versions and the testing team needs to maintain all test case versions against multiple system code versions.In this case, one test case will have multiple test scripts(one for each version).
2.When a test case produces different results in different environments(Operating system or technology.. etc), a test case will be mapped to multiple test scripts which have the expected results change but entire test case remains same.
In either of the above cases, while creating test plan we need to first decide on which version of the test case(in other terms, test script) for execution based on code version or the environment.
Hope this helps to answer your question.