Prior to creating a test program flow with tests that contain patterns there is what is commonly referred to as the 'test list'. This test list is commonly stored in Excel or as Jira tickets, and it determines what is run through simulation or emulation prior to first silicon. Given all of the test pattern generation capabilities of Origen, users must deal with this test list in some manner. I don't see any IP or top level $dut modeling for tests that would yield an API like this:
$dut.ddr.has_tests? # => true
$dut.ddr.tests # => [:ddr4_2133_dataeye, :ddr4_1867_dataeye]
Anyone who is creating patterns with Origen, can you explain how Origen interacts with your 'test list'?
thx
We don't tend to handle that in the model, see the comments about lists here: http://origen-sdk.org/origen//guides/pattern/running/?highlight=list
Related
We're using cucumberJS on a fairly large codebase with hundreds of cucumber scenarios and we've been running into issues with steps reuse.
Since all the steps in Cucumber are global, it's quite difficult to write steps like "and I select the first item in the list" or similar that would be similarly high-level. We end up having to append "on homepage" (so: "I select the first item in the list of folders on homepage") which just feels wrong and reads wrong.
Also, I find it very hard to figure out what the dependencies between steps are. For example we use a "and I see " pattern for storing a page object reference on the world cucumber instance to be used in some later steps. I find that very awkward since those dependencies are all but invisible when reading the .feature files.
What's your tips on how to use cucumber within a large team? (Including "ditch cucumber and use instead" :) )
Write scenarios/steps that are about what you are doing and why you are doing it rather than about how you do things. Cucumber is a tool for doing BDD. The key word here is Behaviour, and its interpretation. The fundamental idea behind Cucumber and steps is that each piece of behaviour (the what) has a unique name and place in the application, and in the application context you can talk about that behaviour using that name without ambiguity.
So your examples should never be in steps because they are about HOW your do something. Good steps never talk about clicking or selecting. Instead they talk about the reason Why you are clicking or selecting.
When you follow this pattern you end up with fewer steps at a higher level of abstraction that are each focused on a particular topic.
This pattern is easy to implement, and moderately easy to maintain. The difficulty is that to write the scenarios you have to have a profound understanding of what you are doing and why its important so you can discover/uncover the language you need to express yourself distinctly, clearly and simply.
I'll give my standard example about login. I use this because we share an understanding of What login is and Why its important. Realise before you can login that you have to be registered and that is complex.
Scenario: Login
Given I am registered
When I login
Then I should be logged in
The implementation of this is interesting in that I delegate all work to helper methods
Given I am registered
#i = create_registered_user
end
When I login
login_as(user: #i)
end
Then I should be logged in
should_be_logged_in
end
Now your problem becomes one of managing helper methods. What you have is a global namespace with a large number of helper methods. This is now a code and naming problem and All you have to do is
keep the number of helper methods as small as possible
keep each helper method simple
ensure there is no ambiguity between method names
ensure there is no duplication
This is still a hard problem, but
- its not as hard as what you are dealing with
- getting to this point has a large number of additional benefits
- its now a code problem, lots of people have experience of managing code.
You can do all these things with
- naming discipline (all my methods above have login in their name)
- clever but controlled use of arguments
- frequent refactoring and code cleaning
The code of your helper methods will have
- the highest churn of all your application code
- the greatest need to be simple and clear
So currently your problem is not about Cucumber its about debt you have with your existing scenarios and their implementation. You have to pay of your debt if you want things to improve, good luck
I am reading a lot about Gherkin, and I had already read that it was not good to repeat steps, and for this it is necessary to use the keyword "Background", but in the example of this page they are repeating the same "Given" again and again, Could it be that I am doing wrong? I need to know your opinion about it:
Like with several things, this a topic that will generate different opinions. On this particular example I would have moved the "Given that I select the post" to the Background section as this seems to be a pre-requisite to all scenarios on this feature. Of course this would leave the scenarios in the feature without an actual Given section but those would be incorporated from the Background section on execution.
I have also seen cases where sometimes the decision of moving steps to the Background is a trade-off between having more or less feature files and how these are structured. For example, if there are 10 scenarios for a particular feature with a lot of similar steps between them - but there are 1 or 2 scenarios which do not require a particular step, then those 1 or 2 scenarios would have to moved into a new feature file in order to have the exact same steps on the Background section of the original feature.
Of course it is correct to keep the scenarios like this. From a tester's perspective, the Scenarios/Test cases should run independently, therefore, you can keep these tests separately for each functionality.
But in case you are doing an integration testing, then some of these test cases can be merged, thus you can cover multiple test cases in one scenario.
And the "given" statement is repeating, therefore you can put that in the background, so you don't have to call it in each scenarios.
Note: These separate scenarios will be handy when you run the scripts separately with annotation tags, when you just have to check for a specific functionality, or a bug fix.
I have two doubts about BDD with Cucumber related to best practices.
I have a page to automate user registration.
You enter your personal data, such as: name, email, and confirmation
After that you select the options of your interest of the site (there are 10 comboboxes, can be as many as you want).
Insert a recaptcha and send.
I need to validate all cases of success, as well as failure.
So, here are my questions:
1) Page with recaptcha.
Since it is not possible to automate a recaptcha and this step naturally comes into my test, should I make a scenario for invalid recaptcha validation?
2) Is there any clever way for me to write a scenario exploring all the possible combinations of site interest options?
In my page:
( ) Economy
( ) Education
( ) Sports
( ) Recreation
( ) Travels
( ) ...
I want to be able to submit the test several times by testing 1 option selected, 2 options, 3 options, ..., all options.
But I just want to do it if there is a lean way to do it.
In other words: In Scenario Outline examples can I pass a null value in this case?
In line with what Thomas mentioned on Captcha I would say that this is one of the few things which cannot be automated to test (except the negative path).
I also agree with Thomas that you should not want to test every single possibility of the options using executable specifications, but rather use integration testing or possible even unit testing if the architecture of this part of the code allows that.
As for an actual executable scenario in Gherkin format I see something like the following for this functionality:
Given Paul supplied the incorrect Captcha
When he wants to register himself
Then he should not be registered
You can wonder whether we should use the implementation word Captcha in the scenario since it will be incorrect when we would substitute Captcha in our implementation for something else.
There could be a potential other scenario depending on whether or not someone is allowed to register when no options are selected:
Given Paul has not chosen any of the possible interest topics
When he wants to register himself
Then he should not be registered
notice the reuse of the sentences for the when and then part to allow for less test code.
Regarding captcha, I would probably verify that a broken captcha stops the user. Verifying the positive path is obviously hard since the captcha is there to stop bots and an automated verification is the same as a bot.
Regarding verifying all your options, I would see if I could do that below the surface. Doing this from the UI using a browser is slow and you are talking about 2^10 combinations. That's a lot of cases. If all combinations needs to be tested, test them against the controller instead. This is a case where a tool like Cucumber may not be your best option. A programming language may be better than Gherkin.
If you still want to use Cucumber, at least make it fast and avoid the browser. I wrote a blogpost about the right tool for the job. It might help you to understand why you don't have to go through to UI for all scenarios.
Recently, while looking for a job change on manual QA, I had interviewed on regular testing concepts questions. But, in a few companies,they gave some scenario and asked to write test case enumerations for it. Is it like test steps I need to write.? As per my knowledge, enumeration means complete, ordered list of all the items in a collection, so, is it writing all the test steps with description.?
Listing all possible test case names which could be extracted out of the scenario provided and classifying them in terms of priority and positive/negative/types is test case enumeration.
Kindly comment if you need anything, here is an example for better understanding.
Enumerate test cases for Login:(Classifying priorities into P1>P2>P3)
Positive cases include:
P1-Verify the login dialog box
P1-Verify the login id
P1-Verify the password
P1-Verify the submit button
Negative cases include:
1. P3-Verify logging in with empty id and password fields
Note: Haven't covered all the test cases.
Test Enumeration orders those scripts one by one- like 1,2,3... etc present in the test suite.It is just like defining the priority with which you want to run a specific script in a test suite.
For me enumeration means give for each test case identifier which is no 1, 2, 3 etc but which can tell you something, for example in very simple project you have three modules Users, Orders, Reports you can enumerate your use cases User.Accounts.1, User.Accounts.2..., User.Roles.1, User.Roles.2, Orders.Add.1, Orders.Edit.1, Orders.Edit.2, etc.
I gave long identifiers but you can short it or even replace names by numbers.
Other way (which is even much clear) you can gave names to use cases:
User.Accounts.Add account
User.Accounts.Edit account
User.Accounts.Remove account
User.Accounts.Remove account - negative (cannot remove)
User.Roles.Add role
etc...
This helps you (and others) to see if list of test cases you planned is full or you should add some new.
If Automation is excluded and from manual testing point of view, what is diffrerence between Test Strategy, Test Scenario, Test case and Test Script
**
Test Strategy
A Test Strategy document is a high level document and normally developed by project manager. This document defines “Software Testing Approach” to achieve testing objectives. The Test Strategy is normally derived from the Business Requirement Specification document.
Some companies include the “Test Approach” or “Strategy” inside the Test Plan, which is fine and it is usually the case for small projects. However, for larger projects, there is one Test Strategy document and different number of Test Plans for each phase or level of testing.
Components of the Test Strategy document
1)Scope and Objectives
2)Business issues
3)Roles and responsibilities
4)Communication and status reporting
5)Test deliverability
6)Industry standards to follow
7)Test automation and tools
8)Testing measurements and metrices
9)Risks and mitigation
10)Defect reporting and tracking
11)Change and configuration management
12)Training plan
**
Test Scenario
A scenario is a story that describes a hypothetical situation. In testing, you check how the program copes with this hypothetical situation.
The ideal scenario test is credible, motivating, easy to evaluate, and complex.
Scenarios are usually different from test cases in that test cases are single steps and scenarios cover a number of steps. Test suites and scenarios can be used in concert for complete system tests.
A Scenario is any functionality that can be tested. It is also called Test Condition ,or Test Possibility.
**
Test Cases
In software engineering, a test case is a set of conditions or variables under which a tester will determine if a requirement upon an application is partially or fully satisfied. It may take many test cases to determine that a requirement is fully satisfied. In order to fully test that all the requirements of an application are met, there must be at least one test case for each requirement unless a requirement has sub requirements. In that situation, each sub requirement must have at least one test case .
A test case is also defined as a sequence of steps to test the correct behavior of a functionality/feature of an application.
A sequence of steps consisting of actions to be performed on the system under test. (These steps are sometimes called the test procedure or test script). These actions are often associated with some set of data (preloaded or input during the test). The combination of actions taken and data provided to the system under test leads to the test condition. This condition tends to produce results that the test can compare with the expected results; I.e assess quality under the given test condition. The actions can be performed serially, in parallel, or in some other combination of consecution.
**
Test Script
Test Script is a set of instructions (written using a scripting/programming language) that is performed on a system under test to verify that the system performs as expected. Test scripts are used in automated testing.
Sometimes, a set of instructions (written in a human language), used in manual testing, is also called a Test Script but a better term for that would be a Test Case.
Test Scenario means " What to be tested" and test case means " How to be tested".
Test case: It consist of test case name, Precondition, steps / input condition, expected result.
Test Scenario: Test scenario consists of a detailed test procedure. We can also say that a test scenario has many test cases associated with it. Before executing the test scenario we need to think of test cases for each scenario.
Test Script: A Test Script is a set of instructions (written using a programming language) that is performed on a system under test to verify that the system performs as expected.
Test scripts is the term used when referring to automated testing. When you're creating a test script, you are using an automation tool to create your script.
Test strategy
outlines the testing approach and everything else that surrounds it. It is different from the test plan, in the sense that a Test strategy is only a sub set of the test plan. It is a hard core test document that is to an extent generic and static. There is also an argument about at what levels test strategy or plan is used- but I really do not see any discerning difference.
Example: Test plan gives the information of who is going to test at what time. For example: Module 1 is going to be tested by “X tester”. If tester Y replaces X for some reason, the test plan has to be updated.
On the contrary, test strategy is going to have details like – “Individual modules are to be tested by test team members. “ In this case, it does not matter who is testing it- so it’s generic and the change in the team member does not have to be updated, keeping it static.
Test scenario
This is a one line pointer that testers create as an initial, transitional step into the test design phase. This is mostly a one line definition of “What” we are going to test with respect to a certain feature. Usually, test scenarios are an input for the creation of test cases. In agile projects, Test scenarios are the only test design outputs and no test cases are written following these. A test scenario might result in multiple tests.
Examples test scenarios:
Validate if a new country can be added by the Admin
Validate if an existing country can be deleted by the admin
Validate if an existing country can be updated
Test Case:
Test Case is a commonly used term for a specific test. This is usually the smallest unit of testing. A Test Case will consist of information such as requirements testing, test steps, verification steps, prerequisites, outputs, test environment, etc.
A set of inputs, execution preconditions, and expected outcomes developed for a particular objective, such as to exercise a particular program path or to verify compliance with a specific requirement.
Test Script:
Commonly used to refer to the instructions for a particular test that will be carried out by an automated test tool
Test Scenarios: A high-level/simple/individual test panorama of actual system capability. We no need to define a clear step-by-step way of validation at this stage as we define test scenarios at very early stages of software life cycle. This will not be considered for test plan as this is a non-defined item in terms resource allocation.
Test Case: Is a document which consists of system specific prerequisites, but no step-by-step validation. In test case traceability we use a test case document against requirements. This is how we will define the test coverage matrix against requirements. In most of the cases, a test case will cover multiple test scenarios. A test case will carry complexity. Test cases are used for calculation of testing efforts for a particular release with respect to code version.
Test Script(without Automation/programming language context): Every one aware of the fact that a test script is an automation program which is uniquely mapped to a test case. But without automation as well we can use this term especially when you are using Rational Quality Manager(RQM) as your test repo.
1.When a test case has multiple versions and the testing team needs to maintain all test case versions against multiple system code versions.In this case, one test case will have multiple test scripts(one for each version).
2.When a test case produces different results in different environments(Operating system or technology.. etc), a test case will be mapped to multiple test scripts which have the expected results change but entire test case remains same.
In either of the above cases, while creating test plan we need to first decide on which version of the test case(in other terms, test script) for execution based on code version or the environment.
Hope this helps to answer your question.