I am evaluating using the test_ids gem and had a few questions:
Can test numbers be left un-configured, only configuring bins and softbins?
Can the test interface query another object for the TestId config? We have lots of test modules that have standardized hardbins but the softbins are always product specific due to test count variance. Would like to have the test interface query the current active test module for the binning config.
Can you explain why there are 3 'b's in the softbin config? Seems like you would only need one to create 100, 200, and 300.
config.softbins = :bbb00
thx
1) I think so, if it doesn't work it should. If you don't configure the test numbers in the TestIds config then it should not generate a test number.
2) Sure, up to application logic how to select between the different configs. It is definitely intended to work like that though. The goal of this plugin is to allow you to write a test flow which is bin/test number agnostic, then depending on what TestIds config you generate it with you can have completely different numbering schemes being output.
3) You would only need one in that case, its just saying that potentially the bin can be between 1 and 999, and from memory I think it will raise an error if it encounters a bin > 999 when presented with a config like that with only 3 places allocated.
Related
I have 135 Test Cases (and growing), and I can't seem to figure out how I can divide my test cases into different sections.
For example, I want my regression test cases to be part of a separate table, and stats test cases to be part of a separate table instead of a single long table. How would I go about doing something like this?
You can achieve this using allure reporting
https://docs.qameta.io/allure/
What is considered current best practice to test own strategies which are based on hypothesis? There are e.g. tests about how good examples shrink HypothesisWorks/hypothesis-python/tests/quality/test_shrink_quality.py. However I could not find examples so far which test the data generation functionality of strategies (in general, performance, etc.).
Hypothesis runs a series of health checks on each strategy you use, including for time taken to generate data and the proportion of generation attempts that succeed - try e.g. none().map(lambda x: time.sleep(2)).example() or integers().map(lambda x: x % 17 == 0).example() to see them in action!
In most cases you do not need to test your own strategies beyond using these healthchecks. Instead, I would check that your tests are sufficient by using a code coverage library.
I see that there is a TestNumberer class in the OrigenTesters at https://github.com/Origen-SDK/origen_testers/blob/master/lib/origen_testers/generator/test_numberer.rb . However, it looks pretty bare, and doesn't look like its being used internally anywhere. So, my question is does this TestNumberer... do anything? I don't see anything in the guides about automatically generating test numbers. What I'd like is something like:
test_numberer.set_base(1000) # for example
test_numberer.set_offset(5)
func (..., test_number: test_numberer.next) #=> test_number = 1000
func (..., test_number: test_numberer.next) #=> test_number = 1005
Possibly even embed incrementing the test number into the func function itself in the interface.
Thanks!
(For the record, I actually already have this in one of my apps for personal use, but am wondering if OrigenTesters already has one, and if not, if it could use one)
No, that is old and dead code which should be removed.
There is a solution for generating test numbers though, and that is the TestIds plugin: http://origen-sdk.org/test_ids/
I'm still not totally happy with how it works, but I use it in production today in a large test flow module.
I would say it does solve these problems effectively:
How do you assign test (and/or bin) numbers within an IP test block, in such a way that it can be included in different SoC test programs which may each want to assign a different range or even a different numbering scheme to the tests for that IP.
How do you automatically assign test (and/or bin) numbers in such a way that they will stick and won't change when other tests are added or removed from the flow in future.
I don't really have anything specific that I know is wrong with it, just some niggles that have come up from time to time and it would be good to get other people using it and involved with it to help iron these out.
One of the things that I have come to realize is that it is easier to manage if you explicitly give tests a number (or bin) ID in the test flow like this:
func :blah, number: :blah_test1
func :blah, number: :blah_test2
This makes it easier to control when you want same-named tests to have the same number or not, whilst not locking down to any particular number.
Anyway, you should find the documentation of it pretty good and obviously ask further questions here if you have any.
Recently, while looking for a job change on manual QA, I had interviewed on regular testing concepts questions. But, in a few companies,they gave some scenario and asked to write test case enumerations for it. Is it like test steps I need to write.? As per my knowledge, enumeration means complete, ordered list of all the items in a collection, so, is it writing all the test steps with description.?
Listing all possible test case names which could be extracted out of the scenario provided and classifying them in terms of priority and positive/negative/types is test case enumeration.
Kindly comment if you need anything, here is an example for better understanding.
Enumerate test cases for Login:(Classifying priorities into P1>P2>P3)
Positive cases include:
P1-Verify the login dialog box
P1-Verify the login id
P1-Verify the password
P1-Verify the submit button
Negative cases include:
1. P3-Verify logging in with empty id and password fields
Note: Haven't covered all the test cases.
Test Enumeration orders those scripts one by one- like 1,2,3... etc present in the test suite.It is just like defining the priority with which you want to run a specific script in a test suite.
For me enumeration means give for each test case identifier which is no 1, 2, 3 etc but which can tell you something, for example in very simple project you have three modules Users, Orders, Reports you can enumerate your use cases User.Accounts.1, User.Accounts.2..., User.Roles.1, User.Roles.2, Orders.Add.1, Orders.Edit.1, Orders.Edit.2, etc.
I gave long identifiers but you can short it or even replace names by numbers.
Other way (which is even much clear) you can gave names to use cases:
User.Accounts.Add account
User.Accounts.Edit account
User.Accounts.Remove account
User.Accounts.Remove account - negative (cannot remove)
User.Roles.Add role
etc...
This helps you (and others) to see if list of test cases you planned is full or you should add some new.
I want to create a feature file that will call other feature files. Is this possible? For example, I'd like a high level feature file that says something like this (only in proper formatting):
call feature1
call feature2
call feature3
and each call goes to a feature file containing, for example:
Scenario Outline:
Given this
Then that
And more
So with this example, you'd have 1 high level feature file, with 3 lower level feature files. Each sub-feature will probably be a scenario outline.
Thanks.
There is no Cucumber feature that allows you to run a feature from another feature. You can't even call a scenario from another scenario. (You could do that in early versions of Cucumber, but it was removed.)
You can share lists of steps among scenarios in a couple of ways, however:
you can run a list of steps before every scenario in a feature file by putting them in the Background section: https://www.relishapp.com/cucumber/cucumber/docs/background
you can write a high-level step that calls a list of low-level steps: Reuse Cucumber steps
Maybe you can restructure your problem to use one of these methods.