I have a soapui framework which is modular. This means that I can execute test cases based upon business operations which are organized into different suites. With this in mind, I will need data from other test cases to use in my current test case (which is in a different suite). To accomplish this, I use a Run TestCase step in my current test case which runs the test case in suite 1 and brings the needed data into my current test case (suite 2) via project properties. After I run the current test case, I need the project properties to be cleared. I have the groovy code to do that. Here’s the issue: Since this is modular, I need to ONLY clear the project properties after the CURRENT test case is run. Using a teardown script within the test case level, isn’t working because it will always clear the project properties EVEN IF this is not the current test case being run. Meaning, my current suite is suite 2. And all the test cases in suite 2 have a teardown script that removes the project properties. When I run a test case in suite 3, and need data from a test case in suite 2, the properties will not be present due to the teardown scripts found in suite 2 (at the test case level). Again, I only need it to clear when the last step is run from the current test case, but not effect any other test cases when doing the modular execution. I hope that makes sense.
As a side note, this framework allows me to test business operations by suite for ad hoc testing. It also allows me to run a full regression from beginning to end (testing all suites in a row). I need the solution to not ruin the full regression run as well.
Any ideas on how to do this?
In order to do this I had to create a setup and tear down script at every level: Project, Suite, and Test Case.
Within the setup script, I created a variable called Is_Running. I then create an if statement which says: If “Is_Running” is NULL, then fill that variable with the name of the project, suite, or test case that is currently being executed. For example, if I’m executing at the project level, this code first checks to see if there is anything in the container Is_running, and if not it writes the project name in that variable.
Then I use the teardown script in each level which says that if the Is_Running variable is equal to the current name of what ever level I’m running, then erase the project properties. This ensures that the project properties are only erased once the current level is finished executing and not during the middle of a test (when using other suites).
For example: If I start my testing at the suite level, and I choose to run “Suite3”, the setup script will write “Suite3” in the Is_Running variable. Once Suite3 engages Suite2 to run the needed test cases, Suite2’s setup script see’s that the Is_Running variable is not null so it does NOT write it’s name to the Is_Running container. As such, the Suite2 teardown script does not erase the project properties since the name does not match. Once Suite3 has completed all it’s test steps, the teardown script sees that the Is_Running is filled with Suite3, so it deletes the project properties.
This approach allows me to run the project at any level and for the project properties to be deleted only after the current suite is finished running. I needed to know groovy well enough to do all the work mentioned above, but the approach is what I was looking for in this question. If you know of a less complicated way, please leave me a note!
Related
I'm new with multithreading, here is my problem statement,
I have an xml file (TestCase.xml) where each tag resembles a test case something like below,
TestCase.xml
In turn, each main tag has a child-tag that links to another xml(TestStep.xml) which dictates the steps of the test case, it’s TS in the above example.
TestStep.xml
The execution always starts from the TestCase.xml based on the id provided. With this overview, I have 100 test cases in my suite & I want to execute them in parallel, i.e. execute at least 5-6 test cases at the same time. I’m not able to use external plug-ins like Testng, Junit, BDD or mavensurefire etc. After a lot of R&D we have ended up with Multithreading. I would need assistance on how to implement the same.
in my Jest test suite,
i use
jest.retryTimes(4)
because of the particular and instable architecture of the software. And this works as expected.
There are some test that must to pass at the first time so for these particular test i need to set
jest.retryTimes(1)
at the beginning of the test, restoring
jest.retryTimes(4)
at the end.
There are two problems :
The problem is this configuration is global, and test are executed
in parallel, so when this test start, it put 1 to jest retry for all
the test running at this moment. I would like to make only this
particular test fail the first time.
Jest Circus ignore the update of jest.retryTimes at the beginning and at the end
of the test, it keep considering 4 temptative before of raise the failure.
I read the documentation but I think I cannot obtain this result.
any suggestion?
Thanks
If there are two separated repositories including webdriverio/node automation tests for different projects A and B, but some of the tests from project A require to run tests that covers area from project B, can we perform a test call inside test belonging to project A, that would run test from the project B, located on other repository?
I know for this particular case the quickest solution would be just copy/paste the necessary code, but I'm wondering whether I'm able not to duplicate the test code by running particular test cases from other repositories.
Well, you can creates testcases as functions and after that you can import it into your project A/B/whatever.
///// project A
function loginTestcases(){
describe('Login', ()=>{
it('Login with valid credentials', ()=>{
//some tests
})
})
}
///// project B
// use it in your another project as
logiTestcases();
But personally I don't recommend this approach. It isn't clear what you test, what body testcase has etc. Also your test should be independent - so no dependencies on other testcases.
Check this:
Rule 10: Write Independent and Isolated Tests
An important methodology of test authoring is creating self contained, independent flows. This allows to run tests in high parallelism, which is crucial for scaling the test suites. If, for example, you have 1,000 tests that run for a minute each, running them one by one will take more than 16 hours. Minutes, at full concurrency, can cut this testing time to 1 minute.
https://devops.com/10-rules-for-writing-automated-tests/
I use the AAA syntax (Arrange, Act, Assert) in all my automatic tests (Unit tests, system tests, etc).
Recently I started to write Coded UI tests. Now I wonder whether AAA syntax fits here. Unlike unit tests, where each test has one act and assert (or more than one assert) and I can have hundreds of them which will run less than couple of minutes, the Coded UI tests will run much longer. So if I write coded UI tests the same way I write my unit tests it will take them couple of hours (if not longer) to run.
If I compare Coded UI tests with manual UI tests then the menual tests don't use the AAA syntax in order to save time (not doing the same 'Arrange' action over and over just to check a field's value after another click).
What do you do in your apps? how do you recommend write Coded UI tests?
Yep. Use the same approach here also. A manual tester does a lot of verification in a single test. When automating it is better to split the multiple verifications test case to smaller test case and do minimum assertion per test case only. This will make your test case maintenance easy in future. Its not a good practice to proceed with another set of AAA once you have already done an Assertion in a test method.
Time is not an issue. UI automation is supposed to run slow. CodedUI tests are usually run on test lab having test controller and test agents. Here your hundreds of test will run in parallel on all test agents, thus reducing the overall execution time.
My CodedUI test method goes like this:
[TestMethod]
public void VerifyWhenThisThenThis()
{
// Prepare test data, Perform prerequisite action
// Do test
// One or more assertions to verify one major requirement only.
// If other major verification exist then split the test case
// and write a new test method for it.
}
If test case is huge then ask the manual tester to split it(or split it yourself informing the tester). Maintain a separate automation test pack having short test cases than the one manual testers have.
I am writing a Rails 3.1 app, and I have a set of three cucumber feature files. When run individually, as with:
cucumber features/quota.feature
-- or --
cucumber features/quota.feature:67 # specifying the specific individual test
...each feature file runs fine. However, when all run together, as with:
cucumber
...one of the tests fails. It's odd because only one test fails; all the other tests in the feature pass (and many of them do similar things). It doesn't seem to matter where in the feature file I place this test; it fails if it's the first test or way down there somewhere.
I don't think it can be the test itself, because it passes when run individually or even when the whole feature file is run individually. It seems like it must be some effect related to running the different feature files together. Any ideas what might be going on?
It looks like there is a coupling between your scenarios. Your failing scenario assumes that system is in some state. When scenarios run individually system is in this state and so scenario passes. But when you run all scenarios, scenarios that ran previously change this state and so it fails.
You should solve it by making your scenarios completely independent. Work of any scenario shouldn't influence results of other scenarios. It's highly encouraged in Cucumber Book and Specification by Example.
I had a similar problem and it took me a long time to figure out the root cause.
I was using #selenium tags to test JQuery scripts on a selenium client.
My page had an ajax call that was sending a POST request. I had a bug in the javascript and the post request was failing. (The feature wasn't complete and I hadn't yet written steps to verify the result of the ajax call.)
This error was recorded in Capybara.current_session.server.error.
When the following non-selenium feature was executed a Before hook within Capybara called Capybara.reset_sessions!
This then called
def reset!
driver.reset! if #touched
#touched = false
raise #server.error if #server and #server.error
ensure
#server.reset_error! if #server
end
#server.error was not nil for each scenario in the following feature(s) and Cucumber reported each step as skipped.
The solution in my case was to fix the ajax call.
So Andrey Botalov and Doug Noel were right. I had carry over from an earlier feature.
I had to keep debugging until I found the exception that was being raised and investigate what was generating it.
I hope this helps someone else that didn't realise they had carry over from an earlier feature.