I have a WDIO project that has many tests. Some tests need to be run consecutively while other tests can run in parallel.
I cannot run all tests in parallel because the tests that need to be run consecutively will fail, and I cannot run all tests consecutively because it would take far too long for the execution to finish.
For these reasons I need to find a way to run these tests both consecutively and in parallel. Is it possible to configure this WDIO project to accomplish this?
I run these tests through SauceLabs and understand that I can set the maxInstances variable to as many VMs as I'd like to run in parallel. Is it possible to set certain tests to use a high maxInstance while other tests have a maxInstance of 1?
Or perhaps there is a way to use logic logic via the test directories to run certain tests in parallel and others consecutively?
For example, if I have these tests:
'./tests/parallel/one.js',
'./tests/parallel/two.js',
'./tests/consecutive/three.js',
'./tests/consecutive/four.js',
Could I create some logic such as:
if(spec.includes('/consecutive/'){
//Do not run until other '/consecutive/' test finishes execution
} else {
//Run in parallel
}
How can I configure this WDIO project to run tests both consecutively and in parallel? Thank you!
You could create 2 separate conf.js files.
//concurrent.conf.js
exports.config = {
// ==================
// Specify Test Files
// ==================
specs: [
'./test/concurrent/**/*.js'
],
maxInstances: 1,
and have one for parallel. To reduce duplication, create a shared conf.js and then simply override the appropriate values.
//parallel.conf.js
const {config} = require('./shared.conf');
config.specs = [
'./test/parallel/**/*.js'
],
config.maxInstances = 100,
And then when you run your tests you can do:
//parallel
wdio test/configs/parallel.conf.js
//concurrent
wdio test/configs/concurrent.conf.js
Here's an example of how to have a shared config file. And other config files using the shared one
Related
I am writing some tests for a Node/MongoDB project that runs various modules via command line entries. My question is, for the tests, is there a way I can simulate a command line entry? For instance, if what I write in the command line is:
TASK=run-comparison node server
... is there a way I can effectively simulate that within my tests?
The common practice here as far as I know, is to wrap as much of your app as you can within a function/class where you pass the arguments, so you can easily test it with unit tests:
function myApp(args, env){
// My app code with given args and env variables
}
In your test file:
// Run app with given env variable
myApp("", { TASK: "run-comparison"});
In your particular case, if all your tasks are set through env variables, through editing of process.env, mocks, or .env files you may be able to test that without modifications on your code.
If that is not enough for your case (i.e. you really need to exactly simulate command line execution) I wrote a small library to solve this exact issue some time ago: https://github.com/angrykoala/yerbamate (I'm not sure if there are other alternatives available now).
With the example you provided, A test case could be something like this:
const yerbamate = require('yerbamate');
// Gets the package.json information
const pkg = yerbamate.loadPackage(module);
//Test the given command in the package.json root dir
yerbamate.run("TASK=run-comparison node server", pkg.dir, {}, function(code, out, errs) {
// This callback will be called once the script finished
if (!yerbamate.successCode(code)) console.log("Process exited with error code!");
if (errs.length > 0) console.log("Errors in process:" + errs.length);
console.log("Output: " + out[0]); // Stdoutput
});
In the end, this is a fairly simple wrapper of native child_process which you could also use to solve your problem by directly executing subprocesses.
According to the docs one can increase the default async timeout from 5000ms using the jest-object
More specifically, by using the jestsettimeouttimeout
The issue I am facing is I am running a series of tests against an API that is very slow, 5-15 second response times, configuring this jest object at the top of each test is painfully annoying.
Is it possible to declare these settings once before all test files are run?
Jest offers a testTimeout configuration option you can add to your package.json:
"jest": {
"testTimeout": 15000,
}
OK, putting bits together:
Option "setupTestFrameworkScriptFile" was replaced by configuration "setupFilesAfterEnv", which supports multiple paths
https://jestjs.io/docs/en/jest-object#jestsettimeouttimeout
https://jestjs.io/docs/en/jest-object#jestdisableautomock
The Jest search box doesn't actually return anything when you search for: setupFilesAfterEnv
And docs talk about: setupTestFrameworkScriptFile (which also doesn't return anything on the search:/ )
Anyway, the docs leave you scratching your head but this works:
jest.config.js:
module.exports = {
setupFilesAfterEnv: ['./setup.js'],
setup.js:
jest.setTimeout(10000); // in milliseconds
The jest folks should make it easier to find this information.
Use testTimeout. In yourjest.config.js (or similar), add the following:
export SECONDS = 1000;
module.exports = {
testTimeout: 60 * SECONDS
}
If you are working with react and initializing you app using create-react-app, then under your src/ directory you should have a file named setupTests.js. Here you can setup a global timeout for all of your tests just by insert this line after the import statement for #testing-libary
jest.setTimeout(15000); // in milliseconds
I read the document and got confused on the difference between these two. I know codes in setupFiles would be executed before codes in setupTestFrameworkScriptFile. What else differences do they have?
I guess codes in these two would be executed before each test. Does that mean if I have 10 it(); they are executed 10 times?
setupTestFrameworkScriptFile and setupFiles are executed before each file containing tests. If you have 10 tests in one file - no mater how many describe's - it will run once. If in 2 separate files - it will run twice.
In both setupTestFrameworkScriptFile and setupFiles you can initiate globals, like this:
global.MY_GLOBAL = 42
setupFiles run before test framework is installed in the environment.
In setupTestFrameworkScriptFile you have also access to installed test environment, methods like describe, expect and other globals. You can for example add your custom matchers there:
expect.extend({
toHaveLength(received, argument) {
// ...
}
})
... or set a new maximum timeout interval:
jest.setTimeout(12000)
Is it possible in Jest to run cleanup or teardown tasks that run after all other tests have completed? Similar to how setupFiles allows one to set up tasks after before any test has run. Bonus points if this can also run regardless if the test had any errors.
Putting afterAll(() => {}) at the top level of a file (outside any describe function) appears only to run after tests from that particular file have finished.
The use case is I have many test files that will create users in a a development database, and I don't want to make each test file responsible for cleaning up and removing the user afterwards. Errors can also happen while writing tests, so if the cleanup happens regardless of errors that would be preferable.
There's a sibling hook to setupFiles that will too fire before every test suite but right after your test runner (by default Jasmine2) has initialised global environment.
It's called setupFilesAfterEnv. Use it like this:
{
"setupFilesAfterEnv": ["<rootDir>/setup.js"]
}
Example setup.js:
beforeAll(() => console.log('beforeAll'));
afterAll(() => console.log('afterAll'));
setup.js doesn't need to export anything. It will be executed before every test suite (every test file). Because test runner is already initialised, global functions like beforeAll and afterAll are in the scope just like in your regular test file so you can call them as you like.
In jest.config.js:
module.exports = {
// ...
setupFilesAfterEnv: [
"./test/setup.js",
// can have more setup files here
],
}
In ./test/setup.js:
afterAll(() => { // or: afterAll(async () => { }); to support await calls
// Cleanup logic
});
Note:
I am using Jest 24.8
Reference:
setupFilesAfterEnv
To do some tasks after all test suites finish, use globalTeardown. Example:
In package.json:
{
"jest": {
"globalTeardown": "<rootDir>/teardownJest.js"
},
}
In teardownJest.js:
const teardown = async () => {
console.log('called after all test suites');
}
module.exports = teardown;
Keep in mind that jest imports every module from scratch for each test suit and teardown file. From official documentation:
By default, each test file gets its own independent module registry
So, you cannot share the same DB module's instance for each test suite or teardown file. Therefore, If you wanted to close db connection after all test suits, this method would not work
There looks like there is a feature called a reporter that just does exactly this:
How do I force a Mochajs test to end completely without continuing on to the next tests. A scenario could be prevent any further tests if the environment was accidentally set to production and I need to prevent the tests from continuing.
I've tried throwing Errors but those don't stop the entire test because it's running asynchronously.
The kind of "test" you are talking about --- namely checking whether the environment is properly set for the test suite to run --- should be done in a before hook. (Or perhaps in a beforeEach hook but before seems more appropriate to do what you are describing.)
However, it would be better to use this before hook to set an isolated environment to run your test suite with. It would take the form:
describe("suite", function () {
before(function () {
// Set the environment for testing here.
// e.g. Connect to a test database, etc.
});
it("blah", ...
});
If there is some overriding reason that makes it so that you cannot create a test environment with a hook and you must perform a check instead you could do it like this:
describe("suite", function () {
before(function () {
if (production_environment)
throw new Error("production environment! Aborting!");
});
it("blah", ...
});
A failure in the before hook will prevent the execution of any callbacks given to it. At most, Mocha will perform the after hook (if you specify one) to perform cleanup after the failure.
Note that whether the before hook is asynchronous or not does not matter. (Nor does it matter whether your tests are asynchronous.) If you write it correctly (and call done when you are done, if it is asynchronous), Mocha will detect that an error occurred in the hook and won't execute tests.
And the fact that Mocha continues testing after you have a failure in a test (in a callback to it) is not dependent on whether the tests are asynchronous. Mocha does not interpret a failure of a test as a reason to stop the whole suite. It will continue trying to execute tests even if an earlier test has failed. (As I said above, a failure in a hook is a different matter.)
I generally agree with Glen, but since you have a decent use case, you should be able to trigger node to exit with the process.exit() command. See http://nodejs.org/api/process.html#process_process_exit_code. You can use it like so:
process.exit(1);
As of the mocha documentation, you can add --exit flag when you are executing the tests.
It will stop the execution whenever all the tests have been executed successfully or not.
ex:
mocha **/*.spec.js --exit