Kicking off mocha describes in parallel - node.js

I want to be able to have all my describe statements in Mocha get kicked off in parallel. Can someone help me figure out how to do that?

You can't do this directly with mocha because it creates a list of it() callbacks and invokes them in order.
mocha-parallel-tests can do this if you're willing to move your describes into separate .js files. To convince yourself, install it somewhere and invoke it with a short --slow so it reports each time:
laptop:/tmp/delme$ npm install mocha-parallel-tests
laptop:/tmp/delme$ cd node_modules/mocha-parallel-tests
laptop:/tmp/delme/node_modules/mocha-parallel-tests$ ./bin/mocha-parallel-tests test/parallel/tests --timeout 10000 --slow 100
You will see that it ran three (very simple) tests suites in the time it took to run the longest.
If your tests don't depend on side-effects of earlier tests, you can make them all asynchronous.
A simple way to do this is to initiate the stuff that takes a while before the describe and use the regular mocha apparatus to evaluate it. Here, I create a bunch of promises which take a while to resolve and then iterate through the tests again, examining their results in a .then() function:
var expect = require("chai").expect;
var SlowTests = [
{ name: "a" , time: 250 },
{ name: "b" , time: 500 },
{ name: "c" , time: 750 },
{ name: "d" , time:1000 },
{ name: "e" , time:1250 },
{ name: "f" , time:1500 }
];
SlowTests.forEach(function (test) {
test.promise = takeAWhile(test.time);
});
describe("SlowTests", function () {
// mocha defaults to 2s timeout. change to 5s with: this.timeout(5000);
SlowTests.forEach(function (test) {
it("should pass '" + test.name + "' in around "+ test.time +" mseconds.",
function (done) {
test.promise.then(function (res) {
expect(res).to.be.equal(test.time);
done();
}).catch(function (err) {
done(err);
});
});
});
});
function takeAWhile (time) {
return new Promise(function (resolve, reject) {
setTimeout(function () {
resolve(time);
}, time);
});
}
(Save this as foo.js and invoke with mocha foo.js.)
Meta I disagree with the assertion that tests should be primarily be synchronous. before and after pragmas are easier but it's rare that one test invalidates all remaining tests. All discouraging asynchronous tests does is discourage extensive testing of network tasks.

Mocha does not support what you are trying to do out of the box. It runs tests sequentially. This has a big advantage when dealing with an unhandled exception: Mocha can be sure that it happened in the test that it is currently running. So it ascribes the exception to the current test. It is certainly possible to support parallel testing but it would complicate Mocha quite a bit.
And I tend to agree with David's comment. I would not do it. At the level at which Mocha usually operates, parallelism does not seem to me particularly desirable. Where I have used test parallelism before is at the level of running end-to-end suites. For instance, run a suite against Firefox in Windows 8.1 while at the same time running the same suite against Chrome in Linux.

Just to update this question, Mocha version 8+ now natively supports parallel runs. You can use the --parallel flag to run your tests in parallel.
Parallel tests should work out-of-the box for many use cases. However, you must be aware of some important implications of the behavior
1 thing to note, some reporters don't currently support this execution (mocha-junit-reporter for example)

If you are using karma to start your tests, you can use karma-parallel to split up your tests across multiple browser instances. It runs specs in different browser instances in parallel and is very simple and easy to install:
npm i karma-parallel
and then add the 'parallel' to the frameworks list in karma.conf.js
module.exports = function(config) {
config.set({
frameworks: ['parallel', 'mocha']
});
};
karma-parallel
Disclosure: I am the author

Related

"top level" test in jest

While writing integration tests in jest I would like to reproduce the same behaviour I have achieved in mocha by:
mocha -r ts-node/register tests/integration/topLevelTest.test.ts 'tests/integration/**/*.test.ts'.
topLevelTest.test.ts :
let importantVariable;
describe("should do something with my variable", () => {
importantVariable = returnSomethingImportant();
it("should important variable exists", () => {
should.exist(importantVariable)
})
})
after(() => {
importantVariable.cleanUp()
})
Behaviour was simple: firstly topLevelTest executed describe, then other test suites executed themselves, and in the end after within topLevelTest were executed.
In my attempt of rewriting it to jest I wrote something very similar. Only difference is I used afterAll instead of after. The result is: firstly topLevelTest executed describe, then afterAll, and then other test suites. Is it possible to make afterAll run after other test suites?
This is what setup files are for, more specifically setupFilesAfterEnv because Jest environment is already initialized there with globals being available.
Top-level afterAll that wasn't grouped with describe applies to all tests within current test suite. Since Jest tests run in parallel (unless runInBand option was specified) in different threads, it obviously won't affect other test suites.
In case tests need to not proceed if a setup failed and data from setup needs to not be, globalSetup and globalTeardown configuration options should be used for that. This is not a test but the main difference is that describe and separate test blocks are unavailable. Global expect is not available but can be imported, this results in meaningful errors in case a setup fails:
// setup.js
let expect = require('expect');
module.exports = async () => {
let server = ...;
expect(server)...;
global.__MYSERVER__ = server;
};
// teardown.js
module.exports = async function () {
// close __MYSERVER__
};
Since global setup and teardown run in parent process, __MYSERVER__ cannot be accessed in tests.

Is there any way to add callbacks to Jest when all tests succeed or fail?

I'd like to run a callback when all the tests in a describe block pass (or fail), is there some hook or something in the Jest API to do this? I could not find anything applicable in the docs.
I'm making several API requests to collect data in order to compare it to data in a CSV file, in order to diff the contents. When the tests have all passed, I would like to save all the API responses in a file, therefore I need some sort of 'all tests passed' callback
You can run jest programmatically. Note that this approach is "hack" because there is no official support for running jest like this.
see: https://medium.com/web-developers-path/how-to-run-jest-programmatically-in-node-js-jest-javascript-api-492a8bc250de
There is afterAll that is aware of describe but runs regardless of test results. It can be used as a part of function to aggregate data from tests:
let responses;
testAndSaveResponses((name, fn) => {
if (!responses) {
responses = [];
} else {
afterAll(async () => {
if (!responses.includes(null)) {
// no errors, proceed with response processing
}
});
}
test(name, async () => {
try {
responses.push(await fn());
} catch (err) {
responses.push(null);
throw err;
}
});
});
It's supposed to be used instead of Jest test and be enhanced to support multiple describe scopes.
There is custom environment. Circus runner allows to hook test events, finish_describe_definition in particular. It is applied to all tests, unaware of custom data (e.g. responses that need to be saved) and should interact with them through global variables.
There is custom reporter, it receives a list of passed and failed tests. It is applied to all tests, unaware of custom data defined in tests and doesn't have access to globals from test scope so cannot be used to collect responses.

Cypress: interrupt all tests on first failure

How to interrupt all Cypress tests on the first test failure?
We are using semaphore to launch complete e2e tests with Cypress for each PR. But it takes too much time.
I'd like to interrupt all tests on the first test failure.
Getting the complete errors is each developer's business when they develop. I just want to be informed ASAP if there is anything wrong prior to deploy, and don't have to wait for the full tests to complete.
So far the only solution I came up with was interrupting the tests on the current spec file with Cypress.
afterEach(() => {
if (this.currentTest.state === 'failed') {
Cypress.runner.end();
}
});
But this is not enough since it only interrupts the tests located on the spec file, not ALL the other files. I've done some intensive search on this matter today and it doesn't seem like this is a thing on Cypress.
So I'm trying other solutions.
1: with Semaphore
fail_fast:
stop:
when: "true"
It is supposed to interrupt the script on error. But it doesn't work: tests keep running after error. My guess is that Cypress will throw an error only when all tests are complete.
2: maybe with the script launching Cypress, but I'm out of ideas
Right now here are my scripts
"cy:run": "npx cypress run",
"cy:run:dev": "CYPRESS_env=dev npx cypress run",
"cy:test": "start-server-and-test start http-get://localhost:4202 cy:run"
EDIT: It seems like this feature was introduced, but it requires paid version of Cypress (Business Plan). More about it: Docs, comment in the thread
Original answer:
This has been a long-requested feature in Cypress for some reason still has not been introduced. There are some workarounds proposed by the community, however it is not guaranteed they will work. Check this thread on Cypress' Github for more details, maybe you will find a workaround that works for your case.
The solution by #user3504541 is excellent! Thanks a ton. I already started giving up on using Cypress since these issues keep popping up. But in any case, here's my config:
support/index.ts
declare global {
// eslint-disable-next-line
namespace Cypress {
interface Chainable {
interrupt: () => void
}
}
}
function abortEarly() {
if (this.currentTest.state === 'failed') {
return cy.task('shouldSkip', true)
}
cy.task('shouldSkip').then(value => {
if (value) return cy.interrupt()
})
}
commands/index.ts
Cypress.Commands.add('interrupt', () => {
eval("window.top.document.body.querySelector('header button.stop').click()")
})
In my case the Cypress tests were left pending indefinitely on the CI (Github action workflow) but with this fix they interrupt properly.
A little hack that worked for me
Cypress.Commands.add('interrupt', () => {
eval("window.top.document.body.querySelector('header button.stop').click()");
});
This is available as the Auto Cancelation feature, which is part of Smart Orchestration, but is only available to Business Plan. From the Auto Cancelation docs:
Continuous Integration (CI) pipelines are typically costly processes that can demand significant compute time. When a test failure occurs in CI, it often does not make sense to continue running the remainder of a test suite since the process has to start again upon merging of subsequent fixes and other code changes. When Auto Cancellation is enabled, once the number of failed tests goes over a preset threshold, the entire test run is canceled. Note that any in-progress specs will continue to run to completion.

Testing Involving Database

well before specifying my problem, i want to tell that i'm new to the field of testing, so here is my problem:
i developed a rest api using express + sequelize(mysql), and i want to write some test for my api. i choosed to use jasmine library for testing.
so right know i want to test the create and update rest endpoint, i will need access to a database, but the problem is that the test cases are run in parallel, and there is only one database, so if i want to delete all item from a table in a test case, and another test case have create a row in that table, there will be a problem.
const request = require('superagent');
const models = require('../../src/models');
const Station = models.Station;
describe("station testing", function () {
before(() => {
// delete and recreate all database table
// before running any test
});
describe("crud station", function () {
it('should create model', () => {
Station.create({
'name': 'test',
lat: 12,
long: 123,
}).then( model => {
expect(model).toBeTruthy();
});
});
it('should delete evrything', () => {
Station.deleteAll().then( () => {
// problem here if after the first model is created and before create model except is executed
expect(Station.Count()).toEqual(0);
}
});
});
});
Your problem is that you are not writing unit tests here.
You need to understand the most important rule of unit testing - only test one unit at a time. A unit can be thought of as an area of your code. In a traditional desktop project (Java, C#, etc), a unit would be one class. In the case of Javascript, a unit is harder to define, but it certainly will only include the Javacript. If you are including any server code (for example, the database) in your tests, then you are not unit testing, you are doing integration testing (which is also very important, but much harder).
Your Javascript will have dependencies (ie other code that it calls, say via Ajax calls), which in your case will include the server code that is called. In order to unit test, you need to make sure that you are only testing the Javascript, which means that when running the tests, you don't want the server code to be called at all. That way, you isolate any errors in that unit of code, and can be confident that any problems found are indeed in that unit. If you include other units, then it could be the other units that have the problem.
In a strongly-typed language (like Java, C#, etc), there are frameworks that allow you to set up a mock for each dependency. Whilst I haven't tried any myself (that's this week's job), there are mocking frameworks for Javascript, and you would probably need to use one of them to do real unit testing. You mock out the server code, so when you run the test, it doesn't actually hit the database at all. Apart from solving your problem, it avoids a whole load of other issues that you will likely hit at some point with your current approach.
If you don't want to use a mocking framework, one other way to do it is to change your Javascript so that the function you are testing takes an extra parameter, which is a function that does the actual server call. So, instead of...
deleteCustomer(42);
deleteCustomer(id) {
validate(id);
$.ajax(...);
}
...your code would look like this...
deleteCustomer(42, callServer);
deleteCustomer(id, serverCall) {
validate(id);
serverCall(id);
}
...where serverCall() contains the Ajax call.
Then, to unit test, you would test something like this...
deleteCustomer(42, function(){});
...so that instead of calling the server, nothing is actually done.
This is obviously going to require some rewriting of your code, which could be avoided by mocking, but would work. My advice would be to spend some time learning how to use a mocking framework. It will pay off in the long run.
Sorry this has been a bit long. Unfortunately, you're getting into a complex area of unit testing, and it's important to understand what you're doing. I strongly recommend you read up about unit testing before you go any further, as a good understanding of the basics will save you a lot of trouble later on. Anything by Robert Martin (aka Uncle Bob) on the subject will be good, but there are plenty of resources around the web.
Hope this helps. If you want any more info, or clarification, ask away.
Jasmine supports a function for beforeEach which run before each spec in a describe block.
You can use that.
describe("A spec using beforeEach and afterEach", function() {
var foo = 0;
beforeEach(function() { foo += 1; });
afterEach(function() { foo = 0; });
it("is just a function, so it can contain any code", function() {
expect(foo).toEqual(1);
});
it("can have more than one expectation", function() {
expect(foo).toEqual(1)
expect(true).toEqual(true);
});
});
So you could let the beforeEach take care of the delete operation.

How to avoid timeouts in mocha testcases?

Here I am attaching my code, I am passing done callback and using supertest for request. Since I am using assert/expect in my testcase inside request.end block why I need to worry about timeout? What is mistake I am making here.
it('should get battle results ', function(done) {
request(url)
.post('/compare?vf_id='+vf_id)
.set('access_token',access_token)
.send(battleInstance)
.end(function(err, res){ // why need timeout
if (err) return done(err);
console.log(JSON.stringify(res.body));
expect(res.body.status).to.deep.equal('SUCCESS');
done();
});
});
Testcase results following response:
Error: timeout of 2000ms exceeded. Ensure the done() callback is being called in this test.
If I am running my testcases with mocha command then its show this error while If I am running test mocha --timeout 15000 then testcase is passing correctly. But I want to avoid timeout, How can I do that?
If I am running my testcases with mocha command then its show this error while If I am running test mocha --timeout 15000 then testcase is passing correctly. But I want to avoid timeout, How can I do that?
You can't avoid timeouts, since it looks like you're testing a remote service. If, for whatever reason, the request to that service takes a long time, you will run into timeouts.
You can tell Mocha to disable for timeout checking by setting the timeout to 0, but that's probably also not ideal because it may cause each test case to take an excessive amount of time.
As an alternative, you can mock request (which I assume is superagent) so you can control the entire HTTP request/response flow, but since it looks like you're testing a remote service (one which you have no control over) that would make this particular test case moot.
In mocha a default timeout of 2 seconds (2000ms) is set by default.
You can extend the default (global) timeout from the command line using the --timeout xxxx flag.
If you want instead to change the timeout for a specific test case you can use the this.timeout( xxxx ) function - note it does not work for arrow functions - (where xxxx is a number like 20000 representing milliseconds).
it('My test', function(){
this.timeout(5000);
//... rest of your code
});
You can also set a timeout of a set of test cases (wrapped by a describe):
describe("My suite", function(){
// this will apply for both "it" tests
this.timeout(5000);
it( "Test 1", function(){
...
});
it( "Test 2", function(){
...
});
});
It also works for before, beforeEach, after, afterEach blocks.
More documentation is available here: https://mochajs.org/#timeouts
Consider that 2 seconds is usually a good amount of time to run your tests so I would say that extend the default timeout should be an exception, not the common rule in your tests.
Also if your test is not async and you have to extend the timeout I would strongly suggest to review the function that is taking so long before extending the timeout.
Mocha : Timeouts
Test-specific timeouts may also be applied, or the use of this.timeout(0) to disable timeouts all together:
To disable the timeout from happening simply set it to 0. I use mocha <file> --timeout 0 when I'm debugging so the timeout error does not get thrown.
Here is what you need
mocha timeouts
describe('a suite of tests', function() {
this.timeout(500);
it('should take less than 500ms', function(done){
setTimeout(done, 300);
});
it('should take less than 500ms as well', function(done){
setTimeout(done, 250);
});
})

Resources