We are using mocha on our Open Source project https://opentermsarchive.org and ended up in the CI deploying our code despite errors in the tests.
Problem is that those errors failed silently with an exit code of 0, which is really bad.
This means
tests are launched
tests fail with no error
CI considers tests have passed even though they did not run completely
We are using mocha programmatically with the following config
const mocha = new Mocha({
delay: true, // as the validation script performs an asynchronous load before running the tests, the execution of the tests are delayed until run() is called
failZero: true, // consider that being called with no service to validate is a failure
});
const VALIDATE_PATH = path.resolve(__dirname, '../scripts/validation/validate.js');
(async () => {
mocha.addFile(VALIDATE_PATH); // As `delay` has been called, this statement will not load the file directly, `loadFilesAsync` is required.
await mocha.loadFilesAsync() // Load files previously added to the Mocha cache with `addFile`.
.catch(error => {
console.error(error);
process.exit(2);
});
let hasFailedTests = false;
mocha.run()
.on('fail', () => { hasFailedTests = true; })
.on('end', () => {
if (hasFailedTests) {
process.exit(1);
}
process.exit(0);
});
})();
Well
this is due to the fact that mocha is silenting unhandledRejection when you are using the delay property
You can see that here https://github.com/mochajs/mocha/blob/master/lib/runner.js#L198
So a way to bypass this is to add
process.on('unhandledRejection', reason => {
// Mocha catch unhandled rejection from the user code and re-emit them to the process (see https://github.com/mochajs/mocha/blob/master/lib/runner.js#L198)
// Re-throw them so that the validation command fails in these cases (for example, if there is a syntax error when parsing JSON declaration files)
console.error(reason); // keep the console error to have the whole stack trace
throw reason;
});
before the instantiation of mocha
Related
I am writing a grapqhl server that has a simple logout mutation. Everything works as expected when I run the server and I can log out by destroying the session and clearing the cookie just fine.
Here is the resolver:
export default async (root, args, context) => {
console.log("THIS WILL LOG")
await new Promise((res, rej) =>
context.req.session.destroy(err => {
if (err) {
return rej(false);
}
context.res.clearCookie("qid");
return res(true);
})
);
console.log("NEVER HERE BEFORE TIMEOUT");
// 4. Return the message
return {
code: "OK",
message: "You have been logged out.",
success: true,
item: null
};
};
I am attempting to write a simple test just to verify that the req.session.destroy and res.clearCookie functions are actually called. At this point I AM NOT attempting to test if a cookie is actually cleared, as I am not actually starting up the server, I am just testing that the graphql resolver was ran correctly and that it called the right functions.
Here is a portion of my test:
describe("confirmLoginResolver", () => {
test("throws error if logged in", async () => {
const user = await createTestUser();
const context = makeTestContext(user.id);
context.req.session.destroy = jest
.fn()
.mockImplementation(() => Promise.resolve(true));
context.res.clearCookie = jest.fn();
// this function is just a helper to process my graphql request.
// it does not actually start up the express server
const res = await graphqlTestCall(
LOGOUT_MUTATION, // the graphql mutation stored in a var
null, // no variables needed for mutation
null // a way for me to pass in a userID to mock auth state,
context // Context override, will use above context
);
console.log(res);
expect(context.req.session.destroy).toHaveBeenCalled();
// expect(res.errors.length).toBe(1);
// expect(res.errors).toMatchSnapshot();
});
});
Again, everything works correctly when actually running the server. The problem is that when I attempt to run the above test, I always get a jest timeout:
Timeout - Async callback was not invoked within the 5000ms timeout specified by jest.setTimeout.
The reason is that the await section of above resolver will hang because it's promise.resolve() is never being executed. So my console will show "THIS WILL LOG", but will never get to "NEVER HERE BEFORE TIMEOUT".
I suspect I need to write a better jest mock to more accurately simulate the callback inside of context.req.session.destroy, but I can not figure it out.
Any ideas how I can write a better mock implementation here?
context.req.session.destroy = jest
.fn()
.mockImplementation(() => Promise.resolve(true));
Is not cutting it. Thoughts?
Try
context.req.session.destroy = jest
.fn()
.mockImplementation((fn) => fn(false));
I am using MongoDB, NodeJs,and Chai to test the code.
connecting to mongodb :
if (!!connection) {
model = connection.model('client', clientSchema, collections.clients[env]);
} else {
model = mongoose.model('client', clientSchema, collections.clients[env]);
}
This is the unit tests:
describe('Bad VSA card test', () => {
it('should contains VSA accounts', async (done) => {
const vsaCard = await updatedVSA();
console.log(vsaCard);
expect(vsaCard.length).to.equal(5);
done();
});
afterAll(() => {
client.disconnect();
});
});
When I run this test code, I get an error:
Jest did not exit one second after the test run has completed.
This usually means that there are asynchronous operations that weren't stopped in your tests. Consider running Jest with `--detectOpenHandles` to troubleshoot this issue.
Actually, I searched for this error first. And I found a similar issue. His error was fixed by closing db connection. Why am I still getting this error by closing MongoDB connection?
I'm migrating an app from Sails.js v0.12 to v1.0, which includes upgrading mocha from ^3.5.0 to ^5.2.0. I assume that is the root of the issue, but I can't seem to find a fix.
Before, when a test failed, there would be a summary of the error at the end of the mocha reporter output: the specific assertion that failed, file name, line number, error message, etc. Now, the reporter is coloring the it block in red, but no additional details are show.
I've tried changing the reporter in mocha.opts, which works for the actual execution output, but nothing is enabling the summary at the end. What am I missing?
// ./test/integration/models/User.test.js
describe('User', () => {
describe('find()', () => {
it('should return an array of users', () => {
return User.find()
.then((users) => {
users.should.be.a('array');
true.should.be.false; // No problems if this is removed
});
});
});
});
In the console:
> node ./node_modules/mocha/bin/mocha test/lifecycle.test.js test/integration/**/*.test.js
√ OtherModel method() should do something: 17ms
1) User find() should return an array of users
'Done.'
PS C:\repos\myproject>
Turns out that Mocha is fine and my test definitions are fine, I had simply fixed a totally different migration issue incorrectly. Since version 4, mocha will no longer automatically kill itself when it thinks all tests are complete. This means that one of these two options must be used:
Add --exit to mocha.opts or to wherever the command is called
Run process.exit() in your test suite's JavaScript
I tried the second option by adding process.exit() to the after() block in my test bootstrap file. This is a bad idea and resulted in the confusing behavior above.
To solve my issue, I removed my changes to the after() block and added --exit to mocha.opts. Thanks to Dhruv Choudhary for pointing me in the right direction.
you can use done callback. The first strategy suggested in the mocha documentation is using the ‘done’ callback. This is an extra argument to the callback in the it . You call it after the last assertion in your test.
I have seen many people using done() method in wrong way. for example, look at below code,
describe('User', () => {
describe('find()', () => {
it('should return an array of users', (done) => {
return User.find()
.then((users) => {
users.should.be.a('array');
true.should.be.false; // No problems if this is removed
done();
});
});
});
});
The above test will work fine and show the test passing, But calling done() in the same then callback is a bad idea because The above code works well until your expectation fails, you might get error like
The above failure is not very useful . If we want to utilize the mocha’s error we shouldn’t call done() from the same then() callback. See the below test
describe('User', () => {
describe('find()', () => {
it('should return an array of users', (done) => {
return User.find()
.then((users) => {
users.should.be.a('array');
true.should.be.false;
})
.then(() => done(), done)
.catch((error) => {
done(error);
});
});
});
});
do not forget to wrap your error with catch block.
Now see the difference in the mocha’s failure message with actual and expected..
Please try adding done callback to your test case .
describe('User', () => {
describe('find()', () => {
it('should return an array of users', (done) => {
return User.find()
.then((users) => {
users.should.be.a('array');
true.should.be.false; // No problems if this is removed
done();
});
});
});
});
When running tests that utilize domains for error handling, Mocha still appears to be throwing an error even if a domain handler inside a library should have caught the error. If I execute the code outside of Mocha, it functions correctly leading me to believe the problem is Mocha.
Example:
foo.js
module.exports = function(done) {
var domain = require("domain");
var d = domain.create();
d.on("error", function() {
done();
});
d.run(function() {
throw new Error("foo");
});
}
test.js - Error thrown inside foo.js is not being caught by the domain.
describe("test", function() {
it("should succeed", function(done) {
var foo = require("./foo.js");
foo(function() {
console.log("done");
done();
});
});
});
result : error thrown
script.js - error is being properly caught by the domain and bubbled up.
var foo = require("./foo.js");
foo(function() {
console.log("done");
});
result : done
As you can see above, if I node straight to script.js it functions as desired, the error is caught by the domain handler and the code continues. If I run the same block of code inside a Mocha test, the error halts the test and a failure is given. I believe this is because the error is being sent on an uncaughtException handler, or something like it. The other complication is that it works properly in Mocha, if I have a process.nextTick() around the function call, leading me to believe that Mocha is only unable to handle synchronous errors, but works just fine with async errors.
There is some talk of this problem here: https://groups.google.com/forum/#!msg/nodejs/n-W9BSfxCjI/SElI1DJ_6u0J and https://github.com/joyent/node/issues/4375 .
The confusion I have is that all of this discussion seems to state the problem has been resolved months ago. Anyone know of either a simple work-around for the issue, or a why I'm not seeing a bug fixed which other people seem to believe is fixed at this point in time.
I am running node v0.10.18 and Mocha 1.13.0 on CentOS 6.3 Vagrant VirtualBox on Windows 7.
Found the problem. NodeJS domains catch synchronous errors but the event continues to bubble to a try/catch. If you wrap a domain.run() in a try/catch then the domain error handler AND the catch will be executed.
Thus, it seems the best practice is to use process.nextTick inside all domain.run(). This is shown in the docs example, but isn't expressed as explicitly as I would prefer.
Example:
d.run(function() {
process.nextTick(function() {
// do stuff
});
});
In this case, the flaw is not in Mocha.
Proof of NodeJS domains not catching synchronous errors in try/catch: https://gist.github.com/owenallenaz/7141699
nodejs domains do definitely catch synchronous errors
See this simple test case
var domain = require("domain");
var d = domain.create();
d.on("error", function() {
console.log("domain caught");
});
d.run(function() {
throw new Error("foo");
});
// result: domain caught
EDIT: Since writing this answer I have written a blog post describing what is going on with domains and try catch and whether you can use domains as a wholesale replacement for try catch. It summarises most of what has been discussed here.
http://www.lighthouselogic.com/node-domains-as-a-replacement-for-try-catch/
ORIGINAL ANSWER:
Actually there is a problem with Mocha.
I wrote the following test function:
function error(callback){
var d = domain.create().on('error', function(err){
console.log("Caught Error in Domain Handler")
return callback(err);
});
d.enter();
throw new Error("TestError");
d.exit();
}
Then I wrote a simple test without mocha:
error(function(err){
if(err)
{
console.log("Error was returned");
}else
{
console.log("Error was not returned")
}
})
The output I received was:
Caught Error in Domain Handler
Error was returned
When I tested using Mocha:
describe('Domain Tests', function(){
it('Should return an error when testing', function(done){
error(function(err){
if(err)
{
console.log("Error was returned");
}else
{
console.log("Error was not returned")
}
return done();
})
});
});
I received the following output:
․
0 passing (4ms)
1 failing
1) Domain Tests Should return an error when testing:
Error: TestError
at error (/Users/bensudbury/Documents/node_projects/testMochaDomains/test.js:9:11)
at Context.<anonymous> (/Users/bensudbury/Documents/node_projects/testMochaDomains/testMocha.js:6:3)
at Test.Runnable.run (/usr/local/share/npm/lib/node_modules/mocha/lib/runnable.js:194:15)
at Runner.runTest (/usr/local/share/npm/lib/node_modules/mocha/lib/runner.js:358:10)
at /usr/local/share/npm/lib/node_modules/mocha/lib/runner.js:404:12
at next (/usr/local/share/npm/lib/node_modules/mocha/lib/runner.js:284:14)
at /usr/local/share/npm/lib/node_modules/mocha/lib/runner.js:293:7
at next (/usr/local/share/npm/lib/node_modules/mocha/lib/runner.js:237:23)
at Object._onImmediate (/usr/local/share/npm/lib/node_modules/mocha/lib/runner.js:261:5)
at processImmediate [as _immediateCallback] (timers.js:330:15)
As you can see: the domain error handler was short circuited.
This problem seems to be related to the following issues:
https://github.com/visionmedia/mocha/issues/513
While the Node issue has been closed, the issue in mocha is still open.
The workaround that was suggested at: https://gist.github.com/mcollina/4443963 didn't resolve the issue in this case.
I dug through the code of Mocha and found that the problem occurs because mocha wraps the tests in a try catch block. This means that the exception is caught and never sent to the uncaughtException or _fatalException handler depending upon the version of node that you are using.
Your workaround is good, but nodejs domains do definitely catch synchronous errors so I wouldn't change your code but instead change your test. Your new test should look like:
describe("test", function() {
it("should succeed", function(done) {
process.nextTick(function(){
var foo = require("./foo.js");
foo(function() {
console.log("done");
done();
});
})
});
});
I haven't tested this code, but similar code for my example works properly:
it('Should return an error when testing', function(done){
process.nextTick(function(){
error(function(err){
if(err)
{
console.log("Error was returned");
}else
{
console.log("Error was not returned")
}
return done();
});
})
});
I have added a comment to the end of the issue in Mocha to see if it can be resolved:
https://github.com/visionmedia/mocha/issues/513
In my node application I'm using mocha to test my code. While calling many asynchronous functions using mocha, I'm getting timeout error (Error: timeout of 2000ms exceeded.). How can I resolve this?
var module = require('../lib/myModule');
var should = require('chai').should();
describe('Testing Module', function() {
it('Save Data', function(done) {
this.timeout(15000);
var data = {
a: 'aa',
b: 'bb'
};
module.save(data, function(err, res) {
should.not.exist(err);
done();
});
});
it('Get Data By Id', function(done) {
var id = "28ca9";
module.get(id, function(err, res) {
console.log(res);
should.not.exist(err);
done();
});
});
});
You can either set the timeout when running your test:
mocha --timeout 15000
Or you can set the timeout for each suite or each test programmatically:
describe('...', function(){
this.timeout(15000);
it('...', function(done){
this.timeout(15000);
setTimeout(done, 15000);
});
});
For more info see the docs.
I find that the "solution" of just increasing the timeouts obscures what's really going on here, which is either
Your code and/or network calls are way too slow (should be sub 100 ms for a good user experience)
The assertions (tests) are failing and something is swallowing the errors before Mocha is able to act on them.
You usually encounter #2 when Mocha doesn't receive assertion errors from a callback. This is caused by some other code swallowing the exception further up the stack. The right way of dealing with this is to fix the code and not swallow the error.
When external code swallows your errors
In case it's a library function that you are unable to modify, you need to catch the assertion error and pass it onto Mocha yourself. You do this by wrapping your assertion callback in a try/catch block and pass any exceptions to the done handler.
it('should not fail', function (done) { // Pass reference here!
i_swallow_errors(function (err, result) {
try { // boilerplate to be able to get the assert failures
assert.ok(true);
assert.equal(result, 'bar');
done();
} catch (error) {
done(error);
}
});
});
This boilerplate can of course be extracted into some utility function to make the test a little more pleasing to the eye:
it('should not fail', function (done) { // Pass reference here!
i_swallow_errors(handleError(done, function (err, result) {
assert.equal(result, 'bar');
}));
});
// reusable boilerplate to be able to get the assert failures
function handleError(done, fn) {
try {
fn();
done();
} catch (error) {
done(error);
}
}
Speeding up network tests
Other than that I suggest you pick up the advice on starting to use test stubs for network calls to make tests pass without having to rely on a functioning network. Using Mocha, Chai and Sinon the tests might look something like this
describe('api tests normally involving network calls', function() {
beforeEach: function () {
this.xhr = sinon.useFakeXMLHttpRequest();
var requests = this.requests = [];
this.xhr.onCreate = function (xhr) {
requests.push(xhr);
};
},
afterEach: function () {
this.xhr.restore();
}
it("should fetch comments from server", function () {
var callback = sinon.spy();
myLib.getCommentsFor("/some/article", callback);
assertEquals(1, this.requests.length);
this.requests[0].respond(200, { "Content-Type": "application/json" },
'[{ "id": 12, "comment": "Hey there" }]');
expect(callback.calledWith([{ id: 12, comment: "Hey there" }])).to.be.true;
});
});
See Sinon's nise docs for more info.
If you are using arrow functions:
it('should do something', async () => {
// do your testing
}).timeout(15000)
A little late but someone can use this in future...You can increase your test timeout by updating scripts in your package.json with the following:
"scripts": {
"test": "test --timeout 10000" //Adjust to a value you need
}
Run your tests using the command test
For me the problem was actually the describe function,
which when provided an arrow function, causes mocha to miss the
timeout, and behave not consistently. (Using ES6)
since no promise was rejected I was getting this error all the time for different tests that were failing inside the describe block
so this how it looks when not working properly:
describe('test', () => {
assert(...)
})
and this works using the anonymous function
describe('test', function() {
assert(...)
})
Hope it helps someone, my configuration for the above:
(nodejs: 8.4.0, npm: 5.3.0, mocha: 3.3.0)
My issue was not sending the response back, so it was hanging. If you are using express make sure that res.send(data), res.json(data) or whatever the api method you wanna use is executed for the route you are testing.
Make sure to resolve/reject the promises used in the test cases, be it spies or stubs make sure they resolve/reject.