Jest tests hang due to open Sequelize connections - node.js

The Setup
I have a NodeJS project that uses:
Jest for testing.
Sequelize as an ORM.
Sequelize is instantiated when loading the models module which gets imported by a few of the files that are being tested with Jest.
The Problem
Jest tests pass but then hang with the message:
Jest did not exit one second after the test run has completed.
This usually means that there are asynchronous operations that weren't
stopped in your tests. Consider running Jest with
--detectOpenHandles to troubleshoot this issue.
Note: Adding --detectOpenHandles to the test call does not affect the output.
I do not actually invoke the sequelize object from any test paths, however some of the tested files import the models module and therefore Sequelize is instantiated.
(I will also note that this only occurs on my TravisCI environment, but I suspect this is a red herring.)
The Context
Because Jest is running tests in parallel, the models module is loaded multiple times during the entire test process. I confirmed this with debug output saying SEQUELIZE LOADED which appears multiple times when I run the tests.
The Attempts
I did attempt to invoke sequelize.close() inside of a globalTeardown but this appears to simply open (and then close) a new sequelize connection.
Since none of the tests actually rely on a database connection, I attempted running sequelize.close() within the models module immediately before export. This fixed the issue (though obviously is not a solution).
I have attempted to configure the test connection pools to aggressively end connections.
const sequelizeConfig = {
...
pool: {
idle: 0,
evict: 0,
}
}
This did nothing.
The Requirements
I don't want to use a brute force solution such as running --forceExit via Jest when I run my tests. This feels like it is ignoring the root issue and might expose me to other kinds of mistakes down the line.
My tests are spread across dozens of files, which means needing to invoke something in an afterAllTests would require a lot of redundancy and would introduce a code smell.
The Question
How can I ensure that sequelize connections are closed after tests finish, so they don't cause Jest to hang?

Jest provides a way to create universal setup before every test suite. This means it is possible to leverage Jest's afterAll to close the sequelize connection pool without having to manually include it in every single test suite.
Example Jest config in package.json
"jest": {
...
"setupFilesAfterEnv": ["./src/test/suiteSetup.js"]
}
Example suiteSetup.js
import models from '../server/models'
afterAll(() => models.sequelize.close())
// Note: in my case sequelize is exposed as an attribute of my models module.
Since the setupFilesAfterEnv is loaded before every test suite, this ensures that every Jest thread that opened a connection ultimately has that connection closed.
This doesn't violate DRY and it doesn't rely on a clunky --forceExit.
It does mean that connections will be closed that may never have been opened (which is a bit brute force) but that might be the best option.

Related

How to ensure mongoose.dropDatabase() can ONLY be called when connected to mongo-memory-server

We're using mongodb-memory-server for unit tests. After every unit test suite we execute:
await connection.dropDatabase();
await collection.deleteMany({});
To setup mongoose we have two different methods:
setupMongoose(); <--- Connects to our dev database in the cloud (Atlas)
setupMongooseWithMemoryServer(); <---- Connects mongoose to memory server.
We're a team of developers, and my worst fear is that someone uses "setupMongoose()" to setup unit tests by mistake some day. If that happens, dropDatabase() will be called for our "real" dev database. That would be a catastrophy.
So how can I ensure that dropDatabase() and maybe collection.deleteMany({}) can NEVER ever be called on our cloud database?
Some thoughts:
I have thought about setting up env variables and check for it before calling the dangerous methods. I've also already made a run time check:
checkForUnitTestEnv() {
if (!this.init || process.env.JEST_WORKER_ID === undefined) {
console.error('FATAL TRIED TO DROP DATABSE WITHOUT JEST!');
throw 'FATAL TRIED TO DROP DATABSE WITHOUT JEST!';
}
}
(this.init is only true if memory-server has been initialized).
But these methods are not fool proof. Errors can still happen if our developers are not careful. So I was hoping to either make it "illegal operations" with our database provider (Atlas) if possible, or check the mongoose connection uri on run time before calling the dangerous methods (but I haven't found a good way to do this yet).

Jest tests leaking due to improper teardown

While doing testing with Jest I am getting a warning saying "A worker process has failed to exit gracefully and has been force exited. This is likely caused by tests leaking due to improper teardown. Try running with --detectOpenHandles to find leaks." I realize that this is coming because inside of one of functions I use Bull https://github.com/OptimalBits/bull which uses Redis. So when adding a task to the queue it results in this warning. I use default Bull configuration (no configuration). I do have a mockup for the add function on the queue which is used by Jest, however it didn't help.
const notificationQueue = {
add: jest.fn().mockImplementation((data: any, opts?: JobOptions) => {}),
};
I'd like to know if there is a way to avoid this warning. If it helps I use in memory mongo for testing but redis is an actual one. As a side note when I run each test suite separately I am not seeing this warning, only when I run all tests.
As suggested in the warning, add --detectOpenHandles option to jest's script in package.json file:
"scripts": {
"test": "jest --watchAll --detectOpenHandles"
}
Dont forget to stop then start the server !
This solution can work whatever your problem. But, according to your case, your problem is coming from the redis connection. You need to close redis at the end of the test:
import { redis } from "redis_file_path";
afterAll(async () => {
await redis.quit();
});

Test worker thread with mocks

My Node application uses AWS SDK, and in order to mock it locally I am using Jest's manual mocks. Basically I am placing in __mocks__ folder barebone mock of AWS SDK that simulates methods I use - and during Jest tests this "AWS-SDK" is being called by the main code.
This works fine for the regular code flow. But when I invoke a worker thread (I am using sync-thread library but this is true for regular worker usage) - code inside of the worker tries to call real AWS-SDK when running Jest tests.
Is there a way to make it use mocked one as well? Also environmental variables set in the worker don't persist for the caller thread - is this by design or could they be set this way?

MongoError: Cannot create collection users - database is in the process of being dropped

I have a REST API and I'm writing TDD for this project. My TDD is consisted of two parts: route and service. I chose to use Jest. I have a MongoDB database that I use for testing. When each test is completed, I reset my database using the afterAll() method. In this method, I run the mongoose.connection.dropDatabase function.
There is no error when I ran only one test file but when I run multiple test files, I get an error. The error message:
MongoError: Cannot create collection auth-db.users - database is in
the process of being dropped.
I share sample codes with you:
users.route.test.ts:
https://gist.github.com/mksglu/8c4c4a3ddcb0e56782725d6457d97a0e
users.service.test.ts:
https://gist.github.com/mksglu/837202c1048687ad33b4d1dee01bd29c
When all my tests run, "sometimes" gives errors. I wrote the above error message. The reason for this error is that the reset process still continues. I can't solve this problem. I'd appreciate it if you could help.
Thanks.
https://jestjs.io/docs/en/cli.html#runinband
What you are looking for is --runInBand command. Which makes jest to run serially instead of creating a worker pool of child processes that run tests

How to unit test a method which connects to mongo, without actually connecting to mongo?

I'm trying to write a test to test a method that connects to mongo, but I don't actually want to have to have mongo running and actually make a connection to it to have my tests pass successfully.
Here's my current test which is successful when my mongo daemon is running.
describe('with a valid mongo string parameter', function() {
it('should return a rejected promise', function(done) {
var con = mongoFactory.getConnection('mongodb://localhost:27017');
expect(con).to.be.fulfilled;
done();
});
});
mongoFactory.getConnection code:
getConnection: function getConnection(connectionString) {
// do stuff here
// Initialize connection once
MongoClient.connect(connectionString, function(err, database) {
if (err) {
def.reject(err);
}
def.resolve(database);
});
return def.promise;
}
There are a couple of SO answers related to unit testing code that uses MongoDB as a data store:
Mocking database in node.js?
Mock/Test Mongodb Database Node.js
Embedded MongoDB when running integration tests
Similar: Unit testing classes that have online functionality
I'll make an attempt at consolidating these solutions.
Preamble
First and foremost, you should want MongoDB to be running while performing your tests. MongoDB's query language is complex, so running legitimate queries against a stable MongoDB instance is required to ensure your queries are running as planned and that your application is responding properly to the results. With this in mind, however, you should never run your tests against a production system, but instead a peripheral system to your integration environment. This can be on the same machine as your CI software, or simply relatively close to it (in terms of process, not necessarily network or geographically speaking).
This ENV could be low-footprint and completely run in memory (resource 1) (resource 2), but would not necessarily require the same performance characteristics as your production ENV. (If you want to performance test, this should be handled in a separate environment from your CI anyway.)
Setup
Install a mongod service specifically for CI. If repl sets and/or sharding are of concern (e.g. write concern, no use of $isolated, etc.), it is possible to mimic a clustered environment by running multiple mongod instances (1 config, 2x2 data for shard+repl) and a mongos instance on the same machine with either some init.d scripts/tweaks or something like docker.
Use environment-specific configurations within your application (either embedded via .json files, or in some place like /etc, /home/user/.your-app or similar). Your application can load these based on a node environment variable like NODE_ENV=int. Within these configurations your db connection strings will differ. If you're not using env-specific configs, start doing this as a means to abstract the application runtime settings (i.e. "local", "dev", "int", "pre", "prod", etc.). I can provide a sample upon request.
Include test-oriented fixtures with your application/testing suite. As mentioned in one of the linked questions, MongoDB's Node.js driver supports some helper libraries: mongodb-fixtures and node-database-cleaner. Fixtures provide a working and consistent data set for testing: think of them as a bootstrap.
Builds/Tests
Clean the associated database using something like node-database-cleaner.
Populate your fixtures into the now empty database with the help of mongodb-fixtures.
Perform your build and test.
Repeat.
On the other hand...
If you still decide that not running MongoDB is the correct approach (and you wouldn't be the only one), then abstracting your data store calls from the driver with an ORM is your best bet (for the entire application, not just testing). For example, something like model claims to be database agnostic, although I've never used it. Utilizing this approach, you would still require fixtures and env configurations, however you would not be required to install MongoDB. The caveat here is that you're at the mercy of the ORM you choose.
You could try tingodb.
TingoDB is an embedded JavaScript in-process filesystem or in-memory database upwards compatible with MongoDB at the API level.

Resources