I am writing integration tests for application written in NodeJS with MongoDB.
On CI server I would like to have some sort of embedded MongoDB for faster performance and easier control.
Currently I have MongoDB on other server, but tests are slow. Before each test I need to drop all collections. I am using mongoose as ORM.
So far I have only found embedded MongoDB for Java.
As of this writing, I'd recommend using mongodb-memory-server. The package downloads a mongod binary to your home directory and instantiates a new memory-backed MondoDB instance as needed. This should work well for your CI setup because you can spin up a new server for each set of tests, which means you can run them in parallel.
See the documentation for details on how to use it with mongoose.
For readers using jest and the native mongodb driver, you may find this class useful:
const { MongoClient } = require('mongodb');
const { MongoMemoryServer } = require('mongodb-memory-server');
// Extend the default timeout so MongoDB binaries can download
jest.setTimeout(60000);
// List your collection names here
const COLLECTIONS = [];
class DBManager {
constructor() {
this.db = null;
this.server = new MongoMemoryServer();
this.connection = null;
}
async start() {
const url = await this.server.getUri();
this.connection = await MongoClient.connect(url, { useNewUrlParser: true });
this.db = this.connection.db(await this.server.getDbName());
}
stop() {
this.connection.close();
return this.server.stop();
}
cleanup() {
return Promise.all(COLLECTIONS.map(c => this.db.collection(c).remove({})));
}
}
module.exports = DBManager;
Then in each test file you can do the following:
const dbman = new DBManager();
afterAll(() => dbman.stop());
beforeAll(() => dbman.start());
afterEach(() => dbman.cleanup());
Following the "don’t use test doubles for types you don’t own" principle, consider continue using a real MongoDB instance for your integration test. Look at this nice article for details.
Our team has been stubbing out the mongo skin calls. Depending on your testing packages you can do the same thing. It takes a little bit of work but it is worth it. Create a stub function and then just declare what you need in your test.
// Object based stubbing
function createObjStub(obj) {
return {
getDb: function() {
return {
collection: function() {
var coll = {};
for (var name in obj) {
var func = obj[name];
if (typeof func === 'object') {
coll = func;
} else {
coll[name] = func;
}
}
return coll;
}
};
}
}
};
// Stubbed mongodb call
var moduleSvc = new ModulesService(createObjStub({
findById: function(query, options, cb) {
return cb({
'name': 'test'
}, null);
//return cb(null, null);
}
}),{getProperties: function(){return{get: function(){} }; } });
Related
I'm learning to develop a Rest API using NodeJs with Express JS. I build a controller to do my stuff inside it. I want to call a local function inside the controller but it is not working. I always get not defined error.
Hear is my Controller,
const db = require('../config/db');
class TransactionController {
constructor(){
}
generateCustomerTransaction(req, res) {
const program_id = req.params.program_id;
const customerList = getCustomerList(program_id); //error here
//Do some business logics
return res.json(result);
};
getCustomerList(program_id) {
//Do some query to get a list of result
return results;
}
}
module.exports = SeatExcelController;
Everything seems simple like others languages but I get
ReferenceError: getCustomerList is not defined
I have no idea how to simply call a local function.
Please help. Thanks a lot.
To be able to access your function that way it needs to be defined on the package scope outside the class as in:
const db = require('../config/db');
const getCustomerList(program_id) = () => {
// Do some query to get a list of result
return results;
}
class TransactionController {
generateCustomerTransaction(req, res) {
const program_id = req.params.program_id;
const customerList = getCustomerList(program_id); //error here
//Do some business logics
return res.json(result);
};
}
module.exports = SeatExcelController;
Or call your function with this before the call as in:
const db = require('../config/db');
class TransactionController {
generateCustomerTransaction(req, res) {
const program_id = req.params.program_id;
const customerList = this.getCustomerList(program_id); //error here
//Do some business logics
return res.json(result);
};
function getCustomerList(program_id) {
// Do some query to get a list of result
return results;
}
}
module.exports = SeatExcelController;
I would go for the first option if the function doesn't need to access any class variables.
You could also make this a static function if it is not related to the class:
static getCustomerList(program_id) {
//Do some query to get a list of result
return results;
}
and call the function like this:
TransactionController.getCustomerList(program_id)
or just call the function using the this keyword. Because the way you code it now, the function is belonging to your class and class dependent.
: this.getCustomerList(program_id)
I am wondering how to properly test Azure Functions with Jest. I have read the online documentation provided by MSoft but it's very vague, and brief. There are also some outdated articles I found that don't really explain much. Here is what I understand: I understand how to test normal JS async functions with Jest. And I understand how to test very simple Azure Functions. However I am not sure how to go about properly testing more complex Azure Functions that make multiple API calls, etc.
For example I have an HTTP Function that is supposed to make a few API calls and mutate the data and then return the output. How do I properly mock the API calls in the test? We only have one point of entry for the function. (Meaning one function that is exported module.exports = async function(context,req). So all of our tests enter through there. If I have sub functions making calls I can't access them from the test. So is there some clever way of mocking the API calls? (since actually calling API's during tests is bad practice/design)
Here is a sample of code to show what I mean
module.exports = async function (context, req)
{
let response = {}
if (req.body && req.body.id)
{
try
{
//get order details
response = await getOrder(context, req)
}
catch (err)
{
response = await catchError(context, err);
}
}
else
{
response.status = 400
response.message = 'Missing Payload'
}
//respond
context.res =
{
headers: { 'Content-Type': 'application/json' },
status: response.status,
body: response
}
};
async function getOrder(context, req)
{
//connection to db
let db = await getDb() // <- how to mock this
//retrieve resource
let item = await db.get...(id:req.body.id)... // <- and this
//return
return {'status':200, 'data':item}
}
Consider this (simplified) example.
src/index.js (Azure Function entry point):
const { getInstance } = require('./db')
module.exports = async function (context) {
// assuming we want to mock getInstance and db.getOrder
const db = await getInstance()
const order = await db.getOrder()
return order
}
src/db.js:
let db
async function getInstance() {
if (db === undefined) {
// connect ...
db = new Database()
}
return db
}
class Database {
async getOrder() {
return 'result from real call'
}
}
module.exports = {
getInstance,
Database,
}
src/__tests__/index.test.js:
const handler = require('../index')
const db = require('../db')
jest.mock('../db')
describe('azure function handler', () => {
it('should call mocked getOrder', async () => {
const dbInstanceMock = new db.Database() // db.Database is already auto-mocked
dbInstanceMock.getOrder.mockResolvedValue('result from mock call')
db.getInstance.mockResolvedValue(dbInstanceMock)
const fakeAzureContext = {} // fake the context accordingly so that it triggers "getOrder" in the handler
const res = await handler(fakeAzureContext)
expect(db.getInstance).toHaveBeenCalledTimes(1)
expect(dbInstanceMock.getOrder).toHaveBeenCalledTimes(1)
expect(res).toEqual('result from mock call')
})
})
> jest --runInBand --verbose
PASS src/__tests__/index.test.js
azure function handler
✓ should call mocked getOrder (4 ms)
For a complete quickstart, you may want to check my blog post
I use the mssql (https://www.npmjs.com/package/mssql) module for my database. Normally I use postgres databases which lead to pg (https://www.npmjs.com/package/pg).
I want to setup prepared statements for the mssql database. When using the pg module it's quite easy.
This is how I do it with pg:
I setup my databaseManager
const { Pool } = require('pg');
const db = require('../config/database.js');
const pool = new Pool(db);
function queryResponse(result, err) {
return {
result,
err
};
}
module.exports = async (text, values) => {
try {
const result = await pool.query(text, values);
return queryResponse(result.rows, null);
} catch (err) {
return queryResponse(null, err);
}
};
and whenever I want to query the database I can call this module and pass in my statement and values. An example for todo apps would be
todos.js (query file)
const db = require('../databaseManager.js');
module.exports = {
getAllTodosFromUser: values => {
const text = `
SELECT
id,
name,
is_completed AS "isCompleted"
FROM
todo
WHERE
owner_id = $1;
`;
return db(text, values);
}
};
I wanted to create an mssql equivalent. From the docs I see that the module differs from the pg module.
I changed my databaseManager to
const sql = require('mssql');
const config = require('../config/database.js');
const pool = new sql.ConnectionPool(config).connect();
module.exports = async (queryBuilder) => {
try {
const preparedStatement = await new sql.PreparedStatement(pool);
return queryBuilder(sql, preparedStatement, async (query, values) => {
await preparedStatement.prepare(query);
const result = await preparedStatement.execute(values);
await preparedStatement.unprepare();
return {
result: result.rows,
err: null
};
});
} catch (err) {
return {
result: null,
err
}
}
};
and my query file would pass in the required parameters for the preparedStatement object
const db = require('../databaseManager.js');
module.exports = {
getUserByName: username => db((dataTypes, statementConfigurator, processor) => {
statementConfigurator.input('username', dataTypes.VarChar);
const query = `
SELECT
*
FROM
person
WHERE
username = #username;
`;
return processor(query, { username });
})
};
I was hoping that this approach would return the desired result but I get the error
this.parent.acquire is not a function
and don't know if my code is wrong. If it is, how can I setup my prepared statements correctly?
Edit:
I just found out that the error comes from this line of code
await preparedStatement.prepare(query);
but I think I took it correctly from the docs
https://tediousjs.github.io/node-mssql/#prepared-statement
I thought this question deserved a little bit more explanation than the answer what OP gave. The solution is no different than what OP already answered.
The issue remains same, pool mustn't have resolved from its promise pending state. So it just has to be awaited.
module.exports = async queryBuilder => {
try {
await pool; // Waiting for pool resolve from promise pending state.
const preparedStatement = await new sql.PreparedStatement(pool);
// ..
} catch (err) {
// ..
}
};
When you try to build a prepared statement, you pass the pool as an argument to its constructor. In its constructor is the below line
this.parent = parent || globalConnection
After which when you prepare the statement the flow leads to this line which would cause the issue since at that time this.parent's value was still a promise which was yet to be resolved.
this.parent.acquire(this, (err, connection, config) => {
So I'm following the example here https://www.mongodb.com/blog/post/optimizing-aws-lambda-performance-with-mongodb-atlas-and-nodejs, to optimize my lambda functions.
I've tried two approaches and tested them locally using serverless-offline and both don't seem to work.
First Approach
// endpoint file
import {connectToDatabase} from "lib/dbUtils.js";
let cachedDb = null;
export function post(event, context, callback) {
let response;
context.callbackWaitsForEmptyEventLoop = false;
connectToDatabase()
.then(//do other stuff
// lib/dbUtils.js
export async function connectToDatabase() {
if (cachedDb && cachedDb.serverConfig.isConnected()) {
console.log(" using cached db instance");
return cachedDb;
}
cachedDb = await mongoose.createConnection(
process.env.DB_URL,
async err => {
if (err) {
throw err;
}
}
);
return cachedDb;
}
Second Approach
global.cachedDb = null;
export function post(event, context, callback) {
let response;
context.callbackWaitsForEmptyEventLoop = false;
connectToDatabase()
.then(connection => createUser(event.body, connection))
// lib/dbUtils.js
export async function connectToDatabase() {
// eslint-disable-next-line
if (global.cachedDb && global.cachedDb.serverConfig.isConnected()) {
// eslint-disable-next-line
console.log(" using cached db instance");
// eslint-disable-next-line
return global.cachedDb;
}
// eslint-disable-next-line
global.cachedDb = await mongoose.createConnection(
process.env.DB_URL,
async err => {
if (err) {
throw err;
}
}
);
// eslint-disable-next-line
return global.cachedDb;
}
In both cases the using cached db instance console log does not run.
Why does this not work? Is this because of serverless-offline?
The answer is simple: serverless-offline doesn't simulate the full AWS. Use the AWS console to to make a real Lambda
The MongoDB Atlas guide is OK, but it's also worth checking the official AWS Lambda documentation describing the context option in each lambda:
callbackWaitsForEmptyEventLoop – Set to false to send the response right away when the callback executes, instead of waiting for the Node.js event loop to be empty. If false, any outstanding events will continue to run during the next invocation.
It's possible to run your code on a real Lambda and see using cached db instance on the console. Since MongoDB's JavaScript code is fairly poor, I've written out my own version below:
var MongoClient = require("mongodb").MongoClient
let db = null
var log = console.log.bind(console)
var print = function(object) {
return JSON.stringify(object, null, 2)
}
// Use your own credentials (and better yet, put them in environment variables)
const password = `notactuallyapassword`
const uri = `mongodb+srv://lambdauser:${password}#fakedomain.mongodb.net/test?retryWrites=true`
exports.handler = function(event, context, callback) {
log(`Calling MongoDB Atlas from AWS Lambda with event: ${print(event)}`)
var document = JSON.parse(JSON.stringify(event))
const databaseName = "myDatabase",
collectionName = "documents"
// See https://www.mongodb.com/blog/post/optimizing-aws-lambda-performance-with-mongodb-atlas-and-nodejs
// and https://docs.aws.amazon.com/lambda/latest/dg/nodejs-prog-model-context.html#nodejs-prog-model-context-properties
context.callbackWaitsForEmptyEventLoop = false
return createDoc(databaseName, collectionName, document)
}
async function createDoc(databaseName, collectionName, document) {
var isConnected = db && db.serverConfig.isConnected()
if (isConnected) {
log(`Already connected to database, warm start!`)
} else {
log(`Connecting to database (cold start)`)
var client = await MongoClient.connect(uri)
db = client.db(databaseName)
}
var result = await db.collection(collectionName).insertOne(document)
log(`just created an entry into the ${collectionName} collection with id: ${result.insertedId}`)
// Don't close the connection thanks to context.callbackWaitsForEmptyEventLoop = false - this will re-use the connection on the next called (if it can re-use the same Lambda container)
return result
}
Use the Test button to run the lambda above twice in the AWS Lambda console.
The first time you run it you'll see Connecting to database (cold start)
The second time you'll see Already connected to database, warm start!
See the log output section in screenshot below:
Basically to run the tests, I need the database connection to MongoDb to be ready. The database manager has a connect method to get a connection and a get method to get a reference to that connection. The connect method is supposed to be called only once, at startup. This design works fine when running the app because the app starts only if the connect method has been called and returned a connection.
The database manager using mongo native (no mongoskin, no mongoose) (database.js):
var config = require('../config')
var MongoClient = require('mongodb').MongoClient
var connection = null
module.exports.connect = function () {
return MongoClient.connect(config.mongodb.uri + config.mongodb.db)
.then(function (db) {
connection = db
})
}
module.exports.get = function () {
if (!connection) {
throw new Error('Call connect first!')
}
return connection
}
Start the app:
var db = require('./services/database');
db.connect()
.then(function() {
logger.info("mongodb is running");
require('./main')
});
In the application, there are service modules and repository modules. The service modules require repository modules when they have to use the database. I have a BaseRepository module where I define the common queries findOne, findAll, etc. For all the MongoDb collections I have a specific repository that inherits from the BaseRepository. In the constructor, I set the collection by calling the database connection.
It means that when a service module which has a dependency on a repository module, the database connect method has to be called. This is the issue because I don't know how to start all the tests after the call to that connect method. I am open to suggestions if you think it's a design flow.
A test:
var usersService = require('../../services/rest/UsersService');
describe('IT Test', function () {
})
The service:
var usersRepository = require('../dao/UsersRepository');
The repository:
var db = require('../database');
var BaseRepository = require('./BaseRepository');
var util = require('util');
util.inherits(UsersRepository, BaseRepository);
function UsersRepository() {
this.collection = db.get().collection('users'); // <====== The connect method has not been called yet
}
The base repository:
function BaseRepository() {
this.collection = undefined;
}
BaseRepository.prototype.findOne = function (filters) {
return this.collection.findOne(filters)
}
I ended up writing a wrapper for all the tests
'use strict'
var logger = require('../services/logger')()
var db = require('../services/database')
describe('All tests wrapper', function () {
it('should pass', function () {
return db.connect()
.then(function () {
logger.info('MongoDB is running')
require('./agenda/AgendaJobsIT')
require('./dashboard/DashboardsServiceIT')
require('./twitter/TwitterServiceTest')
require('./authentication/TokenServiceTest')
})
})
})
The best way is mocha --delayed switch. mocha doc says
If you need to perform asynchronous operations before any of your suites are run (e.g., for dynamically generating tests), you may delay the root suite. Run mocha with the --delay flag.
For example, use mocha in this way mocha --recursive --exit --delay --ui tdd tests.js and --delayed enable you to trigger running the root suite via calling run() explicitly.
const fn = async x => {
return new Promise(resolve => {
setTimeout(resolve, 1500, 2 * x);
});
};
(async function() {
const z = await fn(3);
suite("won't run until run() executes", () => {})
run();
})();
For more information, please read https://mochajs.org/#delayed-root-suite.