Mongoose too many connection and commands - node.js

I'm here to request help with mongo/mongoose. I use AWS lambda that accesses a mongo database and I'm having problems sometimes my connections reach the limit of 500. I'm trying to fix this problem and I did some things like this https://dzone.com/articles/how-to-use-mongodb-connection-pooling-on-aws-lambd and https://www.mongodb.com/blog/post/optimizing-aws-lambda-performance-with-mongodb-atlas-and-nodejs. That basically is to use a singleton-like and set context.callbackWaitsForEmptyEventLoop = false, which indeed helped but is still, rarely, open 100 connections in less than a minute, it looks like there is some connection that is not being reused even tho our logs show that they are being reused. So I realized a weird behavior, whenever mongoatlas shows me an increased number of commands, my mongo connections increase heavily. The first chart is operations and the second is the connections.
Looking at operations, there are too many commands and just a few queries. I have no idea what are those commands, my theory is that those commands are causing the problem but I did not find anything that explained what is the difference between query and command exactly for me to know if that is a valid theory or not. Another thing is, how to choose correctly the number of pool size, we have really simple queries.
Here is our singleton class because maybe this is what we are doing wrong:
class Database {
options: [string, mongoose.ConnectionOptions];
instance?: typeof mongoose | null;
constructor(options = config) {
console.log('[DatabaseService] Created database instance...');
this.options = options;
this.instance = null;
}
async checkConnection() {
try {
if (this.instance) {
const pingResponse = await this.instance.connection.db.admin().ping();
console.log(`[DatabaseService] Connection status: ${pingResponse.ok}`);
return pingResponse.ok === 1;
}
return false;
} catch (error) {
console.log(error);
return false;
}
}
async init() {
const connectionActive = await this.checkConnection();
if (connectionActive) {
console.log(`[DatabaseService] Already connected, returning instance`);
return this.instance;
}
console.log('[DatabaseService] Previous connection was not active, creating new connection...');
this.instance = await mongoose.connect(...this.options);
const timeId = Date.now();
console.log(`Connection opened ${timeId}`);
console.time(`Connection started at ${timeId}`);
this.instance?.connection.on('close', () => {
console.timeEnd(`Connection started at ${timeId}`);
console.log(`Closing connection ${timeId}`);
});
return this.instance;
}
async getData(id: string) {
await this.init();
const response = await Model.findOne({ 'uuid': id });
return response;
}
}
I hope that is enough information. My main question is if my theory of commands causing too many connections is possible and what are exactly commands because every explanation that I found look like is the same than query.

Based on the comment written by Matt I have changed my init function and now my connections are under control.
async init() {
if (this.instance) {
console.log(`[DatabaseService] Already connected, returning instance`);
return this.instance;
}
console.log('[DatabaseService] Previous connection was not active, creating new connection...');
this.instance = await mongoose.connect(...this.options);
const timeId = Date.now();
console.log(`Connection opened ${timeId}`);
console.time(`Connection started at ${timeId}`);
this.instance?.connection.on('close', () => {
console.timeEnd(`Connection started at ${timeId}`);
console.log(`Closing connection ${timeId}`);
});
return this.instance;
}

Related

Lambda + pg-promise + transaction = random timeout

I've been struggling with this issue for the past 48 hours and after reading lots of answers here, blogs, articles, documentation, etc... I still cannot find a solution!
Basically, I have a lambda function with a 2-minute timeout. Based on logs and insights, it processes fine most of the requests, but it randomly fails with a "timed out" error when trying to execute the transaction below.
Lambda code (chopped for legibility):
import pgp from 'pg-promise'
import logger from '../lib/logger'
const Database = pgp()
const db = Database({
connectionString: process.env.DATABASE_URL,
max: 3,
idleTimeoutMillis: 10000,
})
db.connect()
.then(() => logger.info('Successfully connected to the PG database'))
.catch(err => logger.error({ err }))
export const handler = async (event, context) => {
logger.info('transaction start...')
await db.tx(async tx => {
await tx.none(
`
INSERT INTO...`,
[someValue1, someValue2]
)
const updatedRow = await tx.one(
`
UPDATE Something...`,
[somethingId]
)
return someFunction(updatedRow)
})
logger.info('transaction end...')
}
const someFunction = async (data) => {
return db.task('someTask', async ctx => {
const value = await ctx.oneOrNone(
`SELECT * FROM Something...`,
[data.id]
)
if (!value) {
return
}
const doStuff = async (points) =>
ctx.none(
`UPDATE Something WHERE id =.....`,
[points]
)
// increment points x miles
if (data.condition1) {
await doStuff(10)
}
if (data.condition2) {
await doStuff(20)
}
if (data.condition3) {
await doStuff(30)
}
})
}
I see that the transaction starts but never ends, so the function is inevitably killed by timeout.
I read the whole wiki in pg-promise and understood everything about tweaks, performance, good practices, etc. But still, something is very wrong.
You can see that I also changed the pool size and max timeout for experimenting, but it didn't fix the issue.
Any ideas?
Thanks!
Most likely you are running out of connections. You are not using them correctly, while at the same time you are setting a very low connection limit of 3.
The first issue, you are testing a connection by calling connect, without following it with done, which permanently occupies, and thus wastes your initial/primary connection.
See the example here where we are releasing the connection after we have tested it.
The second problem - you are requesting a new connection (by calling .task on the root db level) while inside a transaction, which is bad for any environment, while particularly critical when you have very few connections available.
The task should be reusing connection of the current transaction, which means your someFunction should either require the connection context, or at least take it as optional parameter:
const someFunction = async (data, ctx) => {
return (ctx || db).task('someTask', async tx => {
const value = await tx.oneOrNone(
Task <-> Transaction interfaces in pg-promise can be fully inter-nested, you see, propagating the current connection through all levels.
Also, I suggest use of pg-monitor, for a good query+context visualization.

what is the best way or pattern to create a Wrapper class to abstract the use of RabbitMQ in node js

so I'm using RabbitMQ for some Projects and i noticed that i ll use some duplicate code all the Time that's why i decided to make a Wrapper Class or Interface that have some function to use RabbitMQ direct without repeating the code all the time. i began to do this yesterday and i already had some Problems since i wanted to use OOP and Javascript can be complicated when using OOP (at least i think so)
I began with creating a class IRAbbitMQ with function init to initialize a connection and create a channel, i knew that i cant use nested classes so instead i wanted to use Factory functions, i tried to make the connection and channel a part of the class IRabbitMQ properties but i dont know why that gave me undefined when i create an instance of it
class IRabbitMQ {
constructor() {
this.init(rabbitMQServer); // rabbitMQServer for example 'localhost//5672'
}
// establish a Connection to RAbbitMQ Server
async init(host) {
try {
let connection = await amqplib.connect(host);
let channel = await connection.createChannel();
channel.prefetch(1);
console.log(' [x] Awaiting RPC requests');
this.connection = connection;
this.channel = channel;
}
catch(err) {
console.error(err);
}
}
// Close the Connection with RabbitMQ
closeConnection() {
this.connection.close();
}
log() {
console.log(this.connection);
}
EventPublisher() {
function init(IRabbit, publisherName) {
if(!IRabbit.connection) {
throw new Error('Create an Instance of IRabbitMQ to establish a Connection');
}
let ch = IRabbit.channel;
console.log(ch);
}
return {
init : init
}
}
}
var r = new IRabbitMQ();
r.log();
when i run the code the output is undefined, i dont know why since i m initializing the connection and channel properties in the init function and then called that function in the constructor so that should be initialized when i create an object of the Wrapper class. i wanted also to take some advices from you wether it is good to use classes or is there any other better way to create a Wrapper class or Interface for RabbitMQ to make it easy to use it and not have to duplicate Code.
Not really an answer, but I was able to successfully log the connection with this example code. I trimmed out other code to just focus on the .log() part that was logging a undefined.
Code is far from perfect, but works at least
const amqplib = require('amqplib');
class IRabbitMQ {
constructor() { }
async init(host) {
try {
const connection = await amqplib.connect(host);
const channel = await connection.createChannel();
channel.prefetch(1);
console.log(' [x] Awaiting RPC requests');
this.connection = connection;
this.channel = channel;
}catch(err) {
console.error(err);
}
}
log() {
console.log(this.connection);
}
}
async function createInstance(){
const instance = new IRabbitMQ();
try {
await instance.init('amqp://localhost');
}catch (e) {
throw new Error('OOPS!');
}
return instance;
}
async function runLogic() {
const r = await createInstance();
r.log();
}
runLogic().catch(console.log);
Just comment if you'd want me to give additional advice/tips, but this seems to work for me.

Firestore trigger timeouts occasionally

I have a Cloud Firestore trigger that takes care of adjusting the balance of a user's wallet in my app.
exports.onCreateTransaction = functions.firestore
.document('accounts/{accountId}/transactions/{transactionId}')
.onCreate(async (snap, context) => {
const { accountId, transactionId } = context.params;
const transaction = snap.data();
// See the implementation of alreadyTriggered in the next code block
const alreadyTriggered = await firestoreHelpers.triggers.alreadyTriggered(context);
if (alreadyTriggered) {
return null;
}
if (transaction.status === 'confirmed') {
const accountRef = firestore
.collection('accounts')
.doc(accountId);
const account = (await accountRef.get()).data();
const balance = transaction.type === 'deposit' ?
account.balance + transaction.amount :
account.balance - transaction.amount;
await accountRef.update({ balance });
}
return snap.ref.update({ id: transactionId });
});
As a trigger may actually be called more than once, I added this alreadyTriggered helper function:
const alreadyTriggered = (event) => {
return firestore.runTransaction(async transaction => {
const { eventId } = event;
const metaEventRef = firestore.doc(`metaEvents/${eventId}`);
const metaEvent = await transaction.get(metaEventRef);
if (metaEvent.exists) {
console.error(`Already triggered function for event: ${eventId}`);
return true;
} else {
await transaction.set(metaEventRef, event);
return false;
}
})
};
Most of the time everything works as expected. However, today I got a timeout error which caused data inconsistency in the database.
Function execution took 60005 ms, finished with status: 'timeout'
What was the reason behind this timeout? And how do I make sure that it never happens again, so that my transaction amounts are successfully reflected in the account balance?
That statement about more-than-once execution was a beta limitation, as stated. Cloud Functions is out of beta now. The current guarantee is at-least-once execution by default. you only get multiple possible events if you enable retries in the Cloud console. This is something you should do if you want to make sure your events are processed reliably.
The reason for the timeout may never be certain. There could be any number of reasons. Perhaps there was a hiccup in the network, or a brief amount of downtime somewhere in the system. Retries are supposed to help you recover from these temporary situations by delivering the event potentially many times, so your function can succeed.

A better way to structure a Mongoose connection module

I have refactored some code to place all my mongoose.createConnection(...) in a single file. This file is then required in other files that use connections to the various databases specified. The connections are lazily created and are used in both an http server and in utility scripts.
The connection file looks like this:
var mongoose = require("mongoose");
var serverString = "mongodb://localhost:27017";
var userDBString = "/USER";
var customerDBString = "/CUSTOMER";
var userConnection = null;
exports.getUserConnection = function () {
if (userConnection === null) {
userConnection = mongoose.createConnection(serverString + userDBString, {server: { poolSize: 4 }});
}
return userConnection;
};
var customerConnection = null;
exports.getCustomerConnection = function () {
if (customerConnection === null) {
customerConnection = mongoose.createConnection(serverString + customerDBString, { server: { poolSize: 4 }});
}
return customerConnection;
};
My models are stored in a separate files (based on their DB) that looks a bit like this:
exports.UserSchema = UserSchema; //Just assume I know how to define a valid schema
exports.UserModel = connection.getUserConnection().model("User", UserSchema);
Later , I use the getUserConnection() to refer to the connection I have created to actually do work the model.
TL;DR
In utilities that use this connection format, I have to call
connection.getUserConnection().on("open", function() {
logger.info("Opened User DB");
//Do What I Need To Do
});
It is possible that in some scenarios the task processor will have already broadcast the open event. In some, it won't be guaranteed to have happened yet. I noticed that it doesn't queue work if the connection isn't open (specifically, dropCollection) so I feel stuck.
How can I be certain that the connection is open before proceeding given that I can't depend on subscribing to the open event before the task processor runs?
Is there a better pattern for centralizing the managing of multiple connections?
I can answer part of my own question
How can I be certain that the connection is open before proceeding
given that I can't depend on subscribing to the open event before the
task processor runs?
if (connection.getUserConnection().readyState!==1) {
logger.info("User connection was not open yet. Adding open listener");
connection.getSR26Connection().on("open", function () {
logger.info("User open event received");
doStuff();
});
} else {
logger.info("User is already open.");
doStuff();
}
function doStuff() {
logger.info("Doing stuff");
}
If you see a better way then please comment or offer up an answer. I would still like to hear how other people manage connections without rebuilding the connection every time.

Keeping open a MongoDB database connection

In so many introductory examples of using MongoDB, you see code like this:
var MongoClient = require('mongodb').MongoClient;
MongoClient.connect("mongodb://localhost:port/adatabase", function(err, db)
{
/* Some operation... CRUD, etc. */
db.close();
});
If MongoDB is like any other database system, open and close operations are typically expensive time-wise.
So, my question is this: Is it OK to simply do the MongoClient.connect("... once, assign the returned db value to some module global, have various functions in the module do various database-related work (insert documents into collections, update documents, etc. etc.) when they're called by other parts of the application (and thereby re-use that db value), and then, when the application is done, only then do the close.
In other words, open and close are done once - not every time you need to go and do some database-related operation. And you keep re-using that db object that was returned during the initial open\connect, only to dispose of it at the end, with the close, when you're actually done with all your database-related work.
Obviously, since all the I/O is asynch, before the close you'd make sure that the last database operation completed before issuing the close. Seems like this should be OK, but i wanted to double-check just in case I'm missing something as I'm new to MongoDB. Thanks!
Yes, that is fine and typical behavior. start your app, connect to db, do operations against the db for a long time, maybe re-connect if the connection ever dies unexpectedly, and then just never close the connection (just rely on the automatic close that happens when your process dies).
mongodb version ^3.1.8
Initialize the connection as a promise:
const MongoClient = require('mongodb').MongoClient
const uri = 'mongodb://...'
const client = new MongoClient(uri)
const connection = client.connect() // initialized connection
And then call the connection whenever you wish you perform an action on the database:
// if I want to insert into the database...
const connect = connection
connect.then(() => {
const doc = { id: 3 }
const db = client.db('database_name')
const coll = db.collection('collection_name')
coll.insertOne(doc, (err, result) => {
if(err) throw err
})
})
The current accepted answer is correct in that you may keep the same database connection open to perform operations, however, it is missing details on how you can retry to connect if it closes. Below are two ways to automatically reconnect. It's in TypeScript, but it can easily be translated into normal Node.js if you need to.
Method 1: MongoClient Options
The most simple way to allow MongoDB to reconnect is to define a reconnectTries in an options when passing it into MongoClient. Any time a CRUD operation times out, it will use the parameters passed into MongoClient to decide how to retry (reconnect). Setting the option to Number.MAX_VALUE essentially makes it so that it retries forever until it's able to complete the operation. You can check out the driver source code if you want to see what errors will be retried.
class MongoDB {
private db: Db;
constructor() {
this.connectToMongoDB();
}
async connectToMongoDB() {
const options: MongoClientOptions = {
reconnectInterval: 1000,
reconnectTries: Number.MAX_VALUE
};
try {
const client = new MongoClient('uri-goes-here', options);
await client.connect();
this.db = client.db('dbname');
} catch (err) {
console.error(err, 'MongoDB connection failed.');
}
}
async insert(doc: any) {
if (this.db) {
try {
await this.db.collection('collection').insertOne(doc);
} catch (err) {
console.error(err, 'Something went wrong.');
}
}
}
}
Method 2: Try-catch Retry
If you want more granular support on trying to reconnect, you can use a try-catch with a while loop. For example, you may want to log an error when it has to reconnect or you want to do different things based on the type of error. This will also allow you to retry depending on more conditions than just the standard ones included with the driver. The insert method can be changed to the following:
async insert(doc: any) {
if (this.db) {
let isInserted = false;
while (isInserted === false) {
try {
await this.db.collection('collection').insertOne(doc);
isInserted = true;
} catch (err) {
// Add custom error handling if desired
console.error(err, 'Attempting to retry insert.');
try {
await this.connectToMongoDB();
} catch {
// Do something if this fails as well
}
}
}
}
}

Resources