I read this article on connection pooling with mongodb/nodejs. There he opens the connection once, and leaves it at that.
This is how I set up the database connection in my app.js file:
mongodb.MongoClient.connect(MONGODB_URI, function (error, database) {
if (error) throw error;
db = database; // db is defined outside this callback
coll = db.collection('testData'); // coll is defined outside this callback
});
This is going to leave the db connection open as long as the server is running. Shouldn't you close the connection at some point? Or is it of no consequence to leave it open?
If your app has support for controlled shutdown, then you should close the connection pool at that time. Otherwise you just leave it open.
The connection pool manages the number of actual connections for you, adding more in times of heavy load and closing them when your app is idle.
Related
We are using Mongoose, Nodejs, Serverless, and AWS Lambda. For making use of the same connection instead of opening and closing the connection each time whenever required, I have created a connection pool of size 10 (Which seems to be sufficient for our use-case right now).
But the thing is, when I see the Cloudwatch logs for Lambda, it's not the same connection that is being used.
Every time a new Lambda is called, a new connection is created, while the subsequent calls to that Lambda use the same connection that was opened in the first call.
Resulting in an increase in the number of connections open at a time. At MongoDB Atlas, I can see the number of open connections is way much.
Below is the code I am using for creating a connection if there is no cached connection available. In case it is available, the cached one will be used and a new connection will not be created.
let cached_db;
exports.createConnection = async () => {
if(cached_db == null){
return await mongoose.connect(
connection_uri,
{ 'useUnifiedTopology': true ,
'useNewUrlParser': true,
'useFindAndModify': false ,
'useCreateIndex': true,
'socketTimeoutMS': 60000,
'connectTimeoutMS': 60000,
'poolSize': 10
}
).then(conn => {
cached_db = conn;
return conn;
}).catch((err) => {
console.error('Something went wrong', err);
throw err;
});
} else {
console.log("Cached db in use.");
return cached_db;
}
}
Can the same connection be used across Lambdas? Is there a way to do it?
You should define the client to the MongoDB server outside the AWS Lambda handler function. Don't define a new MongoClient object each time you invoke your function. Doing so causes the driver to create a new database connection with each function call. This can be expensive and can result in your application exceeding database connection limits.
As an alternative, do the following:
Create the MongoClient object once.
Store the object so your function can reuse the MongoClient across function invocations.
Step 1
Isolate the call to the MongoClient.connect() function into its own module so that the connections can be reused across functions. Let's create a file mongo-client.js for that:
mongo-client.js:
const { MongoClient } = require('mongodb');
// Export a module-scoped MongoClient promise. By doing this in a separate
// module, the client can be shared across functions.
const client = new MongoClient(process.env.MONGODB_URI);
module.exports = client.connect();
Step 2
Import the new module and use it in function handlers to connect to database.
some-file.js:
const clientPromise = require('./mongodb-client');
// Handler
module.exports.handler = async function(event, context) {
// Get the MongoClient by calling await on the connection promise. Because
// this is a promise, it will only resolve once.
const client = await clientPromise;
// Use the connection to return the name of the connected database for example.
return client.db().databaseName;
}
Pool Size
Connection pool size is a cache of database connections maintained so these connections can be reused when future requests to the database are required. Connection pools are used to enhance the performance of executing commands on a database.
Note: maxPoolSize and poolSize are the same, except they relate to whether you are using the useUnifiedTopology: true setting.
If you are using useUnifiedTopology: true, maxPoolSize is the spec-compliant setting to manage how large connection pools can be.
But if you are using useUnifiedTopology: false (or omits it), poolSize is the same thing but from before we had the unified topology.
Note: Each connection consumes about 1MB of RAM.
Value of the Pool Size
The connection pool is on a per-mongod/mongos basis, so when connecting to a 3-member replica there will be three connection pools (one per mongod), each with a maxPoolSize. Additionally, there is a required monitoring connection for each node as well, so you end up with (maxPoolSize+1)*number_of_nodes TCP connections.
In my opinion, if you don't care about CPU and RAM, you should use all available connections (why not if we already have them, right?).
For example: You have Atlas free cluster with 3 replica sets, that supports maximum number of 500 connections, and you have only one application that connects to it, give all connections to that one application. In order to set the value of poolSize, you can use above calculation of connections:
poolSize = (maximum_connections/number_of_nodes) - 1
poolSize = (500/3) - 1
poolSize = 165
If you would have 2 applications that will connect to that same cluster, give each application half of connections.
If you have limited RAM memory, check how much you can spear and calculate poolSize based on that (as I said in the note, you can assume that one connection will consume about 1MB of RAM).
Resources
For more info, check this official MongoDB Docs.
For connection pool, check this and this.
I found from this blog that Lambda may use same connection if restore the same snapshot and creates new connection if new snapshot generation.
So Lambda can't give assurance that to use same connection if we use outside the handle function.
So in my opinion best approach to optimise number of connection to Mongodb is to close connection before lambda complete so your other service can use free connection.
Use below method to close connection after database interaction finishes.
createConnection.close()
I have an Express App which connects to a MongoDB server at startup and serves requests on-demand (I don't disconnect - it's a single threaded server so no pooling - fairly simple stuff)
Problem is that it's possible the MongoDB server will be unavailable for periods of time (it's not on-site) and whilst the Express App doesn't crash, it seems that any requests made to the server will run indefinately until the connection is restored!
I'd like to limit that (e.g. throw an error back after a period of time) but I can't seem to make that happen...
I'm using connect options "{server: {auto_reconnect: true}}" which seems to ensure that once the MongoDB server reappears, requests complete (without it, requests made during downtime seem to run forever...) - and I don't have access to the client code so I can't fix it there...
I'd assumed a combination of 'connectTimeoutMS' or 'socketTimeoutMS' would allow me to terminate requests when MongoDB is unavailable for longer periods, but I just can't get those to work (I've tried them as connect options, passing them in the URI etc. etc.)
Any attempt to open a Collection and Find/Insert/Update just 'hangs' until the MongoDB reappears - I've left it over 30 mins and everything was just sitting these (and completed AOK when the network was restored!)
What's the best way around this? Should I open a connection specifically for each request (not really a performance issue - it's not a high volume app) or is there something else I'm missing?
Updated to add the connect code
var myDB
var mongodb = require('mongodb')
var uri = // some env vars and stuff
mongodb.MongoClient.connect(uri, {server: {auto_reconnect: true}}, function (err, db) {
myDB = db
})
myDB is then used elsewhere to open collections - and the handle from that is used to find/insert etc.
If the connection to the DB is interrupted, myDB.collection() calls (or calls to find/insert on their handles) will simply hang until the connection is restored - nothing I've tried will cause them to 'time out' sooner!?
I assume that you are using mongoose as a driver.
You'd catch the error by this.
var db = require('domain').create();
db.on('error', function(err) {
console.log('DB got a problem');
});
db.run(function() {
mongoose.connect(config, options);
});
or you can directly access
mongoose.connection.readyState
to check the statement of your DB.
Connection ready state
0 = disconnected
1 = connected
2 = connecting
3 = disconnecting
Each state change emits its associated event name.
http://mongoosejs.com/docs/api.html
When I connect to Rexster graph server with Grex should I keep the database connection open?
var grex = require('grex');
var client = grex.createClient();
client.connect({ graph: 'graph'}, function(err, client) {
if (err) { console.error(err); }
...
});
I think I should because nodejs is single threaded so there's no chance of different requests trying to use the one connection at the same time.
Yes, you should. There 's no reason to have the overhead of connecting on every request. There will not be any issue of "mangling", as your code will be run in a single thread anyway.
Furthermore, you could even have a pool of connections waiting to serve your requests in case you have a heavy usage application. Some adapters do it for you automatically, for example, MongoClient has a default pool of 5 connections.
Is it a good practice in nodejs to open mongodb connection per each request, and close it in the callback?
app.get('/some_route', function(){
MongoClient.connect(url,function(err, db){
//some db query with callback
db.collection("some_collection").findOne(doc, function(err,item){
if(err){
res.send(err);
//close db connection
db.close();
}else{
//do something with item
res.send(item);
//close db connection
db.close();
}
});
});
Some said that opening/closing mongodb connection on each request isn't necessary, since once opened, a pool of connections can be shared.
The question is how to maintain and share that pool? Is mongoose doing that automatically already?
Especially, on mongodb timeout or disconnect, does it need to be reconnected?
I find contradictory answers here close mongodb connection per request or not
Almost all the online doc nodejs mongodb native driver and example code I read, a db.open() is paired with db.close() somewhere in the callback.
Because if a pool of connection is shared, one might code
According to christkv's answer, one might code:
var p_db=null;
var c_opt = {server:{auto_reconnect:true}};
app.get('/some_route', function(){
//pseudo code
if (!p_db){
MongoClient.connect(url, c_opt, function(err,db){
p_db = db;
p_db.collection("some_collection").findOne(doc, function(err,item){
if(err){
res.send(err);
}else{
//do something with item
res.send(item);
}
});
});
}else {
p_db.collection("some_collection").findOne(doc, function(err,item){
if(err){
res.send(err);
}else{
//do something with item
res.send(item);
}
});
});
According to a major contributor to the driver source, It's best to connect to the database at startup and keep reusing that same connection for each request.
The mongodb native driver has a connection pool which it maintains internally and currently defaults to a maximum of 5 open connections. You can configure the maximum number of connections via the maxPoolSize option. You can also configure the connection to auto reconnect with the auto_reconnect option.
See the documentation here
You don't have to do anything to reconnect as the driver will attempt to reconnect on failure. While it waits for the reconnect to happen it will buffer all operations happening in between and replay them once the connection is up. If you want to control this yourself you can listen to the "close" event on the db instance and handle reconnection manually. On reconnect the db object will still be viable as a db is really just a wrapper around the shared connection pool and does not contain it's own separate connection logic.
I am trying to use node.js with mongodb and following the tutorial at http://howtonode.org/express-mongodb
The code for opening the connection is
ArticleProvider = function(host, port) {
this.db= new Db('node-mongo-blog', new Server(host, port, {auto_reconnect: true}, {}));
this.db.open(function(){});
};
However i cannot see any connections being closed.
But when i see the logs on the mongo console, i can see that are connections which open and they close after some time.
Does the connection close automatically? Will it be a problem when a large no of clients try to access the server? Where should the connection be closed?
Thanks
Tuco
In that example application, only a single ArticleProvider object is created for the application to share when serving requests. That object's constructor opens a db connection that won't be closed until the application terminates (which is fine).
So what you should see is that you get a new mongo connection each time you start your app, but no additional connections made no matter how many clients access the server. And shortly after you terminate your app you should see its connection disappear on the mongo side.
node-mongodb-native provides a close method for Db objects and you can close your connection when you are finished by calling it.
var that = this;
this.db.open(function(){
// do db work here
// close the connection
that.db.close();
});
If you don't close your connection, event loop keeps the connection open and your process doesn't exit. If you are building a web server where your process will not be terminated, it's not necessary for you to close the connection.
A better reference for node-mongodb-native can be found on https://github.com/mongodb/node-mongodb-native.
Remember to put the db.close in the last callback that gets executed so the connection is open until all callbacks are finished. Otherwise, it gives an error like
/usr/local/lib/node_modules/mongodb/lib/utils.js:97
process.nextTick(function() { throw err; });
^
Error
at Error.MongoError (/usr/local/lib/node_modules/mongodb/node_modules/mongodb-core/lib/error.js:13:17)
at Server.destroy (/usr/local/lib/node_modules/mongodb/node_modules/mongodb-core/lib/topologies/server.js:629:47)
at Server.close (/usr/local/lib/node_modules/mongodb/lib/server.js:344:17)
at Db.close (/usr/local/lib/node_modules/mongodb/lib/db.js:267:19)