mongodb nodejs native driver close connection or not - node.js

Is it a good practice in nodejs to open mongodb connection per each request, and close it in the callback?
app.get('/some_route', function(){
MongoClient.connect(url,function(err, db){
//some db query with callback
db.collection("some_collection").findOne(doc, function(err,item){
if(err){
res.send(err);
//close db connection
db.close();
}else{
//do something with item
res.send(item);
//close db connection
db.close();
}
});
});
Some said that opening/closing mongodb connection on each request isn't necessary, since once opened, a pool of connections can be shared.
The question is how to maintain and share that pool? Is mongoose doing that automatically already?
Especially, on mongodb timeout or disconnect, does it need to be reconnected?
I find contradictory answers here close mongodb connection per request or not
Almost all the online doc nodejs mongodb native driver and example code I read, a db.open() is paired with db.close() somewhere in the callback.
Because if a pool of connection is shared, one might code
According to christkv's answer, one might code:
var p_db=null;
var c_opt = {server:{auto_reconnect:true}};
app.get('/some_route', function(){
//pseudo code
if (!p_db){
MongoClient.connect(url, c_opt, function(err,db){
p_db = db;
p_db.collection("some_collection").findOne(doc, function(err,item){
if(err){
res.send(err);
}else{
//do something with item
res.send(item);
}
});
});
}else {
p_db.collection("some_collection").findOne(doc, function(err,item){
if(err){
res.send(err);
}else{
//do something with item
res.send(item);
}
});
});

According to a major contributor to the driver source, It's best to connect to the database at startup and keep reusing that same connection for each request.
The mongodb native driver has a connection pool which it maintains internally and currently defaults to a maximum of 5 open connections. You can configure the maximum number of connections via the maxPoolSize option. You can also configure the connection to auto reconnect with the auto_reconnect option.
See the documentation here

You don't have to do anything to reconnect as the driver will attempt to reconnect on failure. While it waits for the reconnect to happen it will buffer all operations happening in between and replay them once the connection is up. If you want to control this yourself you can listen to the "close" event on the db instance and handle reconnection manually. On reconnect the db object will still be viable as a db is really just a wrapper around the shared connection pool and does not contain it's own separate connection logic.

Related

Mongoose close connection issue

In my node js program, After opening mongo connection I trying to run certain query and finally trying to close connection.
group.find({group_name: "music"}, function (doc) {
// do your stuff
console.log(doc);
mongoose.connection.close();
});
But while I trying to see mongotop log, after closing the connection, its still show as connected
Here finananceManagement is the the database, which I tried to connect, If I restart the mongo service it will disapper from the log, help me to resolve this issue

The efficiency of continuously polling MongoDB in Node

I need to continuously update data on the client based on DB changes. I'm thinking about having a 5 second interval function that repeatedly gathers all the DB information and use Socket.IO to emit the data to the client.
Currently, I'm doing this on the client itself without socket.io, just repeatedly doing a REST call to the server which then handles the data.
My question is: Are either of these methods efficient or inefficient and is there a better solution to solve what I'm trying to achieve?
Ryan, you can try using MongoDB's collection.watch() which fires an event every time an update is made to a collection. You would need to do that within the socket connection event for it to work though. Something along these lines:
io.sockets.on('connection', function(socket) {
// when the socket is connected, start listening to MongoDB
const MongoClient = require("mongodb").MongoClient;
MongoClient.connect("mongodb://192.168.1.201")
.then(client => {
console.log("Connected correctly to server");
// specify db and collections
const db = client.db("your_db");
const collection = db.collection("your_collection");
const changeStream = collection.watch();
// start listening to changes
changeStream.on("change", function(change) {
console.log(change);
// this is where you can fire the socket.emit('the_change', change)
});
})
.catch(err => {
console.error(err);
});
});
Note that using this approach will require you to set up a replica set. You can follow those instructions or use a Dockerised replica set such as this one.
I need more details to make sure but it doesn't sound like a good solution.
If the data you need does not change rapidly, like let's say in seconds, each of your connection still polling every 5 seconds and that's kind of wasting.
In that case you might just trigger an event where the data got changed, then you can push the message through sockets that are active.

Should I keep database connection open?

When I connect to Rexster graph server with Grex should I keep the database connection open?
var grex = require('grex');
var client = grex.createClient();
client.connect({ graph: 'graph'}, function(err, client) {
if (err) { console.error(err); }
...
});
I think I should because nodejs is single threaded so there's no chance of different requests trying to use the one connection at the same time.
Yes, you should. There 's no reason to have the overhead of connecting on every request. There will not be any issue of "mangling", as your code will be run in a single thread anyway.
Furthermore, you could even have a pool of connections waiting to serve your requests in case you have a heavy usage application. Some adapters do it for you automatically, for example, MongoClient has a default pool of 5 connections.

Postgresql connection timed out in node.js and pg

I am new to node, postgresql, and to the whole web development business. I am currently writing a simple app which connects to a postgres database and display the content of a table in a web view. The app will be hosted in OpenShift.
My main entry is in server.js:
var pg = require('pg');
pg.connect(connection_string, function(err, client) {
// handle error
// save client: app.client = client;
});
Now, to handle the GET / request:
function handle_request(req, res){
app.client.query('...', function(err, result){
if (err) throw err; // Will handle error later, crash for now
res.render( ... ); // Render the web view with the result
});
}
My app seems to work: the table is rendered in the web view correctly, and it works for multiple connections (different web clients from different devices). However, if there is no request for a couple of minutes, then subsequent request will crash the app with time out information. Here is the stack information:
/home/hai/myapp/server.js:98
if (err) throw err;
^
Error: This socket is closed.
at Socket._write (net.js:474:19)
at Socket.write (net.js:466:15)
at [object Object].query (/home/hai/myapp/node_modules/pg/lib/connection.js:109:15)
at [object Object].submit (/home/hai/myapp/node_modules/pg/lib/query.js:99:16)
at [object Object]._pulseQueryQueue (/home/hai/myapp/node_modules/pg/lib/client.js:166:24)
at [object Object].query (/home/hai/myapp/node_modules/pg/lib/client.js:193:8)
at /home/hai/myapp/server.js:97:17
at callbacks (/home/hai/myapp/node_modules/express/lib/router/index.js:160:37)
at param (/home/hai/myapp/node_modules/express/lib/router/index.js:134:11)
at pass (/home/hai/myapp/node_modules/express/lib/router/index.js:141:5)
Is there a way to keep the connection from timed out (better)? Or to reconnect on demand (best)? I have tried to redesign my app by not connecting to the database in the beginning, but upon the GET / request. This solution works only for the first request, then crashed on the second. Any insight is appreciated.
Have you looked into the postgres keepalive setting values? It sends packets to keep idle connections from timing out.
http://www.postgresql.org/docs/9.1/static/runtime-config-connection.html
I also found this similar question:
How to use tcp_keepalives settings in Postgresql?
You could also perform really minor queries from the db at a set interval. However, this method is definitely more hacked.
Edit: You could also try initiating the client like this:
var client = new pg.Client(conString);
Before you make your queries, you can check if the client is still connected. I believe you can use:
if(client.connection._events != null)
client.connect();
faced the same problem.. telling the client to close connection upon the end event
query.on('end', function() {
client.end();
});
did the trick for me...
You can also change the default idle timeout of 30 seconds to whatever value you need. E.g.
pg.defaults.poolIdleTimeout = 600000; // 10 mins
I'm using the parameter keepAlive in true and it works.
This is my configuration and it is solved.
const client_pg = new Client({
connectionString,
keepAlive: true,
keepAliveInitialDelayMillis: 10000
});

How to handle Mongoose DB connection interruptions

I've been evaluating Mongoose (an ORM for node.js which uses MongoDB for persistent storage).
What I'd like to do is make sure that the app can run when the DB is not up when the app starts, and also handles the DB going down intelligently.
Currently my test app which does not work in either case does this:
var mongoose_connection = mongoose.createConnection(DATABASE_URL, {server:{poolSize:4}});
Then I use that connection when making Models.
In the case when the DB is down at the start of the app then any save() calls on instances silently fail with no error. If the DB comes back up they never get written.
So I would need to detect that the connection never happened and have the app be able to tell that at runtime so that I can handle it somehow.
When the DB goes down after the app has started though the save() calls still do not cause errors but they are queued and the written when the DB comes back.
That seems fine except I'd like to hook into the API to get events when the DB is down and to query how many save's are queued. At some point I might have so many queued events that I would want to just stop making new ones and have the app back off.
Case #1: db is down on app startup. there is a minor bug preventing this use case that I'm fixing now. However, here is the work-around:
var db = mongoose.createConnection();
db.on('error', function (err) {
if (err) // couldn't connect
// hack the driver to allow re-opening after initial network error
db.db.close();
// retry if desired
connect();
});
function connect () {
db.open('localhost', 'dbname');
}
connect();
https://gist.github.com/2878607
An ugly but working gist. First shutdown mongo, then run this gist.
Notice the connection failures.
Then start mongo and see all of the queued up inserts complete and dumped to the console.
Writes and finds will begin.
Shutdown mongo, notice that the inserts and finds are being attempted but no callbacks are executed.
Restart mongo. Notice all of the queued inserts and finds are completed.
If you would rather fail all the requests to the server when the db is down the native driver does emit the reconnect event which can be sensed in a middleware.
This works and emits the reconnect event fine (mongodb native driver 1.3.23)
mongoose.connection.db.on('reconnect', function (ref) {
connected=true;
console.log('reconnect to mongo server.');
});
So my dbconnection middleware looks for connected/error/reconnect
(some of the events are redundant but does not harm !)
PS. the initial connect failure needs to still be handled by a retry as
aaronheckmann's answer above.
mongoose.connection.on('open', function (ref) {
connected=true;
console.log('open connection to mongo server.');
});
mongoose.connection.on('connected', function (ref) {
connected=true;
console.log('connected to mongo server.');
});
mongoose.connection.on('disconnected', function (ref) {
connected=false;
console.log('disconnected from mongo server.');
});
mongoose.connection.on('close', function (ref) {
connected=false;
console.log('close connection to mongo server');
});
mongoose.connection.on('error', function (err) {
connected=false;
console.log('error connection to mongo server!');
console.log(err);
});
mongoose.connection.db.on('reconnect', function (ref) {
connected=true;
console.log('reconnect to mongo server.');
});

Resources