Context:
Mongoose v4.7.6
MongoDB v3.2.11
I'm trying to handle errors related to my database in my software.
I'm stuck in the following problem: When the database is disconnected, mongoose request hang until it get reconnected.
Here is what happend:
I launch my software
It connect to the database though mongoose
I Ctrl+C the mongod process
I get the "Disconnect" and "Close" event from mongoose
I launch a find(...) request
Find request hang
What I've tried so far:
I tried to use in my schema the option bufferCommands who according to the documentation was supposed to make mongoose return an error if there is no available connection, but the result is the same.
What is my code?
mongoose.createConnection(..., {
server: {
// We disable reconnect from mongoose
auto_reconnect: false,
socketOptions: {
// For long running applictions it is often prudent to enable keepAlive.
// Without it, after some period of time you may start to
// see "connection closed" errors for what seems like no reason.
// From mongoose documentation
keepAlive: 1,
},
},
})
The errors are thrown from the mongoose connection directly whenever there is a connection issue, the main server where you make the connection and there are several ways to handle it depending on what you want.
The find query you make is specifically for schema which in the end makes use of main connection object. You'll have to handle it that way for yourself and for users you'll have to configure a timeout for the request being made by them and send them appropriate response.
Cancelling Request based on timeout
This can be done on several levels, your server's logic, your client's end or the mongoose itself.
Follow this person's answer for setting timeout with mongoose, apparently it is not documented properly by mongoose.
https://stackoverflow.com/a/32609226/5225363
For server's logic, you can make a system for specific request that if there is no this then send a response back to client with something else.
On client if no response is received for a specific time then be assured that there is some problem.
p.s By default there is timeout setting for requests
Related
I want to know how to handle mongo connections with express.
My questions are:
Is it a good idea create a middleware to handle every request and before execute it, connect to mongo?
Is it a good idea create a middleware to handle every request and after make response, close mongo connection?
Best practice is to call mongoose.connect during your application's startup and just leave it open. That call creates a connection pool that is shared across all Mongoose models, by default.
The default pool size is 5, but you can tweak that in your call's options:
const options = {
poolSize: 10
};
mongoose.connect('mongodb://localhost/test', options);
I'm taking my first dip into the MEAN stack, and I'm working with the database through Mongoose. I'm not very familiar with Mongoose or MongoDB, so I don't know how they like to be configured. I also don't really know whether Mongoose matters or this is purely a MongoDB question.
The last time I wrote data access logic directly (without an ORM or injected repositories to handle connection management for me), it was in .NET with System.Data.SqlClient. I remember having to always make sure a SqlConnection was open no longer than necessary, and to always close it explicitly when I was done.
I've read lots of how-tos about writing MEAN apps, and nobody mentions this. From the code I've seen, I get the impression that MongoDB/Mongoose connections like to live at the application level, and that I should only be calling mongoose.connect once for the whole application.
Here's my connection code, which is called once on application startup:
mongoose = require "mongoose"
connection = mongoose.connection
mongoose.connect process.env.MONGO_URI
connection.on 'error', (err) ->
console.error 'DB connection error', err
.once 'open', ->
console.log 'DB open'
gracefulExit = ->
connection.close ->
console.log 'Mongoose default connection disconnected through app termination'
process.exit 0
process.on('SIGINT', gracefulExit)
.on('SIGTERM', gracefulExit)
module.exports = (name, dataStructure) ->
schema = new Schema dataStructure
return mongoose.model(name, schema)
JavaScript translation if you need it
This is the only place mongoose.connect is called, and the resulting connection object is reused throughout the application. Am I right to do it this way, or should I be creating, opening, closing, and destroying it on each request? What other scalability issues do I need to be aware of?
I realize this sounds a little open-ended, but I'm hoping for objective information about MongoDB and Mongoose internals and configuration. Are there going to be problems when I do this in a production environment with lots of concurrent requests?
What you're creating when calling mongoose.connect is not a single connection, but a connection pool that is intended to exist for the life of your application and be shared by all code via your registered Mongoose models.
So you're already doing things correctly, and if you want to alter the default size (5) of the connection pool, you can do that via the options parameter of mongoose.connect:
// Use a pool of 3 connections.
mongoose.connect(process.env.MONGO_URI, { server: { poolSize: 3 }});
I have an Express App which connects to a MongoDB server at startup and serves requests on-demand (I don't disconnect - it's a single threaded server so no pooling - fairly simple stuff)
Problem is that it's possible the MongoDB server will be unavailable for periods of time (it's not on-site) and whilst the Express App doesn't crash, it seems that any requests made to the server will run indefinately until the connection is restored!
I'd like to limit that (e.g. throw an error back after a period of time) but I can't seem to make that happen...
I'm using connect options "{server: {auto_reconnect: true}}" which seems to ensure that once the MongoDB server reappears, requests complete (without it, requests made during downtime seem to run forever...) - and I don't have access to the client code so I can't fix it there...
I'd assumed a combination of 'connectTimeoutMS' or 'socketTimeoutMS' would allow me to terminate requests when MongoDB is unavailable for longer periods, but I just can't get those to work (I've tried them as connect options, passing them in the URI etc. etc.)
Any attempt to open a Collection and Find/Insert/Update just 'hangs' until the MongoDB reappears - I've left it over 30 mins and everything was just sitting these (and completed AOK when the network was restored!)
What's the best way around this? Should I open a connection specifically for each request (not really a performance issue - it's not a high volume app) or is there something else I'm missing?
Updated to add the connect code
var myDB
var mongodb = require('mongodb')
var uri = // some env vars and stuff
mongodb.MongoClient.connect(uri, {server: {auto_reconnect: true}}, function (err, db) {
myDB = db
})
myDB is then used elsewhere to open collections - and the handle from that is used to find/insert etc.
If the connection to the DB is interrupted, myDB.collection() calls (or calls to find/insert on their handles) will simply hang until the connection is restored - nothing I've tried will cause them to 'time out' sooner!?
I assume that you are using mongoose as a driver.
You'd catch the error by this.
var db = require('domain').create();
db.on('error', function(err) {
console.log('DB got a problem');
});
db.run(function() {
mongoose.connect(config, options);
});
or you can directly access
mongoose.connection.readyState
to check the statement of your DB.
Connection ready state
0 = disconnected
1 = connected
2 = connecting
3 = disconnecting
Each state change emits its associated event name.
http://mongoosejs.com/docs/api.html
I'm going through a few error scenarios, trying to understand how to handle those.
In the case where there is no database connection, a Mongoose Model.find(...) call seems to hang. Below the example code. I would have assumed that the callback is invoked with an err object, but it is not.
How can I prevent the model call to hang? Do I manually have to check the readyState each time I access a model?
// app.js
// Let's use a non-existing host so connecting fails:
// (callback is invoked with err object)
mongoose.connect('mongodb://localhostXXX/blog', function(err){ ... });
BlogPost = mongoose.model('BlogPost', BlogPostSchema);
// api.js
exports.list_posts = function(req, res) {
// Ready state is '0' = disconnected (since we used a wrong hostname)
console.log('DB ready state: ' + BlogPost.db.readyState);
// This will not invoke the callback:
BlogPost.find(function(err, threads) {
// Never called...
});
}
It is not an answer, but hope it will help you to find solution. Had very similar issue with
mongoose.createConnection
while using passport module, found out that it works fine with
mongoose.connect
Since you're already using an error handler in the connect call, a sensible thing would be to quit your app when the DB is not up, or activate some middleware that responds with a pretty 500 Internal Server Error.
Mongoose uses node-mongodb-native under the hood for connections to mongodb, you might find other useful connection options in there. :)
EDIT: try setting socketOptions.socketTimeoutMS. Apparently there is no default timeout set. See http://mongodb.github.com/node-mongodb-native/api-generated/server.html.
I don't have node on my work machine to try out the exact syntax for you, but you will probably have to use mongoose.Connection, which has an open() method that accepts options to pass through to node-mongodb-native. I don't think mongoose.connect() accepts these options, but I could be wrong.
In order to solve this problem you need to do 3 tasks:
Configure the bufferMaxEntries:0 in the options.db section (for more details see here)
So when disable bufferMaxEntries this causes mongoose to stop buffer commands and retrying to send them when the server is down.
Configure the autoReconnect:false in the options.db section
disable autoReconnet in the db level. (see more information here)
if you are working with mongodb replicaset then you need to disable bufferCommands in the schema level (for each schema you create)
var schema = new Schema({..}, { bufferCommands: false });
In production, I have a game which uses connection-local variables to hold game state. However I notice that if I idle for a certain time on the connection, it disconnects and reconnects which loses the current state. During my tests on a local host, I never noticed this behavior. Is this the norm behavior for socket connections or is something else causing the connections to drop.
If it is a normal behavior how is this typically handled? Should connection values be stored globally so they can be restored should a user drop/reconnect?
Your problem is around socket timeouts. If there's no activity on a certain socket, socket.io will close it automatically.
An easy (and hackish) fix is to send a heartbeat to the connected client to create activity and stop the socket from timing out.
Server:
function sendHeartbeat(){
setTimeout(sendHeartbeat, 8000);
io.sockets.emit('ping', { beat : 1 });
}
io.sockets.on('connection', function (socket) {
socket.on('pong', function(data){
console.log("Pong received from client");
});
}
setTimeout(sendHeartbeat, 8000);
Client:
socket.on('ping', function(data){
socket.emit('pong', {beat: 1});
});
More Information:
You can get more information on configuring socket.io here.
EDIT: Mark commented that if the user does lose the connection (connection drops on his end because of internet troubles), you should be able to restore the user to his last state.
To do that, the best way would be to use a already widely used method for storing user data, cookies and sessions.
An extremely well done tutorial on how to do this located here. Although he uses express to set cookies, you can do this using anything (I do it using rails). Using this method, you can store the user data in a cookie and fetch it during the handshake. From there you can just access the data using socket.handshake.data.
What you need to do is create or identify the session per (re-) connection. You may reduce the number of reconnections per Moox's answer above but it is still not failsafe - e.g. a user loses wifi connection for a bit, etc. In other words - maintain user metadata per session and not per socket, and expect occasional disconnects and reconnects.