I use node.js, mongodb, mongoose 2.3 for my app.
I have following code:
console.log(config);
db = mongoose.connect(
config[MONGO_HOST_CONFIG],
config[MONGO_ACCOUNTS_DB_CONFIG],
config[MONGO_PORT_CONFIG],
function(err) {
if (err) throw err;
}
);
Config looks like:
{ 'mongo.host': 'google.com',
'mongo.port': '27',
'mongo.accounts.db': 'accounts',
'mongo.sessions.db': 'sessions' }
It times out after some time. Question: How do I find out the timeout interval?
If I change HOST to: example.com: It fails immediately which is good.
If I change host to aruunt.com, it never times out and goes into wait state. The connection is also not established. aruunt.com is some random domain owned by me.
What is the issue here?
Default driver is never to timeout or whatever is the os default timeout (I think it's 30 min on linux bot not completely sure). You can tweak it yourself as the link above show. But don't set it to small or you'll continuously get timeouts and reconnects
Related
I've been using node-redis for a while and so far so good. However, upon setting up a new environment, I had a typo on the hostname (or password) and it wouldn't connect. But because this was an already working application I developed some time ago, it was kind of hard to track the actual issue. When you made requests against this server, it would just take up to the server's timeout which was 5 minutes and come back with error 500.
At the end I found out that it was the credentials for the redis server. I use redis to make my app faster by preventing revalidating security tokens for up to an hour (since the validation process can take up to 2000ms), so I store the token on redis for future requests.
This has worked fine for years, however, just because this time I had a typo on the hostname or password, I noticed that if the redis server can't connect (for whaterver reason) the whole application goes down. The idea is that redis should be used if available, if not it should fallback to just take the long route but fulfill the request anyway.
So my question is, how to tell node-redis to throw an error as soon as possible, and not wait until ETIMEOUT error comes?
For example:
const client = redis.createClient(6380, "redis.host.com", { password: "verystrongone" } });
client.on("error", err => {
console.log(err)
})
Based on this code, I get the console.log error AFTER it reaches timeout (around 30-40 seconds). This is not good, because then my application is AT LEAST 30 seconds unresponsive. What I want to achieve is that if the redis is down or something, it should just give up after 2-5 seconds. I use a very fast and reliable redis server from Azure. It takes less than a second to connect, and has never failed, I believe, but if it does, it will take the whole application with it.
I tried stuff like retry_strategy but I believe that option kicks in only after the initial ~30 seconds attempt.
Any suggestions?
So here's an interesting thing I observed.
When I connect to the redis cache instance using the following options, I am able to reproduce the error you're getting.
port: 6380,
host: myaccount.redis.cache.windows.net,
auth_pass: mysupersecretaccountkey
When I specify incorrect password, I get an error after 1 minute.
However, if I specify tls parameter I get an error almost instantaneously:
port: 6380,
host: myaccount.redis.cache.windows.net,
auth_pass: mysupersecretaccountkey,
tls: {
servername: myaccount.redis.cache.windows.net
}
Can you try with tls option?
I am still not able to reproduce the error if I specify incorrect account name. I get the following error almost instantaneously:
Redis connection to
myincorrectaccountname.redis.cache.windows.net:6380 failed -
getaddrinfo ENOTFOUND myincorrectaccountname.redis.cache.windows.net
Context:
Mongoose v4.7.6
MongoDB v3.2.11
I'm trying to handle errors related to my database in my software.
I'm stuck in the following problem: When the database is disconnected, mongoose request hang until it get reconnected.
Here is what happend:
I launch my software
It connect to the database though mongoose
I Ctrl+C the mongod process
I get the "Disconnect" and "Close" event from mongoose
I launch a find(...) request
Find request hang
What I've tried so far:
I tried to use in my schema the option bufferCommands who according to the documentation was supposed to make mongoose return an error if there is no available connection, but the result is the same.
What is my code?
mongoose.createConnection(..., {
server: {
// We disable reconnect from mongoose
auto_reconnect: false,
socketOptions: {
// For long running applictions it is often prudent to enable keepAlive.
// Without it, after some period of time you may start to
// see "connection closed" errors for what seems like no reason.
// From mongoose documentation
keepAlive: 1,
},
},
})
The errors are thrown from the mongoose connection directly whenever there is a connection issue, the main server where you make the connection and there are several ways to handle it depending on what you want.
The find query you make is specifically for schema which in the end makes use of main connection object. You'll have to handle it that way for yourself and for users you'll have to configure a timeout for the request being made by them and send them appropriate response.
Cancelling Request based on timeout
This can be done on several levels, your server's logic, your client's end or the mongoose itself.
Follow this person's answer for setting timeout with mongoose, apparently it is not documented properly by mongoose.
https://stackoverflow.com/a/32609226/5225363
For server's logic, you can make a system for specific request that if there is no this then send a response back to client with something else.
On client if no response is received for a specific time then be assured that there is some problem.
p.s By default there is timeout setting for requests
I have an Express App which connects to a MongoDB server at startup and serves requests on-demand (I don't disconnect - it's a single threaded server so no pooling - fairly simple stuff)
Problem is that it's possible the MongoDB server will be unavailable for periods of time (it's not on-site) and whilst the Express App doesn't crash, it seems that any requests made to the server will run indefinately until the connection is restored!
I'd like to limit that (e.g. throw an error back after a period of time) but I can't seem to make that happen...
I'm using connect options "{server: {auto_reconnect: true}}" which seems to ensure that once the MongoDB server reappears, requests complete (without it, requests made during downtime seem to run forever...) - and I don't have access to the client code so I can't fix it there...
I'd assumed a combination of 'connectTimeoutMS' or 'socketTimeoutMS' would allow me to terminate requests when MongoDB is unavailable for longer periods, but I just can't get those to work (I've tried them as connect options, passing them in the URI etc. etc.)
Any attempt to open a Collection and Find/Insert/Update just 'hangs' until the MongoDB reappears - I've left it over 30 mins and everything was just sitting these (and completed AOK when the network was restored!)
What's the best way around this? Should I open a connection specifically for each request (not really a performance issue - it's not a high volume app) or is there something else I'm missing?
Updated to add the connect code
var myDB
var mongodb = require('mongodb')
var uri = // some env vars and stuff
mongodb.MongoClient.connect(uri, {server: {auto_reconnect: true}}, function (err, db) {
myDB = db
})
myDB is then used elsewhere to open collections - and the handle from that is used to find/insert etc.
If the connection to the DB is interrupted, myDB.collection() calls (or calls to find/insert on their handles) will simply hang until the connection is restored - nothing I've tried will cause them to 'time out' sooner!?
I assume that you are using mongoose as a driver.
You'd catch the error by this.
var db = require('domain').create();
db.on('error', function(err) {
console.log('DB got a problem');
});
db.run(function() {
mongoose.connect(config, options);
});
or you can directly access
mongoose.connection.readyState
to check the statement of your DB.
Connection ready state
0 = disconnected
1 = connected
2 = connecting
3 = disconnecting
Each state change emits its associated event name.
http://mongoosejs.com/docs/api.html
I am new to node, postgresql, and to the whole web development business. I am currently writing a simple app which connects to a postgres database and display the content of a table in a web view. The app will be hosted in OpenShift.
My main entry is in server.js:
var pg = require('pg');
pg.connect(connection_string, function(err, client) {
// handle error
// save client: app.client = client;
});
Now, to handle the GET / request:
function handle_request(req, res){
app.client.query('...', function(err, result){
if (err) throw err; // Will handle error later, crash for now
res.render( ... ); // Render the web view with the result
});
}
My app seems to work: the table is rendered in the web view correctly, and it works for multiple connections (different web clients from different devices). However, if there is no request for a couple of minutes, then subsequent request will crash the app with time out information. Here is the stack information:
/home/hai/myapp/server.js:98
if (err) throw err;
^
Error: This socket is closed.
at Socket._write (net.js:474:19)
at Socket.write (net.js:466:15)
at [object Object].query (/home/hai/myapp/node_modules/pg/lib/connection.js:109:15)
at [object Object].submit (/home/hai/myapp/node_modules/pg/lib/query.js:99:16)
at [object Object]._pulseQueryQueue (/home/hai/myapp/node_modules/pg/lib/client.js:166:24)
at [object Object].query (/home/hai/myapp/node_modules/pg/lib/client.js:193:8)
at /home/hai/myapp/server.js:97:17
at callbacks (/home/hai/myapp/node_modules/express/lib/router/index.js:160:37)
at param (/home/hai/myapp/node_modules/express/lib/router/index.js:134:11)
at pass (/home/hai/myapp/node_modules/express/lib/router/index.js:141:5)
Is there a way to keep the connection from timed out (better)? Or to reconnect on demand (best)? I have tried to redesign my app by not connecting to the database in the beginning, but upon the GET / request. This solution works only for the first request, then crashed on the second. Any insight is appreciated.
Have you looked into the postgres keepalive setting values? It sends packets to keep idle connections from timing out.
http://www.postgresql.org/docs/9.1/static/runtime-config-connection.html
I also found this similar question:
How to use tcp_keepalives settings in Postgresql?
You could also perform really minor queries from the db at a set interval. However, this method is definitely more hacked.
Edit: You could also try initiating the client like this:
var client = new pg.Client(conString);
Before you make your queries, you can check if the client is still connected. I believe you can use:
if(client.connection._events != null)
client.connect();
faced the same problem.. telling the client to close connection upon the end event
query.on('end', function() {
client.end();
});
did the trick for me...
You can also change the default idle timeout of 30 seconds to whatever value you need. E.g.
pg.defaults.poolIdleTimeout = 600000; // 10 mins
I'm using the parameter keepAlive in true and it works.
This is my configuration and it is solved.
const client_pg = new Client({
connectionString,
keepAlive: true,
keepAliveInitialDelayMillis: 10000
});
I'm using nodejs and a mongoDB - and I'm having some connection issues.
Well, actually "wake" issues! It connects perfectly well - is super fast and I'm generally happy with the results.
My problem: If i don't use the connection for a while (i say while, because the timeframe varies 5+ mins) it seems to stall. I don't get disconnection events fired - it just hangs.
Eventually i get a response like Error: failed to connect to [ * .mongolab.com: * ] - ( * = masked values)
A quick restart of the app, and the connection's great again. Sometimes, if i don't restart the app, i can refresh and it reconnects happily.
This is why i think it is "wake" issues.
Rough outline of code:
I've not included the code - I don't think it's needed. It works (apart from the connection dropout)
Things to note: There is just the one "connect" - i never close it. I never reopen.
I'm using mongoose, socketio.
/* constants */
var mongoConnect = 'myworkingconnectionstring-includingDBname';
/* includes */
/* settings */
/* Schema */
var db = mongoose.connect(mongoConnect);
/* Socketio */
io.configure(function (){
io.set('authorization', function (handshakeData, callback) {
});
});
io.sockets.on('connection', function (socket) {
});//sockets
io.sockets.on('disconnect', function(socket) {
console.log('socket disconnection')
});
/* The Routing */
app.post('/login', function(req, res){
});
app.get('/invited', function(req, res){
});
app.get('/', function(req, res){
});
app.get('/logout', function(req, res){
});
app.get('/error', function(req, res){
});
server.listen(port);
console.log('Listening on port '+port);
db.connection.on('error', function(err) {
console.log("DB connection Error: "+err);
});
db.connection.on('open', function() {
console.log("DB connected");
});
db.connection.on('close', function(str) {
console.log("DB disconnected: "+str);
});
I have tried various configs here, like opening and closing all the time - I believe though, the general consensus is to do as i am with one open wrapping the lot. ??
I have tried a connection tester, that keeps checking the status of the connection... even though this appears to say everthing's ok - the issue still happens.
I have had this issue from day one. I have always hosted the MongoDB with MongoLab.
The problem appears to be worse on localhost. But i still have the issue on Azure and now nodejit.su.
As it happens everywhere - it must be me, MongoDB, or mongolab.
Incidentally i have had a similar experience with the php driver too. (to confirm this is on nodejs though)
It would be great for some help - even if someone just says "this is normal"
thanks in advance
Rob
UPDATE: Our support article for this topic (essentially a copy of this post) has moved to our connection troubleshooting doc.
There is a known issue that the Azure IaaS network enforces an idle timeout of roughly thirteen minutes (empirically arrived at). We are working with Azure to see if we can't make things more user-friendly, but in the meantime others have had success by configuring their driver options to work around the issue.
Max connection idle time
The most effective workaround we've found in working with Azure and our customers has been to set the max connection idle time below four minutes. The idea is to make the driver recycle idle connections before the firewall forces the issue. For example, one customer, who is using the C# driver, set MongoDefaults.MaxConnectionIdleTime to one minute and it cleared up their issues.
MongoDefaults.MaxConnectionIdleTime = TimeSpan.FromMinutes(1);
The application code itself didn't change, but now behind the scenes the driver aggressively recycles idle connections. The result can be seen in the server logs as well: lots of connection churn during idle periods in the app.
There are more details on this approach in the related mongo-user thread, SocketException using C# driver on azure.
Keepalive
You can also work around the issue by making your connections less idle with some kind of keepalive. This is a little tricky to implement unless your driver supports it out of the box, usually by taking advantage of TCP Keepalive. If you need to roll your own, make sure to grab each idle connection from the pool every couple minutes and issue some simple and cheap command, probably a ping.
Handling disconnects
Disconnects can happen from time to time even without an aggressive firewall setup. Before you get into production you want to be sure to handle them correctly.
First, be sure to enable auto reconnect. How to do so varies from driver to driver, but when the driver detects that an operation failed because the connection was bad turning on auto reconnect tells the driver to attempt to reconnect.
But this doesn't completely solve the problem. You still have the issue of what to do with the failed operation that triggered the reconnect. Auto reconnect doesn't automatically retry failed operations. That would be dangerous, especially for writes. So usually an exception is thrown and the app is asked to handle it. Often retrying reads is a no-brainer. But retrying writes should be carefully considered.
The mongo shell session below demonstrates the issue. The mongo shell by default has auto reconnect enabled. I insert a document in a collection named stuff then find all the documents in that collection. I then set a timer for thirty minutes and tried the same find again. It failed, but the shell automatically reconnected and when I immediately retried my find it worked as expected.
% mongo ds012345.mongolab.com:12345/mydatabase -u *** -p ***
MongoDB shell version: 2.2.2
connecting to: ds012345.mongolab.com:12345/mydatabase
> db.stuff.insert({})
> db.stuff.find()
{ "_id" : ObjectId("50f9b77c27b2e67041fd2245") }
> db.stuff.find()
Fri Jan 18 13:29:28 Socket recv() errno:60 Operation timed out 192.168.1.111:12345
Fri Jan 18 13:29:28 SocketException: remote: 192.168.1.111:12345 error: 9001 socket exception [1] server [192.168.1.111:12345]
Fri Jan 18 13:29:28 DBClientCursor::init call() failed
Fri Jan 18 13:29:28 query failed : mydatabase.stuff {} to: ds012345.mongolab.com:12345
Error: error doing query: failed
Fri Jan 18 13:29:28 trying reconnect to ds012345.mongolab.com:12345
Fri Jan 18 13:29:28 reconnect ds012345.mongolab.com:12345 ok
> db.stuff.find()
{ "_id" : ObjectId("50f9b77c27b2e67041fd2245") }
We're here to help
Of course, if you have any questions please feel free to contact us at support#mongolab.com. We're here to help.
Thanks for all the help guys - I have managed to solve this issue on both localhost and deployed to a live server.
Here is my now working connect code:
var MONGO = {
username: "username",
password: "pa55W0rd!",
server: '******.mongolab.com',
port: '*****',
db: 'dbname',
connectionString: function(){
return 'mongodb://'+this.username+':'+this.password+'#'+this.server+':'+this.port+'/'+this.db;
},
options: {
server:{
auto_reconnect: true,
socketOptions:{
connectTimeoutMS:3600000,
keepAlive:3600000,
socketTimeoutMS:3600000
}
}
}
};
var db = mongoose.createConnection(MONGO.connectionString(), MONGO.options);
db.on('error', function(err) {
console.log("DB connection Error: "+err);
});
db.on('open', function() {
console.log("DB connected");
});
db.on('close', function(str) {
console.log("DB disconnected: "+str);
});
I think the biggest change was to use "createConnection" over "connect" - I had used this before, but maybe the options help now. This article helped a lot http://journal.michaelahlers.org/2012/12/building-with-nodejs-persistence.html
If I'm honest I'm not overly sure on why I have added those options - as mentioned by #jareed, i also found some people having success with "MaxConnectionIdleTime" - but as far as i can see the javascript driver doesn't have this option: this was my attempt at trying to replicate the behavior.
So far so good - hope this helps someone.
UPDATE: 18 April 2013 note, this is a second app with a different setup
Now I thought i had this solved but the problem rose it's ugly head again on another app recently - with the same connection code. Confused!!!
However the set up was slightly different…
This new app was running on a windows box using IISNode. I didn't see this as significant initially.
I read there were possibly some issues with mongo on Azure (#jareed), so I moved the DB to AWS - still the problem persisted.
So i started playing about with that options object again, reading up quite a lot on it. Came to this conclusion:
options: {
server:{
auto_reconnect: true,
poolSize: 10,
socketOptions:{
keepAlive: 1
}
},
db: {
numberOfRetries: 10,
retryMiliSeconds: 1000
}
}
That was a bit more educated that my original options object i state.
However - it's still no good.
Now, for some reason i had to get off that windows box (something to do with a module not compiling on it) - it was easier to move than spend another week trying to get it to work.
So i moved my app to nodejitsu. Low and behold my connection stayed alive! Woo!
So…. what does this mean… I have no idea! What i do know is is those options seem to work on Nodejitsu…. for me.
I believe IISNode uses some kind of "forever" script for keeping the app alive. Now to be fair the app doesn't crash for this to kick in, but i think there must be some kind of "app cycle" that is refreshed constantly - this is how it can do continuous deployment (ftp code up, no need to restart app) - maybe this is a factor; but i'm just guessing now.
Of course all this means now, is this isn't solved. It's still not solved. It's just solved for me in my setup.
A couple of recommendations for people still having this issue:
Make sure you are using the latest mongodb client for node.js. I noticed significant improvements in this area when migrating from v1.2.x to v1.3.10 (the latest as of today)
You can pass an options object to the MongoClient.connect. The following options worked for me when connecting from Azure to MongoLab:
options = {
db: {},
server: {
auto_reconnect: true,
socketOptions: {keepAlive: 1}
},
replSet: {},
mongos: {}
};
MongoClient.connect(dbUrl, options, function(err, dbConn) {
// your code
});
See this other answer in which I describe how to handle the 'close' event which seems to be more reliable. https://stackoverflow.com/a/20690008/446681
Enable the auto_reconnect Server option like this:
var db = mongoose.connect(mongoConnect, {server: {auto_reconnect: true}});
The connection you're opening here is actually a pool of 5 connections (by default) so you're right to just connect and leave it open. My guess is that you intermittently lose connectivity with mongolab and your connections die when that occurs. Hopefully, enabling auto_reconnect resolves that.
Increasing timeouts may help.
"socketTimeoutMS" : How long a send or receive on a socket can take
before timing out.
"wTimeoutMS" : It controls how many milliseconds the server waits for
the write concern to be satisfied.
"connectTimeoutMS" : How long a connection can take to be opened
before timing out in milliseconds.
$m = new MongoClient("mongodb://127.0.0.1:27017",
array("connect"=>TRUE, "connectTimeoutMS"=>10, "socketTimeoutMS"=>10,
"wTimeoutMS"=>10));
$db= $m->mydb;
$coll = $db->testData;
$coll->insert($paramArr);
I had a similar problem being disconnected from MongoDB periodically. Doing two things fixed it:
Make sure your computer never sleeps (that'll kill your network connection).
Bypass your router/firewall (or configure it properly, which I haven't figured out how to do yet).