How do you kill a redis client when there is no connection? - node.js

I have a valid server configuration in which redis can't be accessed, but the server can function correctly (I simply strip away features when redis can't be found).
However, I can't manage the connection errors well. I'd like to know when a connection error fails and shutdown the client in that case.
I've found that the connection retry will never stop. And quit() is actually swallowed - "Queueing quit for next server connection." - when called.
Is there a way to kill the client in the case where no connection can be established?
var redis = require("redis"),
client = redis.createClient();
client.on("error", function(err) {
logme.error("Bonk. The worker framework cannot connect to redis, which might be ok on a dev server!");
logme.error("Resque error : "+err);
client.quit();
});
client.on("idle", function(err) {
logme.error("Redis queue is idle. Shutting down...");
});
client.on("end", function(err) {
logme.error("Redis is shutting down. This might be ok if you chose not to run it in your dev environment");
});
client.on("ready", function(err) {
logme.info("Redis up! Now connecting the worker queue client...");
});
ERROR - Resque error : Error: Redis connection to 127.0.0.1:6379 failed - connect ECONNREFUSED
ERROR - Redis is shutting down. This might be ok if you chose not to run it in your dev environment
ERROR - Resque error : Error: Redis connection to 127.0.0.1:6379 failed - connect ECONNREFUSED
ERROR - Resque error : Error: Redis connection to 127.0.0.1:6379 failed - connect ECONNREFUSED
ERROR - Resque error : Error: Redis connection to 127.0.0.1:6379 failed - connect ECONNREFUSED
ERROR - Resque error : Error: Redis connection to 127.0.0.1:6379 failed - connect ECONNREFUSED
One thing that is interesting is the fact that the 'end' event gets emitted. Why?

For v3.1.2 of the library
The right way to have control on the client's reconnect behaviour is to use a retry_strategy.
Upon disconnection the redisClient will try to reconnect as per the default behaviour. The default behaviour can be overridden by providing a retry_strategy while creating the client.
Example usage of some fine grained control from the documentation.
var client = redis.createClient({
retry_strategy: function (options) {
if (options.error && options.error.code === 'ECONNREFUSED') {
// End reconnecting on a specific error and flush all commands with
// a individual error
return new Error('The server refused the connection');
}
if (options.total_retry_time > 1000 * 60 * 60) {
// End reconnecting after a specific timeout and flush all commands
// with a individual error
return new Error('Retry time exhausted');
}
if (options.attempt > 10) {
// End reconnecting with built in error
return undefined;
}
// reconnect after
return Math.min(options.attempt * 100, 3000);
}
});
Ref: https://www.npmjs.com/package/redis/v/3.1.2
For the purpose of killing the client when the connection is lost, we could use the following retry_strategy.
var client = redis.createClient({
retry_strategy: function (options) {
return undefined;
}
});
Update June 2022 (Redis v4.1.0)
The original answer was for an earlier version of Redis client. Since v4 things have changed in the client configuration. Specifically, the retry_strategy is now called reconnectStrategy and is nested under the socket configuration option for createClient.

You might want to just forcibly end the connection to redis on error with client.end() rather than using client.quit() which waits for the completion of all outstanding requests and then sends the QUIT command which as you know requires a working connection with redis to complete.

Related

Handling Mongoose Disconnect

I am using mongoose 4.13.7
I want to connect to mongo but if an error occurs I want to reconnect.
But after 5 reconnects, if an error occurs, the process should exit.
This is the code:
var count = 0;
handleDisconnect();
function handleDisconnect(){
count++;
console.log('Trying to connect to mongo. Attempt : ' + count);
mongoose.connect(config.mongo.uri,{useMongoClient:true});
mongoose.connection.on('error',(error)=>{
if (count >= 5){
console.log('Mongo ERROR');
console.error(error);
process.exit(1);
}
else{
setTimeout(handleDisconnect,1000);
}
});
mongoose.connection.on('open',()=>{
console.log('Connected to mongo at ' + Date.now());
});
}
I have posted the output of the code as well. I don't understand how the attempt count is exceeding 5? There is also a memory leak warning and node:6804 error message. What am I doing wrong?
Output of the code
You attach an event listener for the error event each time the event is emitted. This causes the memory leak warning and also makes the callback function run multiple times per each event ocurrance. You should not add the event handler inside the handleDisconnect() funciton.
Example:
function handleDisconnect(mongoError) {
// check error reason, increment counters, check if errors limit reached
}
mongoose.connection.on('error', handleDisconnect);
You need to check disconnected event when Mongoose lost connection to the MongoDB server. This event may be due to your code explicitly closing the connection, the database server crashing, or network connectivity issues.
mongoose.connection.on('disconnected', () => console.log('Server disconnected from mongoDB'));
error Event: Emitted if an error occurs on a connection, like a parseError due to malformed data or a payload larger than 16MB.
https://mongoosejs.com/docs/connections.html#connection-events

Error: Redis connection to localhost:6379 failed - getaddrinfo EMFILE localhost:6379

I am getting the below error.
Error: Redis connection to localhost:6379 failed - getaddrinfo EMFILE localhost:6379
at Object.exports._errnoException (util.js:870:11)
at errnoException (dns.js:32:15)
at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:78:26)
Node.js with MySQL db and redis concept is used.
There are too many requests for fetching the data from MySQL so data is cached for 2 minutes by syncing with db. So when new requests arrives it checks in redis if found its serves from redis else data is retrieved from MySQL and cached in redis and sent as response. This keeps happening.
After some time probably 1 hour or 2 hours the server crashes resulting in above error.
But as of now pm2 is used which restarts the server.
But need to know the reason for it.
Redis installation followed the instructions from here.
https://www.digitalocean.com/community/tutorials/how-to-install-and-use-redis
Please let me know how to solve the issue...
Redis Connection File Code
var Promise = require('bluebird');
var redisClient; // Global (Avoids Duplicate Connections)
module.exports =
{
OpenRedisConnection : function()
{
if (redisClient == null)
{
redisClient = require("redis").createClient(6379, 'localhost');
redisClient.selected_db = 1;
}
},
GetRedisMultiConnection: function ()
{
return require("redis").createClient(6379, 'localhost').multi();
},
IsRedisConnectionOpened : function()
{
if (redisClient && redisClient.connected == true)
{
return true;
}
else
{
if(!redisClient)
redisClient.end(); // End and open once more
module.exports.OpenRedisConnection();
return true;
}
}
};
What I usually do with code like this is write a very thin module that loads in the correct Redis driver and returns a valid handle with a minimum of fuss:
var Redis = require("redis");
module.exports = {
open: function() {
var client = Redis.createClient(6379, 'localhost');
client.selected_db = 1;
return client;
},
close: function(client) {
client.quit();
}
};
Any code in your Node application that needs a Redis handle acquires one on-demand and it's also understood that code must close it no matter what happens. If you're not catching errors, or if you're catching errors and skipping the close you'll "leak" open Redis handles and your app will eventually crash.
So, for example:
var Redis = require('./redis'); // Path to module defined above
function doStuff() {
var redis = Redis.open();
thing.action().then(function() {
redis.ping();
}).finally(function() {
// This code runs no matter what even if there's an exception or error
Redis.close(redis);
});
}
Due to the concurrent nature of Node code having a single Redis handle that's shared by many different parts of code will be trouble and I'd strongly advise against this approach.
To expand on this template you'd have a JSON configuration file that can override which port and server to connect to. That's really easy to require and use instead of the defaults here. It also means that you don't have to hack around with any actual code when you deploy your application to another system.
You can also expand on the wrapper module to keep active connections in a small pool to avoid closing and then immediately opening a new one. With a little bit of attention you can even check that these handles are in a sane state, such as not stuck in the middle of a MULTI transaction, before handing them out, by doing a PING and testing for an immediate response. This weeds out stale/dead connections as well.

Node.js connectListener still called on socket error

I'm having a weird issue with a TCP client - I use socket.connect() to connect to the server instance. However, since the server is not running, I receive an error of ECONNREFUSED (so far so good).
I handle it using on('error') and set a timeout to try and reconnect in 10 seconds. This should continue to fail as long as the server is down. which is the case.
However, as soon as the server is running, it looks like all of the previous sockets are still active, so now I have several client sockets connected to the server.
I tried to call the destroy at the beginning of the on('error') handler function.
Any ideas how to deal with that?
Thanks!
EDIT: Code snippet:
var mySocket;
var self = this;
...
var onError = function (error) {
mySocket.destroy(); // this does not change anything...
console.log(error);
// Wait 10 seconds and try to reconnect
setTimeout(function () {
console.log("reconnecting...");
self.once('InitDone', function () {
// do something
console.log("init is done")
});
self.init();
}, 10000);
}
Inside init function:
...
console.log("trying to connect");
mySocket = tls.connect(options, function () {
console.log("connected!");
self.emit('InitDone');
});
mySocket.setEncoding('utf8');
mySocket.on('error', onError);
...
The result of this is something like the following:
trying to connect
ECONNREFUSED
reconnecting...
trying to connect
ECONNREFUSED
reconnecting...
trying to connect
ECONNREFUSED
reconnecting...
--> Starting the server here
trying to connect
connected
init is done
connected
init is done
connected
init is done
connected
init is done
However I would expect only one connection since the previous sockets failed to connect. Hope this clarifies the question.
Thanks!

Should MongooseJS be emitting events on replica set disconnection?

With a single server setup, I receive events from the driver.
mongoose.connect('mongodb://localhost/mydb');
mongoose.connection.on('disconnected', function() {...});
mongoose.connection.on('error', function(err) {...});
When using a replica set (mongoose.connect('mongodb://localhost:27017/mydb,mongodb://localhost:27018/mydb');), shutting down all connected set members doesn't trigger those same events.
I'm not very familiar with the internals of the native driver and I'm wondering if this is a bug or if I need to manually detect this condition.
I'm using Mongoose 3.6.17 (mongodb driver 1.3.18)
Sans mongoose, I tried this with the same results (no events from a replica set).
require('mongodb').MongoClient.connect("mongodb://localhost:27017,localhost:27018/mydb", function(err, db) {
if (db) {
db.on('disconnected', function() {
console.log('disconnected');
}).on('error', function(err) {
console.log('error');
});
}
});
I've been having similar problems with Mongoose, asked on SO also. More recently, I've found this issue on Mongoose GitHub repository which led to this issue on the driver repository.
The Mongo driver wasn't emitting any of these events more than once, and today this has been fixed for single connections on v1.3.19.
It seems that it's a "won't fix" for now.
I ended up doing the following:
I set auto_reconnect=true
Until the application has connected to the database for the first time, i disconnect and reconnect. if i don't disconnect and reconnect, any queued queries won't run. after a connection has been established at least once, those queued queries do complete and then...
for single connections:
1. forked mongoose (to use mongodb to 1.3.19) so errors get triggered more than once.
2. catch the connection error and make the app aware of the disconnection, retrying until i give up and panic or the app is reconnected. how that's done is by pinging the server every x milliseconds with a command that will not queue:
var autoReconnect = mongoose.connection.db.serverConfig.isAutoReconnect;
mongoose.connection.db.serverConfig.isAutoReconnect = function(){return false;};
mongoose.connection.db.executeDbCommand({ping:1}, {failFast: true}, function(err) {
if (!err) {
// we've reconnected.
}
});
mongoose.connection.db.serverConfig.isAutoReconnect = autoReconnect;
for a replica set, i ended up polling the mongoose connection with the above ping every x milliseconds until i detect an error, in which case i set my local app state to disconnected and enter the reconnect poll loop above (2.).
here's a gist with the relevant bits. https://gist.github.com/jsas/6299412
This is a nasty inconsistency/oversight in mongoose.
Especially when developing a microservice where you're using a single server setup for development and replica set in production.
This is the way I ended up accurately tracking the status of my mongoose connection.
let alive = false;
function updateAlive() {
return mongoose.connection
&& mongoose.connection.readyState === mongoose.STATES.connected
// This is necessary because mongoose treats a dead replica set as still "connected".
&& mongoose.connection.db.topology.connections().length > 0;
}
mongoose.connection.on('connected', () => {
updateAlive();
// I think '.topology' is available on even single server connections.
// The events just won't be emitted.
mongoose.connection.db.topology.on('joined', () => updateAlive());
mongoose.connection.db.topology.on('left', () => updateAlive());
});
mongoose.connection.on('disconnected', () => {
updateAlive();
});

Node socket.io server crash when starts

I have a socket.io server and a client runing correctly. Each time that the server is down, the client try to connect again each 5 seconds. When the server is up again they connect without problems.
But the problem comes when I wait long time before up the server again, when the server is up, it crashes showing :
info - socket.io started
debug - client authorized
info - handshake authorized DqN4t2YVP7NiqQi8zer9
debug - setting request GET /socket.io/1/websocket/DqN4t2YVP7NiqQi8zer9
debug - set heartbeat interval for client DqN4t2YVP7NiqQi8zer9
debug - client authorized for
debug - websocket writing 1::
buffer.js:287
pool = new SlowBuffer(Buffer.poolSize);
^
RangeError: Maximum call stack size exceeded
Client reconnection (Executed each 5 seconds while is not connected):
function socket_connect() {
if (!socket) {
socket = io.connect('http://192.168.1.25:8088', { 'reconnect':false, 'connect timeout': 5000 });
} else {
socket.socket.connect();
}
socket.on("connect", function () {
clearInterval(connect_interval);
connect_interval = 0;
socket.emit('player', { refresh_data:true });
});
}
On server side, only with the socket instance, it crashes:
var io = require('socket.io').listen(8088);
I think that the problem is:
When the server goes up, it recive all the connections emited by the client each 5 seconds, (15 hours disconnected * 60 m * 60 s / 5 seconds reconnection) and it crashes.
What can i do to close the connections that the server try to do?
PS:If i reload the client, and after up the server, it works
The main idea for socket.io.js is to reuse an existing connection.
You should only connect it once and then exchange messages by using socket.emit()
I am not sure why you are creating a new connection between your client and server for every 5 seconds. There is a limit on the number of connections the server can create, but that should be more than enough. If you put it in a loop then eventually the server will run out of sockets.
io.connect has to be executed once on the client, then may be you can socket.emit() every 5 seconds. Remove the { 'reconnect':false, 'connect timeout': 5000 } and you will be fine.
I founded the problem...
Each time that the function socket_connect() is called, a "socket.on("connect" ..." function is created. So when the server turns up, a new connection is created, but the event "socket.on("connect" is fired multiple times...
The solution was:
function socket_connect() {
if (!socket) {
socket = io.connect('http://192.168.1.25:8088', { 'reconnect':false, 'connect timeout': 5000 });
} else {
socket.socket.connect();
}
}
socket.on("connect", function () {
clearInterval(connect_interval);
connect_interval = 0;
socket.emit('player', { refresh_data:true });
});

Resources