In our redis configuration we have set timeout: 7 seconds
In node_redis We handle redis connection ready and end event as
client.on("ready", function() {
logger.info("Connection Successfully Established to ", this.host, this.port);
}
client.on("end", function() {
logger.fatal("Connection Terminated to ", this.host, this.port);
}
Sample log
[2012-07-11 08:21:29.545] [FATAL] Production - Connection Terminated
on end to 'x.x.x.9' '6399'
[2012-07-11 08:21:29.803] [INFO] Production - Connection Successfully Established to 'x.x.x.9' '6399'
But in some cases (most probably redis is closing the connection without notifying the client) we see the command queue getting piled up and requests are taking too much time to get the response [till the time node-redis client able to sense the close event]. In all such cases command callback is returned with this error Redis connection gone from close event. even after so much waiting. It looks as if this is not an issue because of timeout since the usual end event wasn't triggered.
Issue seems to be similar to this - http://code.google.com/p/redis/issues/detail?id=368
Is this a known thing happening in redis?
Is there a way to specify that execution of a command [sending and receiving a reply back] should not exceed the threshold and reply with an error in that case, instead of making the client stall?
Or is there anyother way of triggering close event in such cases like socket_timeout?
Or should we check something from our redis side? We monitored our redis log at debug level and we found nothing useful related to this issue
When we run the node-redis on debug mode we are clearly able to see the client getting stalled with the requests getting piled up in the command queue. We logged the why and queue length inside flush_on_error function. We have kept offline_queuing disabled.
Sample Log
Redis connection is gone from close event.
offline queue 0
command queue 8
Response time of failed a request: 30388 ms [this varies as per the waiting in the command queue. First queued guy has the max response time and the ones following him lesser]
Usual Resonse time: 1 ms
PS: We have filed an issue in node_redis too
We had a bunch of connection trouble with Redis as well. It seems like it would close the connection without it telling the client. We noticed that it was possibly a timeout problem on the server. This is the solution that we use and we haven't had a problem since July.
var RETRY_EVERY = 1000 * 60 * 3;
var startTimer = function(){
console.log('Begin the hot tub!')
setInterval(function(){
try{
client.set('hot',new Date());
console.log(client.get('hot'))
}
catch(e){
console.log(e);
}
},RETRY_EVERY)
}();
Considering it's only one call every 3 minutes, it shouldn't be a problem for performance ;)
With regards to oconnecp's answer, can't you just do:
setInterval(client.ping(), 1000 * 60 * 30);
Related
I have a node.js, the monitors a queue on Firebase, to send GCM Push notifications. Works fine.
It also updates an "online" status on firebase via .onDisconnect(), so one can easily see if the Node.js server is online and running.
Problem: after some time it will show "disconnected" even when the listener is still connected and running fine.
const NODESERVERONLINE="NodeSeverStatus";
var ref = new Firebase(FBURL+FBKEY_GCM_QUEUE);
ref.child("NODESERVERONLINE").set("Online");
ref.child("NODESERVERONLINE").onDisconnect().set("Offline!");
ref.on("child_added",function(snapshot, prevChild){
If (snapshot.key()!=NODESERVERONLINE) DO_GCM_PUSH(snapshot.val());
}, function(errorObject){
console.log("Error reading Firebase: " + errorObject.code);
});
Initially, the listener is running and -NodeSeverStatus shows Online.
However "after some time" (several hours), the listener is still running fine, and the queue is being processed, but NodeServerStatus now shows Offline.
I could move the online/offline code inside the listener itself, but that would appear to just be an ugly hack, and would presumably still have the same issue if there were no new queue posts within the timeout period.
What is best practice here? Thankyou.
A quick guess is that your network connection gets interrupted briefly.
If you network connection flaps, the server will detect the disconnect and set Offline!.
The client will automatically reconnect, but you never set Online again.
So you'll want to listen for .info/connected and set Online there.
var ref = new Firebase("https://yours.firebaseio.com");
ref.child("NODESERVERONLINE").onDisconnect().set("Offline!");
ref.child(".info/connected").on("value", function(snapshot) {
if (snapshot.val() === true) {
ref.child("NODESERVERONLINE").set("Online");
}
});
See https://www.firebase.com/docs/web/guide/offline-capabilities.html
I can't figure out one problem I got.
I'm using the Net module on my Node.JS server which is used to listen to client connections.
The client do connect to the server correctly and the connection remains available to read/write data. So far, so good. But when the client unexpectedly disconnects (ed. when internet falls away at client side) I want to fire an event server side.
In socket.io it would be done with the 'disconnect' event, but this event doesn't seem to exist for the Net module. How is it possible to do?
I've searched on Google/StackOverflow and in the Net documentation (https://nodejs.org/api/net.html) but I couldn't find anything usefull. I'm sry if I did mis something.
Here is a code snippet I got:
var net = require('net');
var server = net.createServer(function(connection) {
console.log('client connected');
connection.wildcard = false;//Connection must be initialised with a configuration stored in the database
connection.bidirectional = true;//When piped this connection will be configured as bidirectional
connection.setKeepAlive(true, 500);
connection.setTimeout(3000);
connection.on('close', function (){
console.log('Socket is closed');
});
connection.on('error', function (err) {
console.log('An error happened in connection' + err.stack);
});
connection.on('end', function () {
console.log('Socket did disconnect');
});
connection.on('timeout', function () {
console.log('Socket did timeout');
connection.end();
});
connection.on('data', function (data) {
//Handling incoming data
});
});
serverUmrs.listen(40000, function () {
console.log('server is listening');
});
All the events(close, end, error, timeout) don't fire when I disconnect the client(by pulling out the UTP cable).
Thanks in advance!
EDIT:
I did add a timeout event in the code here above but the only thing that happens is that the socket does timeout after 3 seconds everytime the client does connect again. Isn't KeepAlive enough to make the socket not Idle? How is it possible to make the socket not idle without to much overhead. It may be possible that there are more than 10,000 connections at the same time which must remain alive as long as they are connected (ie respond to the keepalive message).
Update:
I think the KeepAlive is not related with the Idle state of socket, sort of.
Here is my test, I remove the following code in your example.
//connection.setKeepAlive(true, 500);
Then test this server with one client connect to it var nc localhost 40000. If there is no message sending to server after 3 seconds, the server logs as below
Socket did timeout
Socket did disconnect
Socket is closed
The timeout event is triggered without KeepAlive setting.
Do further investigation, refer to the Node.js code
function onread(nread, buffer) {
//...
self._unrefTimer();
We know timeout event is triggered by onread() operation of socket. Namely, if there is no read operation after 3 seconds, the timeout event will be emitted. To be more precisely, not only onread but also write successfully will call _unrefTimer().
In summary, when the write or read operation on the socket, it is NOT idle.
Actually, the close event is used to detect the client connection is alive or not, also mentioned in this SO question.
Emitted when the server closes. Note that if connections exist, this event is not emitted until all connections are ended.
However, in your case
disconnect the client(by pulling out the UTP cable).
The timeout event should be used to detective the connection inactivity. This is only to notify that the socket has been idle. The user must manually close the connection. Please refer to this question.
In TCP connection, end event fire when the client sends 'FIN' message to the server.
If the client side is not sending 'FIN' message that event is not firing.
For example, in your situation,
But when the client unexpectedly disconnects (ed. when internet falls away at client side) I want to fire an event server side.
There may not be a 'FIN' message because internet is gone.
So you should handle this situation in timeout without using keepAlive. If there is no data coming data, you should end or destroy the socket.
EDIT: I did add a timeout event in the code here above but the only
thing that happens is that the socket does timeout after 3 seconds
everytime the client does connect again. Isn't KeepAlive enough to
make the socket not Idle? How is it possible to make the socket not
idle without to much overhead. It may be possible that there are more
than 10,000 connections at the same time which must remain alive as
long as they are connected (ie respond to the keepalive message).
For your edit, your devices should send to the server some heartbeat message between a time period. So that, server understands that that device is alive and that timeout event will not fire because you get some data. If there is no heartbeat message such cases you cannot handle this problem.
I have a node.js script which allows a client to connect and receive some realtime data from an external script.
I have just upgraded node.js & socket.io to the current versions (from <0.9) and am trying to get to grips with what happens when a client quits, times out or disconnects from the server.
Here is my current node.js script;
var options = {
allowUpgrades: true,
pingTimeout: 50000,
pingInterval: 25000,
cookie: 'k1'
};
var io = require('socket.io')(8002, options);
cp = require('child_process');
var tail = cp.spawn('test-scripts/k1.rb');
//On connection do the code below//
io.on('connection', function(socket) {
console.log('************ new client connected ****************', io.engine.clientsCount);
//Read from mongodb//
var connection_string = '127.0.0.1:27017/k1-test';
var mongojs = require('mongojs');
var db = mongojs(connection_string, ['k1']);
var k1 = db.collection('k1');
db.k1.find({}, {'_id': 0, "data.time":0}).forEach(function(err, doc) {
if (err) throw err;
if (doc) { socket.emit('k1', doc); }
});
//Run Ruby script & Listen to STDOUT//
tail.stdout.on('data', function(chunk) {
var closer = chunk.toString()
var sampArray = closer.split('\n');
for (var i = 0; i < sampArray.length; i++) {
try {
var newObj = JSON.parse(sampArray[i]);
// DO SOCKET //
socket.emit('k1', newObj);
} catch (err) {}
}
});
socket.on('disconnect', function(){
console.log('****************** user disconnected *******************', socket.id, io.engine.clientsCount);
socket.disconnect();
});
});
In the old version of socket.io when a client exits I get the following logged in debug;
info - transport end (undefined)
debug - set close timeout for client Owb_B6I0ZEIXf6vOF_b-
debug - cleared close timeout for client Owb_B6I0ZEIXf6vOF_b-
debug - cleared heartbeat interval for client Owb_B6I0ZEIXf6vOF_b-
debug - discarding transport
then everything goes quite and all is well.
With the new (1.3.7) version of socket.io when a client exits I get the following logged in debug;
socket.io:client client close with reason transport close +2s
socket.io:socket closing socket - reason transport close +1ms
socket.io:client ignoring remove for -0BK2XTmK98svWTNAAAA +1ms
****************** user disconnected ******************* -0BK2XTmK98svWTNAAAA
note the line socket.io:client ignoring remove for -0BK2XTmK98svWTNAAAA
but after that and with no other clients connected to the server I'm still seeing it trying to write data to a client that already left. (in the example below this is what I get after I've had 2 clients connected, both of which have since disconnected.
socket.io:client ignoring packet write {"type":2,"data":["k1",{"item":"switch2","datapoint":{"type":"SWITCH","state":"0"}}],"nsp":"/"} +1ms
socket.io:client ignoring packet write {"type":2,"data":["k1",{"item":"switch2","datapoint":{"type":"SWITCH","state":"0"}}],"nsp":"/"} +3ms
I'm trying to stop this apparently new behaviour so that once a client has disconnected and the server is idle its not still trying to send data out.
I've been playing about with socket.disconnect and delete socket["id"] but I'm still left with the same thing.
I tried with io.close() which sort of worked - it booted any clients who where actually connected and made them re-connect but still left the server sitting there trying to send updates to the client that had left.
Am I missing something obvious, or has there been a change in the way this is done with the new version of socket.io? There is nothing in the migration doc about this. The only other result I found was this bug report from June 2014 which has been marked as closed. From my reading of it - it appears to be the same problem I'm having but with the current version.
Update: I've done some more testing and added io.engine.clientsCount to both instances of console.log to track what it's doing. It appears when I connect 1 client it gives me 1 (as expected) and when I close that client it changes to 0 (as expected) this leads me to believe that the client connection has been closed and engine.io know this. So why am I still seeing all the 'ignoring packet write' lines and more with every client who has disconnected.
Update 2: I've updated the code above to include the parser section and the DB section - this represents the full node script as there was a thought that I may need to clean up my own clients. I have tried adding the following code to the script in the hope it would but alas not :(
In the connection event I added clients[socket.id] = socket; and the disconnection event I added delete clients[socket.id]; but it didn't change anything (that I could see)
Update 3: Answer thanks to #robertklep It was an 'event handler leak' that I was actually looking for. Having found that I also found this post.
My guess is that the newer socket.io is just showing you (by way of debug messages) a situation that was already happening in the old socket.io, where it just wasn't being logged.
I think the main issue is this setup:
var tail = cp.spawn('test-scripts/k1.rb');
io.on('connection', function(socket) {
...
tail.stdout.on('data', function(chunk) { ... });
...
});
This adds a new handler for each incoming connection. However, these won't miraculously disappear once the socket is disconnected, so they keep on trying to push new data through the socket (whether it's disconnected or not). It's basically an event handler leak, as they aren't getting cleaned up.
To clean up the handlers, you need to keep a reference to the handler function and remove it as a listener in the disconnect event handler:
var handler = function(chunk) { ... }:
tail.stdout.on('data', handler)
socket.on('disconnect', function() {
tail.stdout.removeListener('data', handler);
});
There's also a (slight) chance that you will get ignored packet writes from your MongoDB code, if the socket is closed before the forEach() has finished, but that may be acceptable (since the amount of data is finite).
PS: eventually, you should consider moving the processing code (what handler is doing) to outside the socket code, as it's now being run for each connected socket. You can create a separate event emitter instance that will emit the processed data, and subscribe to that from each new socket connection (and unsubscribe again when they disconnect), so they only have to pass the processed data to the clients.
This is most probably due to your connection is established via polling transport, which is sooo painful for developer. The reason is that this transport uses timeout to determine if the client is here or not.
The behavior you see is due to the client has left but next polling session opening moment has not come yet, and due to it server still thinks that client "it out there".
I have tried to "fight" this problem in many ways (like adding a custom onbeforeunload event on client side to force disconnect) but they all just do not work in 100% cases when polling is used as transport.
In production, I have a game which uses connection-local variables to hold game state. However I notice that if I idle for a certain time on the connection, it disconnects and reconnects which loses the current state. During my tests on a local host, I never noticed this behavior. Is this the norm behavior for socket connections or is something else causing the connections to drop.
If it is a normal behavior how is this typically handled? Should connection values be stored globally so they can be restored should a user drop/reconnect?
Your problem is around socket timeouts. If there's no activity on a certain socket, socket.io will close it automatically.
An easy (and hackish) fix is to send a heartbeat to the connected client to create activity and stop the socket from timing out.
Server:
function sendHeartbeat(){
setTimeout(sendHeartbeat, 8000);
io.sockets.emit('ping', { beat : 1 });
}
io.sockets.on('connection', function (socket) {
socket.on('pong', function(data){
console.log("Pong received from client");
});
}
setTimeout(sendHeartbeat, 8000);
Client:
socket.on('ping', function(data){
socket.emit('pong', {beat: 1});
});
More Information:
You can get more information on configuring socket.io here.
EDIT: Mark commented that if the user does lose the connection (connection drops on his end because of internet troubles), you should be able to restore the user to his last state.
To do that, the best way would be to use a already widely used method for storing user data, cookies and sessions.
An extremely well done tutorial on how to do this located here. Although he uses express to set cookies, you can do this using anything (I do it using rails). Using this method, you can store the user data in a cookie and fetch it during the handshake. From there you can just access the data using socket.handshake.data.
What you need to do is create or identify the session per (re-) connection. You may reduce the number of reconnections per Moox's answer above but it is still not failsafe - e.g. a user loses wifi connection for a bit, etc. In other words - maintain user metadata per session and not per socket, and expect occasional disconnects and reconnects.
I'm writing a node.js application that needs to talk to a server. It establishes an http connection with the following code:
var client = http.createClient(u.port, u.hostname, u.secure);
client.on("error", function(exception) {
logger.error("error from client");
});
var request = client.request(method, u.path, headers);
I don't see any option in the node.js documentation for setting a timeout on the connection, and it seems to be set to 20 seconds by default. The problem I'm having is that I have users in China on what appears to be a slow or flaky network, who sometimes hit the timeout connecting to our datacenter in the US. I'd like to increase the timeout to 1 minute, to see if that fixes it for them.
Is there a way to do that in node.js?
Try
request.socket.setTimeout(60000); // 60 sec
I think you can do something like:
request.connection.setTimeout(60000)
request.connection returns the net.Stream object associated with the connection.
and net.Stream has a setTimeout method.
There is no capability in Node to increase connect timeout. Since usually connect timeout (i.e. connection establishing timeout) is OS-wide setting for all applications (e.g., 21 seconds in Windows, from 20 to 120 seconds in Linux). See also Timouts in Request package.
In contrast, Node allows to set decreased timeout and abort connecting even in case when the connection is not yet established.
The further timeouts (in case of connection has been established) can be controlled according to the documentation (see request.setTimeout, socket.setTimeout).
You have to wait for the client socket connection to be established first, before setting the timeout. To do this, add a callback for the 'socket' event:
req.on('socket', function (socket) {
myTimeout = 500; // millis
socket.setTimeout(myTimeout);
socket.on('timeout', function() {
console.log("Timeout, aborting request")
req.abort();
});
}).on('error', function(e) {
console.log("Got error: " + e.message);
// error callback will receive a "socket hang up" on timeout
});
See this answer.