If I have a (synchronous) process writing to a substream on a socket which disconnects during code execution, what's the best way to keep it from throwing the exception?
(I'm listening to the socket's close/end/etc events to remove the block from flow, but these event handlers wouldn't fire until after the code has finished)
Should I do this:
if (clientStream.stream) {
clientStream.write(bufferData);
} else {
console.log('*** Client FAILURE *****')
}
or use a try/catch?
try {
clientStream.write(bufferData);
} catch (err) {
console.log('*** Client FAILURE *****')
}
I know try/catches are expensive, but I haven't found any info on checking the clientStream.stream object to verify it exists. Maybe it's been deprecated like stream.readyState?
With a single server setup, I receive events from the driver.
mongoose.connect('mongodb://localhost/mydb');
mongoose.connection.on('disconnected', function() {...});
mongoose.connection.on('error', function(err) {...});
When using a replica set (mongoose.connect('mongodb://localhost:27017/mydb,mongodb://localhost:27018/mydb');), shutting down all connected set members doesn't trigger those same events.
I'm not very familiar with the internals of the native driver and I'm wondering if this is a bug or if I need to manually detect this condition.
I'm using Mongoose 3.6.17 (mongodb driver 1.3.18)
Sans mongoose, I tried this with the same results (no events from a replica set).
require('mongodb').MongoClient.connect("mongodb://localhost:27017,localhost:27018/mydb", function(err, db) {
if (db) {
db.on('disconnected', function() {
console.log('disconnected');
}).on('error', function(err) {
console.log('error');
});
}
});
I've been having similar problems with Mongoose, asked on SO also. More recently, I've found this issue on Mongoose GitHub repository which led to this issue on the driver repository.
The Mongo driver wasn't emitting any of these events more than once, and today this has been fixed for single connections on v1.3.19.
It seems that it's a "won't fix" for now.
I ended up doing the following:
I set auto_reconnect=true
Until the application has connected to the database for the first time, i disconnect and reconnect. if i don't disconnect and reconnect, any queued queries won't run. after a connection has been established at least once, those queued queries do complete and then...
for single connections:
1. forked mongoose (to use mongodb to 1.3.19) so errors get triggered more than once.
2. catch the connection error and make the app aware of the disconnection, retrying until i give up and panic or the app is reconnected. how that's done is by pinging the server every x milliseconds with a command that will not queue:
var autoReconnect = mongoose.connection.db.serverConfig.isAutoReconnect;
mongoose.connection.db.serverConfig.isAutoReconnect = function(){return false;};
mongoose.connection.db.executeDbCommand({ping:1}, {failFast: true}, function(err) {
if (!err) {
// we've reconnected.
}
});
mongoose.connection.db.serverConfig.isAutoReconnect = autoReconnect;
for a replica set, i ended up polling the mongoose connection with the above ping every x milliseconds until i detect an error, in which case i set my local app state to disconnected and enter the reconnect poll loop above (2.).
here's a gist with the relevant bits. https://gist.github.com/jsas/6299412
This is a nasty inconsistency/oversight in mongoose.
Especially when developing a microservice where you're using a single server setup for development and replica set in production.
This is the way I ended up accurately tracking the status of my mongoose connection.
let alive = false;
function updateAlive() {
return mongoose.connection
&& mongoose.connection.readyState === mongoose.STATES.connected
// This is necessary because mongoose treats a dead replica set as still "connected".
&& mongoose.connection.db.topology.connections().length > 0;
}
mongoose.connection.on('connected', () => {
updateAlive();
// I think '.topology' is available on even single server connections.
// The events just won't be emitted.
mongoose.connection.db.topology.on('joined', () => updateAlive());
mongoose.connection.db.topology.on('left', () => updateAlive());
});
mongoose.connection.on('disconnected', () => {
updateAlive();
});
I know socket.io has a built in feature for reconnecting and everything, however I don't think that it is working - as I have seen from others it's also not working for them either.
If a user puts their computer to sleep, it disconnects them, and then when they open it back up they are no longer connected so they don't any of the notifications or anything until they refresh the page. Perhaps it's just something that I'm not doing correctly?
var io = require('socket.io').listen(8080);
var users = {};
////////////////USER CONNECTED/////
console.log("Sever is now running");
io.sockets.on('connection', function (socket) {
//Tell the client that they are connected
socket.emit('connected');
//Once the users session is recieved
socket.on('session', function (session) {
//Add users to users variable
users[socket.id] = {userID:session, socketID:socket};
//When user disconnects
socket.on('disconnect', function () {
//socket.socket.connect();
var count= 0;
for(var key in users){
if(users[key].userID==session)++count;
if(count== 2) break;
}
if(count== 1){
socket.broadcast.emit('disconnect', { data : session});
}
//Remove users session id from users variable
delete users[socket.id];
});
socket.on('error', function (err) {
//socket.socket.connect();
});
socket.emit("connection") needs to be called when the user reconnects, or at least the events that happen in that event need to be called.
Also socket.socket.connect(); doesn't work, it returns with an error and it shuts the socket server down with an error of "connect doesn't exist".
The problem is related to io.connect params.
Look at this client code (it will try to reconnect forever, with max delay between attempts 3sec):
ioParams = {'reconnection limit': 3000, 'max reconnection attempts': Number.MAX_VALUE, 'connect timeout':7000}
socketAddress = null
socket = io.connect(socketAddress, ioParams)
There are two important parameters out there, related to your problem:
reconnection limit - limit the upper time of delay between reconnect attemts. Normally it's getting bigger and bigger in time of server outage
max reconnection attempts - how many times you want to try. Default is 10. In most cases this is the problem why the client stops trying.
I've written up a minimal example of this. The code is posted here: https://gist.github.com/1524725
I start my server, start my client, verify that the connection between the two is successful, and finally kill the server with CTRL+C. When the server dies, the client immediately runs to completion and closes without printing the message in either on_client_close or on_client_disconnect. There is no perceptible delay.
From the reading I've done, because the client process is terminating normally there isn't any chance that the STDOUT buffer isn't being flushed.
It may also be worth noting that when I kill the client instead of the server, the server responds as expected, firing the on_ws_disconnect function and removing the client connection from its list of active clients.
32-bit Ubuntu 11.10
Socket.io v0.8.7
Socket.io-client v0.8.7
NodeJS v0.6.0
Thanks!
--- EDIT ---
Please note that both the client and the server are Node.js processes rather than the conventional web browser client and node.js server.
NEW ANSWER
Definitely a bug in io-client. :(
I was able to fix this by modifying socket.io-client/libs/socket.js. Around line 433, I simply moved the this.publish('disconnect', reason); above if (wasConnected) {.
Socket.prototype.onDisconnect = function (reason) {
var wasConnected = this.connected;
this.publish('disconnect', reason);
this.connected = false;
this.connecting = false;
this.open = false;
if (wasConnected) {
this.transport.close();
this.transport.clearTimeouts();
After pressing ctrl+c, the disconnect message fires in roughly ten seconds.
OLD DISCUSSION
To notify client of shutdown events, you would add something like this to demo_server.js:
var logger = io.log;
process.on('uncaughtException', function (err) {
if( io && io.socket ) {
io.socket.broadcast.send({type: 'error', msg: err.toString(), stack: err.stack});
}
logger.error(err);
logger.error(err.stack);
//todo should we have some default resetting (restart server?)
app.close();
process.exit(-1);
});
process.on('SIGHUP', function () {
logger.error('Got SIGHUP signal.');
if( io && io.socket ) {
io.socket.broadcast.send({type: 'error', msg: 'server disconnected with SIGHUP'});
}
//todo what happens on a sighup??
//todo if you're using upstart, just call restart node demo_server.js
});
process.on('SIGTERM', function() {
logger.error('Shutting down.');
if( io && io.socket ) {
io.socket.broadcast.send({type: 'error', msg: 'server disconnected with SIGTERM'});
}
app.close();
process.exit(-1);
});
Of course, what you send in the broadcast.send(...) (or even which command you use there) depends on your preference and client structure.
For the client side, you can tell if the server connection is lost using on('disconnect', ...), which you have in your example:
client.on('disconnect', function(data) {
alert('disconnected from server; reconnecting...');
// and so on...
});
I am working on a websocket oriented node.js server using Socket.IO. I noticed a bug where certain browsers aren't following the correct connect procedure to the server, and the code isn't written to gracefully handle it, and in short, it calls a method to an object that was never set up, thus killing the server due to an error.
My concern isn't with the bug in particular, but the fact that when such errors occur, the entire server goes down. Is there anything I can do on a global level in node to make it so if an error occurs it will simply log a message, perhaps kill the event, but the server process will keep on running?
I don't want other users' connections to go down due to one clever user exploiting an uncaught error in a large included codebase.
You can attach a listener to the uncaughtException event of the process object.
Code taken from the actual Node.js API reference (it's the second item under "process"):
process.on('uncaughtException', function (err) {
console.log('Caught exception: ', err);
});
setTimeout(function () {
console.log('This will still run.');
}, 500);
// Intentionally cause an exception, but don't catch it.
nonexistentFunc();
console.log('This will not run.');
All you've got to do now is to log it or do something with it, in case you know under what circumstances the bug occurs, you should file a bug over at Socket.IO's GitHub page:
https://github.com/LearnBoost/Socket.IO-node/issues
Using uncaughtException is a very bad idea.
The best alternative is to use domains in Node.js 0.8. If you're on an earlier version of Node.js rather use forever to restart your processes or even better use node cluster to spawn multiple worker processes and restart a worker on the event of an uncaughtException.
From: http://nodejs.org/api/process.html#process_event_uncaughtexception
Warning: Using 'uncaughtException' correctly
Note that 'uncaughtException' is a crude mechanism for exception handling intended to be used only as a last resort. The event should not be used as an equivalent to On Error Resume Next. Unhandled exceptions inherently mean that an application is in an undefined state. Attempting to resume application code without properly recovering from the exception can cause additional unforeseen and unpredictable issues.
Exceptions thrown from within the event handler will not be caught. Instead the process will exit with a non-zero exit code and the stack trace will be printed. This is to avoid infinite recursion.
Attempting to resume normally after an uncaught exception can be similar to pulling out of the power cord when upgrading a computer -- nine out of ten times nothing happens - but the 10th time, the system becomes corrupted.
The correct use of 'uncaughtException' is to perform synchronous cleanup of allocated resources (e.g. file descriptors, handles, etc) before shutting down the process. It is not safe to resume normal operation after 'uncaughtException'.
To restart a crashed application in a more reliable way, whether uncaughtException is emitted or not, an external monitor should be employed in a separate process to detect application failures and recover or restart as needed.
I just did a bunch of research on this (see here, here, here, and here) and the answer to your question is that Node will not allow you to write one error handler that will catch every error scenario that could possibly occur in your system.
Some frameworks like express will allow you to catch certain types of errors (when an async method returns an error object), but there are other conditions that you cannot catch with a global error handler. This is a limitation (in my opinion) of Node and possibly inherent to async programming in general.
For example, say you have the following express handler:
app.get("/test", function(req, res, next) {
require("fs").readFile("/some/file", function(err, data) {
if(err)
next(err);
else
res.send("yay");
});
});
Let's say that the file "some/file" does not actually exist. In this case fs.readFile will return an error as the first argument to the callback method. If you check for that and do next(err) when it happens, the default express error handler will take over and do whatever you make it do (e.g. return a 500 to the user). That's a graceful way to handle an error. Of course, if you forget to call next(err), it doesn't work.
So that's the error condition that a global handler can deal with, however consider another case:
app.get("/test", function(req, res, next) {
require("fs").readFile("/some/file", function(err, data) {
if(err)
next(err);
else {
nullObject.someMethod(); //throws a null reference exception
res.send("yay");
}
});
});
In this case, there is a bug if your code that results in you calling a method on a null object. Here an exception will be thrown, it will not be caught by the global error handler, and your node app will terminate. All clients currently executing requests on that service will get suddenly disconnected with no explanation as to why. Ungraceful.
There is currently no global error handler functionality in Node to handle this case. You cannot put a giant try/catch around all your express handlers because by the time your asyn callback executes, those try/catch blocks are no longer in scope. That's just the nature of async code, it breaks the try/catch error handling paradigm.
AFAIK, your only recourse here is to put try/catch blocks around the synchronous parts of your code inside each one of your async callbacks, something like this:
app.get("/test", function(req, res, next) {
require("fs").readFile("/some/file", function(err, data) {
if(err) {
next(err);
}
else {
try {
nullObject.someMethod(); //throws a null reference exception
res.send("yay");
}
catch(e) {
res.send(500);
}
}
});
});
That's going to make for some nasty code, especially once you start getting into nested async calls.
Some people think that what Node does in these cases (that is, die) is the proper thing to do because your system is in an inconsistent state and you have no other option. I disagree with that reasoning but I won't get into a philosophical debate about it. The point is that with Node, your options are lots of little try/catch blocks or hope that your test coverage is good enough so that this doesn't happen. You can put something like upstart or supervisor in place to restart your app when it goes down but that's simply mitigation of the problem, not a solution.
Node.js has a currently unstable feature called domains that appears to address this issue, though I don't know much about it.
I've just put together a class which listens for unhandled exceptions, and when it see's one it:
prints the stack trace to the console
logs it in it's own logfile
emails you the stack trace
restarts the server (or kills it, up to you)
It will require a little tweaking for your application as I haven't made it generic as yet, but it's only a few lines and it might be what you're looking for!
Check it out!
Note: this is over 4 years old at this point, unfinished, and there may now be a better way - I don't know!)
process.on
(
'uncaughtException',
function (err)
{
var stack = err.stack;
var timeout = 1;
// print note to logger
logger.log("SERVER CRASHED!");
// logger.printLastLogs();
logger.log(err, stack);
// save log to timestamped logfile
// var filename = "crash_" + _2.formatDate(new Date()) + ".log";
// logger.log("LOGGING ERROR TO "+filename);
// var fs = require('fs');
// fs.writeFile('logs/'+filename, log);
// email log to developer
if(helper.Config.get('email_on_error') == 'true')
{
logger.log("EMAILING ERROR");
require('./Mailer'); // this is a simple wrapper around nodemailer http://documentup.com/andris9/nodemailer/
helper.Mailer.sendMail("GAMEHUB NODE SERVER CRASHED", stack);
timeout = 10;
}
// Send signal to clients
// logger.log("EMITTING SERVER DOWN CODE");
// helper.IO.emit(SIGNALS.SERVER.DOWN, "The server has crashed unexpectedly. Restarting in 10s..");
// If we exit straight away, the write log and send email operations wont have time to run
setTimeout
(
function()
{
logger.log("KILLING PROCESS");
process.exit();
},
// timeout * 1000
timeout * 100000 // extra time. pm2 auto-restarts on crash...
);
}
);
Had a similar problem. Ivo's answer is good. But how can you catch an error in a loop and continue?
var folder='/anyFolder';
fs.readdir(folder, function(err,files){
for(var i=0; i<files.length; i++){
var stats = fs.statSync(folder+'/'+files[i]);
}
});
Here, fs.statSynch throws an error (against a hidden file in Windows that barfs I don't know why). The error can be caught by the process.on(...) trick, but the loop stops.
I tried adding a handler directly:
var stats = fs.statSync(folder+'/'+files[i]).on('error',function(err){console.log(err);});
This did not work either.
Adding a try/catch around the questionable fs.statSynch() was the best solution for me:
var stats;
try{
stats = fs.statSync(path);
}catch(err){console.log(err);}
This then led to the code fix (making a clean path var from folder and file).
I found PM2 as the best solution for handling node servers, single and multiple instances
One way of doing this would be spinning the child process and communicate with the parent process via 'message' event.
In the child process where the error occurs, catch that with 'uncaughtException' to avoid crashing the application. Mind that Exceptions thrown from within the event handler will not be caught. Once the error is caught safely, send a message like: {finish: false}.
Parent Process would listen to the message event and send the message again to the child process to re-run the function.
Child Process:
// In child.js
// function causing an exception
const errorComputation = function() {
for (let i = 0; i < 50; i ++) {
console.log('i is.......', i);
if (i === 25) {
throw new Error('i = 25');
}
}
process.send({finish: true});
}
// Instead the process will exit with a non-zero exit code and the stack trace will be printed. This is to avoid infinite recursion.
process.on('uncaughtException', err => {
console.log('uncaught exception..',err.message);
process.send({finish: false});
});
// listen to the parent process and run the errorComputation again
process.on('message', () => {
console.log('starting process ...');
errorComputation();
})
Parent Process:
// In parent.js
const { fork } = require('child_process');
const compute = fork('child.js');
// listen onto the child process
compute.on('message', (data) => {
if (!data.finish) {
compute.send('start');
} else {
console.log('Child process finish successfully!')
}
});
// send initial message to start the child process.
compute.send('start');