NodeJS crash with multiple request - node.js

Question

My nodejs server crashes randomly in real time ( and always on Web Stress Tool with 10+ thread request). Below is the code that I believe to be the root cause.
main.js
--------
app = express():
---------
app.get('/image/*', actions.download);
actions.js
var request = require('request');
exports.download = function(req, res){
var url = <Amazon s3 URL>;
req.pipe(request(url)).pipe(res);
};
When server crashes, I am getting below error in nohup
stream.js:94
throw er; // Unhandled stream error in pipe.
^
Error: socket hang up
at createHangUpError (http.js:1476:15)
at Socket.socketOnEnd [as onend] (http.js:1572:23)
at Socket.g (events.js:180:16)
at Socket.emit (events.js:117:20)
at _stream_readable.js:943:16
at process._tickCallback (node.js:419:13)
Detailed log when I tried with sudo NODE_DEBUG=net node main.js and subjected to stress test with 10 threads
NET: 3017 Socket._read readStart
NET: 3017 afterWrite 0 { domain: null, bytes: 335, oncomplete: [Function: afterWrite] }
NET: 3017 afterWrite call cb
NET: 3017 onread ECANCELED 164640 4092 168732
NET: 2983 got data
NET: 2983 onSocketFinish
NET: 2983 oSF: not ended, call shutdown()
NET: 2983 destroy undefined
NET: 2983 destroy
NET: 2983 close
NET: 2983 close handle
Error: read ECONNRESET
at errnoException (net.js:904:11)
at TCP.onread (net.js:558:19)

Answer 1

This is cased by libuv in src/unix/stream.c. Here we have:
if (stream->shutdown_req) {
/* The UV_ECANCELED error code is a lie, the shutdown(2) syscall is a
* fait accompli at this point. Maybe we should revisit this in v0.11.
* A possible reason for leaving it unchanged is that it informs the
* callee that the handle has been destroyed.
*/
uv__req_unregister(stream->loop, stream->shutdown_req);
uv__set_artificial_error(stream->loop, UV_ECANCELED);
stream->shutdown_req->cb(stream->shutdown_req, -1);
stream->shutdown_req = NULL;
}
I've found the reason of this problem:
stream->shutdown_req was assigned by int uv_shutdown(. So someone called uv_shutdown. Who called uv_shutdown?
uv_shutdown is not a simple function. See here. The name of this function is StreamWrap::Shutdown.
StreamWrap::Shutdown is used in nodejs: SET_INSTANCE_METHOD("shutdown", StreamWrap::Shutdown, 0). shutdown method is a part of wrappers/pipe_wrap and wrappers/tcp_wrap
So someone called shutdown from nodejs/lib. It can be lib/net and lib/tls.
So shutdown is called from function onCryptoStreamFinish or function onSocketFinish.
So you need to find who sent shutdown request in your case. onread ECANCELED means that stream (for example socket1.pipe(socket2)) has been killed.
BTW I think that you can workaround your issue by using special technique to destroy piped sockets from lib/tls:
pair.encrypted.on('close', function() {
process.nextTick(function() {
// Encrypted should be unpiped from socket to prevent possible
// write after destroy.
pair.encrypted.unpipe(socket);
socket.destroy();
});
});