I'm running an Express.js application using Socket.io for a chat webapp
and I get the following error randomly around 5 times during 24h.
The node process is wrapped in forever and it restarts itself immediately.
The problem is that restarting Express kicks my users out of their rooms
and nobody wants that.
The web server is proxied by HAProxy. There are no socket stability issues,
just using websockets and flashsockets transports.
I cannot reproduce this on purpose.
This is the error with Node v0.10.11:
events.js:72
throw er; // Unhandled 'error' event
^
Error: read ECONNRESET //alternatively it s a 'write'
at errnoException (net.js:900:11)
at TCP.onread (net.js:555:19)
error: Forever detected script exited with code: 8
error: Forever restarting script for 2 time
EDIT (2013-07-22)
Added both socket.io client error handler and the uncaught exception handler.
Seems that this one catches the error:
process.on('uncaughtException', function (err) {
console.error(err.stack);
console.log("Node NOT Exiting...");
});
So I suspect it's not a Socket.io issue but an HTTP request to another server
that I do or a MySQL/Redis connection. The problem is that the error stack
doesn't help me identify my code issue. Here is the log output:
Error: read ECONNRESET
at errnoException (net.js:900:11)
at TCP.onread (net.js:555:19)
How do I know what causes this? How do I get more out of the error?
Ok, not very verbose but here's the stacktrace with Longjohn:
Exception caught: Error ECONNRESET
{ [Error: read ECONNRESET]
code: 'ECONNRESET',
errno: 'ECONNRESET',
syscall: 'read',
__cached_trace__:
[ { receiver: [Object],
fun: [Function: errnoException],
pos: 22930 },
{ receiver: [Object], fun: [Function: onread], pos: 14545 },
{},
{ receiver: [Object],
fun: [Function: fireErrorCallbacks],
pos: 11672 },
{ receiver: [Object], fun: [Function], pos: 12329 },
{ receiver: [Object], fun: [Function: onread], pos: 14536 } ],
__previous__:
{ [Error]
id: 1061835,
location: 'fireErrorCallbacks (net.js:439)',
__location__: 'process.nextTick',
__previous__: null,
__trace_count__: 1,
__cached_trace__: [ [Object], [Object], [Object] ] } }
Here I serve the flash socket policy file:
net = require("net")
net.createServer( (socket) =>
socket.write("<?xml version=\"1.0\"?>\n")
socket.write("<!DOCTYPE cross-domain-policy SYSTEM \"http://www.macromedia.com/xml/dtds/cross-domain-policy.dtd\">\n")
socket.write("<cross-domain-policy>\n")
socket.write("<allow-access-from domain=\"*\" to-ports=\"*\"/>\n")
socket.write("</cross-domain-policy>\n")
socket.end()
).listen(843)
Can this be the cause?
You might have guessed it already: it's a connection error.
"ECONNRESET" means the other side of the TCP conversation abruptly closed its end of the connection. This is most probably due to one or more application protocol errors. You could look at the API server logs to see if it complains about something.
But since you are also looking for a way to check the error and potentially debug the problem, you should take a look at "How to debug a socket hang up error in NodeJS?" which was posted at stackoverflow in relation to an alike question.
Quick and dirty solution for development:
Use longjohn, you get long stack traces that will contain the async operations.
Clean and correct solution:
Technically, in node, whenever you emit an 'error' event and no one listens to it, it will throw. To make it not throw, put a listener on it and handle it yourself. That way you can log the error with more information.
To have one listener for a group of calls you can use domains and also catch other errors on runtime. Make sure each async operation related to http(Server/Client) is in different domain context comparing to the other parts of the code, the domain will automatically listen to the error events and will propagate it to its own handler. So you only listen to that handler and get the error data. You also get more information for free.
EDIT (2013-07-22)
As I wrote above:
"ECONNRESET" means the other side of the TCP conversation abruptly closed its end of the connection. This is most probably due to one or more application protocol errors. You could look at the API server logs to see if it complains about something.
What could also be the case: at random times, the other side is overloaded and simply kills the connection as a result. If that's the case, depends on what you're connecting to exactly…
But one thing's for sure: you indeed have a read error on your TCP connection which causes the exception. You can see that by looking at the error code you posted in your edit, which confirms it.
A simple tcp server I had for serving the flash policy file was causing this. I can now catch the error using a handler:
# serving the flash policy file
net = require("net")
net.createServer((socket) =>
//just added
socket.on("error", (err) =>
console.log("Caught flash policy server socket error: ")
console.log(err.stack)
)
socket.write("<?xml version=\"1.0\"?>\n")
socket.write("<!DOCTYPE cross-domain-policy SYSTEM \"http://www.macromedia.com/xml/dtds/cross-domain-policy.dtd\">\n")
socket.write("<cross-domain-policy>\n")
socket.write("<allow-access-from domain=\"*\" to-ports=\"*\"/>\n")
socket.write("</cross-domain-policy>\n")
socket.end()
).listen(843)
I had a similar problem where apps started erroring out after an upgrade of Node. I believe this can be traced back to Node release v0.9.10 this item:
net: don't suppress ECONNRESET (Ben Noordhuis)
Previous versions wouldn't error out on interruptions from the client. A break in the connection from the client throws the error ECONNRESET in Node. I believe this is intended functionality for Node, so the fix (at least for me) was to handle the error, which I believe you did in unCaught exceptions. Although I handle it in the net.socket handler.
You can demonstrate this:
Make a simple socket server and get Node v0.9.9 and v0.9.10.
require('net')
.createServer( function(socket)
{
// no nothing
})
.listen(21, function()
{
console.log('Socket ON')
})
Start it up using v0.9.9 and then attempt to FTP to this server. I'm using FTP and port 21 only because I'm on Windows and have an FTP client, but no telnet client handy.
Then from the client side, just break the connection. (I'm just doing Ctrl-C)
You should see NO ERROR when using Node v0.9.9, and ERROR when using Node v.0.9.10 and up.
In production, I use v.0.10. something and it still gives the error. Again, I think this is intended and the solution is to handle the error in your code.
Had the same problem today.
After some research i found a very useful --abort-on-uncaught-exception node.js option. Not only it provides much more verbose and useful error stack trace, but also saves core file on application crash allowing further debug.
I also get ECONNRESET error during my development, the way I solve it is by not using nodemon to start my server, just use "node server.js" to start my server fixed my problem.
It's weird, but it worked for me, now I never see the ECONNRESET error again.
I was facing the same issue but I mitigated it by placing:
server.timeout = 0;
before server.listen. server is an HTTP server here. The default timeout is 2 minutes as per the API documentation.
Yes, your serving of the policy file can definitely cause the crash.
To repeat, just add a delay to your code:
net.createServer( function(socket)
{
for (i=0; i<1000000000; i++) ;
socket.write("<?xml version=\"1.0\"?>\n");
…
… and use telnet to connect to the port. If you disconnect telnet before the delay has expired, you'll get a crash (uncaught exception) when socket.write throws an error.
To avoid the crash here, just add an error handler before reading/writing the socket:
net.createServer(function(socket)
{
for(i=0; i<1000000000; i++);
socket.on('error', function(error) { console.error("error", error); });
socket.write("<?xml version=\"1.0\"?>\n");
}
When you try the above disconnect, you'll just get a log message instead of a crash.
And when you're done, remember to remove the delay.
Another possible case (but rare) could be if you have server to server communications and have set server.maxConnections to a very low value.
In node's core lib net.js it will call clientHandle.close() which will also cause error ECONNRESET:
if (self.maxConnections && self._connections >= self.maxConnections) {
clientHandle.close(); // causes ECONNRESET on the other end
return;
}
ECONNRESET occurs when the server side closes the TCP connection and your request to the server is not fulfilled. The server responds with the message that the connection, you are referring to a invalid connection.
Why the server sends a request with invalid connection?
Suppose you have enabled a keep-alive connection between client and server. The keep-alive timeout is configured to 15 seconds. This means that if keep-alive is idle for 15 seconds, it will send connection close request. So after 15 seconds, server tells the client to close the connection. BUT, when server is sending this request, client is sending a new request which is already on flight to the server end. Since this connection is invalid now, server will reject with ECONNRESET error. So the problem occurs due to fewer requests to the server end. So please disable keep-alive and it will work fine.
I had this Error too and was able to solve it after days of debugging and analysis:
my solution
For me VirtualBox (for Docker) was the Problem. I had Port Forwarding configured on my VM and the error only occured on the forwarded port.
general conclusions
The following observations may save you days of work I had to invest:
For me the problem only occurred on connections from localhost to localhost on one port. -> check changing any of these constants solves the problem.
For me the problem only occurred on my machine -> let someone else try it.
For me the problem only occurred after a while and couldn't be reproduced reliably
My Problem couldn't be inspected with any of nodes or expresses (debug-)tools. -> don't waste time on this
-> figure out if something is messing around with your network (-settings), like VMs, Firewalls etc., this is probably the cause of the problem.
I solved the problem by simply connecting to a different network. That is one of the possible problems.
As discussed above, ECONNRESET means that the TCP conversation abruptly closed its end of the connection.
Your internet connection might be blocking you from connecting to some servers. In my case, I was trying to connect to mLab ( cloud database service that hosts MongoDB databases). And my ISP is blocking it.
I had resolved this problem by:
Turning off my wifi/ethernet connection and turn on.
I typed: npm update in terminal to update npm.
I tried to log out from the session and log in again
After that I tried the same npm command and the good thing was it worked out. I wasn't sure it is that simple.
I am using CENTOS 7
I just figured this out, at least in my use case.
I was getting ECONNRESET. It turned out that the way my client was set up, it was hitting the server with an API call a ton of times really quickly -- and it only needed to hit the endpoint once.
When I fixed that, the error was gone.
I had the same issue and it appears that the Node.js version was the problem.
I installed the previous version of Node.js (10.14.2) and everything was ok using nvm (allow you to install several version of Node.js and quickly switch from a version to another).
It is not a "clean" solution, but it can serve you temporarly.
Try adding these options to socket.io:
const options = { transports: ['websocket'], pingTimeout: 3000, pingInterval: 5000 };
I hope this will help you !
Node JS socket is non-blocking io. Consider using a non-blocking io connection from other sources. For instance, if you use a blocking Java socket with node it will only work for a few seconds after which the error will be served. Mitigate this by implementing a non-blocking connection I.e. socketchannel with the selector.
First I run my app I got ECONNRESET after that I got error like ECONNREFUSED . I had faced both of this problem while running my node app.For both of the Problem, I found that this was occuring because of not starting the wampserver.I am using mysql database in my app for getting the data with the help of wampserver. I resolve this by starting the wampserver and then after running my node app. It works fine.You can use node or nodemon for running the node application It's not the problem in my case.
Few options I tried and worked as a temporary solutions
If using node, try to switch between different node versions using node use #version#. Worked for me
Try switching internet connection
Hello I'm new to all this IOT thing and I want to be able to send and receive data from an Arduino via mqtt using https://myqtthub.com as our broker. I used the following code in order to connect:
import React, { Component } from 'react';
var mqtt = require('mqtt');
var options={
clientId:"mqttjs01",
username:"user",
password:"password",
clean:true};
var client = mqtt.connect("mqtt://node02.myqtthub.com",options)
client.on('connect', () => {
alert('Connected!');
console.log('Connected');
});
class Garage extends Component {
render() {
return (
<div>
<h1>HEllooo</h1>
</div>
);
}
}
export default Garage;
I'm having the error:
"WebSocket connection to 'ws://node02.myqtthub.com/' failed: Error during WebSocket handshake: Unexpected response code: 301"
Any help will be appreciated.
Assuming this is all running in the browser (mainly because MQTT.js is forcing the connection to be over ws:// as that is the ONLY option in the browser).
Looking at the docs for myqtthub, they do NOT claim any support for MQTT over Websockets (and the support forum says they don't support it yet). This means your application will NOT work with this broker provider until they add Websockets support..
Ugh....Where to start....
First off, go read the docs at https://github.com/mqttjs/MQTT.js
You CONNECT, but you never SUBSCRIBE...not to mention you didn't post any code that you are PUBLISHing any data. Your client can both send (PUBLISH) and receive (SUBSCRIBE) data, but you have to set them both up.
As for your initial problem, you didn't define the port to use on the connect. You are also specifying mqtt:// as the protocol, not ws://, so that will also cause you problems if you really do want a WebSocket connection. How you are getting that websocket error is a mystery, since your code example shows it to be using the MQTT protocol.
If you are just starting out, stick with mqtt:// and work up from that.
Edit: After thinking about the issue, the real question is what is an example of connecting to digitalocean's managed redis with node-redis using tls?
I'm able to connect just fine with redisinsight GUI client using username / password, but cannot connect with nodejs. It's on the same computer so no firewall issues.
var redis = require('redis');
var client = redis.createClient(process.env.REDIS_PORT, process.env.REDIS_URL, {no_ready_check: true});
client.auth('password', function (err) {
if (err) {
console.log(err);
return
}
console.log('auth')
});
One thing I'm confused about is where to enter the username? It's just 'default' but the documentation for node_redis doesn't provide a way to give a username during auth.
Error is: AbortError: Redis connection lost and command aborted. It might have been processed.
Here's my working lightly anonymized redisinsight connection screen.
How do I do the same in node-redis?
The AUTH command, as stated in the docs:
When ACLs are used, the single argument form of the command,
where only the password is specified, assumes that the implicit username is "default".
So even if you are using Redis 6, where additional users are supported, your authentication for default should work.
The error you're seeing is the result of a broken connection, e.g. you somehow lost connection with the Redis server. node-redis is dealing with one of two scenarios (or both) - the connection has timed out or the the reconnect attempts have exceeded the maximum number specified in a config. I would double check your connection information and how your redis server is configured.
I see you are using TLS, you may find this useful: Securing Node Redis
If you want to authenticate node-redis client with a different user, when using Redis 6, you will have to use send_command, but before you need to remove the current AUTH command, as currently node-redis doesn't support the new command AUTH <username> <password>.
client['auth'] = null;
client.send_command('AUTH', ['<username>', '<password>'], redis.print);
I am making a chat-bot that is based on a HTTP Webhook API that sends requests via POST with JSON message in it's body. It runs in multithread across multiple servers. So here's the problem - when users send multiple commands at the same time, commands are being handled asynchronously, resulting in a mess - for example, if you send a command that increments some counter in database (I'm using MongoDB) twice at the same moment, the bot will answer twice with the same counter value, incrementing it only once in database.
So I've came up with an idea of some custom reverse-proxy with a queueing logic in it. This proxy would accept an HTTP request, transfer it to a chat-bot thread and remember the chat where the message came from. If another request will come from the same chat, it will put the request in some sort of queue and transfer it to the bot when the first one will complete.
I made this proxy on Node.js using uWebSockets.js as a HTTP server and native http.request as HTTP client to transfer requests, but it does not perform very well. When it's being used at the "real" load (2-3 requests/sec) it starts to spit out errors and the whole chatbot becomes unresponsive.
There's nothing special about request sending code - it just makes a request and then responses with status, headers and body that it got from backend.
In general these errors do occur:
Error: connect ECONNREFUSED x.x.x.x:3xxx
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1056:14) {
errno: 'ECONNREFUSED',
code: 'ECONNREFUSED',
syscall: 'connect',
address: 'x.x.x.x',
port: 3xxx
}
Error: socket hang up
at connResetException (internal/errors.js:559:14)
at Socket.socketCloseListener (_http_client.js:376:25)
at Socket.emit (events.js:208:15)
at TCP.<anonymous> (net.js:588:12) {
code: 'ECONNRESET'
}
I would like to hear any ideas about how can I fix my reverse-proxy or how can I use a ready one. Maybe there is a way to make a queueing mechanism without a custom reverse-proxy at all. I wanted to try bouncy module, but it seems to be deprecated with the last commit on GitHub made at 2014.
try this
// include dependencies
var express = require('express')
var proxy = require('http-proxy-middleware')
// proxy middleware options
var options = {
target: 'http://www.example.org', // target host
changeOrigin: true, // needed for virtual hosted sites
ws: true, // proxy websockets
pathRewrite: {
'^/api/old-path': '/api/new-path', // rewrite path
'^/api/remove/path': '/path' // remove base path
},
router: {
// when request.headers.host == 'dev.localhost:3000',
// override target 'http://www.example.org' to 'http://localhost:8000'
'dev.localhost:3000': 'http://localhost:8000'
}
}
// create the proxy (without context)
var exampleProxy = proxy(options)
// mount `exampleProxy` in web server
var app = express()
app.use('/api', exampleProxy)
app.listen(3000)
Use the Caching technique. This is likely because your server processes too much code and leads the socket server to fail.
Can anyone help me, how t disconnect ssh connection using node-ssh module in node js. Also how to handle error.
my code is
driver = require('node-ssh');
ssh = new driver({
host: '192.168.*.*',
username: 'user',
password: 'password',
privateKey : require('fs').readFileSync('/tmp/my_key')
});
ssh.connect().then(function() {
/*
some code
*/
},function(error) {
console.log(error);
});
Pls help.
Use the dispose method. Example:
const node_ssh = require('node-ssh');
const ssh = new node_ssh();
ssh.connect({
host: 'XXX',
username: 'YYY',
privateKey: 'ZZZ'
}).then(resp => {
console.log(resp);
ssh.dispose();
});
The 'node-ssh' wrapper doesn't seem to provide an 'end' function, or a way to access the underlying ssh2 connection object. That leaves you with a couple options:
Fork the source, write an 'end' method, and issue a pull request to the node-ssh wrapper so that future users can use the wrapper and end the connections. Alternatively, you can create an issue and wait for someone else to create the functionality if/when they deem it necessary.
Use the underlying ssh2 library instead. It exposes a lot more functionality to you, including an 'end' method, to close the connection, but uses callbacks instead of promises, which would require you to refactor your code.
Additionally, you could add the following to your code, though this is very heavily not recommended, as it's messing with a prototype you don't own and could ruin compatibility with future versions of the node-ssh module:
driver.prototype.end = function() {
this.Connection.end()
this.Connected = false
}
you can then call ssh.end() to close the connection.
I've asked directly the creator of the library, here i report the answer:
"Note: If anything of the ssh2 object is not implemented and I am taking more time than I should, You can always use MySSH.Connection.close()"
Issue#9
Then he has done a commit and now you have your method!
ssh.end();
I struggle for these the ssh to a remote server frequently, and the connection can not be closed by ssh.compose() in the ssh-node package, and I find remote windows ssh server still generate many processes named sshd process and not be closed, and after ssh.compose, the node.js will throw the error Error: read ECONNRESET at TCP.onStreamRead which can not be caught by code.I find many references, it could be the TCP connection is still working on and not be closed. So I try to refer to the ssh-node code and use the
ssh.connection.destroy()
which point to the Client.prototype.destroy method in ssh2 package， so the error Error: read ECONNRESET at TCP.onStreamRead will disappeared, and the connection will be closed totally.