Node.js + ws - websocket is unexpectedly closing - node.js

I have a Node.js + ws server and a Qt client. I'm having an issue with the websocket connection closing unexpectedly in certain conditions. In my client, I iterate through a list of objects and send a series of requests to the server. The server then accesses a database and then writes the response back to the client. Here is the (simplified) basic server loop:
// ...
var httpServer = http.createServer(//params);
var WebSocketServer = require('ws').Server;
var wss = new WebSocketServer({ server: httpServer });
// ...
wss.on('connection', function connection(ws) {
ws.on('message', function incoming(msg) {
handleRequest(JSON.parse(msg), function (err, result) {
if (err) throw err;
ws.send(JSON.stringify(result), function ack(error) {
if (error) throw error;
}
}
});
});
If I limit the number of request from the client, everything works fine. But if I reach a critical number of requests, data stops coming back from the server and the websocket connection times out.
I'm wondering if I need to flush the websocket on the server before or after each write? QtWebSocket has a flush() method, but I don't see this capability in the ws docs. Any ideas would be appreciated.
UPDATE: I should also note that no errors are sent to either ws or wss. Data just stops flowing.
UPDATE 2: Okay, this seems pretty clearly to be a problem with transmitting oversized packets from my client over the Internet to the node+ws server. Based on Wireshark traces, once the oversize threshold is reached, the socket craps out, but only when I'm connected remotely to the server. If the server is on my local subnet, oversized packets are properly transmitted to the server. Any ideas how to debug this?

This problem resolved after I replaced one of my routers. I am chalking this up to a hardware issue with the old router. This was consumer-grade hardware, so I'm hoping this shouldn't be an issue in production. It does make me wonder how many of these kinds of bugs might be hanging out on the Internet...

Related

Authenticating using ws WebSocket

I'm using ws webSocket (can't use socket.io) and I'm a bit new to it. I know how it works, but don't fully get a couple of things.
How can I authenticate using jwt? The docs say that using the upgrade event is the correct way to go about it, but when the client connected, the code in the upgrade function didn't run.
What's the correct way to add an event type? In socket.io, one can listen using client.on('eventType', function). What's the webSocket ws equivalent?
When and why would one use paths in webSocket ws?
Update
The update event doesn't get fired on my end. Here's the code I have:
File 1
const server = require('http').createServer(app);
require('./socket/socket')(server);
File 2
module.exports = function(server) {
const WebSocket = require('ws');
const wss = new WebSocket.Server({ port: 8080 });
server.on('upgrade', function(request, socket, head) {
console.log('Upgraded');
wss.handleUpgrade(request, socket, head, function(ws) {
wss.emit('connection', ws, request);
});
});
wss.on('connection', function connection(ws) {
ws.on('message', function incoming(message) {
console.log('received: %s', message);
});
});
};
How can I authenticate using jwt? The docs say that using the upgrade event is the correct way to go about it, but when the client connected, the code in the upgrade function didn't run.
Well, there are lots of different choices. You could pass a token as a query parameter of a custom header when first making the connection. You could require that the first data sent over the webSocket is a token and disconnect if such a token does not arrive soon and first. If the client was already authenticated and there's an http cookie that indicates that, you could examine that cookie upon websocket connection.
What's the correct way to add an event type? In socket.io, one can listen using client.on('eventType', function). What's the webSocket ws equivalent?
The socket.io message types (or event types as you call them) are something that socket.io adds on top of webSocket. The webSocket protocol itself does not have such a thing. You send a data packet and that data packet arrives at the other end. All data packets are of the same webSocket "type". If you want to identify your data packets as being a certain type of message, then you have to invent a way of doing that inside your data back. For example, if your data was JSON formatted, you could add a type: "someType" property to event message and then examine that property upon receipt (this is essentially what socket.io does although it does it outside of the socket.io payload).
When and why would one use paths in webSocket ws?
You may have two completely separate parts of code that use a webSocket for different things that have nothing in common and reside in separate modules and it's just more straightforward for each to have its own webSocket connection. And, they may be used by two completely separate types of clients.
How to access the orignal URL when a webSocket client connects to your server is shown here on your server's upgrade event used with the ws library.
You will note in that example that the upgrade event occurs on the http server, not on the websocket server instance.
ws provide auth examples.
These work when a HTTP server first handle the auth requests. Then pass a
HTTP upgrade request to ws rather than ws itself listening on a port. Note
the noServer: true option they include in the setup:
const wss = new WebSocket.Server({ clientTracking: false, noServer: true });
The jwt component will be easier using koa or express in HTTP
first then doing the upgrade in this way. Otherwise you would need to
write a ws message handler to look for some token data and verify it.
The message is the event in ws. You get to write anything more specific on top of the message event, which is a big reason to use socket.io that has done all that for you (including client callback functions, which are super helpful).
Using a URI path in the ws:// or http upgrade request would usually be to connect to a separate ws context on the server side. A simple way to namespace or separate connection into discreet groups.

emitting data via socket on browser close /window close

I need to send data to nodejs server via socket.io when the user closes the browser tab .
I tried doing :
var data={};
window.onbeforeunload = function() {
// i have a object to be sent
data.data1='abcd';
data.data2=1234;
socket.emit("senddata",data);
}
This code works when the user navigates around clicking links on the site but doesnot work when the user closes the browser tab
I also tried configuring the socket io on server side as below .. thinking the error may be due to socket connection being closed before emitting data:
var io = require('socket.io').listen(app.listen(port));
io.configure(function () {
io.set('close timeout',12000);
});
It also didnt work most of the time.
I also tried this on client side:
var socket = require('socket.io').listen(80, {
"sync disconnect on unload":false
});
It also did not work
I had tried receiving data like this
var io = require('socket.io').listen(app.listen(port));
io.sockets.on('connection', function (socket) {
socket.on('senddata', function (data) {
// data processing
});
});
please ..help me with this problem..thanks in advance..
When user connects - register on server side the time it happened.
Socket.IO has heart beating and pinging functionality. So just subscribe to disconnect event on server side and once it happens - means client disconnected - and you have on server time when client had connection. So that way you have timespan of client session.
Do not trust client to tell you any time or important data, as client can simply 'lie' - which leads to hacks/cheats/bugs on your server-side.
There is no reliable way to send data just before disconnect from client-side at all. There is nothing in Socket.IO for that, nor in just one of transports (WebSockets). As well as there is too many different scenarios of disconnection (power off, force close, ethernet cable out, wifi lose, etc).

NodeJS - What does "socket hang up" actually mean?

I'm building a web scraper with Node and Cheerio, and for a certain website I'm getting the following error (it only happens on this one website, no others that I try to scrape.
It happens at a different location every time, so sometimes it's url x that throws the error, other times url x is fine and it's a different url entirely:
Error!: Error: socket hang up using [insert random URL, it's different every time]
Error: socket hang up
at createHangUpError (http.js:1445:15)
at Socket.socketOnEnd [as onend] (http.js:1541:23)
at Socket.g (events.js:175:14)
at Socket.EventEmitter.emit (events.js:117:20)
at _stream_readable.js:910:16
at process._tickCallback (node.js:415:13)
This is very tricky to debug, I don't really know where to start. To begin, what IS a socket hang up error? Is it a 404 error or similar? Or does it just mean that the server refused a connection?
I can't find an explanation of this anywhere!
EDIT: Here's a sample of code that is (sometimes) returning errors:
function scrapeNexts(url, oncomplete) {
request(url, function(err, resp, body) {
if (err) {
console.log("Uh-oh, ScrapeNexts Error!: " + err + " using " + url);
errors.nexts.push(url);
}
$ = cheerio.load(body);
// do stuff with the '$' cheerio content here
});
}
There is no direct call to close the connection, but I'm using Node Request which (as far as I can tell) uses http.get so this is not required, correct me if I'm wrong!
EDIT 2: Here's an actual, in-use bit of code that is causing errors. prodURL and other variables are mostly jquery selectors that are defined earlier. This uses the async library for Node.
function scrapeNexts(url, oncomplete) {
request(url, function (err, resp, body) {
if (err) {
console.log("Uh-oh, ScrapeNexts Error!: " + err + " using " + url);
errors.nexts.push(url);
}
async.series([
function (callback) {
$ = cheerio.load(body);
callback();
},
function (callback) {
$(prodURL).each(function () {
var theHref = $(this).attr('href');
urls.push(baseURL + theHref);
});
var next = $(next_select).first().attr('href');
oncomplete(next);
}
]);
});
}
There are two cases when socket hang up gets thrown:
When you are a client
When you, as a client, send a request to a remote server, and receive no timely response. Your socket is ended which throws this error. You should catch this error and decide how to handle it: whether retry the request, queue it for later, etc.
When you are a server/proxy
When you, as a server, perhaps a proxy server, receive a request from a client, then start acting upon it (or relay the request to the upstream server), and before you have prepared the response, the client decides to cancel/abort the request.
This stack trace shows what happens when a client cancels the request.
Trace: { [Error: socket hang up] code: 'ECONNRESET' }
at ClientRequest.proxyError (your_server_code_error_handler.js:137:15)
at ClientRequest.emit (events.js:117:20)
at Socket.socketCloseListener (http.js:1526:9)
at Socket.emit (events.js:95:17)
at TCP.close (net.js:465:12)
Line http.js:1526:9points to the same socketCloseListener mentioned by #Blender, particularly:
// This socket error fired before we started to
// receive a response. The error needs to
// fire on the request.
req.emit('error', createHangUpError());
...
function createHangUpError() {
var error = new Error('socket hang up');
error.code = 'ECONNRESET';
return error;
}
This is a typical case if the client is a user in the browser. The request to load some resource/page takes long, and users simply refresh the page. Such action causes the previous request to get aborted which on your server side throws this error.
Since this error is caused by the wish of a client, they don't expect to receive any error message. So, no need to consider this error as critical. Just ignore it. This is encouraged by the fact that on such error the res socket that your client listened to is, though still writable, destroyed.
console.log(res.socket.destroyed); //true
So, no point to send anything, except explicitly closing the response object:
res.end();
However, what you should do for sure if you are a proxy server which has already relayed the request to the upstream, is to abort your internal request to the upstream, indicating your lack of interest in the response, which in turn will tell the upstream server to, perhaps, stop an expensive operation.
Take a look at the source:
function socketCloseListener() {
var socket = this;
var parser = socket.parser;
var req = socket._httpMessage;
debug('HTTP socket close');
req.emit('close');
if (req.res && req.res.readable) {
// Socket closed before we emitted 'end' below.
req.res.emit('aborted');
var res = req.res;
res.on('end', function() {
res.emit('close');
});
res.push(null);
} else if (!req.res && !req._hadError) {
// This socket error fired before we started to
// receive a response. The error needs to
// fire on the request.
req.emit('error', createHangUpError());
req._hadError = true;
}
}
The message is emitted when the server never sends a response.
One case worth mentioning: when connecting from Node.js to Node.js using Express, I get "socket hang up" if I don't prefix the requested URL path with "/".
below is a simple example where I got the same error when I missed to add the commented code in below example. Uncommenting the code req.end() will resolve this issue.
var fs = require("fs");
var https = require("https");
var options = {
host: "en.wikipedia.org",
path: "/wiki/George_Washington",
port: 443,
method: "GET"
};
var req = https.request(options, function (res) {
console.log(res.statusCode);
});
// req.end();
I used require('http') to consume https service and it showed "socket hang up".
Then I changed require('http') to require('https') instead, and it is working.
Expanding on Blender's answer, this happens in a number of situations. The most common ones I run into are:
The server crashed.
The server refused your connection, most likely blocked by User-Agent.
socketCloseListener, as outlined in Blender's answer, is not the only place that hangup errors are created.
For example, found here:
function socketOnEnd() {
var socket = this;
var req = this._httpMessage;
var parser = this.parser;
if (!req.res) {
// If we don't have a response then we know that the socket
// ended prematurely and we need to emit an error on the request.
req.emit('error', createHangUpError());
req._hadError = true;
}
if (parser) {
parser.finish();
freeParser(parser, req);
}
socket.destroy();
}
You could try curl with the headers and such that are being sent out from Node and see if you get a response there. If you don't get a response with curl, but you do get a response in your browser, then your User-Agent header is most likely being blocked.
Another case worth mentioning (for Linux and OS X) is that if you use a library like https for performing the requests, or if you pass https://... as a URL of the locally served instance, you will be using port 443 which is a reserved private port and you might be ending up in Socket hang up or ECONNREFUSED errors.
Instead, use port 3000, f.e., and do an http request.
For request module users
Timeouts
There are two main types of timeouts: connection timeouts and read timeouts. A connect timeout occurs if the timeout is hit while your client is attempting to establish a connection to a remote machine (corresponding to the connect() call on the socket). A read timeout occurs any time the server is too slow to send back a part of the response.
Note that connection timeouts emit an ETIMEDOUT error, and read timeouts emit an ECONNRESET error.
This caused me issues, as I was doing everything listed here, but was still getting errors thrown. It turns out that calling req.abort() actually throws an error, with a code of ECONNRESET, so you actually have to catch that in your error handler.
req.on('error', function(err) {
if (err.code === "ECONNRESET") {
console.log("Timeout occurs");
return;
}
//handle normal errors
});
I had the same problem while using Nano library to connect to Couch DB. I tried to fine tune connection pooling with use of keepaliveagent library and it kept failing with socket hang up message.
var KeepAliveAgent = require('agentkeepalive');
var myagent = new KeepAliveAgent({
maxSockets: 10,
maxKeepAliveRequests: 0,
maxKeepAliveTime: 240000
});
nano = new Nano({
url : uri,
requestDefaults : {
agent : myagent
}
});
After some struggling I was able to nail the problem - as it came out it was very, very simple mistake. I was connecting to the database via HTTPS protocol, but I kept passing to my nano object a keepalive agent created as the examples for use of this library show (they rely on some defaults that use http).
One simple change to use HttpsAgent did the trick:
var KeepAliveAgent = require('agentkeepalive').HttpsAgent;
I think "socket hang up" is a fairly general error indicating that the connection has been terminated from the server end. In other words, the sockets being used to maintain the connection between the client and the server have been disconnected. (While I'm sure many of the points mentioned above are helpful to various people, I think this is the more general answer.)
In my case, I was sending a request with a payload in excess of 20K. This was rejected by the server. I verified this by removing text and retrying until the request succeeded. After determining the maximum acceptable length, I verified that adding a single character caused the error to manifest. I also confirmed that the client wasn't the issue by sending the same request from a Python app and from Postman. So anyway, I'm confident that, in my case, the length of the payload was my specific problem.
Once again, the source of the problem is anecdotal. The general problem is "Server Says No".
I had the same problem during request to some server. In my case, setting any value to User-Agent in headers in request options helped me.
const httpRequestOptions = {
hostname: 'site.address.com',
headers: {
'User-Agent': 'Chrome/59.0.3071.115'
}
};
It's not a general case and depends on server settings.
This error also can happen when working with http.request, probably your request is not finished yet.
Example:
const req = https.request(options, res => {})
And you always need to add this line: req.end()
With this function we will order to finish sending request.
As in documentation is said:
With http.request() one must always call req.end() to signify the end of the request - even if there is no data being written to the request body.
Also reason can be because of using app instance of express instead of server from const server = http.createServer(app) while creating server socket .
Wrong
const express = require('express');
const http = require('http');
const WebSocket = require('ws');
const app = express();
app.use(function (req, res) {
res.send({ msg: "hello" });
});
const wss = new WebSocket.Server({ server: app }); // will throw error while connecting from client socket
app.listen(8080, function listening() {
console.log('Listening on %d', server.address().port);
});
Correct
const express = require('express');
const http = require('http');
const WebSocket = require('ws');
const app = express();
app.use(function (req, res) {
res.send({ msg: "hello" });
});
const server = http.createServer(app);
const wss = new WebSocket.Server({ server });
server.listen(8080, function listening() {
console.log('Listening on %d', server.address().port);
});
it's been a long time but another case is when performing requests which takes a long time on the server side (more then 2 minutes which is the default for express) and the timeout parameter was not configured in the server side. In my case I was doing client->server->server request (Node.js express) and I should set the timeout parameter on each request router on the server and on the client.
So in both servers I needed to set the request timeout by using
req.setTimeout([your needed timeout])
on the router.
I do both web (node) and Android development, and open Android Studio device simulator and docker together, both of them use port 8601, it complained socket hang up error, after close Android Studio device simulator and it works well in node side. Don’t use Android Studio device simulator and docker together.
There seems to be one additional case here, which is Electron not being a fan of the "localhost" domain name. In my case I needed to change this:
const backendApiHostUrl = "http://localhost:3000";
to this:
const backendApiHostUrl = "http://127.0.0.1:3000";
After that the problem just went away.
This means that DNS resolution (local or remote) might be causing some problems too.
I got a similar error when using CouchDB on OCP cluster.
const cloudantSessionStore = sessionStore.createSessionStore(
{
type: 'couchdb',
host: 'https://' + credentials['host'],
port: credentials['port'],
dbName: 'sessions',
options: {
auth: {
username: credentials['username'],
password: credentials['password']
},
cache: false
}
}
Which should be "http", not "https", to connect with my CouchDB instance. Hope it could be helpful for anyone who is faced with similar issue.
In my case, it was because a application/json response was badly formatted (contains a stack trace). The response was never send to the server.
That was very tricky to debug because, there were no log. This thread helps me a lot to understand what happens.
In case you're using node-http-proxy, please be aware to this issue, which will result a socket hang-up error : https://github.com/nodejitsu/node-http-proxy/issues/180.
For resolution, also in this link, simply move declaring the API route (for proxying) within express routes before express.bodyParser().
Ran into this issue yesterday running my web application and node.js server through IntelliJ IDEA 2016.3.6. All I had to do was clear my cookies and cache in my Chrome browser.
If you are experiencing this error over a https connection and it's happening instantly it could be a problem setting up the SSL connection.
For me it was this issue https://github.com/nodejs/node/issues/9845 but for you it could be something else. If it is a problem with the ssl then you should be able to reproduce it with the nodejs tls/ssl package just trying to connect to the domain
I think worth noting...
I was creating tests for Google APIs. I was intercepting the request with a makeshift server, then forwarding those to the real api. I was attempting to just pass along the headers in the request, but a few headers were causing a problem with express on the other end.
Namely, I had to delete connection, accept, and content-length headers before using the request module to forward along.
let headers = Object.assign({}, req.headers);
delete headers['connection']
delete headers['accept']
delete headers['content-length']
res.end() // We don't need the incoming connection anymore
request({
method: 'post',
body: req.body,
headers: headers,
json: true,
url: `http://myapi/${req.url}`
}, (err, _res, body)=>{
if(err) return done(err);
// Test my api response here as if Google sent it.
})
I my case it's was not an error, but expected behavior for chrome browser. Chrome keeps tls connection alive (for speed i think), but node.js server stop it after 2 min and you get an error.
If you try GET request using edge browser, there will be no error at all.
If you will close chrome window - you will get error right away.
So what to do?
1)You can filter this errors, because they are not really errors.
2)Maybe there is a better solution :)
After a long debug into node js code, mongodb connection string, checking CORS etc, For me just switching to a different port number server.listen(port); made it work, into postman, try that too. No changes to proxy settings just the defaults.
I was using nano, and it took me a long time to figure out this error. My problem was I was using the wrong port. I had port 5948 instead of 5984.
var nano = require('nano')('http://localhost:5984');
var db = nano.use('address');
var app = express();
Might be your server or Socket connection crashes unexpectedly.
I had this error when running two applications on the same port by mistake.
I had a next.js app and another one in nest.js, running both on port 8080, when I looked at the .env files I realized that they had the same port, so I changed the one from nest.js to 3000 and everything worked.
I'm not saying that this is the reason for the error but it's a possibility.
Your problem might also come from an attempt to connect to an HTTP URL while your service is only published on HTTPS...
Definitely a time-consuming mistake!
Got "[GET] localhost:4200, Socket hang up" during Azure Static Web App (SWA) Emulator for Angular app.
Solution is to remove this from angular.json:
"headers": {"cross-origin-opener-policy": "same-origin-allow-popups"}

Node.js Outgoing Http request connection limit (cannot make connections more than five)

I'm building a data transfer proxy server using node.js.
It pipes client's request to swift object storage server using http(s) REST API.
It works fine for the individual request but when the outgoing
ESTABLISHED tcp connection for the same destination and port(443)
reaches five, it cannot create any new connection.
It does not seem to be a problem of O/S, because I've tried to create more than 10 connections using java servlet and it works fine.
I've tried to set maximum sockets for globalAgent like below, but it does not change anything.
http.globalAgent.maxSockets = 500;
https.globalAgent.maxSockets = 500;
Here is a part of my source code.
app.post('/download*', function(req, res){
/***********************************************************
* Some codes here to create new request option
***********************************************************/
var client = https.request(reqOptions, function(swiftRes) {
var buffers = [];
res.header('Content-Length', swiftRes.headers['content-length']);
res.header('Content-Type', swiftRes.headers['content-type']);
res.header('range', swiftRes.headers['range']);
res.header('connection', swiftRes.headers['connection']);
swiftRes.pipe(res);
swiftRes.on('end', function(err){
res.end();
});
});
client.on('error', function(err) {
callback && callback(err);
client.end(err);
clog.error('######### Swift Client Error event occurred. Process EXIT ');
});
client.end();
});
I hope I can get the solution for this problem.
Thanks in advance.
Usually, the change of the maxSockets should solve your problem, try it with a value a little bit lower.
https.globalAgent.maxSockets=20;
If that does not solve your problem, try to turn off pooling for the connections. Add the key agent with the value false to the options to the request. Keep in mind that Node.js uses the pooling to use keep-alive connection.
//Your option code
reqOptions.agent=false;

Postgresql connection timed out in node.js and pg

I am new to node, postgresql, and to the whole web development business. I am currently writing a simple app which connects to a postgres database and display the content of a table in a web view. The app will be hosted in OpenShift.
My main entry is in server.js:
var pg = require('pg');
pg.connect(connection_string, function(err, client) {
// handle error
// save client: app.client = client;
});
Now, to handle the GET / request:
function handle_request(req, res){
app.client.query('...', function(err, result){
if (err) throw err; // Will handle error later, crash for now
res.render( ... ); // Render the web view with the result
});
}
My app seems to work: the table is rendered in the web view correctly, and it works for multiple connections (different web clients from different devices). However, if there is no request for a couple of minutes, then subsequent request will crash the app with time out information. Here is the stack information:
/home/hai/myapp/server.js:98
if (err) throw err;
^
Error: This socket is closed.
at Socket._write (net.js:474:19)
at Socket.write (net.js:466:15)
at [object Object].query (/home/hai/myapp/node_modules/pg/lib/connection.js:109:15)
at [object Object].submit (/home/hai/myapp/node_modules/pg/lib/query.js:99:16)
at [object Object]._pulseQueryQueue (/home/hai/myapp/node_modules/pg/lib/client.js:166:24)
at [object Object].query (/home/hai/myapp/node_modules/pg/lib/client.js:193:8)
at /home/hai/myapp/server.js:97:17
at callbacks (/home/hai/myapp/node_modules/express/lib/router/index.js:160:37)
at param (/home/hai/myapp/node_modules/express/lib/router/index.js:134:11)
at pass (/home/hai/myapp/node_modules/express/lib/router/index.js:141:5)
Is there a way to keep the connection from timed out (better)? Or to reconnect on demand (best)? I have tried to redesign my app by not connecting to the database in the beginning, but upon the GET / request. This solution works only for the first request, then crashed on the second. Any insight is appreciated.
Have you looked into the postgres keepalive setting values? It sends packets to keep idle connections from timing out.
http://www.postgresql.org/docs/9.1/static/runtime-config-connection.html
I also found this similar question:
How to use tcp_keepalives settings in Postgresql?
You could also perform really minor queries from the db at a set interval. However, this method is definitely more hacked.
Edit: You could also try initiating the client like this:
var client = new pg.Client(conString);
Before you make your queries, you can check if the client is still connected. I believe you can use:
if(client.connection._events != null)
client.connect();
faced the same problem.. telling the client to close connection upon the end event
query.on('end', function() {
client.end();
});
did the trick for me...
You can also change the default idle timeout of 30 seconds to whatever value you need. E.g.
pg.defaults.poolIdleTimeout = 600000; // 10 mins
I'm using the parameter keepAlive in true and it works.
This is my configuration and it is solved.
const client_pg = new Client({
connectionString,
keepAlive: true,
keepAliveInitialDelayMillis: 10000
});

Resources