Using websocket compression with uWebSockets.js and Websocket-Sharp - node.js

We have a mobile game using websocket for connections. The server is a Node.js app using uWebSockets.js library and the client is a Unity app using Websocket-Sharp library. They both play well together and we didn't have encountered an issue with them.
Recently we wanted to enable websocket compression. Both libraries stated that they support Per-message Compression extension but it seems there's something incompatible with them. Because when we configure to use compression the websocket connection closes immediately on handshake.
We also tested client with ws library and it's provided example for compression with the same result. We tried tinkering with ws compression options and found that when we comment serverMaxWindowBits option (defaults to negotiated value) the connection could be established and sending and receiving messages works without a problem. We also asked about controlling the serverMaxWindowBits in uWebsockets.
The last thing we tried was connecting a minimal uWS server and websocket-sharp client.
Here is the code for the server:
const uWS = require('uWebSockets.js');
const port = 5001;
const app = uWS.App({
}).ws('/*', {
/* Options */
compression: 1, // Setting shared compression method
maxPayloadLength: 4 * 1024,
idleTimeout: 1000,
/* Handlers */
open: (ws, req) => {
console.log('A WebSocket connected via URL: ' + req.getUrl() + '!');
},
message: (ws, message, isBinary) => {
/* echo every message received */
let ok = ws.send(message, isBinary);
},
drain: (ws) => {
console.log('WebSocket backpressure: ' + ws.getBufferedAmount());
},
close: (ws, code, message) => {
console.log('WebSocket closed');
}
}).any('/*', (res, req) => {
res.end('Nothing to see here!');
}).listen(port, (token) => {
if (token) {
console.log('Listening to port ' + port);
} else {
console.log('Failed to listen to port ' + port);
}
});
Here is the client code:
using System;
using WebSocketSharp;
namespace Example
{
public class Program
{
public static void Main (string[] args)
{
using (var ws = new WebSocket ("ws://localhost:5001")) {
ws.OnMessage += (sender, e) =>
Console.WriteLine ("server says: " + e.Data);
ws.Compression = CompressionMethod.Deflate; // Turning on compression
ws.Connect ();
ws.Send ("{\"comm\":\"example\"}");
Console.ReadKey (true);
}
}
}
}
When we ran the server and the client, the client emits the following error:
Error|WebSocket.checkHandshakeResponse|The server hasn't sent back 'server_no_context_takeover'.
Fatal|WebSocket.doHandshake|Includes an invalid Sec-WebSocket-Extensions header.
It seemed the client expected server_no_context_takeover header and didn't received one. We reviewed uWebsockets source (C++ part of uWebsockets.js module) and found a commented condition for sending back server_no_context_takeover header. So we uncommented the condition and built uWebsockets.js and tested again to encounter the following error in the client:
WebSocketSharp.WebSocketException: The header of a frame cannot be read from the stream.
Any suggestions for making these two libraries work together?

Update: Based on my reading of the code in uWebSockets.js, changes would need to be made to enable all the parameters websocket-sharp needs set to enable compression. In Vertx, a high-performance Java server, the following settings work with Unity-compatible websocket-sharp for compression:
vertx.createHttpServer(new HttpServerOptions()
.setMaxWebsocketFrameSize(65536)
.setWebsocketAllowServerNoContext(true)
.setWebsocketPreferredClientNoContext(true)
.setMaxWebsocketMessageSize(100 * 65536)
.setPerFrameWebsocketCompressionSupported(true)
.setPerMessageWebsocketCompressionSupported(true)
.setCompressionSupported(true));
Previously:
The error is real, websocket-sharp only supports permessage-deflate, use DEDICATED_COMPRESSOR (compression: 2) instead.

Related

Socket.Io not emitting immediately after first emit (order important)

Environment:
Backend
node:latest
socket.io | 4.5.2
Frontend
React Native | 0.70.4
socket.io-client | 4.6.0
both Android and iOS
Here is my NodeJs entry file:
const numCPUs = cpus().length
if (cluster.isPrimary) {
const app = express()
const httpServer = http.createServer(app)
setupMaster(httpServer, { loadBalancingMethod: 'least-connection' })
setupPrimary()
for (let i = 0; i < numCPUs; i++) {
cluster.fork()
}
cluster.on('exit', (worker) => {
cluster.fork()
})
} else {
const app = express()
const httpServer = http.createServer(app)
const io = new Server(httpServer, { maxHttpBufferSize: 1e8 })
io.adapter(createAdapter())
setupWorker(io)
API.Socket.init(io, process.pid)
middlewares.forEach((middleware: any) => app.use(middleware))
routes.forEach((route) => app.use(route.path, route.handler))
httpServer.listen(CONFIG.PORT, () => {})
}
I have a simple chat application.
When user A sends message to user B, new chat message and notification is recorded in database. Now that chat message and notification* should be sent to the B user. There are 2 socket emit-functions for that:
sendNewNotification(
notification: BE.Entities.TNotification,
toUser: string,
) {
this.io
?.to(toUser)
.volatile.emit(ECustomEvents.NewNotification, notification)
}
sendPrivateMessage(
toUser: string | Array<string>,
chatMessage: BE.Entities.TChatMessage,
sourceUser: BE.Entities.TUser,
) {
this.io
?.to(toUser)
.volatile.emit(ECustomEvents.PrivateMessage, chatMessage, sourceUser)
}
If I do it like this, the targetUser is not going to receive the event with the newChatMessage however he will receive the savedNotification
API.Socket.sendPrivateMessage(targetUserId, newChatMessage, userToPass)
API.Socket.sendNewNotification(savedNotification, targetUserId)
Now, if I switch these lines:
API.Socket.sendNewNotification(savedNotification, targetUserId)
API.Socket.sendPrivateMessage(targetUserId, newChatMessage, userToPass)
the behavior would be as expected: the target user B will receive both saved notification and new chat message
How is that possible? What could be wrong?
Thank you mates in advance!
With the current information, I'm not so sure the order matters but perhaps that it's a side-effect / coincidence. Are you checking anywhere to make sure the server-side socket is ready before the client emits?
Consider this super simple WebSocket chat sandbox:
One of the issues I noticed when writing this is when the server WebSocket is not ready, I could not emit from the client to the server. To make sure the server is ready, I sent a ping from the server to the client notifying the client that the server is ready:
wss.on("connection", async function connection(client, request) {
console.log("user connected", Date.now());
client.send(JSON.stringify({ ready: true }));
...
});
I also notice you are usingg the volatile.emit which according to the documentation:
Volatile events
Volatile events are events that will not be sent if the underlying connection is not ready (a bit like UDP, in terms of reliability).
This can be interesting for example if you need to send the position of the characters in an online game (as only the latest values are useful).
socket.volatile.emit("hello", "might or might not be received");
The Socket.IO docs have a similar listener which lets you know when the server is ready.
If you prevent the client from emitting until the server is ready, you can avoid this issue. You also should not need to use the volatile.emit for something that must be delivered.

Node js with express return connection closed before receiving a handshake response

I have a socket running in nodejs and using this socket in html page this is working fine and some times I'm receiving the error on developer console as like
failed: Connection closed before receiving a handshake response. In this time my update not getting reflect on the user screen. Actually whenever the changes updated in admin screen I written the login in laravel to store this values into the redis and I have used the laravel event broadcast and in node js socket.io read the redis value change and push the values into the user screens.
I have code in laravel as like,
Laravel Controller,
public function updatecommoditygroup(Request $request)
{
$request_data = array();
parse_str($request, $request_data);
app('redis')->set("ssahaitrdcommoditydata", json_encode($request_data['commodity']));
event(new SSAHAITRDCommodityUpdates($request_data['commodity']));
}
In this above controller when the api call receives just store the values into this redis key and broadcast the event.
In my event class,
public $updatedata;
public function __construct($updatedata)
{
$this->updatedata = $updatedata;
}
public function broadcastOn()
{
return ['ssahaitrdupdatecommodity'];
}
Finally I have written my socket.io file as like below,
var app = require('express')();
var http = require('http').Server(app);
var io = require('socket.io')(http);
var Redis = require('ioredis');
var redis = new Redis({ port: 6379 } );
redis.subscribe('ssahaitrdupdatecommodity', function(err, count) {
});
io.on('connection', function(socket) {
console.log('A client connected');
});
redis.on('pmessage', function(subscribed, channel, data) {
data = JSON.parse(data);
io.emit(channel + ':' + data.event, data.data);
});
redis.on('message', function(channel, message) {
message = JSON.parse(message);
io.emit(channel + ':' + message.event, message.data);
});
http.listen(3001, function(){
console.log('Listening on Port 3001');
});
When I have update the data from admin I'm passing to laravel controller, and controller will store the received data into redis database and pass to event broadcast.And event broadcast pass the values to socket server and socket server push the data whenever the redis key get change to client page.
In client page I have written the code as like below,
<script src="../assets/js/socket.io.js"></script>
var socket = io('http://ip:3001/');
socket.on("novnathupdatecommodity:App\\Events\\NOVNATHCommodityUpdates", function(data){
//received data processing in client
});
Everything working fine in most of the time and some times issue facing like
**VM35846 socket.io.js:7 WebSocket connection to 'ws://host:3001/socket.io/?EIO=3&transport=websocket&sid=p8EsriJGGCemaon3ASuh' failed: Connection closed before receiving a handshake response**
By this issue user page not getting update with new data. Could you please anyone help me to solve this issue and give the best solution for this issue.
I think this is because your socket connection timeout.
new io({
path:,
serveClient:,
orgins:,
pingTimeout:,
pingInterval:
});
The above is the socket configuration. If you are not configuring socket sometime it behaves strangely. I do not know the core reason, but i too have faced similar issues that implementing the socket configuration solved it.
Socket.io Server
Similar configuration should be done on the client side. There is an option of timeout in client side
Socket.io Client
For example.
Say this is your front-end code
You connect to the socket server using the following command:
io('http://ip:3001', { path: '/demo/socket' });
In your server side when creating the connection:
const io = require("socket.io");
const socket = new io({
path: "/demo/socket",
serveClient: false /*whether to serve the client files (true/false)*/,
orgins: "*" /*Supports cross orgine i.e) it helps to work in different browser*/,
pingTimeout: 6000 /*how many ms the connection needs to be opened before we receive a ping from client i.e) If the client/ front end doesnt send a ping to the server for x amount of ms the connection will be closed in the server end for that specific client*/,
pingInterval: 6000 /* how many ms before sending a new ping packet */
});
socket.listen(http);
Note:
To avoid complication start you http server first and then start you sockets.
There are other options available, but the above are the most common ones.
I am just describing what i see in the socket.io document available in github.socket_config. Hope this helps

Websocket 'Sec-WebSocket-Accept' header mismatch between ReactJS client and Node.js server

I'm writing an application using WebSockets with a React client on port 8080 (run using Webpack devServer) and Node server and sockets on port 5000. However, the initial handshake always fails with an error: WebSocket connection to 'ws://localhost:5000/' failed: Error during WebSocket handshake: Incorrect 'Sec-WebSocket-Accept' header value
To make sure, I check the request and response of the React app using Chrome devtools, and see the following:
While on my Node server, I logged the sec-websocket-accept header accept key, as well as the headers for my response, and got the following:
It looks like that indeed, the keys don't match. In fact, they don't seem to be the same keys at all. Is there something in between the React client and Node server (like the Webpack devserver that I'm using for React) that's changing them?
My React code:
componentDidMount(){
this.socket = new WebSocket('ws://localhost:5000', ['json']);
this.socket.onerror = err => {
console.log(err)
}
this.socket.onmessage = e => {
let res = JSON.parse(e.data);
console.log(e, res);
let copyArr = [...this.state.message]
copyArr.push(res);
this.setState({
message: copyArr
});
}
}
My Node.js code:
const server = http.createServer();
server.on('upgrade', (req, socket) => {
if(req.headers['upgrade'] !== "websocket"){
socket.end('HTTP/1.1 400 Bad Request');
return;
}
const acceptKey = req.headers['sec-websocket-key'];
const acceptHash = generateValue(acceptKey);
console.log('accepkey', acceptKey, 'hash', acceptHash);
const resHeaders = [ 'HTTP/1.1 101 Web Socket Protocol Handshake', 'Upgrade: WebSocket', 'Connection: Upgrade', `Sec-WebSocket-Accept: ${acceptHash}` ];
console.log(resHeaders);
let protocols = req.headers['sec-websocket-protocol'];
protocols = !protocols ? [] : protocols.split(',').map(name => name.trim());
if(protocols.includes('json')){
console.log('json here');
resHeaders.push(`Sec-WebSocket-Protocol: json`);
}
socket.write(resHeaders.join('\r\n') + '\r\n\r\n');
})
function generateValue(key){
return crypto
.createHash('sha1')
.update(key + '258EAFA5-E914–47DA-95CA-C5AB0DC85B11', 'binary')
.digest('base64');
}
The correct Accept hash for a key of 'S1cb73xifMvqiIpMjvBabg==' is 'R35dUOuC/ldiVp1ZTchRsiHUnvo='.
Your generateValue() function calculates an incorrect hash because it has an incorrect character in the GUID string '258EAFA5-E914–47DA-95CA-C5AB0DC85B11'. If you look very carefully you'll see that the second dash, in '...14–47...', is different from the other dashes. It should be a plain ASCII dash or hyphen with character code 45, but in fact it is a Unicode en-dash with character code 8211. That different character code throws off the calculation.
Fixing that character will make your WebSocket client much happier.
For anyone wondering, the culprits causing the issues, in my case, were the extra new lines and returns I added after writing my headers in my Node.js server. Taking them out and doing:
socket.write(resHeaders.join('\r\n'));
instead of:
socket.write(resHeaders.join('\r\n') + '\r\n\r\n');
solved the handshake mismatch for me.

Setting response headers in WebSocket server and verify in client using npm WebSocket server

I am creating a mock web server using ws library of node.js:
https://www.npmjs.com/package/ws#api-docs
I need to set a protocol in Sec-WebSocket-Protocol header and send it to client, then verify the header on client side.
I tried below options:
wss.on('headers', function (headers) {
console.log("on headers");
headers.push(`sec-websocket-protocol: ${protocol}`);
})
Also this:
var msg = {
message: message,
"sec-websocket-protocol": protocol
};
ws.send(JSON.stringify(msg));
Nothing seems to work currently. Also on client side I am not sure on how to verify this header?
There is no need to mess with the headers yourself.
On the client side you list the protocols as the second arguemtn to the Websocket constructor.
const ws = new WebSocket(ws_url, ["protocol-1", "protocol-2", "protocol-3"]);
On the server side, you need to pass a handleProtocols function, to chose one of the available protocols.
var wss = new WebSocketServer({
...
handleProtocols: (protocols, client) => {
var protocol = /* choose one from protocols argument */;
return protocol;
},
...
});
Then on the client side you get the chosen protocol on the protocol property on your WebSocket object.
ws.onopen = function() {
console.log("WS opened; protocol chosen:", this.protocol);
};
ws.onmessage = function(data) {
if (this.protocol in protocol_handlers) {
protocol_handlers[this.protocol](data.data);
}
}

webSocketServer node.js how to differentiate clients

I am trying to use sockets with node.js, I succeded but I don't know how to differentiate clients in my code.
The part concerning sockets is this:
var WebSocketServer = require('ws').Server,
wss = new WebSocketServer({port: 8080});
wss.on('connection', function(ws) {
ws.on('message', function(message) {
console.log('received: %s', message);
ws.send(message);
});
ws.send('something');
});
This code works fine with my client js.
But I would like to send a message to a particular user or all users having sockets open on my server.
In my case I send a message as a client and I receive a response but the others user show nothing.
I would like for example user1 sends a message to the server via webSocket and I send a notification to user2 who has his socket open.
In nodejs you can directly modify the ws client and add custom attributes for each client separately. Also you have a global variable wss.clients that can be used anywhere. Please try the following code with at least two clients connected:
var WebSocketServer = require('ws').Server;
var wss = new WebSocketServer({
server: httpsServer
});
wss.getUniqueID = function () {
function s4() {
return Math.floor((1 + Math.random()) * 0x10000).toString(16).substring(1);
}
return s4() + s4() + '-' + s4();
};
wss.on('connection', function connection(ws, req) {
ws.id = wss.getUniqueID();
wss.clients.forEach(function each(client) {
console.log('Client.ID: ' + client.id);
});
});
You can also pass parameters directly in the client connection URL:
https://myhost:8080?myCustomParam=1111&myCustomID=2222
In the connection function you can get these parameters and assign them directly to your ws client:
wss.on('connection', function connection(ws, req) {
const parameters = url.parse(req.url, true);
ws.uid = wss.getUniqueID();
ws.chatRoom = {uid: parameters.query.myCustomID};
ws.hereMyCustomParameter = parameters.query.myCustomParam;
}
You can simply assign users ID to an array CLIENTS[], this will contain all users. You can directly send message to all users as given below:
var WebSocketServer = require('ws').Server,
wss = new WebSocketServer({port: 8080}),
CLIENTS=[];
wss.on('connection', function(ws) {
CLIENTS.push(ws);
ws.on('message', function(message) {
console.log('received: %s', message);
sendAll(message);
});
ws.send("NEW USER JOINED");
});
function sendAll (message) {
for (var i=0; i<CLIENTS.length; i++) {
CLIENTS[i].send("Message: " + message);
}
}
you can use request header 'sec-websocket-key'
wss.on('connection', (ws, req) => {
ws.id = req.headers['sec-websocket-key'];
//statements...
});
This code snippet in Worlize server really helped me a lot. Even though you're using ws, the code should be easily adaptable. I've selected the important parts here:
// initialization
var connections = {};
var connectionIDCounter = 0;
// when handling a new connection
connection.id = connectionIDCounter ++;
connections[connection.id] = connection;
// in your case you would rewrite these 2 lines as
ws.id = connectionIDCounter ++;
connections[ws.id] = ws;
// when a connection is closed
delete connections[connection.id];
// in your case you would rewrite this line as
delete connections[ws.id];
Now you can easily create a broadcast() and sendToConnectionId() function as shown in the linked code.
Hope that helps.
It depends which websocket you are using. For example, the fastest one, found here: https://github.com/websockets/ws is able to do a broadcast via this method:
var WebSocketServer = require('ws').Server,
wss = new WebSocketServer({host:'xxxx',port:xxxx}),
users = [];
wss.broadcast = function broadcast(data) {
wss.clients.forEach(function each(client) {
client.send(data);
});
};
Then later in your code you can use wss.broadcast(message) to send to all. For sending a PM to an individual user I do the following:
(1) In my message that I send to the server I include a username
(2) Then, in onMessage I save the websocket in the array with that username, then retrieve it by username later:
wss.on('connection', function(ws) {
ws.on('message', function(message) {
users[message.userName] = ws;
(3) To send to a particular user you can then do users[userName].send(message);
I'm using fd from the ws object. It should be unique per client.
var clientID = ws._socket._handle.fd;
I get a different number when I open a new browser tab.
The first ws had 11, the next had 12.
You can check the connection object. It has built-in identification for every connected client; you can find it here:
let id=ws._ultron.id;
console.log(id);
One possible solution here could be appending the deviceId in front of the user id, so we get to separate multiple users with same user id but on different devices.
ws://xxxxxxx:9000/userID/<<deviceId>>
By clients if you mean the open connections, then you can use ws.upgradeReq.headers['sec-websocket-key'] as the identifier. And keep all socket objects in an array.
But if you want to identify your user then you'll need to add user specific data to socket object.
If someone here is maybe using koa-websocket library, server instance of WebSocket is attached to ctx along side the request. That makes it really easy to manipulate the wss.clients Set (set of sessions in ws). For example pass parameters through URL and add it to Websocket instance something like this:
const wss = ctx.app.ws.server
const { userId } = ctx.request.query
try{
ctx.websocket.uid = userId
}catch(err){
console.log(err)
}
Use a global counter variable and assign its value for every new connection:
const wss = new WebSocket.Server({server});
let count_clients = 0;
wss.on('connection', function connection(ws){
ws.id=count_clients++;
console.log(`new connection, ws.id=${ws.id}, ${ws._socket.remoteAddress}:${ws._socket.remotePort} #clients=${wss.clients.size}`);
ws.on('close', req => {console.log(`disconnected, ws.id=${ws.id}, ${ws._socket.remoteAddress}:${ws._socket.remotePort} #clients=${wss.clients.size}`);});
...
Here is what I did:
* on connect, server generate an unique id (e.g uuid) for the connection,
* save it in memory, (e.g as key of map),
* send back to client in response,
*
*
* client save the id, on each request will also send the id as part of request data,
* then server identify the client by id, on receive further request,
*
* server maintain client, e.g cleanup on close/error,
*
I've impl the idea, it works well to identify the client.
And, I also achieved group/topic broadcast based on the idea, which need the server to maintain extra info.
There are a lot of interesting answers that do the job, however they mostly seem unclean, that is if you don't mind mutating the ws object. I did it this way because I'm using TypeScript and you can't arbitrarily add properties to objects.
import WebSocket from 'ws'
declare module 'ws' {
interface WebSocket {
id: any
key: string
}
}
The id doesn't have to be type any can be number or string depending on how you ID your connections. I haven't flushed out the system yet but for now when a connection is made, I just assign a random number.
const socketConnection = (socket: WebSocket.WebSocket): void => {
socket.id = Math.random()
console.log(socket.id)
const msg = JSON.stringify({ res: `[open] Welcome to the WebSocket server!` })
socket.send(msg)
}
This can be modified at any point so once I authenticate the connection I plan on assigning a relative ID here and might even add in a key property if I want to do some more fancy stuff.
How this works is explained in the Module Augmentation section of the documentation.
TypeScript: Module Augmentation
You can check that it's still assigned by looking over multiple messages in the onmessage event.
const socketMessage = (socket: WebSocket.WebSocket): void => {
socket.on('message', async (message: WebSocket.RawData) => {
console.log(socket.id)
console.log(socket.key)
})
}
Oh and a note, I made this module declaration in the document where I setup my socket. But the modification does populate across documents. For example in the AuthController I started prototyping I use it this way.
export default class AuthController {
public static connections: DLinkedList = new DLinkedList()
static async validate(request: { id: string, socket: WebSocket.WebSocket }): Promise<void> {
console.log('test', request.socket.id)
this.connections.add(request.socket, request.id)
request.socket.send(JSON.stringify({ res: true }))
console.log(this.connections.size())
}
static getSocket(id: string): WebSocket.WebSocket {
return this.connections.getAtKey(id).data
}
static removeSocket(socket: WebSocket.WebSocket) {
}
}
You can also do this in pure JS just by directly modifying the WebSocket object prototype. Some of the answers here talk about it. I haven't done it myself but the principle is similar.
Add a method to an existing class in typescript?
Hope this is useful.

Resources