While scaling out the azure instance at a high load, many WebSocket requests are not handled properly as these instances are not warmed up. To pre-warm up the instance, we have added a piece of code in the listening function in the WebSocket. In this code, we are trying to create multiple WebSocket connections to the same instance for the warmup of the machine in order to prepare the instance to handle a high load. But we are not able to establish these WebSocket connections to the same machine in the Azure app service while we could make WebSocket connections to the same machine in the local machine.
Really appreciate it if anyone could guide us on this.
function onListening() {
//psuedo code for warming up the instances quickly//
for (let warmupIndex = 0; warmupIndex < 20; warmupIndex++) {
let ws = new WebSocket('ws://' + os.hostname() + ':'+ port+'/');
setTimeout(function () {
logger.infoLog("Inside timeout")
ws.close();
}, 10000);
}
callback(null);
}
Related
I have a Node.js script which is supposed to regularly access a SailsJS application via a socket connection. Client and server run on physically different machines on different networks. The SailsJS application is proxied behind nginx. That works in general. However, at random times, the connection is established but the first post request within the websocket connection never reaches its destination.
The code looks basically like this:
var socketIOClient = require('socket.io-client');
var sailsIOClient = require('sails.io.js');
var io = sailsIOClient(socketIOClient);
io.sails.url = 'https://foo.foo:443';
io.sails.rejectUnauthorized = false;
io.socket.on('connect', function() {
console.log("Connected!")
io.socket.post('/someroute', { someOptions: "foo" } ,
function(data) {
console.log(data);
});
});
io.socket.on('disconnect', function(){
console.log("Disconnected!");
});
io.socket.on('connect_error',function () {
console.log("connect_error!");
});
In case of a failure, simply nothing happens after console.log("Connected!"). Nothing appears in nginx's logs (in contrast to successful cases), the callback of io.socket.post never gets executed.
The most important question for me is: At which side is the problem? Client or server?
How can I debug this and narrow down the problem? Could it be a networking issue? Or something wrong the configuration, implementation or with the script itself?
I have one node server running on EC2 instance and client is also running on same EC2 instance, Client open websocket connection to communicate node server, it is working in QA and Dev AWS environment but same web connection is getting close after 60 seconds of being idle in prod environment ,I am running client and node server behind ELB in aws environment.
Client Code:
ws = new WebSocket('ws://localhost:8443');
ws.onclose = function () {
console.log("Websocket connection has been closed.");
clientObj.emit('LogoffSuccess', 'LogoffSuccessfully');
};
ws.onerror=function(event)
{
console.log(event.data);
};
ws.addEventListener('open', function (event) {
console.log('Websocket connection has been opened');
ws.send(JSON.stringify(loginCreds));
});
Node server Code below:
const wss = new WebSocket.Server({ server: app });
const clients = {};
const idMap = {};
wss.on(`connection`, ws => {
const headers = ws.upgradeReq.headers;
const host = headers.host;
const key = ws.upgradeReq.headers[`sec-websocket-key`];
ctiServer.on(`responseMessage`, message => {
clients[message.AgentId].send(JSON.stringify(message));
});
ws.on(`message`, message => {
log.info(`Message received. Host: ${host}, Msg: ${message}`);
if (JSON.parse(message).EventName === `Login`) {
clients[JSON.parse(message).AgentId] = ws;
idMap[key] = JSON.parse(message).AgentId;
}
ctiServer.processIncomingRequest(message);
});
ws.on(`close`, () => {
log.info(`Connection closed. Host: ${host}`);
const message = {
EventName: `Logoff`,
AgentId: idMap[key],
EventData: {}
};
});
});
By default, Elastic Load Balancing sets the idle timeout value to 60 seconds. Therefore, if the target doesn't send some data at least every 60 seconds while the request is in flight, the load balancer can close the front-end connection. To ensure that lengthy operations such as file uploads have time to complete, send at least 1 byte of data before each idle timeout period elapses, and increase the length of the idle timeout period as needed.
https://docs.aws.amazon.com/elasticloadbalancing/latest/application/application-load-balancers.html#connection-idle-timeout
Note that your interests are best served by periodically sending traffic to keep the connection alive. You can set the idle timeout to up to 4000 seconds in an Application Load Balancer, but you will find that stateful intermediate network infrastructure (firewalls, NAT devices) tends to reset connections before they are actually idle for so long.
PING!
Write a ping implementation (or a nil message implementation)...
...otherwise the AWS proxy (probably nginx) will shut down the connection after a period of inactivity (60 seconds in your case, but it's a bit different on different systems).
Do you use NGINX? Their requests timeout after 60 seconds.
You can extended the timeout in the NGINX configuration file for your websockets specific location.
In your case it could look something like this when extending the timeout to an hour:
...
location / {
...
proxy_pass http://127.0.0.1:8443;
...
proxy_read_timeout 3600;
proxy_send_timeout 3600;
...
}
Also see this website for more information:
https://ubiq.co/tech-blog/increase-request-timeout-nginx/
https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout
https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_send_timeout
I'm trying to make a scalable socket connection handler using node.js, express, socket.io, cluster, socket.io-adapter-mongo, and mubsub. This is my first attempt to use sockets, so forgive me if I reveal my noobness here, but it is my understanding that a cluster worker can only emit to the sockets that are connected to it.
In my dev environment I have cluster forking 8 workers (number of cpus.)
I have my workers subscribe to mubsub db so that they will pick up events that are published from other workers.
if (cluster.isMaster) {
var cpuCount = require("os").cpus().length;
for (var cp = 0; cp < cpuCount; cp++) {
cluster.fork();
}
} else {
io.adapter(mongo({ host: 'localhost', port: 27017, db: 'mubsub' }));
var client = mubsub('mongodb://localhost:27017/mubsub');
var channel = client.channel('test');
channel.subscribe('testEvent', function (message) {
console.log(message);
io.sockets.emit('testEvent', {message: cluster.worker.id});
});
io.sockets.on('connection', function (socket) {
console.log('connected to ' + cluster.worker.id);
channel.publish('testEvent', { message: 'connection3' });
});
...
server.listen(8080);
}
So when I try to connect from the client, the 'connection' event fires and a single console log is written by the worker that receives the connection.
That event is published to the database only once.
Each worker is subscribed to that event, and should emit to all sockets connected to that worker.
For some reason though, my connected client receives 8 messages, one for each worker.
How is the client picking up the emits from workers it should not be connected to? Am I overlooking some cluster magic here?
Not sure what version you are using but this should be true for most current versions.
From the socket.io docs (http://socket.io/docs/server-api/#server#emit):
Server#emit
Emits an event to all connected clients. The following two are equivalent:
var io = require('socket.io')();
io.sockets.emit('an event sent to all connected clients');
io.emit('an event sent to all connected clients');
So the method you are using will broadcast to all connected clients. If you want to split them across the workers this is something you need to manage.
There are a number of ways to address individual sockets(clients) that socket.ios API can help you with but probably best to refer to the docs for this:
http://socket.io/docs/rooms-and-namespaces/
Imagine there is a Node.js chatting app.
It is going to be hosted on AWS that will scale across multiple instances.
The chat server app has this piece of codes that tracks who is online or offline using a variable called "sockets".
This code works on single instance. But will this code fail when the server is scaled across multiple instances?
var sockets = {};
io.configure(function () {
...
});
io.sockets.on('connection', function(socket) {
socket.on('online', function(data){
if(data.userId){
sockets[userId] = socket;
console.log("user " + userId + " connected.");
}
});
socket.on('disconnect', function () {
for(var key in sockets){
if(sockets[key] == socket)
{
delete sockets[key];
console.log("user " + key + " disconnected.");
break;
}
}
});
});
// check if a userId is connected
// will this still work when multiple server instances
function isUserIdOnline(userId)
{
return (sockets[userId] != null);
}
No, it will not work with multiple instances. sockets represent only the users who connected to this particular server, not all the other servers.
The likely answer to this question involves storing who is connected and which server they are connected to in a central database server so any socket server can find out if someone is connected and which server they are connected to by contacting the central database. Since the data is temporal, an in-memory database like redis could be used.
A realated discussion of various architectural options for using multiple servers to handle connections while solving the "who is online problem and how do I contact them" is in this answer.
Ok, this is not going to be easy to explain, but I'll give it my best..
We have a file sharing solution serving (small images to big 3d drawings/videos(++5-10GB) running on a strange network setup (IT Security request).
We have the internal zone (LAN) with a NODE.js server:
Backend ( Windows Server 2008 2x2.60Ghz 16GB Ram and disk transfer speed of up till 122 MB/S read). 0.8.22 (Some MSNodeSQL stuff so cant use the newer Node versions.. Converting to TDS now
We then have a external zone ( DMZ ) where the proxy and web-server lives.
Proxy ( Ubuntu 12.10 1x2.60Ghz 32GB Ram rest is same as above ) node 0.8.22. Proxy is the only server that can access the LAN, but there is only one port open and it has to be initilized by the backend..
Web ( Windows Server 2008 8x2.60Ghz 16GB ram and rest is same as above ) node 0.10.12
So, we need to transfer a file to the client connected to the web-server.
The library I've used now is BinaryJS witch works good, but we have a extreamly slow transfer speed of about 12MB/s and this drops when new clients are connecting (usuall arround 4-6MB/s).
Request flow:
- Client (browser) -> Web-Server over https
Web-Server -> Proxy over BinaryJS
Proxy -> Backend over BinaryJS CreateStream.
Backend -> Proxy with a stream object that are piped to the web-server socket.
Proxy to Web-Server via pipe ( the original request )
Web-Server -> Client (Browser) binaryStream.pipe(res) where res is the ExpressJS response.
From testing it seems like we loose 50% of the transfer speed for each server (backend, proxy, web) we go through and i cant figure out why.. We have enough bandwidth and disk speed to get full speed of the Gigabit connected clients. Also the memory and CPU are not working that hard.
Code:
Backend (BASIC) - DB Query's of filepath etc is removed :
client = BinaryClient('wss://'+config.proxy+':'+config.port+'/binary');
client.on('stream', function(msgStream, msgMeta){
console.log("Got stream with meta: ", msgMeta);
var stream = require('fs').createReadStream(msgMeta.filepath);
stream.pipe(msgStream);
msgStream.on('error', function(e){
console.log("Bin stream failed: ", e);
});
msgStream.on('close', function(){
console.log("Stream closed")
});
stream.on('end', function(){
console.log("Ending file stream of file: ", msgMeta.meta.itemID);
msgStream.end();
});
}
})
Proxy:
var binServer = BinaryServer({port:8001});
binServer.on('connection', function(client){
client.on('stream', function(stream, meta){
if(meta.register == true){
if(meta.who == "master"){
config.masterBinary = client;
}
} else {
// Its a stream request.
if(meta.what == "GET"){
config.masterBinary.createStream(meta).pipe(stream).on('close', function(){
console.log("CLOSED");
}).on('end', function(){
console.log("ENDED");
});
}
}
})
});
Web-Server:
client = binaryClient('wss://'+config.proxy+':'+config.port+'/binary');
client.on('open', function(){
ready = true;
})
client.on('close', function(){
ready = false;
})
exports.get = function(file, res, cb){
// This get called from the Express request.. Client will wait for the stream to start
console.log("Sending request to backend: ", file);
if(file.itemID || file.requestID){
client.createStream({filepath:"C:/SomePathToAFile", register:false, what:"GET"}).pipe(res);
} else {
cb("Parameter not present", false);
}
}
I've been wondering if is should try to write my own TCP-server and clients to better fit what we are trying to do.. Unfortunate I'm quite new to Node and have not figure out how to implement the EventEmitter into TCP so I can send data with meta for identification..
Is there a better way or have I completly missunderstood the BinaryJS server?
Thanks for reading!