Node.JS event loop socket delayed or thread blocked - node.js

I have a problem with node.js when sending a lot of concurrent request. The problem is that sometimes it seems it puts some request at the end of the event pool and give me the response after some serious time (60 seconds+, normal is under 10 seconds).
The story goes like this, i have 3 scripts, a CONSOLE, a SERVER and 10 CLIENTS.
console.js
// sending message
client.connect(port, host, function(connect_socket)
{
client.sendMessage(code:301,... some json-socket message ...);
client.on('message', function(message)
{
if(message.code == 304)
{
console.log(... print data ...)
}
});
}
server.js
server = net.createServer(function(socket)
{
socket = new JsonSocket(socket);
socket.on('message', function(message)
{
if(message.code == 301)
{
var client_random = get_random_client();
client_random.sendMessage(code:302,... some json-socket message ...);
}
if(message.code == 303)
{
var client_return = get_client_by_id(message.return_client_id);
client_return.sendMessage(code:304,... some json-socket message ...);
}
});
});
});
client.js
client.connect(port, host, function(connect_socket)
{
client.on('message', function(message)
{
if(message.code == 302)
{
execute_command(message.data, function(result_command)
{
client.sendMessage(code:303,... some json-socket message (good or bad check) ...)
});
}
});
}
Arhitecture concept, console sends message to server, server to a random client, client executes an external program and sends output back to server, server sends response back to the console and console prints it.
console.js => server.js => client.js => server.js => console.js
I open the server, clients are connecting no problem. I open the console and type the command, i get every time the response under 10 seconds.
Now, i made another PHP script that would simulate 600 requests per second. I do the same thing, open console, send command, and once every 10 runs (1 of 10), the console waits, waits and waits, and after 60 seconds it gives me the result (10 was normal).
I made some debug and it seems that server.js do not trigger message event when reciving from client.js and somehow puts it at the very end of the event pool, but never forget it, runs eventually.
I have double check :
console.js every time sends message to server.js (always instant)
server.js every time sends message to client.js (always instant)
client.js every time sends message to server.js (always instant)
[server.js do not fire the event message event instant, and put it
on the very end of the event pool ]
server.js every time sends message to client.js (always instant)
Also i have checked for the possible I/O main thread block, everything is fine. All operations are async, no sync functions.
It is that kind of bug that sometime it is manifesting, sometimes not. Like after a console.js waiting, you can open another terminal and console.js and send messages and see how it responds right away. As i already told, it is like a probability of 1 from 10.
How can i solve this? I had made a lot of debugging.

Related

SocketIO limit requests on progress watching

I'm using SocketIO for a small app, where users would receive updates whenever a change occurs. However, i'd like to implement it to have real time progress visualization in tasks that are done in server side.
However, if the task progress changes too fastly, this would result in tons of event emissions and i think this could decrease the app performance. Is there a way to limit event emits to a maximum of N per second (Emitting only the last one, with the last percent of the progress) ?
Yes, that can be done. It requires you to hold events for a short time to see if there are more events of the same kind coming and thus combine all of them into one. I will code up an example in a few minutes here.
Here's a general idea for how you could do this:
function emitMessageLast(socket, msg, data) {
const queueTime = 500; // wait for up to 500ms of idle time before sending latest data
const longestWaitTime = 2000; // wait no more than 2 seconds if data is being continuously sent
function stopTimer() {
if (socket._timer) {
clearTimeout(socket._timer);
socket._timer = null;
}
}
function sendNow() {
socket._lastMsg = msg;
socket._lastTime = Date.now();
return socket.emit(msg, data);
}
// if this is the first time we're sending this message
// or it's been awhile since we last sent data
// just send the new data immediately
if (socket._lastMsg !== msg || !socket._lastTime || Date.now() - socket._lastTime > longestWaitTime) {
stopTimer();
return sendNow();
}
// at this point, we know we're sending the same message as has recently been sent
socket._lastMsg = msg;
socket._lastData = data;
stopTimer();
// set a timer so that if no more data has arrived before the timer fires,
// we sent the last data we saved
socket._timer = setTimeout(() => {
socket._timer = null;
sendNow();
}, queueTime);
}
The general idea for this code is as follows:
When you get called with a message to send and no message of the same type has been recently sent, then send this one immediately and record the time it was sent.
When you get called with a message to send and it's been more than longestWaitTime since you last sent a message, then send this one immediately. This means if the server is continuously sending data, the server will wait for up to longestWaitTime before sending the latest value of the data.
When the server is sending data sporadically, it will wait up to queueTime (waiting to see if there's more data coming) before sending the last piece of data. It is essentially buffering the last message until no more messages have been send in the last queueTime and then a timer will fire off that last message.
I've configured the defaults here so that it will delay sending data to the client for up to 500ms (while waiting to see if the server is about to send more data so it can avoid sending all the intermediate values of the data) figuring that if the client updates its status every 500ms, that is plenty often. And, if the server is continuously sending updates, then the server will skip up to 2000ms of updates to send just the one last update. Again, you can set these numbers however you see appropriate.

How does the event: 'close' work in Node.js?

I'm experimenting with the close event in Node.js. I'm very new to Node.js so I'm not sure if this is a decent question or a sad one.
Documentation for close event:
http://nodejs.org/api/http.html#http_event_close_2
I want to output to the console a message if the browser is closed before the end event is reached.
While the server ran and before it got to 15 seconds, I tried closing the browser and killing the process through Chrome Tools. No message is output to the console and if I open up other connections by visiting localhost:8080 with other windows, I quickly get a 'hello' indicating my node server thinks there are at least two connections.
I'm either not understanding how to kill processes in Chrome or how the event close works.
Or if End and Close are the same - node.js https no response 'end' event, 'close' instead? - why isn't my "They got impatient!" message still ouput in the console?
How can you output to a console if the process was ended before the event end was reached?
var http = require('http'),
userRequests = 0;
http.createServer(function(request,response){
userRequests++;
response.writeHead(200, {'Content-Type' : 'text/plain'});
if ( userRequests == 1 ){
response.write('1st \n');
setTimeout ( function() {
response.write('2nd Thanks for waiting. \n');
response.on('close', function() {
console.log("They got impatient!");
});
response.end();
}, 15000);
response.write('3rd \n');
}
else {
// Quick response for all connections after first user
response.write('Hello');
response.end();
}
}).listen(8080, function() {
console.log('Server start');
});
Thank you.
First - move the event handler for the close message outside the timeout function - you're not hooking up the close handler until after your timeout expires, and probably missing the event.
Second, you never decrement userRequests anywhere; shouldn't there be a userRequests--; line somewhere? This would be throwing off your logic, since it'll always look like there's more than one request.

Node socket.io server crash when starts

I have a socket.io server and a client runing correctly. Each time that the server is down, the client try to connect again each 5 seconds. When the server is up again they connect without problems.
But the problem comes when I wait long time before up the server again, when the server is up, it crashes showing :
info - socket.io started
debug - client authorized
info - handshake authorized DqN4t2YVP7NiqQi8zer9
debug - setting request GET /socket.io/1/websocket/DqN4t2YVP7NiqQi8zer9
debug - set heartbeat interval for client DqN4t2YVP7NiqQi8zer9
debug - client authorized for
debug - websocket writing 1::
buffer.js:287
pool = new SlowBuffer(Buffer.poolSize);
^
RangeError: Maximum call stack size exceeded
Client reconnection (Executed each 5 seconds while is not connected):
function socket_connect() {
if (!socket) {
socket = io.connect('http://192.168.1.25:8088', { 'reconnect':false, 'connect timeout': 5000 });
} else {
socket.socket.connect();
}
socket.on("connect", function () {
clearInterval(connect_interval);
connect_interval = 0;
socket.emit('player', { refresh_data:true });
});
}
On server side, only with the socket instance, it crashes:
var io = require('socket.io').listen(8088);
I think that the problem is:
When the server goes up, it recive all the connections emited by the client each 5 seconds, (15 hours disconnected * 60 m * 60 s / 5 seconds reconnection) and it crashes.
What can i do to close the connections that the server try to do?
PS:If i reload the client, and after up the server, it works
The main idea for socket.io.js is to reuse an existing connection.
You should only connect it once and then exchange messages by using socket.emit()
I am not sure why you are creating a new connection between your client and server for every 5 seconds. There is a limit on the number of connections the server can create, but that should be more than enough. If you put it in a loop then eventually the server will run out of sockets.
io.connect has to be executed once on the client, then may be you can socket.emit() every 5 seconds. Remove the { 'reconnect':false, 'connect timeout': 5000 } and you will be fine.
I founded the problem...
Each time that the function socket_connect() is called, a "socket.on("connect" ..." function is created. So when the server turns up, a new connection is created, but the event "socket.on("connect" is fired multiple times...
The solution was:
function socket_connect() {
if (!socket) {
socket = io.connect('http://192.168.1.25:8088', { 'reconnect':false, 'connect timeout': 5000 });
} else {
socket.socket.connect();
}
}
socket.on("connect", function () {
clearInterval(connect_interval);
connect_interval = 0;
socket.emit('player', { refresh_data:true });
});

nodeJS zeroMQ: why cant send message if sock.send not in setInterval

I was using zeroMQ in nodeJS. But it seems that while sending the data from producer to worker, if I do not put it in setInterval, then it does not send the data to the worker. My example code is as follows:
producer.js
===========
var zmq = require('zmq')
, sock = zmq.socket('push');
sock.bindSync('tcp://127.0.0.1:3000');
console.log('Producer bound to port 3000');
//sock.send("hello");
var i = 0;
//1. var timer = setInterval(function() {
var str = "hello";
console.log('sending work', str, i++);
sock.send(str);
//2. clearTimeout(timer);
//3. }, 150);
sock.on('message', function(msg) {
console.log("Got A message, [%s], [%s]", msg);
});
So in the above code, if I add back the lines commented in 1, 2 and 3, then I do receive the message to the worker side, else it does not work.
Can anyone throw light why to send message I need to put it in setInterval? Or am I doing something wrong way?
The problem is hidden in the zmq bindings for node.js . I've just spent some time digging into it and it basically does this on send():
Enqueue the message
Flush buffers
Now the problem is in the flushing part, because it does
Check if the output socket is ready, otherwise return
Flush the enqueued messages
In your code, because you call bind and immediately send, there is no worker connected at the moment of the call, because they simply didn't have enough time to notice. So the message is enqueued and we are waiting for some workers to appear. Now the interesting part - where do we check for new workers? In the send function itself! So unless we call send() later, when there are actually some workers connected, our messages are never flushed and they are enqueued forever. And that is why setInterval works, because workers have enough time to notice and connect and you periodically check if there are any.
You can find the interesting part at https://github.com/JustinTulloss/zeromq.node/blob/master/lib/index.js#L277 .
Cheers ;-)

node.js + socket.IO - socket not reconnecting?

I know socket.io has a built in feature for reconnecting and everything, however I don't think that it is working - as I have seen from others it's also not working for them either.
If a user puts their computer to sleep, it disconnects them, and then when they open it back up they are no longer connected so they don't any of the notifications or anything until they refresh the page. Perhaps it's just something that I'm not doing correctly?
var io = require('socket.io').listen(8080);
var users = {};
////////////////USER CONNECTED/////
console.log("Sever is now running");
io.sockets.on('connection', function (socket) {
//Tell the client that they are connected
socket.emit('connected');
//Once the users session is recieved
socket.on('session', function (session) {
//Add users to users variable
users[socket.id] = {userID:session, socketID:socket};
//When user disconnects
socket.on('disconnect', function () {
//socket.socket.connect();
var count= 0;
for(var key in users){
if(users[key].userID==session)++count;
if(count== 2) break;
}
if(count== 1){
socket.broadcast.emit('disconnect', { data : session});
}
//Remove users session id from users variable
delete users[socket.id];
});
socket.on('error', function (err) {
//socket.socket.connect();
});
socket.emit("connection") needs to be called when the user reconnects, or at least the events that happen in that event need to be called.
Also socket.socket.connect(); doesn't work, it returns with an error and it shuts the socket server down with an error of "connect doesn't exist".
The problem is related to io.connect params.
Look at this client code (it will try to reconnect forever, with max delay between attempts 3sec):
ioParams = {'reconnection limit': 3000, 'max reconnection attempts': Number.MAX_VALUE, 'connect timeout':7000}
socketAddress = null
socket = io.connect(socketAddress, ioParams)
There are two important parameters out there, related to your problem:
reconnection limit - limit the upper time of delay between reconnect attemts. Normally it's getting bigger and bigger in time of server outage
max reconnection attempts - how many times you want to try. Default is 10. In most cases this is the problem why the client stops trying.

Resources