Hello i am running a basic code for a chatserver on node.js. This is almost lifted from the book - Node:Up and Running. The problem is when a client types a message, his message is transmitted at each keystroke, than after a complete line and pressing enter, resulting in output like this -
client1: (Sends) Hello
Client2: (Recieves)127.0.0.1:50672>h 127.0.0.1:50672>e 127.0.0.1:50672>l 127.0.0.1:50672>l 127.0.0.1:50672>o
But This is how it SHOULD come
Client2: (Recieves) 127.0.0.1:50672>hello
What is happening here is messages are being transmitted at each keystroke, than after pressing enter. I lifted another example code from git by someone and same thing happened! here is my code :
var net = require ('net');
var chatServer = net.createServer(),
clientlist =[];
chatServer.on('connection',function(client){
client.name = client.remoteAddress+':'+client.remotePort;
client.write('Welcome '+ client.name + '!\n');
clientlist.push(client);
client.on('data',function(data){
broadcast(data,client);
});
});
function broadcast(message, client){
for(var i=0;i<clientlist.length;i+=1){
if(client !== clientlist[i]){
clientlist[i].write(client.name + ":"+ message);
}
}
}
console.log("Chatserver Started")
chatServer.listen(9000);
Here is the github example which yielded same results:
GITHUB
HOW TO CORRECT THIS BEHAVIOR?
The server event data is fired every time a block of data is received. Probably it fires for every character from the client.
The solution would be to create, in the server's connection callback, a buffer string/array. The data callback adds the received data to that buffer. When the data contains a newline character, the buffer is broadcasted and emptied.
Related
I'm using socket.io like this
Client:
socket.on('response', function(i){
console.log(i);
});
socket.emit('request', whateverdata);
Server:
socket.on('request', function(whateverdata){
for (i=0; i<10000; i++){
console.log(i);
socket.emit('response', i);
}
console.log("done!");
});
I need output like this when putting the two terminals side by side:
Server Client
0 0
1 1
. (etc) .
. .
9998 9998
9999 9999
done!
But instead I am getting this:
Server Client
0
1
. (etc)
.
9998
9999
done!
0
1
.
. (etc)
9998
9999
Why?
Shouldn't Socket.IO / Node emit the message immediately, not wait for the loop to complete before emitting any of them?
Notes:
The for loop is very long and computationally slow.
This question is referring to the socket.io library, not websockets in general.
Due to latency, waiting for confirmation from the client before sending each response is not possible
The order that the messages are received is not important, only that they are received as quickly as possible
The server emits them all in a loop and it takes a small bit of time for them to get to the client and get processed by the client in another process. This should not be surprising.
It is also possible that the single-threaded nature of Javascript in node.js prevents the emits from actually getting sent until your Javascript loop finishes. That would take detailed examination of socket.io code to know for sure if that is an issue. As I said before if you want to 1,1 then 2,2 then 3,3 instead of 1,2,3 sent, then 1,2,3 received you have to write code to force that.
If you want the client to receive the first before the server sends the 2nd, then you have to make the client send a response to the first and have the server not send the 2nd until it receives the response from the first. This is all async networking. You don't control the order of events in different processes unless you write specific code to force a particular sequence.
Also, how do you have client and server in the same console anyway? Unless you are writing out precise timestamps, you wouldn't be able to tell exactly what event came before the other in two separate processes.
One thing you could try is to send 10, then do a setTimeout(fn, 1) to send the next 10 and so on. That would give JS a chance to breathe and perhaps process some other events that are waiting for you to finish to allow the packets to get sent.
There's another networking issue too. By default TCP tries to batch up your sends (at the lowest TCP level). Each time you send, it sets a short timer and doesn't actually send until that timer fires. If more data arrives before the timer fires, it just adds that data to the "pending" packet and sets the timer again. This is referred to as the Nagle's algorithm. You can disable this "feature" on a per-socket basis with socket.setNoDelay(). You have to call that on the actual TCP socket.
I am seeing some discussion that Nagle's algorithm may already be turned off for socket.io (by default). Not sure yet.
In stepping through the process of socket.io's .emit(), there are some cases where the socket is marked as not yet writable. In those cases, the packets are added to a buffer and will be processed "later" on some future tick of the event loop. I cannot see exactly what puts the socket temporarily in this state, but I've definitely seen it happen in the debugger. When it's that way, a tight loop of .emit() will just buffer and won't send until you let other events in the event loop process. This is why doing setTimeout(fn, 0) every so often to keep sending will then let the prior packets process. There's some other event that needs to get processed before socket.io makes the socket writable again.
The issue occurs in the flush() method in engine.io (the transport layer for socket.io). Here's the code for .flush():
Socket.prototype.flush = function () {
if ('closed' !== this.readyState &&
this.transport.writable &&
this.writeBuffer.length) {
debug('flushing buffer to transport');
this.emit('flush', this.writeBuffer);
this.server.emit('flush', this, this.writeBuffer);
var wbuf = this.writeBuffer;
this.writeBuffer = [];
if (!this.transport.supportsFraming) {
this.sentCallbackFn.push(this.packetsFn);
} else {
this.sentCallbackFn.push.apply(this.sentCallbackFn, this.packetsFn);
}
this.packetsFn = [];
this.transport.send(wbuf);
this.emit('drain');
this.server.emit('drain', this);
}
};
What happens sometimes is that this.transport.writable is false. And, when that happens, it does not send the data yet. It will be sent on some future tick of the event loop.
From what I can tell, it looks like the issue may be here in the WebSocket code:
WebSocket.prototype.send = function (packets) {
var self = this;
for (var i = 0; i < packets.length; i++) {
var packet = packets[i];
parser.encodePacket(packet, self.supportsBinary, send);
}
function send (data) {
debug('writing "%s"', data);
// always creates a new object since ws modifies it
var opts = {};
if (packet.options) {
opts.compress = packet.options.compress;
}
if (self.perMessageDeflate) {
var len = 'string' === typeof data ? Buffer.byteLength(data) : data.length;
if (len < self.perMessageDeflate.threshold) {
opts.compress = false;
}
}
self.writable = false;
self.socket.send(data, opts, onEnd);
}
function onEnd (err) {
if (err) return self.onError('write error', err.stack);
self.writable = true;
self.emit('drain');
}
};
Where you can see that the .writable property is set to false when some data is sent until it gets confirmation that the data has been written. So, when rapidly sending data in a loop, it may not be letting the event come through that signals that the data has been successfully sent. When you do a setTimeout() to let some things in the event loop get processed that confirmation event comes through and the .writable property gets set to true again so data can again be sent immediately.
To be honest, socket.io is built of so many abstract layers across dozens of modules that it's very difficult code to debug or analyze on GitHub so it's hard to be sure of the exact explanation. I did definitely see the .writable flag as false in the debugger which did cause a delay so this seems like a plausible explanation to me. I hope this helps.
RouteChat example which is a bi-directional rpc, seems to be buffering messages on write. Is there a way to force writes to the outgoing stream immediately without buffering?
The reason you're seeing this buffering behavior is because writes are actually completed asynchronously, so the server code needs to regularly allow other asynchronous operations to happen between writes to ensure that they are sent as soon as they are computed. You can take advantage of the optional callback argument to write to send each message immediately after the last one is sent. The code in that example can instead be written using async.each:
async.each(notes, function(note, next) {
console.log('Sending message "' + note.message + '" at ' +
note.location.latitude + ', ' + note.location.longitude);
var noteMsg = new messages.RouteNote();
noteMsg.setMessage(note.message);
var location = new messages.Point();
location.setLatitude(note.location.latitude);
location.setLongitude(note.location.longitude);
noteMsg.setLocation(location);
// Note the use of the next callback here
call.write(noteMsg, next);
}, function() {
// End the stream after all messages have been sent
call.end();
});
I am pretty new to Node.Js and I'm using tcp sockets to communicate with a client. Since the received data is fragmented I noticed that it prints "ondata" to the console more than once. I need to be able to read all the data and concatenate it in order to implement the other functions. I read the following http://blog.nodejs.org/2012/12/20/streams2/ and thought I can use socket.on('end',...) for this purpose. But it never prints "end" to the console.
Here is my code:
Client.prototype.send = function send(req, cb) {
var self = this;
var buffer = protocol.encodeRequest(req);
var header = new Buffer(16);
var packet = Buffer.concat([ header, buffer ], 16 + buffer.length);
function cleanup() {
self.socket.removeListener('data', ondata);
self.socket.removeListener('error', onerror);
}
var body = '';
function ondata() {
var chunk = this.read() || '';
body += chunk;
console.log('ondata');
}
self.socket.on('readable', ondata);
self.socket.on('end', function() {
console.log('end');
});
function onerror(err) {
cleanup();
cb(err);
}
self.socket.on('error', onerror);
self.socket.write(packet);
};
The end event will handle the FIN package of the TCP protocol (in other words: will handle the close package)
Event: 'end'#
Emitted when the other end of the socket sends a FIN packet.
By default (allowHalfOpen == false) the socket will destroy its file descriptor once it has written out its pending write queue. However, by setting allowHalfOpen == true the socket will not automatically end() its side allowing the user to write arbitrary amounts of data, with the caveat that the user is required to end() their side now.
About FIN package: https://en.wikipedia.org/wiki/Transmission_Control_Protocol#Connection_termination
The solution
I understand your problem, the network communication have some data transfer gaps and it split your message in some packages. You just want read your fully content.
For solve this problem i will recommend you create a protocol. Just send a number with the size of your message before and while the size of your concatenated message was less than total of your message size, keep concatenating :)
I have created a lib yesterday to simplify that issue: https://www.npmjs.com/package/node-easysocket
I hope it helps :)
I was using zeroMQ in nodeJS. But it seems that while sending the data from producer to worker, if I do not put it in setInterval, then it does not send the data to the worker. My example code is as follows:
producer.js
===========
var zmq = require('zmq')
, sock = zmq.socket('push');
sock.bindSync('tcp://127.0.0.1:3000');
console.log('Producer bound to port 3000');
//sock.send("hello");
var i = 0;
//1. var timer = setInterval(function() {
var str = "hello";
console.log('sending work', str, i++);
sock.send(str);
//2. clearTimeout(timer);
//3. }, 150);
sock.on('message', function(msg) {
console.log("Got A message, [%s], [%s]", msg);
});
So in the above code, if I add back the lines commented in 1, 2 and 3, then I do receive the message to the worker side, else it does not work.
Can anyone throw light why to send message I need to put it in setInterval? Or am I doing something wrong way?
The problem is hidden in the zmq bindings for node.js . I've just spent some time digging into it and it basically does this on send():
Enqueue the message
Flush buffers
Now the problem is in the flushing part, because it does
Check if the output socket is ready, otherwise return
Flush the enqueued messages
In your code, because you call bind and immediately send, there is no worker connected at the moment of the call, because they simply didn't have enough time to notice. So the message is enqueued and we are waiting for some workers to appear. Now the interesting part - where do we check for new workers? In the send function itself! So unless we call send() later, when there are actually some workers connected, our messages are never flushed and they are enqueued forever. And that is why setInterval works, because workers have enough time to notice and connect and you periodically check if there are any.
You can find the interesting part at https://github.com/JustinTulloss/zeromq.node/blob/master/lib/index.js#L277 .
Cheers ;-)
I am trying to learn about streams in node.js!
server.js
var net = require("net");
var server = net.createServer(function(conn) {
conn.write("welcome!");
# echo the user input!
conn.pipe(conn);
});
server.listen("1111", function() {
console.log("port 1111 opened");
});
telnet test
The server currently echos the user's input
$ telnet localhost 1111
welcome!
hello
hello
desired output
To demonstrate where/how I should process the stream on the server side, I would like to wrap the user's input in {} before echoing it back
$ telnet localhost 1111
welcome!
hello
{hello}
This will basically accomplish the exact output you've requested:
var net = require('net');
var server = net.createServer(function(c) {
c.setEncoding('utf8');
c.on('data', function(d) {
c.write('{' + d.trim() + '}\n');
});
});
server.listen(9871);
First let me call your attention to c.setEncoding('utf8'). This will set a flag on the connection that will automatically convert the incoming Buffer to a String in the utf8 space. This works well for your example, but just note that for improved performance between Sockets it would be better to perform Buffer manipulations.
Simulating the entirety of .pipe() will take a bit more code.
.pipe() is a method of the Stream prototype, which can be found in lib/stream.js. If you take a look at the file you'll see quite a bit more code than what I've shown above. For demonstration, here's an excerpt:
function ondata(chunk) {
if (dest.writable) {
if (false === dest.write(chunk) && source.pause) {
source.pause();
}
}
}
source.on('data', ondata);
First a check is made if the destination is writable. If not, then there is no reason to attempt writing the data. Next the check if dest.write === false. From the documentation:
[.write] returns true if the entire data was flushed successfully to the kernel buffer. Returns false if all or part of the data was queued in user memory.
Since Streams live in kernel space, outside of the v8 memory space, it is possible to crash your machine by filling up memory (instead of just crashing the node app). So checking if the message has drained is a safety prevention mechanism. If it hasn't finished draining, then the source will be paused until the drain event is emitted. Here is the drain event:
function ondrain() {
if (source.readable && source.resume) {
source.resume();
}
}
dest.on('drain', ondrain);
Now there is a lot more we could cover with how .pipe() handles errors, cleans up its own event emitters, etc. but I think we've covered the basics.
Note: When sending a large string, it is possible that it will be sent in multiple packets. For this reason it may be necessary to do something like the following:
var net = require('net');
var server = net.createServer(function(c) {
var tmp = '';
c.setEncoding('utf8');
c.on('data', function(d) {
if (d.charCodeAt(d.length - 1) !== 10) {
tmp += d;
} else {
c.write('{' + tmp + d.trim() + '}\n');
tmp = '';
}
});
});
server.listen(9871);
Here we use the assumption that the string is ended by the new line character (\n, or ascii character code 10). We check the end of the message to see if this is the case. If not, then we temporarily store the message from the connection until the new line character is received.
This may not be a problem for your application, but thought it would be worth noting.
you can do something like
conn.on 'data', (d) ->
conn.write "{#{d}}"
the .pipe method is basically just attaching the data event of the input stream to write to the output stream
I'm not sure about net() actually, but I imagine it's quite similar to http:
http.createServer(function (req, res) {
res.writeHead(200, {'Content-Type': 'text/event-stream'});
http.get(options, function(resp){
resp.on('data', function(chunk){
res.write("event: meetup\n");
res.write("data: "+chunk.toString()+"\n\n");
});
}).on("error", function(e){
console.log("Got error: " + e.message);
});
});
https://github.com/chovy/nodejs-stream