I am pretty new to Node.Js and I'm using tcp sockets to communicate with a client. Since the received data is fragmented I noticed that it prints "ondata" to the console more than once. I need to be able to read all the data and concatenate it in order to implement the other functions. I read the following http://blog.nodejs.org/2012/12/20/streams2/ and thought I can use socket.on('end',...) for this purpose. But it never prints "end" to the console.
Here is my code:
Client.prototype.send = function send(req, cb) {
var self = this;
var buffer = protocol.encodeRequest(req);
var header = new Buffer(16);
var packet = Buffer.concat([ header, buffer ], 16 + buffer.length);
function cleanup() {
self.socket.removeListener('data', ondata);
self.socket.removeListener('error', onerror);
}
var body = '';
function ondata() {
var chunk = this.read() || '';
body += chunk;
console.log('ondata');
}
self.socket.on('readable', ondata);
self.socket.on('end', function() {
console.log('end');
});
function onerror(err) {
cleanup();
cb(err);
}
self.socket.on('error', onerror);
self.socket.write(packet);
};
The end event will handle the FIN package of the TCP protocol (in other words: will handle the close package)
Event: 'end'#
Emitted when the other end of the socket sends a FIN packet.
By default (allowHalfOpen == false) the socket will destroy its file descriptor once it has written out its pending write queue. However, by setting allowHalfOpen == true the socket will not automatically end() its side allowing the user to write arbitrary amounts of data, with the caveat that the user is required to end() their side now.
About FIN package: https://en.wikipedia.org/wiki/Transmission_Control_Protocol#Connection_termination
The solution
I understand your problem, the network communication have some data transfer gaps and it split your message in some packages. You just want read your fully content.
For solve this problem i will recommend you create a protocol. Just send a number with the size of your message before and while the size of your concatenated message was less than total of your message size, keep concatenating :)
I have created a lib yesterday to simplify that issue: https://www.npmjs.com/package/node-easysocket
I hope it helps :)
Related
On one end I have a http request (long polling). On the other end is a "game server" dispatching events. These ends are tied together with a duplex non-flowing stream in object mode and that part works fine.
The long poll end listens on readable and drains the stream by calling stream.read repeatedly. Then closes the client's connection.
The game server uses stream.write to push events to the clients.
Some events in the game actually spand several events and here's the problem:
When the game server adds several events at once (calling stream.write repeatedly) the first write triggers readable and the long poll is filled with the event and closed. That's very inconvinient.
The essential of the problem is that I can't silent readable and then trigger it when I'm done writing.
So my question is; can I somehow "pause" the stream and resume afterwards?
Is there a known another solution to this problem?
My best bet is to write an array of events, but I think that's somehow an antipattern.
Here's some code to illustrate my problem:
var stream = require('stream');
var connection = stream.PassThrough({ objectMode: true });
var exhaust = function() {
console.log('exhausting');
var chunk;
while ((chunk = connection.read()) !== null)
console.log(chunk);
console.log('exhausting end');
}
connection.on('readable', function(){
console.log('Ready to read');
exhaust();
});
for (var i = 0;i < 10;i++)
connection.write({ test: true });
I ended up writing an array of events to write, but I'm looking forward to this upcoming feature, which could be the missin solution:
http://strongloop.com/strongblog/performance-node-js-v-0-12-whats-new/
http://nodejs.org/docs/v0.11.10/api/stream.html#stream_writable_cork
I have Node server which use Express as web app.
This server creates a tcp socket connection with other side TCP server.
I'm trying to pipe tcp data to the user http response.
It works fine for a while, but the LAST tcp packet is NOT piped to http response.
So, download status of web browser stopped as 99.9% downloaded.
My source code is below.
Anyone can help me to solve this problem?
Thanks in advance.
app.get('/download/*', function(req, res){
var tcpClient = new net.Socket();
tcpClient.connect(port, ip, function() {
// some logic
});
tcpClient.on('data', function(data) {
/* skip ... */
tcpClient.pipe(res); // This method is called once in the 'data' event loop
/* skip ... */
});
tcpClient.on('close', function() {
clog.debug('Connection closed.');
});
tcpClient.on('end', function() {
clog.debug('Connection Ended.');
});
tcpClient.on('error', function(err){
clog.err(err.stack);
});
});
That's not how you are supposed to use .pipe().
When you pipe a stream into another, you don't have to handle the data events yourself: everything is taken care of by the pipe. Moreover, the data event is emitted on every chunk of data, which means that you are possibly piping() the streams multiple times.
You only need to create and initialize the Socket, and then pipe it to your response stream:
tcpClient.connect(port, ip, function () {
// some logic
this.pipe(res);
});
Edit: As you precised in the comments, the first chunk contains metadata, and you only want to pipe from the second chunk thereon. Here's a possible solution:
tcpClient.connect(port, ip, function () {
// some logic
// Only call the handler once, i.e. on the first chunk
this.once('data', function (data) {
// Some logic to process the first chunk
// ...
// Now that the custom logic is done, we can pipe the tcp stream to the response
this.pipe(res);
});
});
As a side note, if you want to add custom logic to the data that comes from the tcpClient before writing it to the response object, check out the Transform stream. You will then have to:
create a transform stream with your custom transforming logic
pipe all streams together: tcpClient.pipe(transformStream).pipe(res).
I am trying to learn about streams in node.js!
server.js
var net = require("net");
var server = net.createServer(function(conn) {
conn.write("welcome!");
# echo the user input!
conn.pipe(conn);
});
server.listen("1111", function() {
console.log("port 1111 opened");
});
telnet test
The server currently echos the user's input
$ telnet localhost 1111
welcome!
hello
hello
desired output
To demonstrate where/how I should process the stream on the server side, I would like to wrap the user's input in {} before echoing it back
$ telnet localhost 1111
welcome!
hello
{hello}
This will basically accomplish the exact output you've requested:
var net = require('net');
var server = net.createServer(function(c) {
c.setEncoding('utf8');
c.on('data', function(d) {
c.write('{' + d.trim() + '}\n');
});
});
server.listen(9871);
First let me call your attention to c.setEncoding('utf8'). This will set a flag on the connection that will automatically convert the incoming Buffer to a String in the utf8 space. This works well for your example, but just note that for improved performance between Sockets it would be better to perform Buffer manipulations.
Simulating the entirety of .pipe() will take a bit more code.
.pipe() is a method of the Stream prototype, which can be found in lib/stream.js. If you take a look at the file you'll see quite a bit more code than what I've shown above. For demonstration, here's an excerpt:
function ondata(chunk) {
if (dest.writable) {
if (false === dest.write(chunk) && source.pause) {
source.pause();
}
}
}
source.on('data', ondata);
First a check is made if the destination is writable. If not, then there is no reason to attempt writing the data. Next the check if dest.write === false. From the documentation:
[.write] returns true if the entire data was flushed successfully to the kernel buffer. Returns false if all or part of the data was queued in user memory.
Since Streams live in kernel space, outside of the v8 memory space, it is possible to crash your machine by filling up memory (instead of just crashing the node app). So checking if the message has drained is a safety prevention mechanism. If it hasn't finished draining, then the source will be paused until the drain event is emitted. Here is the drain event:
function ondrain() {
if (source.readable && source.resume) {
source.resume();
}
}
dest.on('drain', ondrain);
Now there is a lot more we could cover with how .pipe() handles errors, cleans up its own event emitters, etc. but I think we've covered the basics.
Note: When sending a large string, it is possible that it will be sent in multiple packets. For this reason it may be necessary to do something like the following:
var net = require('net');
var server = net.createServer(function(c) {
var tmp = '';
c.setEncoding('utf8');
c.on('data', function(d) {
if (d.charCodeAt(d.length - 1) !== 10) {
tmp += d;
} else {
c.write('{' + tmp + d.trim() + '}\n');
tmp = '';
}
});
});
server.listen(9871);
Here we use the assumption that the string is ended by the new line character (\n, or ascii character code 10). We check the end of the message to see if this is the case. If not, then we temporarily store the message from the connection until the new line character is received.
This may not be a problem for your application, but thought it would be worth noting.
you can do something like
conn.on 'data', (d) ->
conn.write "{#{d}}"
the .pipe method is basically just attaching the data event of the input stream to write to the output stream
I'm not sure about net() actually, but I imagine it's quite similar to http:
http.createServer(function (req, res) {
res.writeHead(200, {'Content-Type': 'text/event-stream'});
http.get(options, function(resp){
resp.on('data', function(chunk){
res.write("event: meetup\n");
res.write("data: "+chunk.toString()+"\n\n");
});
}).on("error", function(e){
console.log("Got error: " + e.message);
});
});
https://github.com/chovy/nodejs-stream
I connected wavecom GSM modem on ubantu. I use node.js language to communicate with GSM modem.
I send command to modem by Child Process. Here example
var spawn = require("child_process").spawn,
exec = require('child_process').exec;
// Write dev_ttyUSB15.tmp file
var child = exec('cat < /dev/ttyUSB15 > /tmp/dev_ttyUSB15.tmp');
// Read dev_ttyUSB15.tmp file
var m1 = spawn('tail',['-f','/tmp/dev_ttyUSB15.tmp']);
// on data event is emitted when dev_ttyUSB15.tmp file has some data
m1.stdout.on('data', function (data) {
console.log("Data : "+data); // this is executed as output
});
Now When I fire some command on port /dev/ttyUSB15 I do not get output properly.
E.g
Suppose my output should be
Data : abcd1234
but instead of it I got
Data : abc
Data : d1234
In short My output is breaked.
I can not extrapolate from where my output exactly break. It's random.
Can anyone give me any idea?
Thanks in advance.
As all streams in node.js, the reading of data consists of 2 separate events: data and end.
data event is fired when some data is readable in the stream (in your case, twice).
end event is fired when no more data events will be fired.
var blob = "";
m1.stdout.on('data', function (data) {
blob += data;
});
m1.stdout.on('end', function () {
console.log("Data : " + blob); // here you have all the data within one variable
});
It's hard to say without knowing what protocol you are speaking with the modem, but if it's e.g. \n delimited, you will have to buffer the data and split on \n:
var buffer = '';
m1.stdout.on('data', function(data) {
var received = (buffer + data).split('\n');
buffer = received.pop().trim();
console.log(received.join(''));
});
I am trying to create a simple reply server in node.js
The problem I am having, is that when I telnet into the server, and send a hello, the if loop doesn't catch it, and it goes to the else.
Below is my code:
var net = require('net');
var server = net.createServer(function(socket) {
// Server start
socket.write('Welcome\n');
socket.on('data', function(data) {
dataReceived(socket, data);
});
});
server.listen(8250);
function dataReceived(socket, data) {
if(data == 'hello') {
socket.end('Hi');
} else {
socket.write(data);
socket.end('what??\n');
}
}
Thanks.
Data is a binary buffer, not a string. See http://nodejs.org/docs/v0.4.9/api/buffers.html.
Use the buffer.toString method to convert to a string.
Also, a new line will be added when hitting enter in telnet. Not sure if line endings vary by os, but in this case I'm stripping \r\n.
function dataReceived(socket, data) {
data = data.toString('utf8').replace(/\r\n/, '');
if(data == 'hello') {
socket.end('Hi');
} else {
socket.write(data);
socket.end('what??\n');
}
}
As mentioned, main problem is that you compare Buffer object with string.
There is another problem, most probably not visible in your example.
You don't have control how data is split into packets. 'Hello' sent to your server may result dataReceived called with 'Hel' + 'l' + 'o' buffer 3 times
Correct way to handle 'Hello' input us to create state machine or, more simple and less efficient - buffer all incoming data, look for 'Hello' at the beginning of buffered data, then cut handled data from buffer. There are modules aiming to help to unpack/unframe structured data from input stream, for example node-binary