I would like to receive binary data (like .pdf or .doc) from a tcp socket, here is the code :
To send the file :
fs.readFile(path, function (err, data) {
var client = new net.Socket();
client.connect(user_port, user_ip, function () {
client.write(data, 'binary');
client.destroy();
});
});
To receive the file :
net.createServer(function(socket){
socket.setEncoding('binary');
socket.on('data', function (data) {
var file_data = new Buffer(data, 'binary');
fs.appendFile(utils.getUserDir() + '/my_file.doc', file_data);
});
socket.on('error', function(err){
console.log(err.message);
})
}).listen(utils.getPort(), utils.getExternalIp());
As the files are too big for only one tcp socket, they are sent with multiples packets, in fact there is multiple 'data' event for the same file.
I thought it was possible to append each Buffer() data to a file, but when I open the .doc, he is corrupted or has binary things in it.
PS: I can't use Buffer().concat and save the file after since I don't know which packet is the last one...
Thank you
For sending files like this, it's better to stream them instead of buffer it all into memory and then sending it. (Also, you don't need the 'binary' encoding argument since fs.readFile() gives you a Buffer by default)
For example:
var client = new net.Socket();
client.connect(user_port, user_ip, function() {
fs.createReadStream(path).pipe(client);
});
// ...
net.createServer(function(socket){
socket.pipe(fs.createWriteStream(utils.getUserDir() + '/my_file.doc'));
socket.on('error', function(err){
console.log(err.message);
});
}).listen(utils.getPort(), utils.getExternalIp());
Related
I have made a simple server and client program where the server reads the data from file and send to the client through TCP socket But the data I am getting is in object and not a simple string ?
So why I cant see the data as plaintext as it is in my data.txt file.
Explanation with example would be appreciated.
Here is my code :-
SERVER CODE
const fs = require('fs');
const net = require('net');
const readableData = fs.createReadStream('data.txt', 'utf8');
const server = net.createServer(socket => {
socket.on('data', chunk => {
console.log(chunk.toString());
socket.write(JSON.stringify(readableData));
});
socket.on('end', () => {
console.log("done");
})
socket.on('close', () => {
console.log("closed")
})
});
server.listen(3000);
CLIENT CODE
const fs = require('fs');
const net = require('net');
const client = new net.Socket();
client.connect('3000', () => {
console.log("connected");
client.write("Server please send the data");
});
client.on('data', chunk => {
console.log("Data recieved:" + chunk.toString());
});
client.on('finish', () => {
console.log("Work completed");
})
client.on('close', () => {
console.log("connection closed");
})
And here is my data.txt file which has simple data
Hello client how are you ?
And the output I'm getting is here :-
Data recieved:{"_readableState":{"objectMode":false,"highWaterMark":65536,"buffer":{"head":{"data":"Hello client how are you ?","next":null},"tail":{"data":"Hello client how are you ?","next":null},"length":1},"length":26,"pipes":null,"pipesCount":0,"flowing":null,"ended":true,"endEmitted":false,"reading":false,"sync":false,"needReadable":false,"emittedReadable":false,"readableListening":false,"resumeScheduled":false,"paused":true,"emitClose":false,"autoDestroy":false,"destroyed":false,"defaultEncoding":"utf8","awaitDrain":0,"readingMore":false,"decoder":{"encoding":"utf8"},"encoding":"utf8"},"readable":true,"_events":{},"_eventsCount":1,"path":"data.txt","fd":35,"flags":"r","mode":438,"end":null,"autoClose":true,"bytesRead":26,"closed":false}
The question why I won't be able to see the data as plaintext on client side as it is in data.txt file.
Your variable readableData contains a node.js stream object. That's what that variable is. It's only of use in the current node.js instance so it doesn't do anything useful to try to send that stream object to the client.
If you want to get all the data from that 'data.txt' file, you have several choices.
You can just read the whole file into a local variable with fs.readFile() and then send all that data with socket.write().
You can create a new stream attached to the file for each new incoming request and then as the data comes in on the readStream, you can send it out on the socket (this is often referred to as piping one stream into another). If you use higher level server constructs such as an http server, they make piping real easy.
Option #1 would look like this:
const server = net.createServer(socket => {
socket.on('data', chunk => {
console.log(chunk.toString());
fs.readFile('data.txt', 'utf8', (err, data) => {
if (err) {
// insert error handling here
console.log(err);
} else {
socket.write(data);
}
});
});
socket.on('end', () => {
console.log("done");
})
socket.on('close', () => {
console.log("closed")
})
});
FYI, you should also know that socket.on('data', chunk => {...}) can give you any size chunk of data. TCP streams do not make any guarantees about delivering the exact same chunks of data in the same pieces that they were originally sent in. They will come in order, but if you sent three 1k chunks from the other end, they might arrive as three separate 1k chunks, they might arrive as one 3k chunk or they might arrive as a whole bunch of much smaller chunks. How they arrive will often depend upon what intermediate transports and routers they had to travel over and if there were any recoverable issues along that transmission. For example, data sent over a satellite internet connection will probably arrive in small chunks because the needs of the transport broke it up into smaller pieces.
This means that reading any data over a plain TCP connection generally needs some sort of protocol so that the reader knows when they've gotten a full, meaningful chunk that they can process. If the data is plain text, it might be as simple a protocol as every message ends with a line feed character. But, if the data is more complex, then the protocol may need to be more complex.
Code
var websock = net.createServer(function(sock) {
sock.pipe(sock);
sock.setEncoding('utf8');
sock.setKeepAlive(true);
sock.on("data", function(d) {
console.log("websock", d);
});
sock.on('end', function() {
console.log('websock disconnected');
});
});
websock.listen(777, '127.0.0.1');
After few minutes ~15 mins the callback code in sock.on("data", function() {}) seems not to be working. why is it the case? I checked the console.log, there is no log with a string "websock disconnected".
if the socket is not disconnected and if there is no error, what has happened to the socket connection or the data stream?
On the other end, (Server side, data sender) seems to be streaming data continuously while client side (nodejs app) has stopped receiving data.
The issue arises from your use of the pipe mechanism to echo back data which is never consumed on the original side (communication is unidirectional):
sock.pipe(sock);
This makes your code work as an echo server. Your socket "sock" is a duplex stream (i.e. both readable - for the incoming data you receive, and writable - for outgoing data you send back).
A quick fix if you don't need to respond back and you just need to receive data is to simply delete the "sock.pipe(sock);" line. To find out the explanation, read ahead.
Most probably your data source (the MT5 application you mentioned) sends data continuously and it doesn't read what you send back at all. So, your code keeps echoing back the received data using sock.pipe(sock), filling the outgoing buffer which is never consumed. However, the pipe mechanism of Nodejs streams handles backpressure, which means that when two streams (a readable and a writable one) are connected by a pipe, if the outgoing buffer is filling (reaching a high watermark), the readable stream is paused, to prevent the "overflow" of the writable stream.
You can read more about backpressure in the Nodejs docs. This fragment particularly describes how streams are handling backpressure:
In Node.js the source is a Readable stream and the consumer is the Writable stream [...]
The moment that backpressure is triggered can be narrowed exactly to the return value of a Writable's .write() function. [...]
In any scenario where the data buffer has exceeded the highWaterMark or the write queue is currently busy, .write() will return false.
When a false value is returned, the backpressure system kicks in. It will pause the incoming Readable stream from sending any data and wait until the consumer is ready again.
Below you can find my setup to show where backpressure kicks in; there are two files, server.js and client.js. If you run them both, server will write to console "BACKPRESSURE" soon. As the server is not handling backpressure (it ignores that sock.write starts returning false at some point), the outgoing buffer is filled and filled, consuming more memory, while in your scenario, socket.pipe was handling backpressure and thus it paused the flow of the incoming messages.
The server:
// ----------------------------------------
// server.js
var net = require('net');
var server = net.createServer(function (socket) {
console.log('new connection');
// socket.pipe(socket); // replaced with socket.write on each 'data' event
socket.setEncoding('utf8');
socket.setKeepAlive(true);
socket.on("data", function (d) {
console.log("received: ", d);
var result = socket.write(d);
console.log(result ? 'write ok' : 'BACKPRESSURE');
});
socket.on('error', function (err) {
console.log('client error:', err);
});
socket.on('end', function () {
console.log('client disconnected');
});
});
server.listen(10777, '127.0.0.1', () => {
console.log('server listening...');
});
The client:
// ----------------------------------------
// client.js
var net = require('net');
var client = net.createConnection(10777, () => {
console.log('connected to server!' + new Date().toISOString());
var count = 1;
var date;
while(count < 35000) {
count++;
date = new Date().toISOString() + '_' + count;
console.log('sending: ', date);
client.write(date + '\n');
}
});
client.on('data', (data) => {
console.log('received:', data.toString());
});
client.on('end', () => {
console.log('disconnected from server');
});
What i tried to achieve with node.js/io.js, is to send a file from one server to another one via a proxy. To avoid memory buffering i want to use streams.
The proxy should be able to connect to multiple targets dynamically. The target connection information for the proxy should be send prior to the filedata.
With normal socket communication and buffering it is not a problem. But how or in general can this be done with streams??
var net = require('net');
var fs = require('fs');
//create readstream from file
var myFile = fs.createReadStream('E:/sample.tar.gz');
// Proxy server
//####################################################################################################
var proxy = net.createServer(function (socket) {
// Create a new connection to the TCP server
var client = net.connect('9010');
// 2-way pipe between client and TCP server
socket.pipe(client).pipe(socket);
}).listen(9000);
// Targetserver
//####################################################################################################
var server = net.createServer(function (socket) {
// create filestream to write data into file
var destfile = fs.createWriteStream('E:/sample_copy.tar.gz')
socket.on('data', function (buffer) {
console.log('Get data on targetserver...');
// write buffer to file
destfile.write(buffer);
});
socket.on('end', function () {
// release file from writestream
destfile.end();
});
}).listen(9010);
// Client
//####################################################################################################
// Send file to proxy
var client = new net.Socket();
// connect to proxy
client.connect('9000', '127.0.0.1', function () {
console.log('Connection to proxy opened');
});
// send data to proxy
myFile.pipe(client);
// read response from taget
client.on('data', function(data) {
console.log('Response: ' + data);
// close the client socket completely
client.destroy();
});
// Add a 'close' event handler for the client socket
client.on('close', function() {
console.log('Connection to proxy closed');
});
Any hint to a good tutorial is also welcome.
TMOE
socket.write() already uses streams under the hood so you don't need to do anything special. Just send it the usual Buffer object or string and it will use a stream.
From the current source code of io.js, here's what happens when you use socket.write():
Socket.prototype.write = function(chunk, encoding, cb) {
if (typeof chunk !== 'string' && !(chunk instanceof Buffer))
throw new TypeError('invalid data');
return stream.Duplex.prototype.write.apply(this, arguments);
};
And stream is declared like this:
const stream = require('stream');
Apologies if I've misunderstood your question/requirements! By all means, clarify if I have misunderstood you and I'll try again (or delete this answer so it's not a distraction).
My client sends an image file to the server. It works 5 times and then it suddenly stops. I am pretty new using streams and pipe so I am not sure what I am doing wrong.
Server Code
http.createServer(function(req, res) {
console.log("File received");
// This opens up the writeable stream to `output`
var name = "./test"+i+".jpg";
var writeStream = fs.createWriteStream(name);
// This pipes the POST data to the file
req.pipe(writeStream);
req.on('end', function () {
console.log("File saved");
i++;
});
// This is here incase any errors occur
writeStream.on('error', function (err) {
console.log(err);
});
}).listen(3000);
Client code
var request = require('request');
var fs = require('fs');
setInterval(function () {
var readStream = fs.createReadStream('./test.jpg');
readStream.on('open', function () {
// This just pipes the read stream to the response object (which goes to the client)
readStream.pipe(request.post('http://192.168.1.100:3000/test'));
console.log("Send file to server");
});
}, 1000);
Behaves like a resource exhaustion issue. Not sure which calls throw errors and which just return. Does the server connect on the 6th call? Does the write stream open? Does the pipe open?
Try ending the connection and closing the pipe after the image is saved. Maybe close the write stream too, don't remember if node garbage collects file descriptors.
I had to do the following on the server side to make this work :
res.statusCode = 200;
res.end();
My end goal is to stream video from my laptop to a server. I'm trying to accomplish this by using NodeJs on the laptop & the server. I use the OpenCV library to capture the video on the laptop and save it to a jpg file. I then read the file and convert it to base64 so that I can transport it using the Net.socket module in Node. This is a continous process: capture, encode, and send.
Here is the server code for just transmitting one jpg file:
var cv = require('opencv');
var fs = require('fs');
var net = require('net');
var camera = new cv.VideoCapture(0);
var server = net.createServer();
server.listen('50007', '127.0.0.1');
server.on('connection', function(socket){
camera.read(function(image){
image.save('original.jpg');
fs.readFile('original.jpg', 'base64', function(err, image){
socket.write(image, 'base64', function(){
socket.end();
});
});
});
});
On the client I loop until the FIN is received from the server. Here is the client code:
var net = require('net');
var fs = require('fs');
var client = new net.Socket();
var buffer ='';
client.setEncoding('base64');
client.connect('50007', '127.0.0.1', function(){
console.log('Connecting to server...');
});
client.on('data', function(data){
buffer += data;
});
client.on('end', function(){
var dataBuffer = new Buffer(buffer, 'base64');
fs.writeFile('copy.jpg', dataBuffer, function(err){
if(err){
console.log(err);
}
});
});
The problem is that the entire image does not actually get sent. When I open the received file, copy.jpg, there is always a chunk missing at the bottom.
In the final version the goal is to send one jpg after another and delimit the end of each 'jpg' via a keyword such as 'EndOfFile'. I tried to do this by appending the keyword 'EndOfFile' to my base64 encoded image before sending but on the receiving end that really got screwed up.
Sample Advanced Server:
fs.readFile('original.jpg', 'base64', function(err, image){
image += 'EndOfFile';
socket.write(image, 'base64');
});
One the client side the loop would examine each chunk of data for the keyword and if it found it then whatever is in the buffer would be written to file and the buffer reset, ready for the next file.
Sample Advanced Client
client.on('data', function(data){
if(data.indexOf('EndOfFile') > 0){
buffer += data.substr(0, data.indexOf('EndOfLine'));
var dataBuffer = new Buffer(buffer, 'base64');
fs.writeFile('copy.jpg', dataBuffer, function(err){
if(err){
console.log(err);
}
});
buffer = '';
} else {
buffer += data;
}
});
I've gotten this to work in Python so I think my logic is correct but I'm not as confortable in NodeJS.
If someone could tell me if this is a sane way to do this and where may I have gone wrong.
Thanks in advance!
I suspect you're seeing end event while the last bit of data is still buffered.
Try waiting for the close event rather than the end event. I'm not sure about sockets, but in other Node APIs like spawn, the end event is fired early, before related streams are flushed, so there may still be buffered data waiting.
You could avoid managing this yourself by piping. Use fs.createWriteStream() and .pipe() the socket stream to the file.