I am trying to send a (huge) file with a limited amount of data passing every second (using TooTallNate/node-throttle):
var fs = require('fs');
var Throttle = require('throttle');
var throttle = new Throttle(64);
throttle.on('data', function(data){
console.log('send', data.length);
res.write(data);
});
throttle.on('end', function() {
console.log('error',arguments);
res.end();
});
var stream = fs.createReadStream(filePath).pipe(throttle);
If I cancel the download at the clients browser, the stream will just continue until it completly transferred.
I also tested the scenario above with npm node-throttled-stream, same behavour.
How to cancel the stream if the browser closed his request?
Edit:
I am able to obtain the connections close event by using
req.connection.on('close',function(){});
But the stream has neither a destroy nor an end or stop property which I could use to stop the stream from further reading.
I does provide the property pause Doc, but I would rather stop node from reading the whole file than just stopping to recieve the contents (as described in the doc).
I ended up using the following dirty workaround:
var aborted = false;
stream.on('data', function(chunk){
if(aborted) return res.end();
// stream contents
});
req.connection.on('close',function(){
aborted = true;
res.end();
});
As mentioned above, this isn't really a nice solution, but it works.
Any other solution would be highly appreciated!
Related
My node.js script:
var net = require('net');
var fs = require('fs');
var client = new net.Socket()
client.connect(PORT, HOST, function () {
client.write('You can Send file\0');
client.on('data', function (data) {
// console.log(data);
var destinationFile = fs.createWriteStream("destination.txt");
destinationFile.write(data);
});
});
it is coded for receiving a file from remote HOST.
when i use console.log(data) its OK to log remote data file to console.
but for writing data to file its write one part of received data file.
How can write all data to file?
thanks
The cause of what you get:
client.on('data') is called multiple times, as the file is being sent in data-chunks, not as a single whole data. As a result on receiving each piece of data, you create a new file stream and write to it..
Console.log on the other hand works, because it does not create new console window each time you write to it.
Quick solution would be:
client.connect(PORT, HOST, function () {
var destinationFile = fs.createWriteStream("destination.txt");
client.write('You can Send file\0');
client.on('data', function (data) {
// console.log(data);
destinationFile.write(data);
});
});
Also notice in the net Documentation on net.connect method:
Normally this method is not needed, as net.createConnection opens the
socket. Use this only if you are implementing a custom Socket.
The problem you got is because you declare the filestream inside the event on, which is called each time a packet arrive.
client.connect(PORT, HOST, function () {
var destinationFile = fs.createWriteStream("destination.txt"); //Should be here
client.write('You can Send file\0');
client.on('data', function (data) {
destinationFile.write(data);
});
});
thanks. in tiny test its OK. but written file has 16 bit more than real size. first of file has 16 bit of unknown data. how to ignore that?
In your code, since client.on is called multiple time, multiple write happens in the same times, so it is undefined behavior for which write happens, in which order, or if they ll be complete.
The 16 bytes of unknown data are probably bytes from different packet writen one after the other in random order. Your tiny test work because the file can be send in one packet, so the event is called only once.
If you declare the filestream first, then call the write in client.on, the order of write and data are preserved, and the file is written successfully.
I am trying to work with the new Streams API in Node.js, but having troubles when specifying a fixed read buffer size.
var http = require('http');
var req = http.get('http://143.226.75.100/waug_mp3_128k', function (res) {
res.on('readable', function () {
var receiveBuffer = res.read(1024);
console.log(receiveBuffer.length);
});
});
This code will receive a few buffers and then exit. However, if I add this line after the console.log() line:
res.read(0);
... all is well again. My program continues to stream as predicted.
Why is this happening? How can I fix it?
It's explained here.
As far as I understand it, by reading only 1024 bytes with each readable event, Node is left to assume that you're not interested in the rest of the data that's in the stream buffers, and discards it. Issuing the read(0) (in the same event loop iteration) 'resets' this behaviour. I'm not sure why the process exits after reading a couple of 1024-byte buffers though; I can recreate it, but I don't understand it yet :)
If you don't have a specific reason to use the 1024-byte reads, just read the entire buffer for each event:
var receiveBuffer = res.read();
Or instead of using non-flowing mode, use flowing mode by using the data/end events instead:
var http = require('http');
var req = http.get('http://143.226.75.100/waug_mp3_128k', function (res) {
var chunks = [];
res.on('data', function(chunk) {
chunks.push(chunk);
console.log('chunk:', chunk.length);
});
res.on('end', function() {
var result = Buffer.concat(chunks);
console.log('final result:', result.length);
});
});
How to close a readable stream in Node.js?
var input = fs.createReadStream('lines.txt');
input.on('data', function(data) {
// after closing the stream, this will not
// be called again
if (gotFirstLine) {
// close this stream and continue the
// instructions from this if
console.log("Closed.");
}
});
This would be better than:
input.on('data', function(data) {
if (isEnded) { return; }
if (gotFirstLine) {
isEnded = true;
console.log("Closed.");
}
});
But this would not stop the reading process...
Edit: Good news! Starting with Node.js 8.0.0 readable.destroy is officially available: https://nodejs.org/api/stream.html#stream_readable_destroy_error
ReadStream.destroy
You can call the ReadStream.destroy function at any time.
var fs = require("fs");
var readStream = fs.createReadStream("lines.txt");
readStream
.on("data", function (chunk) {
console.log(chunk);
readStream.destroy();
})
.on("end", function () {
// This may not been called since we are destroying the stream
// the first time "data" event is received
console.log("All the data in the file has been read");
})
.on("close", function (err) {
console.log("Stream has been destroyed and file has been closed");
});
The public function ReadStream.destroy is not documented (Node.js v0.12.2) but you can have a look at the source code on GitHub (Oct 5, 2012 commit).
The destroy function internally mark the ReadStream instance as destroyed and calls the close function to release the file.
You can listen to the close event to know exactly when the file is closed. The end event will not fire unless the data is completely consumed.
Note that the destroy (and the close) functions are specific to fs.ReadStream. There are not part of the generic stream.readable "interface".
Invoke input.close(). It's not in the docs, but
https://github.com/joyent/node/blob/cfcb1de130867197cbc9c6012b7e84e08e53d032/lib/fs.js#L1597-L1620
clearly does the job :) It actually does something similar to your isEnded.
EDIT 2015-Apr-19 Based on comments below, and to clarify and update:
This suggestion is a hack, and is not documented.
Though for looking at the current lib/fs.js it still works >1.5yrs later.
I agree with the comment below about calling destroy() being preferable.
As correctly stated below this works for fs ReadStreams's, not on a generic Readable
As for a generic solution: it doesn't appear as if there is one, at least from my understanding of the documentation and from a quick look at _stream_readable.js.
My proposal would be put your readable stream in paused mode, at least preventing further processing in your upstream data source. Don't forget to unpipe() and remove all data event listeners so that pause() actually pauses, as mentioned in the docs
Today, in Node 10
readableStream.destroy()
is the official way to close a readable stream
see https://nodejs.org/api/stream.html#stream_readable_destroy_error
You can't. There is no documented way to close/shutdown/abort/destroy a generic Readable stream as of Node 5.3.0. This is a limitation of the Node stream architecture.
As other answers here have explained, there are undocumented hacks for specific implementations of Readable provided by Node, such as fs.ReadStream. These are not generic solutions for any Readable though.
If someone can prove me wrong here, please do. I would like to be able to do what I'm saying is impossible, and would be delighted to be corrected.
EDIT: Here was my workaround: implement .destroy() for my pipeline though a complex series of unpipe() calls. And after all that complexity, it doesn't work properly in all cases.
EDIT: Node v8.0.0 added a destroy() api for Readable streams.
At version 4.*.* pushing a null value into the stream will trigger a EOF signal.
From the nodejs docs
If a value other than null is passed, The push() method adds a chunk of data into the queue for subsequent stream processors to consume. If null is passed, it signals the end of the stream (EOF), after which no more data can be written.
This worked for me after trying numerous other options on this page.
This destroy module is meant to ensure a stream gets destroyed, handling different APIs and Node.js bugs. Right now is one of the best choice.
NB. From Node 10 you can use the .destroy method without further dependencies.
You can clear and close the stream with yourstream.resume(), which will dump everything on the stream and eventually close it.
From the official docs:
readable.resume():
Return: this
This method will cause the readable stream to resume emitting 'data' events.
This method will switch the stream into flowing mode. If you do not want to consume the data from a stream, but you do want to get to its 'end' event, you can call stream.resume() to open the flow of data.
var readable = getReadableStreamSomehow();
readable.resume();
readable.on('end', () => {
console.log('got to the end, but did not read anything');
});
It's an old question but I too was looking for the answer and found the best one for my implementation. Both end and close events get emitted so I think this is the cleanest solution.
This will do the trick in node 4.4.* (stable version at the time of writing):
var input = fs.createReadStream('lines.txt');
input.on('data', function(data) {
if (gotFirstLine) {
this.end(); // Simple isn't it?
console.log("Closed.");
}
});
For a very detailed explanation see:
http://www.bennadel.com/blog/2692-you-have-to-explicitly-end-streams-after-pipes-break-in-node-js.htm
This code here will do the trick nicely:
function closeReadStream(stream) {
if (!stream) return;
if (stream.close) stream.close();
else if (stream.destroy) stream.destroy();
}
writeStream.end() is the go-to way to close a writeStream...
for stop callback execution after some call,
you have to use process.kill with particular processID
const csv = require('csv-parser');
const fs = require('fs');
const filepath = "./demo.csv"
let readStream = fs.createReadStream(filepath, {
autoClose: true,
});
let MAX_LINE = 0;
readStream.on('error', (e) => {
console.log(e);
console.log("error");
})
.pipe(csv())
.on('data', (row) => {
if (MAX_LINE == 2) {
process.kill(process.pid, 'SIGTERM')
}
// console.log("not 2");
MAX_LINE++
console.log(row);
})
.on('end', () => {
// handle end of CSV
console.log("read done");
}).on("close", function () {
console.log("closed");
})
In my node application im writing data to the file using write method in the createWriteStream method.Now i need to find whether the write for the particular stream is complete or not.How can i find that.
var stream = fs.createWriteStream('myFile.txt', {flags: 'a'});
var result = stream.write(data);
writeToStream();
function writeToStream() {
var result = stream.write(data + '\n');
if (!result) {
stream.once('drain',writeToStream());
}
}
I need to call other method for every time when write completes.How can i do this.
From the node.js WritableStream.write(...) documentation you can give the "write" method a callback that is called when the written data is flushed:
var stream = fs.createWriteStream('myFile.txt', {flags: 'a'});
var data = "Hello, World!\n";
stream.write(data, function() {
// Now the data has been written.
});
Note that you probably don't need to actually wait for each call to "write" to complete before queueing the next call. Even if the "write" method returns false you can still call subsequent writes and node will buffer the pending write requests into memory.
I am using maerics's answer along with error handling. The flag 'a' is used to Open file for appending. The file is created if it does not exist. There Other flags you can use.
// Create a writable stream & Write the data to stream with encoding to be utf8
var writerStream = fs.createWriteStream('MockData/output.txt',{flags: 'a'})
.on('finish', function() {
console.log("Write Finish.");
})
.on('error', function(err){
console.log(err.stack);
});
writerStream.write(outPutData,function() {
// Now the data has been written.
console.log("Write completed.");
});
// Mark the end of file
writerStream.end();
I've read about nodejs event-loop, but what i don't understand well is:
i've created a simple http-server that logs the whole request post-data to a file .
i used apache-ab to flood it with a 700 kb file for-each request it does .
i imagined that each request will write some chunks after each other each tick in the event-loop, but i found that the full post data is written completely after each request and i don't know why , and i cannot understand it .
i'm using something like this
stream = require('fs').createWriteStream('path/to/log.file', {flags: 'a'})
log = function(data){
return stream.write(data)
}
require('http').createServer(function(req, res)
{
// or req.pipe(stream)
req.on('data', function(chunk){
log(chunk.toString() + "\r\n")
})
req.on('end', function(){
res.end("ok")
})
}).listen(8000)
sorry for my bad English :)
I edited the code to output the chunk size and also use an easily identifiable word 'SQUIRREL' to search for in the log file. I also sent the image file using curl instead of apache ab, mainly because I do not have it setup.
If you look at the output of http in the terminal you are running it, you will see the chunk size for each chunk as it is processed, which in my case was 65536 (as you alluded to in your comments we should see). In a text search, I see the word SQUIRREL one time for each chunk that was processed.
There is nothing wrong with your code, hopefully making these changes will allow you to see what you are expecting to see.
stream = require('fs').createWriteStream('./logfile.txt', {flags: 'a'});
log = function(data){
return stream.write(data);
};
require('http').createServer(function(req, res)
{
// or req.pipe(stream)
req.on('data', function(chunk){
log(chunk.toString() + "SQUIRREL ");
console.log(chunk.length);
})
req.on('end', function(){
res.end("ok");
})
}).listen(8000)
curl --data-binary "#IMAG0152.jpg" http://localhost:8000
thanks to everyone tried to help, i found the main issue
the stream.write was returning false i've added the following code to use drain event with .pause() and .resume() and the
problem solved
ok = stream.write(chunk)
if ( ! ok )
{
req.pause()
stream.once('drain', function(){
req.resume()
stream.write(chunk)
})
}