I'm just starting learning about streams in node. I have a string in memory and I want to put it in a stream that applies a transformation and pipe it through to process.stdout. Here is my attempt to do it:
var through = require('through');
var stream = through(function write(data) {
this.push(data.toUpperCase());
});
stream.push('asdf');
stream.pipe(process.stdout);
stream.end();
It does not work. When I run the script on the cli via node, nothing is sent to stdout and no errors are thrown. A few questions I have:
If you have a value in memory that you want to put into a stream, what is the best way to do it?
What is the difference between push and queue?
Does it matter if I call end() before or after calling pipe()?
Is end() equivalent to push(null)?
Thanks!
Just use the vanilla stream API
var Transform = require("stream").Transform;
// create a new Transform stream
var stream = new Transform({
decodeStrings: false,
encoding: "ascii"
});
// implement the _transform method
stream._transform = function _transform(str, enc, done) {
this.push(str.toUpperCase() + "\n";
done();
};
// connect to stdout
stream.pipe(process.stdout);
// write some stuff to the stream
stream.write("hello!");
stream.write("world!");
// output
// HELLO!
// WORLD!
Or you can build your own stream constructor. This is really the way the stream API is intended to be used
var Transform = require("stream").Transform;
function MyStream() {
// call Transform constructor with `this` context
// {decodeStrings: false} keeps data as `string` type instead of `Buffer`
// {encoding: "ascii"} sets the encoding for our strings
Transform.call(this, {decodeStrings: false, encoding: "ascii"});
// our function to do "work"
function _transform(str, encoding, done) {
this.push(str.toUpperCase() + "\n");
done();
}
// export our function
this._transform = _transform;
}
// extend the Transform.prototype to your constructor
MyStream.prototype = Object.create(Transform.prototype, {
constructor: {
value: MyStream
}
});
Now use it like this
// instantiate
var a = new MyStream();
// pipe to a destination
a.pipe(process.stdout);
// write data
a.write("hello!");
a.write("world!");
Output
HELLO!
WORLD!
Some other notes about .push vs .write.
.write(str) adds data to the writable buffer. It is meant to be called externally. If you think of a stream like a duplex file handle, it's just like fwrite, only buffered.
.push(str) adds data to the readable buffer. It is only intended to be called from within our stream.
.push(str) can be called many times. Watch what happens if we change our function to
function _transform(str, encoding, done) {
this.push(str.toUpperCase());
this.push(str.toUpperCase());
this.push(str.toUpperCase() + "\n");
done();
}
Output
HELLO!HELLO!HELLO!
WORLD!WORLD!WORLD!
First, you want to use write(), not push(). write() puts data in to the stream, push() pushes data out of the stream; you only use push() when implementing your own Readable, Duplex, or Transform streams.
Second, you'll only want to write() data to the stream after you've setup the pipe() (or added some event listeners). If you write to a stream with nothing wired to the other end, the data you've written will be lost. As #naomik pointed out, this isn't true in general since a Writable stream will buffer write()s. In your example you do need to write() after pipe() though. Otherwise, the process will end before writing anything to STDOUT. This is possibly due to how the through module is implemented, but I don't know that for sure.
So, with that in mind, you can make a couple simple changes to your example to get it to work:
var through = require('through');
var stream = through(function write(data) {
this.push(data.toUpperCase());
});
stream.pipe(process.stdout);
stream.write('asdf');
stream.end();
Now, for your questions:
The easiest way to get data from memory in to a writable stream is to simply write() it, just like we're doing with stream.wrtie('asdf') in your example.
As far as I know, the stream doesn't have a queue() function, did you mean write()? Like I said above, write() is used to put data in to a stream, push() is used to push data out of the stream. Only call push() in your owns stream implementations.
Only call end() after all your data has been written to your stream. end() basically says: "Ok, I'm done now. Please finish what you're doing and close the stream."
push(null) is pretty much equivalent to end(). That being said, don't call push(null) unless you're doing it inside your own stream implementation (as stated above). It's almost always more appropriate to call end().
Based on the examples for stream (http://nodejs.org/api/stream.html#stream_readable_pipe_destination_options)
and through (https://www.npmjs.org/package/through)
it doesn't look like you are using your stream correctly... What happens if you use write(...) instead of push(...)?
Related
I'm really just looking for clarification on how these work. IMO the documentation on streams is somewhat lacking, and there actually aren't a lot of resources out their that comprehensively explain explain how they're are meant to work and be extended.
My question can be broken down into two parts
One, What is the role of the _read function within the stream module? When I run this code it endlessly prints out "hello world" until null is pushed onto the stream buffer. This seems to indicate that _read is called in some kind of loop that waits for a null in the buffer, but I can't find documentation anywhere that states this in explicit terms.
var Readable = require('stream').Readable
var rs = Readable()
rs._read = function () {
rs.push("hello world")
rs.push(null)
};
rs.on("data", function(data){
console.log("some data", data)
})
Two, what does read actually do? My understanding is that read consumes data from the read stream buffer, and fires the data event. Is that all that's going on here?
read() is something that a consumer of the readStream calls if they want to specifically read some bytes from the stream (when the stream is not flowing).
_read() is an internal method that is part of the internal implementation of the read stream. The internals of the stream call this method (it is NOT to be called from the outside) when the stream is flowing and the stream wants to get more data from the source. When called the _read() method pushes data with .push(data) or if it has no more data, then it does a .push(null).
You can see an explanation and example here in this article.
_read(size) {
if (this.data.length) {
const chunk = this.data.slice(0, size);
this.data = this.data.slice(size, this.data.length);
this.push(chunk);
} else {
this.push(null); // 'end', no more data
}
}
If you were implementing a read stream to some custom source of data, then you would implement the _read() method to fetch up to the size amount of data from your source and .push() that data into the stream.
I'm having trouble understanding, why the following works, that is, why the invocations of the read() function actually return the objects stored in the readable stream.
const { Readable } = require('stream')
var r = new Readable({objectMode: true, read: () => {}}) // dummy read
var a = [1,2,3,4,5,6,7]
r.push(...a)
Now, when I invoke r.read() I get the numbers I pushed into my readable stream r
r.read() // -> 1
r.read() // -> 2
// etc
But I provided a "dummy" read function (read: () => {}) above when creating my readable stream. So, why do I get values back, when calling read?
Help will be much appreciated.
The answer is simple. You're calling the push method which should be called by your read implementation.
The purpose of push is to say: here's what I've read from the source, but it doesn't have to be called from within the internal methods.
In other words in the process:
wait for _read to be called
_read something from source
push the read chunks to stream
return the chunks from read
You simply skipped the two first steps and pushed the data from outside.
I have two PCM-streams (decoder1 + decoder2):
var readable1 = fs.createReadStream("track1.mp3");
var decoder1 = new lame.Decoder({
channels: 2,
mode: lame.STEREO
});
readable1.pipe(decoder1);
and
var readable2 = fs.createReadStream("track2.mp3");
var decoder2 = new lame.Decoder({
channels: 2,
mode: lame.STEREO
});
readable2.pipe(decoder2);
Now I want to pipe the streams into one mix-function, where I can use the buffer-function like:
function mixStream(buf1, buf2, callback) {
// The mixStream-Function is not implemented yet (dummy)
var out = new Buffer(buf1.length);
for (i = 0; i < buf1.length; i+=2) {
var uint = Math.floor(.5 * buf1.readInt16LE(i));
out.writeInt16LE(uint, i);
}
this.push(out);
callback();
}
I need something like
mixStream(decoder1.pipe(), decoder2.pipe(), function() { }).pipe(new Speaker());
for output to speaker. Is this possible?
Well, pipe() function actually means a stream is linked to another, a readable to a writable, for instance. This 'linking' process is to write() to the writable stream once any data chunk is ready on the readable stream, along with a little more complex logic like pause() and resume(), to deal with the backpressure.
So all you have to do is to create a pipe-like function, to process two readable streams at the same time, which drains data from stream1 and stream2, and once the data is ready, write them to the destination writable stream.
I'd strongly recommend you to go through Node.js docs for Stream.
Hope this is what you are looking for :)
I try to catch the completion of writing the canvas stream thusly:
var out = fs.createWriteStream(out_fs);
var stream = canvas.createPNGStream({
bufsize: 2048
});
stream.on('end', function () {
// can we use out_fs now? why not?
});
stream.pipe(out);
But when I try to load out_fs in sub function
Error: Image given has not completed loading
at this line:
fs.readFile(out_fs, function (err, data) {
if (err) throw err;
var img = new Canvas.Image; // Create a new Image
img.src = data;
ctx2.drawImage(img, 0, 50, img.width, img.height); <--
http://nodejs.org/api/stream.html#stream_event_end
But I don't see any other way to continue with the control flow after the stream is written. If I let the entire parent function return, the file then seems readable. I've tried wrapping my child functions in setImmediate(), but that only seems to work intermittently.
What is the definitive way to catch the final usable end result of writing the stream?
The node-canvas documentation claims that the end event signals the final writing of the file: https://www.npmjs.com/package/canvas#canvaspngstream
But this generates the error above if you immediately try to use it.
`finish' does not seem to be implemented at all.
Since you have piped stream to out, out will be close()'d automatically on stream's end event (this is part of what gets setup automatically when you .pipe() a stream). So, to know when file is finished being written, listen to the close event of out stream.
You saw intermittent results because the stream end event is the same event that will be used by out writable stream to finalize the file.
I would put this in a comment (but can't):
You need to close your WriteStream called 'out' - use the event aarosil suggests and do out.close()
I'm attempting to create a Readable file stream that I can read individual bytes from. I'm using the code below.
var rs = fs.createReadStream(file).on('open', function() {
var buff = rs.read(8); //Read first 8 bytes
console.log(buff);
});
Given that file is an existing file of at least 8 bytes, why am I getting 'null' as the output for this?
Event open means that stream has been initialized, it does not mean you can read from the stream. You would have to listen for either readable or data events.
var rs = fs.createReadStream(file);
rs.once('readable', function() {
var buff = rs.read(8); //Read first 8 bytes only once
console.log(buff.toString());
});
It looks like you're calling this rs.read() method. However, that method is only available in the Streams interface. In the Streams interface, you're looking for the 'data' event and not the 'open' event.
That stated, the docs actually recommend against doing this. Instead you should probably be handling chunks at a time if you want to stream them:
var rs = fs.createReadStream('test.txt');
rs.on('data', function(chunk) {
console.log(chunk);
});
If you want to read just a specific portion of a file, you may want to look at fs.open() and fs.read() which are lower level.