I am trying to figure out the best way to read from a piped stream. Basically, I have a file on my computer that I would like to read from, pipe it through a crypto cipher and then upload it to an endpoint. I'm using the form-data package which helps create mutli-part form requests by accepting a ReadableStream. But I'm not sure the best way to pipe the readable stream to the cipher and still give the package a readable stream from that pipe without first writing that readable stream to a file and then reading from that file.
Current Code:
let myStream = fs.createReadStream('./myfile.txt'),
form = new FormData();
form.append('contents', myStream);
Hopefully this makes sense. Let me know if any clarification is needed.
Take a look at Transform Class of the Node js Stream API.
The main thing here is understanding that transform stream is kind of a duplex, that is both readable and writable.
Basically, your "streams-scheme" will look like:
read from file -> transform with cipher -> pipe to form-data
Related
I have an application that streams data to a file, can I use Node.js to read the file while it's being streamed to?
I tried using createReadStrem, but it only read one chunk and the stream ended
You could try watching for file changes with fs.watchFile(filename[, options], listener) or node-watch. In file change you could just read last line with read-last-lines.
Although I'm not sure how efficient it would be.
I have a situation where I receive a base64 encoded image, decode it, then want to use it in some analysis activity.
I can use Buffer to go from base64 to binary but i seem to be unable to use that output as expected (as an image).
The solution now is to convert to binary, write it to a file, then read that file again. The FS output can be used as an image but this approach seems a bit inefficient and additional steps as i would expect the buffer output to also be a usable image as it has the same data?
my question, is how does the fs.readfile output differ from the buffer output? And is there a way i can use the buffer output as i would the fs output?
Buffer from a base64 string:
var bin = new Buffer(base64String, 'base64').toString('binary');
Read a file
var bin = fs.readFileSync('image.jpg');
Many thanks
Well I have a buffer inside mongoDB, and I want to send it like an image.
I tried many different things like res.send(buffer) or fs.createReadStream(buffer).pipe(res) and others. I really don't want to write this to real file in the server and then send it.
How can I send the buffer to the client without needing to write it to a file first?
You could encode the buffer/image to base64 or whatever encoding you want and send that.
var encodedBuffer = buffer.toString('base64');
res.send(encodedBuffer);
I'm trying to read the contents of some .gz log files using streams in node.
I've started simply with: fs.createReadStream('log.gz').pipe(zlib.createUnzip().
This works and I can pipe to process.stdout to verify. I'd like to pipe to this a new writeableStream, that I can have a data event to actually work with the contents. I guess I just don't fully understand how the streams work. I tried just creating a new writable stream, var writer = fs.createWriteStream() but this doesn't work because it requires a path.
Any ideas how I can go about doing this (without creating any other files to write to)?
var unzipStream = zlib.createUnzip()
unzipStream.on('data', myDataHandler).on('end', myEndHandler)
fs.createReadStream('log.gz').pipe(unzipStream)
That will get you the data and end events. Since it's log data you may also find split useful to get events line by line.
I am using jazzlib package in j2me application to compress the xml file in zip format using ZipOutputStream and the send the compress stream to the server as a string . I am able to do unzip the in mobile using ZipInputStream. But in server i am not able to unzip , i got
EOF exception. when i copy the compressed stream from console and put into browser, the empty space put special character like [] in compressed stream. I didnt understand what happened. Plz help
You send the compressed stream as a String? That's your problem(s) right there:
compressed data is binary data (i.e. byte[]).
String is designed to handle textual (Unicode) data and not arbitrary binary data
converting arbitrary binary data to String is bound to lead to problems
So if you want to handle (send/receive/...) binary data, make sure you never use a String/Reader/Writer to handle the data anywhere in the process. Stay with byte[]/InputStream/OutputStream.