I'm trying to download a large XML file and parse it using xml-stream library. I'm using request to download the file and it's capable of streaming the content of the file. Ideally, I'd like to pass that stream directly to xml-stream and have it parsed. But I can't figure out how to connect those two.
Here is the code a have so far:
request('http://example.com/data.xml').pipe(fs.createWriteStream('data.xml'));
...
var stream = fs.createReadStream('data.xml');
var xml = new XmlStream(stream);
Is it possible to connect them directly without a temp data.xml file?
request() returns a Readable stream, so just pass that return value to XmlStream():
var xml = new XmlStream(request('http://example.com/data.xml'));
Related
I am using Gridfs to store large files in mognodb.
Now I am using PDFTron for pdf editing and want to watermark pdf.
The problem is i am not able to read file from Gridfs stream in pdftron nodejs sdk.
also i want to send it back to the client without storing it locally or anywhere else.
I am doing something like this...
const bucket = new mongodb.GridFSBucket(db);
const stream = bucket.openDownloadStream(ObjectId(file_id))
const pdfdoc = await PDFNet.PDFDoc.createFromFilter(stream);
the error i am getting is ...
TypeError: 1st input argument in function 'createFromFilter' is of type 'object'. Expected type 'Filter'. Function Signature: createFromFilter(PDFNet.Filter)
The PDFDoc.createFromFilter API is expecting a PDFNet Filter, not whatever GridFS is returning.
https://www.pdftron.com/api/pdfnet-node/PDFNet.PDFDoc.html#.createFromFilter__anchor
You can see this sample on creating a PDFDoc object from a Filter
https://www.pdftron.com/documentation/samples/node/js/PDFDocMemoryTest
Though the easiest is to write your GridFD stream to a buffer, and then pass that buffer to PDFDoc.createFromBuffer. https://www.pdftron.com/api/pdfnet-node/PDFNet.PDFDoc.html#.createFromBuffer__anchor
I'm trying to create a Read Stream from a remote file without writing it to disc.
var file = fs.createWriteStream('Video.mp4');
var request = http.get('http://url.tld/video.mp4', function(response){
response.pipe(file);
});
Can I create a Read Stream directly from an HTTP response without writing it to disc ? Maybe creating a buffer in chunks and covert it to readable stream ?
Seems like you can use the request module.
Have a look at 7zark7's answer here: https://stackoverflow.com/a/14552721/7189461
I am parsing a JSON file using a parsing stream module and would like to stream results to request like this:
var request = require("request")
fs.createReadStream('./data.json')
.pipe(parser)
.pipe(streamer)
.pipe(new KeyFilter({find:"URL"}) )
.pipe(request)
.pipe( etc ... )
(Note: KeyFilter is a custom transform that works fine when piping to process.stdout)
I've read the docs and source code. This won't work with 'request' or 'new request()' because the constructor wants a URL.
It will work with request.put() as this : yourStream.pipe(request.put('http://example.com/form'))
After more research and experimenting I've concluded that request cannot be used in this way. The simple answer is that request creates a readable stream and .pipe() methods expects a writable stream.
I tried several attempts to wrap request in a transform to get by this with no luck. While you can receive the piped url and create a new request, I can't figure out how to reset the pipe callbacks without some truly unnatural bending of the stream pattern. Any thoughts would be appreciated, but I have moved on to using an event in the url stream to kick off a new request(url).pipe(etc) type stream.
I would like to store some documents in a database as base64 strings. Then when those docs are requested using HTTP, I would like ExpressJS to decode the base64 docs and return them. So something like this:
app.get('/base64', function (req, res) {
//pdf is my base64 encoded string that represents a document
var buffer = new Buffer(pdf, 'base64');
res.send(buffer);
});
The code is simply to give an idea of what I'm trying to accomplish. Do I need to use a stream for this? If so, how would I do that? Or should I be writing these docs to a temp directory and then serving up the file? Would be nice to skip that step if possible. Thanks!
UPDATE: Just to be clear I would like this to work with a typical HTTP request. So the user will click a link in his browser that will take him to a URL that returns a file from the database. Seems like it must be possible, Microsoft SharePoint stores serialized files in a SQL database and returns those files over http requests, and I don't believe it writes all those files to a temp location first. I'm feeling like a nodejs stream may be the answer, but I'm not very familiar with streaming.
Before saving a file representation to the DB you can just use the toString method with base 64 encoding:
var base64pdf = pdf.toString('base64');
After you get the base64 file representation from db use the buffer as follows in order to convert it back to a file:
var decodedFile = new Buffer(base64pdf, 'base64');
More information on Buffer usages can be found here - NodeJS Buffer
As for how to send a buffer from express server to the client, Socket IO should solve this issue.
Using socket.emit -
Emits an event to the socket identified by the string name. Any
other parameters can be included.
All datastructures are supported, including Buffer. JavaScript
functions can’t be serialized/deserialized.
var io = require('socket.io')();
io.on('connection', function(socket){
socket.emit('an event', { some: 'data' });
});
Required documentation on socket.io website.
I have a Buffer with tar contents and I would like to parse this with tar node.js package, but all examples of use of this package are using .pipe() and streams - how can I pass buffer to tar package?
I get Buffer from AWS SDK.
Streams have a standard API, so you can just use write and end directly instead of using pipe.
var data = ... // Buffer
var parser = tar.Parse();
// Bind whatever handlers you would normally bind
parser.on('entry', function(e){
});
// Write the data into the parser, which will parse it and trigger your event handlers
parser.end(data);