Clients can upload files by using a Multi-part form post to my node.js application. To handle the file upload I'm using the node-formidable library.
Now, I manage to upload the file in chunks to node but I don't want it to be buffered before it's written to disk. So, I'm trying to understand how I can write the file data chunks to disk when they're received. I don't fully understand the node-formidable api how to acheive.
Can someone give a simple example of how to listen for an incoming file, create a file stream and then access the data coming in and write that data to the stream and finally closing the stream.
Thanks for help!
There is one example here. It doesn't use formidable library but a similar module (multipart). But it explains everything that you have asked for. You can massage it to fit you needs.
Thanks,
KA
Related
I have a problem with developing a crawler using nodejs/puppeteer. The old crawler was:
Crawl Pages
Store the output file locally with the fs module
Since i'm going to introduce UI on the server, have set up the scenario to upload it to S3 instead of storing it locally, and show the result as a UI.
Crawl Pages
Stream output files to the server with the fs module
Get the output file back and upload it to the S3 bucket
The above is a scenario that i know of as a knowledge, and i'd like to know if it is possible as below.
Crawl Pages
Upload the stream stored data to memory to the S3 bucket
If you have a scenario like this, I would like to receive a guide. I would really appreciate if you would comment or reply :)
This is definitely possible. If you just pipe from your input stream to your server and pipe up to the S3 it should complete the loop.
This is possible because you can stream uploads to S3, even without knowing the size of the file beforehand.
This answer should help you out: S3 file upload stream using node js
If you post some code we can answer this a little bit better. But hope this puts you in the right direction.
I am trying to upload a pdf file from client to my server.
I know how to read a file using node js "fs" module but how to read a file which is not on my server (i.e is on client disk). There is a upload button which chooses the file and then from client side I want to send that file in stream to my server. And then I can write the stream into a file on my server.
How are the packages like ostrio:Files doing it?
Maybe this could help you https://github.com/VeliovGroup/Meteor-Files
Great package to ease the upload and storage of files :)
Hope it helps,
Regards,
Yann
I have a need to create a pdf or html document within a Node.js express API which then sends that document over HTTP to an API managing our CMS.
So functionally I would like to create the document and POST it as part of a multipart-form upload POST request to an external service.
I see how to do this if after I create the file, I then turn around and write it disk. After that point I can do a read stream of the file from that path to format the POST request with the file.
However I'm wondering how I can perform this action without writing the file to disk and then reading it into a read stream. It seems I should be able to accomplish this without that IO.
Anybody able to point me to a good example or library that does something along these lines?
You can extend Writable and/or Readable streams. By the first look this library do what you need, with the same way - extending built-in streams.
I'm trying to save the images from a third-party API to my own S3 bucket using Node.js and MongoDB. The API provides a URL to the image on the third-party servers. I've never done this before but I'm assuming I have to download the image to my own server and then upload it to S3?
Should I save the image to mongodb with GridFS and then delete it once it is on S3? If so, what's the best way to do that?
I've read and re-read this previous question:
Problem with MongoDB GridFS Saving Files with Node.JS
But I can't find good documentation on how I should determine buffer/chunk size and other attributes for a JPEG image file.
Once I've saved the file on my server/database, then it seems like I should use:
https://github.com/appsattic/node-awssum
To upload it to S3. Is that a good idea?
I apologize if this is an obvious answer, I'm pretty green when it comes to databases and server scripting.
The easiest thing to do would be to save the image onto disk and then stream the upload from there using AwsSum's S3 PutObject operation. Alternatively, if you have the file contents in a Buffer you can just use that.
Have a look at the following two examples and they should help you figure out what to do:
https://github.com/appsattic/node-awssum/blob/master/examples/amazon/s3/put-bucket.js
https://github.com/appsattic/node-awssum/blob/master/examples/amazon/s3/put-object-streaming.js
Let me know if you need any more help. :)
Cheers,
Andy
Disclaimer: I'm the author of AwsSum.
Is there a way to stream directly to a remote server or fileshare when handling Node.js file uploads? I'm using formidable and currently I am trying to enable this scenario, but it always ends up writing to a temp file on my server first, which in my mind is unnecessary if the ultimate destination is a remote server.
I also tried connect-form which is built on top of formidable, but didn't find any good documentation for onPart/handlePart, which I assume allows me to do this. Does anybody have a pointer to a good example which enables this scenario?
Turns out the Formidable team is working on making this scenario much easier with Node.js streams. When the issue is fixed this will become much easier.
https://github.com/felixge/node-formidable/issues/61
Not the stream, but you can get the string contents of the uploaded file (it will be stored at req.body.input_name) this way:
app.use(express.bodyParser({keepExtensions: true, _fileName: function() { }}));