I am developing a web app and would like to stream file uploads straight to S3 (or Minio in my development environment). It doesn't look like koa-better-body or koa-body can stream, what might be a way I can do it?
You can access the original request in the context via ctx.request.req: https://github.com/koajs/koa/blob/master/lib/application.js#L153
With this you should be able to use express tools for what you need.
Related
I am having a curious problem where a download request from the browser will be passed onto a microservice via the Express layer that serves the frontend. The service queries the db, and generates a csv on the fly and can stream it.
Right now, I am fetching the file to my Express server, and then streaming it to the browser.
Is there a way I can directly pipe the file to the browser without exposing the micro-service endpoint ?
I need to upload a local file to s3 and save its link in the database. Right now I am converting the image to base64 and sending it to my rails server, which saves it on s3 and returns a url. I send this URL in the next HTTP request. Now, how about I save it via express get a link and then use it for the request. What be the better approach? Using middleware or backend server?
For file upload, i suggest you to use multer middleware, because native multipart implementation is a little bit tricky. For interaction with amazon s3 middleware is used.
To send file somewhere else you could use pipes:
fs.createReadStream(rqPath).pipe(res);
In above example, file is read from local system and piped to response.
All mentioned modules could be find at NPM
If you're still trying to figure this out, I was struggling with the same issue, decided to POST binary data (converted from base64) directly in body without dealing with multipart forms, and whipped up the base64-image-upload package to make this easy.
I have a fineuploader server set up which handles uploads from the browser using the fineuploader library. I can't seem to find anywhere if I can upload from node.js as well. Is this possible? Can I create a Blob and pass it to fineuploader?
Fine Uploader is a browser-based upload tool. It was never designed to work server-side. If you'd like to send requests from a Node server, there are a number of library focused on that particular workflow, such as the request library.
I am new to working with images in web development. We have a Node.js Express server that will run on Heroku and uses Cloudinary to store images.
Ideally we could save images directly to Cloudinary, but I am not sure if that's possible and we are afraid of putting our Cloudinary credentials on the client.
Assuming we must send images data to our server first instead of sending them directly to Cloudinary - if the images are encoded as base64 on the client, is it possible to stream the images from the client to the server - or must we send all the data at once? Either way, what headers do we use to send binary / base64 data?
is it possible to send or even stream binary data from the client to the server?
since it is a Node.js server, it would be ideal to use streams and to stream the file from our server to Cloudinary.
hope this makes sense, and info would be very helpful.
Why not using direct uploads from the client side to Cloudinary using the jQuery plugin?
This method supports both signed or unsigned uploads, when the signature can (and should) be generated on your server before rendering the page, for privacy reasons. Uploading from Base64 URI is also possible with this mechanism.
Note that Cloudinary's client-libraries also wrap this plugin and provide you with "off the shelf" solutions for embedding the upload-fields in your web app, with the signature already inside.
Let us know if you need any further guidance.
How do you send files on node.js/express.
I am using Rackspace Cloudfiles and wanna send images/videos to their remote storage but I am not sure that it's as simple as reading the file (fs.readFileSync()) and send the data in the request body, or is it?
What should the headers be.
What if it's a very large file on a couple of GBs?
Is it possible to use superagent (http://visionmedia.github.com/superagent) for this or is there a better library for sending files?
Please give me some information about this.
Thanks!
app.get('/img/bg.png', function(req, res) {
res.sendFile('public/img/background.png')
})
https://expressjs.com/en/api.html#res.sendFile
use "res.sendFile". "res.sendfile" is deprecated.
For large files, you will want to use node.js's concept of piping IO streams together. You want to open the local file for reading, start the HTTP request to rackspace, and then pipe the data events from the file read process to the HTTP request process.
Here's an article on how to do this.
Superagent is fine for small files, but because the superagent API presumes your entire request body is loaded into memory before starting the request, it's not the best approach for large file transfers.
Normally you won't need to worry specifically about the request headers as node's HTTP request library will send the appropriate headers for you. Just make sure you use whatever HTTP method your API requires (probably POST), and it looks like for rackspace you will need to add the X-Auth-Token extra header with your API token as well.
I am using Rackspace Cloudfiles and wanna send images/videos to their remote storage but I am not sure that it's as simple as reading the file (fs.readFileSync()) and send the data in the request body, or is it?
You should never use fs.readFileSync in general. When you use it, or any other method called somethingSync, you block the entire server for the duration of that call. The only acceptable time to make synchronous calls in a node.js program is during startup.
What should the headers be.
See RackSpace Cloud Files API.
Is it possible to use superagent (http://visionmedia.github.com/superagent) for this or is there a better library for sending files?
While I don't have any experience with superagent, I'm sure it will work fine. Just make sure you read the API documentation and make your requests according to their specification.