Using Twitter as an example: Twitter has an endpoint for uploading file data. https://developer.twitter.com/en/docs/media/upload-media/api-reference/post-media-upload-append
Can anyone provide an example of a real HTTP message containing, for example, image file data, showing how it is supposed to be structured? I'm fairly sure Twitter's documentation is nonsense, as their "example request" is the following:
POST https://upload.twitter.com/1.1/media/upload.json?command=APPEND&media_id=123&segment_index=2&media_data=123
Is the media_data really supposed to go in the URL? What if you have raw binary media data? Would it go in the body? How is the REST service to know how the data is encoded?
You're looking at the chunked uploader - it's intended for sending large files, breaking them into chunks, so a network failure doesn't mean you have to re-upload a 100 MB .mp4. It is, as a result, fairly complicated. (Side note: The file data goes in the request body, not the URL as a GET parameter... as indicated by "Requests should be multipart/form-data POST format.")
There's a far less complicated unchunked uploader that'll be easier to work with if you're just uploading a regular old image.
All of this gets a lot easier if you use one of Twitter's recommended libraries for your language.
to upload a file, you need to send it in a form, in node.js server you save accept the incoming file using formidable.
You can also use express-fileupload or multer
Related
I have a need to create a pdf or html document within a Node.js express API which then sends that document over HTTP to an API managing our CMS.
So functionally I would like to create the document and POST it as part of a multipart-form upload POST request to an external service.
I see how to do this if after I create the file, I then turn around and write it disk. After that point I can do a read stream of the file from that path to format the POST request with the file.
However I'm wondering how I can perform this action without writing the file to disk and then reading it into a read stream. It seems I should be able to accomplish this without that IO.
Anybody able to point me to a good example or library that does something along these lines?
You can extend Writable and/or Readable streams. By the first look this library do what you need, with the same way - extending built-in streams.
I need to upload a local file to s3 and save its link in the database. Right now I am converting the image to base64 and sending it to my rails server, which saves it on s3 and returns a url. I send this URL in the next HTTP request. Now, how about I save it via express get a link and then use it for the request. What be the better approach? Using middleware or backend server?
For file upload, i suggest you to use multer middleware, because native multipart implementation is a little bit tricky. For interaction with amazon s3 middleware is used.
To send file somewhere else you could use pipes:
fs.createReadStream(rqPath).pipe(res);
In above example, file is read from local system and piped to response.
All mentioned modules could be find at NPM
If you're still trying to figure this out, I was struggling with the same issue, decided to POST binary data (converted from base64) directly in body without dealing with multipart forms, and whipped up the base64-image-upload package to make this easy.
I have an image upload view on my client (ember.js) that send the resized image to nodejs rest api;
it works well but it is easy for someone expert to force upload of a non-resized image;
I would like to keep the resize process on the client because this allows users to select heavy-weight images, that are resized locally and uploaded only after that, when they are lightweight;
If someone else uses something like this, I'm interested on how it is possible to make this as safe as possible;
As a rule of thumb when developing web applications is never ever trust any data coming from the client side, always try to do a check in your server side!
Use authentication, this ensures that user only allow to upload data to their own account and not fiddling others files.
Add a special message passing between your server and client, a simple example would be
i. send a post API request first (that contains the image information and targeted compressed size) to your server indicating that your client is starting to compress the picture
ii. when uploading, add a metadata to include the complete compressed image, and check the uploaded image with your server if it is within the accepted threshold, else discard it
You could enhance the security of the message passing to be more complicated!
This would be my simple security, anyone else got better solution? :)
Approaches here also work for file uploads. You can use a combination of checking:
content-length header and/or (i.e. req.headers['content-length'] > x)
reading stream size as it's being read by server. (i.e req.on('data'))
If the stream data exceeds a certain size you can respond accordingly. Check out something like Multer for file uploads, specifically the limits section. Best approach would probably the second option.
I'm writing a node SDK and one of the endpoints allows for a file_upload parameter. I'm currently using the standard https library to make my calls, and I wonder if I should continue using it or move to something the "requests" library given that I need to do file uploads.
Here is an article I was reading through to build multi-part file upload functionality into the https module, but the article doesn't say the best way to combine the multi-part file form data and additional parameters say "test_mode=true" or something like that.
how to upload a file from node.js
Wondering if I should move over to requests complete or if this approach seems good then how can I add the above multi-part form functionality but extend it to allow for additional parameters in the body as well as a file binary.
The request module uses the form-data module for sending multipart/form-data requests. You could also use form-data by itself if you don't want to use request.
How do you send files on node.js/express.
I am using Rackspace Cloudfiles and wanna send images/videos to their remote storage but I am not sure that it's as simple as reading the file (fs.readFileSync()) and send the data in the request body, or is it?
What should the headers be.
What if it's a very large file on a couple of GBs?
Is it possible to use superagent (http://visionmedia.github.com/superagent) for this or is there a better library for sending files?
Please give me some information about this.
Thanks!
app.get('/img/bg.png', function(req, res) {
res.sendFile('public/img/background.png')
})
https://expressjs.com/en/api.html#res.sendFile
use "res.sendFile". "res.sendfile" is deprecated.
For large files, you will want to use node.js's concept of piping IO streams together. You want to open the local file for reading, start the HTTP request to rackspace, and then pipe the data events from the file read process to the HTTP request process.
Here's an article on how to do this.
Superagent is fine for small files, but because the superagent API presumes your entire request body is loaded into memory before starting the request, it's not the best approach for large file transfers.
Normally you won't need to worry specifically about the request headers as node's HTTP request library will send the appropriate headers for you. Just make sure you use whatever HTTP method your API requires (probably POST), and it looks like for rackspace you will need to add the X-Auth-Token extra header with your API token as well.
I am using Rackspace Cloudfiles and wanna send images/videos to their remote storage but I am not sure that it's as simple as reading the file (fs.readFileSync()) and send the data in the request body, or is it?
You should never use fs.readFileSync in general. When you use it, or any other method called somethingSync, you block the entire server for the duration of that call. The only acceptable time to make synchronous calls in a node.js program is during startup.
What should the headers be.
See RackSpace Cloud Files API.
Is it possible to use superagent (http://visionmedia.github.com/superagent) for this or is there a better library for sending files?
While I don't have any experience with superagent, I'm sure it will work fine. Just make sure you read the API documentation and make your requests according to their specification.