I want to make an web app where user able to upload to upload.
So how can i make the api where the video will be stored in mongodb.
Storing an entire file is not an option, you can store using binary data, but is not the best approach.
As Kris said, are better options like AWS S3. On the other hand, remember the limitation of 16MB per document, that is the maximum amount of data a document can store, so in case you decide to use binary data to store your videos you need having in mind this limit.
Related
I'm new to mongoDB,
I want to fill a form for a post model and one of the fields is file-type (image or video), can I store the attached media in the same mongo document ? If not, how should I go about doing this and is there a helpful guide I can follow ?
this is how my backend looks like
this is my Post model that I'm going to fill it in the form
NB : I'm using angular in my front-end
If the media is relatively small like an 8K thumbnail then you can store it as a binary type field as a peer to the rest of your ints, strings, and dates.
However, an individual document cannot be larger than 16MB so such an approach in general is not feasible especially for video content.
You can use the gridFs utils which are bundled with the client side drivers. A comprehensive example is posted here: How to save images from url into mongodb using python?
If you do not require the media to be managed by the database storage subsystem, a practical alternative is to store the media in an AWS S3 bucket (or Azure or GCP equivalent) and store just the path to the content in mongodb.
I am working on web api in node.js and express and I want to enable users to upload images.
My api is using JSON requests and responses but when it comes to uploading images I don't know which option is a better one. I can think of two ideas:
encode images as a base64 strings and send them as a JSON (like {"image": "base64_encoded_image"})
use multipart/form request and handle the request with a help of packages like multer
I've been reading some articles and other questions related to my issue and I'm still struggling to choose one approach over the other. Encoding image and sending it with JSON increases the size of data by about 25% (that's what I've read) but using multipart seems weird to me as all other endpoints on my api use JSON.
The multipart/formdata approach has certain advantages over the Base64 encoding one.
First and foremost disadvantage of using Base64 approach is the 30% increase in size of data, while this may not be significant for files of small size but it will definitely matter if you are sending large files and storing them on storage spaces( will increase your cost/data-consumption ). Also packages like multer provide you with certain functionalities like - checking the type of file(jpg,png etc) and set size limit on files etc. And they are quite easy to implement as well with a lot of tutorials and guides present online.
Furthermore, converting image to Base64 string increases computation overhead on user's machine especially if the file is large.
I would advise you to use multipart/form-data approach for your case.
I would like to know what is the best way to handle image uploading and saving the reference to the database. What I'm mostly interested is what order do you do the process in?
Should you upload the images first in the front-end (say Cloudinary), and then call the API with result links to the images and save it to the database?
Or should you upload the images to the server first, and upload them from the back-end and save the reference afterwards?
OR, should you do the image uploading after you save the record in the database and then update it once the images were uploaded?
It really depends on the resources, timeline, and number of images you need to upload daily.
So basically if you have very few images to upload then you can upload that image to your server then upload it to any cloud storage(s3, Cloudinary,..) you are using. As this will be very easy to implement(you can find code snippet over the internet) and you can securely maintain your secret keys/credential to your cloud platform on the server side.
But, according to me best way of doing this will be something like this. I am taking user registration as an example
Make server call to get a temporary credential to upload files on the cloud(Generally, all the providers give this functionality i.e. STS/Signed URL in AWS).
The user will fill up the form and select the image on the client side. When the user clicks the submit button make one call to save the user in the database and start upload with credentials. If possible keep a predictable path for upload. Like for user upload /users/:userId or something like that. this highly depends on your use case.
Now when upload finishes make a server call for acknowledgment and store some flag in the database.
Now advantages of this approach are:
You are completely offloading your server from handling file operations which are pretty heavy and I/O blocking and you are distributing that load to all clients.
If you want to post process the files after upload you can easily integrate this with serverless platforms and do that on there and again offload that.
You can easily provide retry mechanism to your users in case of file upload fails but they won't need to refill the data, just upload the image/file again
You don't need to expose the URL directly to the client for file upload as you are using temporary Creds.
If the significance of the images in your app is high then ideally, you should not complete the transaction until the image is saved. The approach should be to create an object in your code which you will eventually insert into mongodb, start upload of image to cloud and then add the link to this object. Finally then insert this object into mongodb in one go. Do not make repeated calls. Anything before that, raise an error and catch the exception
You can have many answers,
if you are working with big files greater than 16mb please go with gridfs and multer,
( changing the images to a different format and save them to mongoDB)
If your files are actually less than 16 mb, please try using this Converter that changes the image of format jpeg / png to a format of saving to mongodb, and you can see this as an easy alternative for gridfs ,
please check this github repo for more details..
Expressjs has bodyParser middleware which can handle file-uploads and can even store them in a directory given in options. But in my app I want to store the files in Amazon S3, so I basically want to stream the file straight to S3 without having to store it locally at all.
But the problem is validation of the file. How can I be sure that these files are all images. Checking the content-type isn't good enough option coz that can be faked. I want to know is it ok if I do the validation after streaming the file to S3?? I am asking from the security point of view.
After storing the image, I need to retrieve it for creating thumbnails, How can I do it asynchronously after giving the response after file upload?
You have contradictory goals of not wanting to store it locally during upload but then also wanting to download it needlessly again to make thumbnails. If you want to go for technical slickness awards, you can simultaneously stream the file upload request body to a local temporary file as well as S3. Or you can do what the rest of the industry does and store it in a local temporary file and then thumbnail it, and then upload all sizes to S3. Either of these approaches alleviates any need to immediately download it from S3 to make thumbnails.
How exactly do you intend to validate that it's really an image? You could look at the first chunk of file data and validate for the file type's magic number if that gives you warm fuzzies, but ultimately it's untrusted user data. The second half of the supposed image file could be virus code and that is just as easily faked at the Content-Type header. Sounds like your security concerns are mostly driven by FUD as opposed to specific threats you intend to defend against. As long as you don't take the user's uploaded data, mark it executable and run it as root on your server, any non-image data is just going to be corrupt and fail to render correctly in a browser (and/or cause your thumbnailer program to exit with an error or perhaps crash in an extreme case).
Regarding validation can I just try to create a thumbnail and if I can't then its not a valid image and delete it. Is this way fine?
Most of the time, yes. There will be edge cases where your thumbnailer cannot process an image but a browser can as thumbnailers are not perfect and some images are partially corrupt. For example, I have found some animated GIFs that render and animate fine in a web browser but graphicsmagick crashes trying to process them. Not sure there's anything that can be done about those 0.01% edge cases.
And for uploads part, can I send a response to the user and than carry on with the thumbnail creation and storing it in S3?
Yes, that is generally the best approach so the user knows their upload succeeded. Generally image processing is usually architected as a "work queue" model where you just record that there's work to do and then proceed and a separate process or processes take work off the queue and complete it.
I'm currently considering developing a Meteor node.js app, but am struggling with how best to handle uploading of user images. In particular, I want to create a photography website that will allow the photographer to upload images in an 'admin' section, and these images will then be displayed on the website. I need to create a thumbnail of these images, and save the respective URLs to the database. I'm struggling with how to best accomplish this in meteor.
Is my best bet to use something like s3 combined with an AWS process for generating thumbnails?
Or should I save and host the images directly in the Meteor/node session?
Or should I scrap Meteor and use something like Express.js for this project?
Why don't you just use something like Filepicker.io to handle uploading and hosting images and simply store the image unique url (given to you by filepicker in the callback)?
Thumbnails can also be dynamically generated by Filepicker (using simple url modifications).
Cloudinary is a nicer alternative to filepicker when it comes to images, but integration process will be messier.
I would store the images on the filesystem, not in a database. If you have a unique id, you can use that as part of the url, for example an id of the item the image belongs to. Might look like this:
./uploads/img-<id>-<size>.jpg
You can write to disk and resize if necessary with node-imagemagick and your cdn should just poll these images from time to time. Not exactly sure how that part would work in terms of including the url to the image in the html.