I'm trying to save the images from a third-party API to my own S3 bucket using Node.js and MongoDB. The API provides a URL to the image on the third-party servers. I've never done this before but I'm assuming I have to download the image to my own server and then upload it to S3?
Should I save the image to mongodb with GridFS and then delete it once it is on S3? If so, what's the best way to do that?
I've read and re-read this previous question:
Problem with MongoDB GridFS Saving Files with Node.JS
But I can't find good documentation on how I should determine buffer/chunk size and other attributes for a JPEG image file.
Once I've saved the file on my server/database, then it seems like I should use:
https://github.com/appsattic/node-awssum
To upload it to S3. Is that a good idea?
I apologize if this is an obvious answer, I'm pretty green when it comes to databases and server scripting.
The easiest thing to do would be to save the image onto disk and then stream the upload from there using AwsSum's S3 PutObject operation. Alternatively, if you have the file contents in a Buffer you can just use that.
Have a look at the following two examples and they should help you figure out what to do:
https://github.com/appsattic/node-awssum/blob/master/examples/amazon/s3/put-bucket.js
https://github.com/appsattic/node-awssum/blob/master/examples/amazon/s3/put-object-streaming.js
Let me know if you need any more help. :)
Cheers,
Andy
Disclaimer: I'm the author of AwsSum.
Related
Good day everyone, i was looking for a similar that i can refer to, but sadly i wasn't found yet until now. Hope someone would give me some guidances on it...
I'm using filepond to send the api request, and use the prisma 2 client to store it
Thank you!
It's usually a good practice to store Images in a Blob Storage and images should not directly be stored in the database, so you can use something like AWS S3 or Cloudinary to store the image and store the Image's S3 Bucket Path in your PostgreSQL Database.
Here's a working example of how you could achieve it.
I have a problem with developing a crawler using nodejs/puppeteer. The old crawler was:
Crawl Pages
Store the output file locally with the fs module
Since i'm going to introduce UI on the server, have set up the scenario to upload it to S3 instead of storing it locally, and show the result as a UI.
Crawl Pages
Stream output files to the server with the fs module
Get the output file back and upload it to the S3 bucket
The above is a scenario that i know of as a knowledge, and i'd like to know if it is possible as below.
Crawl Pages
Upload the stream stored data to memory to the S3 bucket
If you have a scenario like this, I would like to receive a guide. I would really appreciate if you would comment or reply :)
This is definitely possible. If you just pipe from your input stream to your server and pipe up to the S3 it should complete the loop.
This is possible because you can stream uploads to S3, even without knowing the size of the file beforehand.
This answer should help you out: S3 file upload stream using node js
If you post some code we can answer this a little bit better. But hope this puts you in the right direction.
I would like to know what is the best way to handle image uploading and saving the reference to the database. What I'm mostly interested is what order do you do the process in?
Should you upload the images first in the front-end (say Cloudinary), and then call the API with result links to the images and save it to the database?
Or should you upload the images to the server first, and upload them from the back-end and save the reference afterwards?
OR, should you do the image uploading after you save the record in the database and then update it once the images were uploaded?
It really depends on the resources, timeline, and number of images you need to upload daily.
So basically if you have very few images to upload then you can upload that image to your server then upload it to any cloud storage(s3, Cloudinary,..) you are using. As this will be very easy to implement(you can find code snippet over the internet) and you can securely maintain your secret keys/credential to your cloud platform on the server side.
But, according to me best way of doing this will be something like this. I am taking user registration as an example
Make server call to get a temporary credential to upload files on the cloud(Generally, all the providers give this functionality i.e. STS/Signed URL in AWS).
The user will fill up the form and select the image on the client side. When the user clicks the submit button make one call to save the user in the database and start upload with credentials. If possible keep a predictable path for upload. Like for user upload /users/:userId or something like that. this highly depends on your use case.
Now when upload finishes make a server call for acknowledgment and store some flag in the database.
Now advantages of this approach are:
You are completely offloading your server from handling file operations which are pretty heavy and I/O blocking and you are distributing that load to all clients.
If you want to post process the files after upload you can easily integrate this with serverless platforms and do that on there and again offload that.
You can easily provide retry mechanism to your users in case of file upload fails but they won't need to refill the data, just upload the image/file again
You don't need to expose the URL directly to the client for file upload as you are using temporary Creds.
If the significance of the images in your app is high then ideally, you should not complete the transaction until the image is saved. The approach should be to create an object in your code which you will eventually insert into mongodb, start upload of image to cloud and then add the link to this object. Finally then insert this object into mongodb in one go. Do not make repeated calls. Anything before that, raise an error and catch the exception
You can have many answers,
if you are working with big files greater than 16mb please go with gridfs and multer,
( changing the images to a different format and save them to mongoDB)
If your files are actually less than 16 mb, please try using this Converter that changes the image of format jpeg / png to a format of saving to mongodb, and you can see this as an easy alternative for gridfs ,
please check this github repo for more details..
went on NPMJS but could't find any libraries useful. It looks like nearly all of them requires it first to be stored on the server then upload to S3. Any chance the file can be uploaded directly to S3?
These two links are good sources of uploads to S3.
http://blog.katworksgames.com/2014/01/26/nodejs-deploying-files-to-aws-s3/
http://docs.aws.amazon.com/AWSJavaScriptSDK/guide/node-examples.html
This blog post explains in detail how you can solve this problem:
https://www.terlici.com/2015/05/23/uploading-files-S3.html
You need to create a signed URL first and after that a user can upload a file directly to S3.
I need to create a function to upload CSV/txt file into mongodb using mean stack.
The function should be like i will upload a file. First it will check whether its in text/csv format than it will upload that to mongodb.
I searched on internet and couldnt find any good material. Anyone have any idea, Please share
I used angular-file-upload and wrote my own handler using multiparty for express.