I'm building a react webapp which allows users to upload a picture and then render the thumbnail, just like facebook chat. But I'm not sure what's the best practice to solve the uploading-resizing-generatingThumbnail time gap before I can render them.
The workflow is like:
1) User uploads a picture to S3, stored in bucket1
2) Lambda function invoked, getting the newly uploaded picture and do the resizing work, then store thumbnail in bucket2
3) Thumbnail rendered in browser(client-side). Here it's a bit tricky - I just hardcode the img url because it's predictable, however, it takes a while before available(generating process).
But I don't know how to let lambda notifies the browser when a thumbnail successfully generated and ready to render. In production it might be lambda tells node server first, and then node server tells the client; but in developing mode, it seems impossible as I'm running an express server on my own laptop. Should lambda do the notification in a proper way or there's other better solution?
AWS Lambda cannot "notify the browser" because it is a process that was independently triggered and has no connection with the web page request.
One option would be to code the web page to keep trying to download the image. You'd need some fancy JavaScript/node code that can check whether the image was successfully downloaded and then retry if necessary.
By the way, there are also services that can resize images on-the-fly so you don't have to create your own thumbnails:
Cloudinary
Imgix
Related
I'm beginner and using MERN. I've successfully deployed frontend and backend separately on render for testing purposes and when a user signs up they can choose a profile picture, now on localhost this works fine and it successfully adds to mongodb and shows in the web application. However when deployed I get an error that I can't 'GET' the image from the specific path.
Now I'm trying to workout with this is the case but could someone explain in ELI5 terms? Also would I need to host my images such as cloudinary? Thank you.
Tried to upload images when deployed but not being fetched.
You should consider using cloudinary or an S3 bucket to store images in production.
First of all, call your /upload api route
Generate a uuid + .extension (png or jpg)
Upload this key as the name with your image file to cloudinary (for exemple)
Then save this key into your database to be able to access it later on.
If you consider using an S3 bucket, you should always configure a CDN between your client and your bucket to avoid billing’s surprises
Context
I am building Stateless REST APIs for a browser-based platform that needs to store some user-generated files. These files could potentially be in the GBs.
I am using AWS S3 for storage. I have used AWS SDK in the past for this to route the file uploads through the NodeJS server (Basically - Upload to Server, Server uploads to S3).
I am trying to figure out how to improve it using the Pre-signed urls. I understand the dynamics and the flow on how to get the presigned urls and how to upload the file to S3 directly.
I cannot use SQS or Lambda to trigger object created event.
The architecture needs to be AWS independent.
Question
The simplest of flows I need to achieve is pretty common -
User --> Opens Profile
Clicks Upload Photo
Client Sends Request to /getSignedUrl
Server Returns the signedURL for the file name/type
The client executes the PUT/POST request to upload the file to the signedUrl
Upload Successful
After this - my understanding is -
Client tells the server - File Uploaded Successfully
Server associates the S3 Url for the Photo to the User.
...and that's my problem. How do I associate the successfully uploaded file back to the user on the server in a secure way?
Not sure what I've been missing. It seems like a trivial use case but I haven't been able to find anything regarding it.
1/ I think for the avatar, you should set it as public-read.
When create signed upload url in the
GET: /signed-upload-url
You need to set the image as public-read. After that you are free to interact with image through the direct url. Since this is an avatar, so you can compress it, reduce the size of image by the AWS Lambda function.
2/ If you don't want to have the public-read, you need to associate with server to get signed-download-url to interact with image
GET: /signed-download-url
I would like to know what is the best way to handle image uploading and saving the reference to the database. What I'm mostly interested is what order do you do the process in?
Should you upload the images first in the front-end (say Cloudinary), and then call the API with result links to the images and save it to the database?
Or should you upload the images to the server first, and upload them from the back-end and save the reference afterwards?
OR, should you do the image uploading after you save the record in the database and then update it once the images were uploaded?
It really depends on the resources, timeline, and number of images you need to upload daily.
So basically if you have very few images to upload then you can upload that image to your server then upload it to any cloud storage(s3, Cloudinary,..) you are using. As this will be very easy to implement(you can find code snippet over the internet) and you can securely maintain your secret keys/credential to your cloud platform on the server side.
But, according to me best way of doing this will be something like this. I am taking user registration as an example
Make server call to get a temporary credential to upload files on the cloud(Generally, all the providers give this functionality i.e. STS/Signed URL in AWS).
The user will fill up the form and select the image on the client side. When the user clicks the submit button make one call to save the user in the database and start upload with credentials. If possible keep a predictable path for upload. Like for user upload /users/:userId or something like that. this highly depends on your use case.
Now when upload finishes make a server call for acknowledgment and store some flag in the database.
Now advantages of this approach are:
You are completely offloading your server from handling file operations which are pretty heavy and I/O blocking and you are distributing that load to all clients.
If you want to post process the files after upload you can easily integrate this with serverless platforms and do that on there and again offload that.
You can easily provide retry mechanism to your users in case of file upload fails but they won't need to refill the data, just upload the image/file again
You don't need to expose the URL directly to the client for file upload as you are using temporary Creds.
If the significance of the images in your app is high then ideally, you should not complete the transaction until the image is saved. The approach should be to create an object in your code which you will eventually insert into mongodb, start upload of image to cloud and then add the link to this object. Finally then insert this object into mongodb in one go. Do not make repeated calls. Anything before that, raise an error and catch the exception
You can have many answers,
if you are working with big files greater than 16mb please go with gridfs and multer,
( changing the images to a different format and save them to mongoDB)
If your files are actually less than 16 mb, please try using this Converter that changes the image of format jpeg / png to a format of saving to mongodb, and you can see this as an easy alternative for gridfs ,
please check this github repo for more details..
Title is my questions.
First. I'm making simple diary app with Node.js and Vue.js. Vue-router using history mode and in backend, using "connect-history-api-fallback" module. I think i did everything i can do, but when i run my app in local, refresh make vue.js's store clear. I googled but can't find same problem. Someone have any idea?
Second. I'm using Multer to upload. Upload is fine, i can see uploaded image. But i don't know how to show that uploaded image.
I mean in vue's template, what path will show uploaded image?
Image uploaded here "/simple-diary/backend/upload/profilePhoto/"
Vue.js component is here "/simple-diary/frontend/src/components/"
#samayo is right.One question per post.So of course when the page refresh the store is refreshed.So if you dont want that you can use plugin like vuex-persistedstate.Or if you want to keep the token and the user that is currently logged then when the user logged in,use localstorage to save the token and the user.Or cookies.Or session storage.You have many options
I'm currently considering developing a Meteor node.js app, but am struggling with how best to handle uploading of user images. In particular, I want to create a photography website that will allow the photographer to upload images in an 'admin' section, and these images will then be displayed on the website. I need to create a thumbnail of these images, and save the respective URLs to the database. I'm struggling with how to best accomplish this in meteor.
Is my best bet to use something like s3 combined with an AWS process for generating thumbnails?
Or should I save and host the images directly in the Meteor/node session?
Or should I scrap Meteor and use something like Express.js for this project?
Why don't you just use something like Filepicker.io to handle uploading and hosting images and simply store the image unique url (given to you by filepicker in the callback)?
Thumbnails can also be dynamically generated by Filepicker (using simple url modifications).
Cloudinary is a nicer alternative to filepicker when it comes to images, but integration process will be messier.
I would store the images on the filesystem, not in a database. If you have a unique id, you can use that as part of the url, for example an id of the item the image belongs to. Might look like this:
./uploads/img-<id>-<size>.jpg
You can write to disk and resize if necessary with node-imagemagick and your cdn should just poll these images from time to time. Not exactly sure how that part would work in terms of including the url to the image in the html.