I am working in application where react js application uploading files to backend ( node js, Loopback 4 based ) and backend then send those files AWS S3 Bucket. Also upload is multiple files based so can lead up any number of files within groups something like below
POST Request
Group 1
Group Type: Text
File 1
File 2
File 3
Group 2
Group Type: Text
File 4
File 5
File 6
Group 3
Group Type: Text
File 7
File 8
File 9
One single Form Data request sending all these details to backend server. For now every this is working file for files.
Now as per production requirements, These files can be very big, like some of files above than 100MB but less than 500MB. I am thinking, on average it is possible to upload GBs of file upload in single POST Request.
This make me to rethink whole solution is feasible or not and if failure chances will increase in production with each such request. While Node Js/Multer application itself do not have limits ( and I can update those limits if required ).
Their is one idea to make whole upload process interactive by showing upload progress, also add some solution where user can work to other pages while files is still uploading ( As these are react SPA pages may be redux can handle this or not) something like AWS create services.
Lastly I am thinking if instead of upload by frontend, some FTP server setup and schedule sync files to S3 by backend scripts is better and safe idea. Later by web sockets we can inform user to sync completion.
If their is some industry standard approach to make such solution.
I am using Shopware 6, I am trying to upload multiple images how to upload multiple images upload by rest Api?
I believe even if you upload multiple images via the backend, it makes separate requests. So I am not sure if there is such a thing like a multi upload.
Additionaly there is usually a size limit due to your web server and PHP settings, so it makes not much sense to combine multiple images into one request.
What keeps you from doing multiple uploads in a loop?
In NodeJS, I am logging every every API request and would get the log through another API request . But I do not want to use MONGODB or any such database. I am currently writing to file each API request in JSON format. But now API requests are much quicker and sometimes file is busy before another request comes.
What would be the best solution in this case?
Actually I am developing a portable and distributable solution. So database will hinder the motive.
On my website I have some categories in the database which don't change often. I am using the NextJS framework. In order to decrease the load time and make category control load faster I am thinking of putting categories in a json file under /static/data/categories.json and let React fetch that json file instead of making a database call. I have read about multiple approaches to importing static json files including json-loader. However, I am thinking of making an api call something like the following from React once on the home page and store the contents in redux state so that I could use them wherever I need. My intention here is whenever categories are modified, I update categories.json and the client will fetch the updated json file. Deployment downtime is not an issue for me.
const categories = yield call(request, `${BASE_URL}/static/data/categories.json`, options);
My questions are:
Is this a right approach to load a json file from the server keeping in mind I would want to update the file every few months without the need to redeploy the whole website.
Can a json file become a bottleneck if thousands of concurrent users try to access it. I am using express with nginx. Would using express.static help at all?
Thanks in advance.
I would like to know what is the best way to handle image uploading and saving the reference to the database. What I'm mostly interested is what order do you do the process in?
Should you upload the images first in the front-end (say Cloudinary), and then call the API with result links to the images and save it to the database?
Or should you upload the images to the server first, and upload them from the back-end and save the reference afterwards?
OR, should you do the image uploading after you save the record in the database and then update it once the images were uploaded?
It really depends on the resources, timeline, and number of images you need to upload daily.
So basically if you have very few images to upload then you can upload that image to your server then upload it to any cloud storage(s3, Cloudinary,..) you are using. As this will be very easy to implement(you can find code snippet over the internet) and you can securely maintain your secret keys/credential to your cloud platform on the server side.
But, according to me best way of doing this will be something like this. I am taking user registration as an example
Make server call to get a temporary credential to upload files on the cloud(Generally, all the providers give this functionality i.e. STS/Signed URL in AWS).
The user will fill up the form and select the image on the client side. When the user clicks the submit button make one call to save the user in the database and start upload with credentials. If possible keep a predictable path for upload. Like for user upload /users/:userId or something like that. this highly depends on your use case.
Now when upload finishes make a server call for acknowledgment and store some flag in the database.
Now advantages of this approach are:
You are completely offloading your server from handling file operations which are pretty heavy and I/O blocking and you are distributing that load to all clients.
If you want to post process the files after upload you can easily integrate this with serverless platforms and do that on there and again offload that.
You can easily provide retry mechanism to your users in case of file upload fails but they won't need to refill the data, just upload the image/file again
You don't need to expose the URL directly to the client for file upload as you are using temporary Creds.
If the significance of the images in your app is high then ideally, you should not complete the transaction until the image is saved. The approach should be to create an object in your code which you will eventually insert into mongodb, start upload of image to cloud and then add the link to this object. Finally then insert this object into mongodb in one go. Do not make repeated calls. Anything before that, raise an error and catch the exception
You can have many answers,
if you are working with big files greater than 16mb please go with gridfs and multer,
( changing the images to a different format and save them to mongoDB)
If your files are actually less than 16 mb, please try using this Converter that changes the image of format jpeg / png to a format of saving to mongodb, and you can see this as an easy alternative for gridfs ,
please check this github repo for more details..