I have deployed a nodejs server on Heroku as free services. So, I have noticed that it only supports two image file to upload. I have an API in my project to upload an image. After upload it appears on my application or on my interface for sometime.
After 5 to 10 minutes images will delete automatically from the server only if it exceeded from two images. Just two images remain in the folder rest of the images will deleted automatically. So, I want to ask why this happen? Because of free deployment or any other reason?
When you upload an image to your Heroku server it is being stored in some tmp location and Heroku may free up that location periodically. So it is not persistent storage.
The more convenient way of storing images is to using AWS S3, google drive or any third party service which can guarantee persistency and availability
Related
I'm building an application that saves a lot of images using the C# Web API of ASP.NET.
The best way seems to save the images in the file system and save their path into the database.
However, I am concerned about load balancing. Since every server will put the image in its own file system, how can another server behind the same load balancer retrieve the image?
If you have resources for it, I would state that:
the best way is to save them in the file system and save the image path into the database
Is not true at all.
Instead, Id say using an existing file server system is probably going to produce the best results, if you are willing to pay for the service.
For dotnet the 'go to' would be Azure Blob Storage, which is ideal for non-streamed data like images.
https://learn.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-dotnet
Otherwise, you can try and create your own file storage service from scratch. In which case you will effectively be creating a separate service API apart from your main cluster that handles your actual app, this secondary API just handles file storage and runs on its own dedicated server.
You then simply just create an association between Id <-> File Data on the File Server, and you're App Servers can request and push files to the File Server via those Ids.
Its possible but a File Server is for sure one of those types of projects that seems straightforward at first but very quickly you realize its a very very difficult task and it may have been easier to just pay for an existing service.
There might be existing self hosted file server options out there as well!
If I used a paid (non-sleep) dyno, can I push code and ignore the uploads directory in the .gitignore file, so my uploaded files are not lost?
No.
A dyno is a container (much like docker) that is built when you deploy your app. It can be started one or many times, and won't share anything.
When you restart/redeploy the app, the filesystem will always be lost and reset to what it was when you deployed the app.
You need to store your uploaded files on a dedicated storage platform such as Amazon S3.
See https://devcenter.heroku.com/articles/s3-upload-node
I made a discordJs bot which saves data in a file.
Everything is hosted on Heroku and all works good.
But everyday, Heroku reset my file.
Why can't i keep my files everyday ?
Here's the full explanation from Heroku docs:
The Heroku filesystem is ephemeral - that means that any changes to
the filesystem whilst the dyno is running only last until that dyno is
shut down or restarted. Each dyno boots with a clean copy of the
filesystem from the most recent deploy. This is similar to how many
container based systems, such as Docker, operate.
In addition, under normal operations dynos will restart every day in a
process known as "Cycling".
These two facts mean that the filesystem on Heroku is not suitable for
persistent storage of data. In cases where you need to store data we
recommend using a database addon such as Postgres (for data) or a
dedicated file storage service such as AWS S3 (for static files). If
you don't want to set up an account with AWS to create an S3 bucket we
also have addons here that handle storage and processing of static
assets https://elements.heroku.com/addons
Source: https://help.heroku.com/K1PPS2WM/why-are-my-file-uploads-missing-deleted
I am a total beginner in web development using MEAN Stack. I am considering to deploy my app on Heroku for this particular project. I am trying to create an application where the user would upload large excel files(>200MB). The application needs to process the excel file and then export the resultant file of similar size to another service. It is very hard for me to figure out the following:
Would it be possible to save a file on disk in Heroku?
For this process do I need to save the file somewhere on server at all, or would it be possible to just process the file keeping in memory and then export?
Would it be possible to process large files from disk or should I try to save the information in a database?
Are there any Heroku constraints that limit the size of file to be processed within memory?
The information I have looked up seems too involved to be understood by a beginner like me. I haven't had much success with what I have tried and I don't want to spend too much time trying all these possibilities.
How should I proceed?
Thanks.
You should not attempt to save your files on the local file system of a Heroku dyno. Your heroku dynos can recycle without warning at any time, and anything you "stored" on the local file system of your dyno before the recycle will disappear.
Instead, you will probably need some kind of external storage for your files, such as Amazon S3.
You could optionally use a service such as Bucketeer to simplify provisioning your Amazon S3 storage from within Heroku.
I am planning to use cloudfoundry paas service (from VMWare) for hosting my node.js application. I have seen that it has support for mongo and redis in the service layer and node.js framework. So far so good.
Now I need to store my mediafiles(images uploaded by users) to a filesystem. I have the metadata stored in Mongo.
I have been searching internet, but have not yet got good information.
You cannot do that for the following reasons:
There are multiple host machines running your application. They each have their own filesystems. Each running process in your application would see a different set of files.
The host machines on which your particular application is running can change moment-to-moment. Indeed, they will change every time you re-deploy your application. Every time a process is started on a new host machine, it will see an empty set of files. Every time a process is stopped on an old host machine, all the files would be permanently deleted.
You absolutely must solve this problem in another way.
Store the media files in MongoDB GridFS.
Store the media files in an object store such as Amazon S3 or Rackspace Cloud Files.
Filesystem in most cloud solutions are "ephemeral", so you can not use FS. You will have to use solutions like S3/ DB for such purpose