I am a total beginner in web development using MEAN Stack. I am considering to deploy my app on Heroku for this particular project. I am trying to create an application where the user would upload large excel files(>200MB). The application needs to process the excel file and then export the resultant file of similar size to another service. It is very hard for me to figure out the following:
Would it be possible to save a file on disk in Heroku?
For this process do I need to save the file somewhere on server at all, or would it be possible to just process the file keeping in memory and then export?
Would it be possible to process large files from disk or should I try to save the information in a database?
Are there any Heroku constraints that limit the size of file to be processed within memory?
The information I have looked up seems too involved to be understood by a beginner like me. I haven't had much success with what I have tried and I don't want to spend too much time trying all these possibilities.
How should I proceed?
Thanks.
You should not attempt to save your files on the local file system of a Heroku dyno. Your heroku dynos can recycle without warning at any time, and anything you "stored" on the local file system of your dyno before the recycle will disappear.
Instead, you will probably need some kind of external storage for your files, such as Amazon S3.
You could optionally use a service such as Bucketeer to simplify provisioning your Amazon S3 storage from within Heroku.
Related
I have deployed a nodejs server on Heroku as free services. So, I have noticed that it only supports two image file to upload. I have an API in my project to upload an image. After upload it appears on my application or on my interface for sometime.
After 5 to 10 minutes images will delete automatically from the server only if it exceeded from two images. Just two images remain in the folder rest of the images will deleted automatically. So, I want to ask why this happen? Because of free deployment or any other reason?
When you upload an image to your Heroku server it is being stored in some tmp location and Heroku may free up that location periodically. So it is not persistent storage.
The more convenient way of storing images is to using AWS S3, google drive or any third party service which can guarantee persistency and availability
I have a server deployed on Heroku through Heroku auto deployment method from GitHub. On that server I have a file named subscription.json which contains user data whenever user is registered. I want to see that file.
How can I access that file?
If that file is in your repository you should be able to read it like any regular file.
However, this isn't going to work on Heroku:
which contains user data whenever user is registered
Heroku's filesystem is dyno-local and ephemeral. Any changes you make to it will be lost the next time your dyno restarts. This happens frequently (at least once per day).
For saving things like user uploads, Heroku recommends using a third-party service like Amazon S3. But in this case I think a database would be a much more appropriate solution.
Heroku has its own PostgreSQL service that is very well supported and available out of the box. If you prefer to use another database there are plenty of options.
I have a node.js app on heroku and I sometimes need to write files to heroku. Do you know how to see those files? Should I delete them after I am finished using them? I do not want to use memory for no reason.
Heroku (and other container based platforms) are different from traditional servers that you might be used to. It's worth bearing in mind that the Heroku filesystem is ephemeral - that means that any changes to the filesystem whilst the dyno is running only last until that dyno is shut down or restarted. Each dyno boots with a clean copy of the filesystem from the most recent deploy.
If you really needed to check a file on a running dyno (let's say to debug an issue with a file upload) it is possible to login using Heroku Exec https://devcenter.heroku.com/articles/exec
That said, you really shouldn't be using the filesystem for anything other than temporary files. Instead you should aim to use external services for persistent storage as described here: https://12factor.net/
For example, if you are handling file uploads you could try storing these on a service like Amazon S3.
I made a discordJs bot which saves data in a file.
Everything is hosted on Heroku and all works good.
But everyday, Heroku reset my file.
Why can't i keep my files everyday ?
Here's the full explanation from Heroku docs:
The Heroku filesystem is ephemeral - that means that any changes to
the filesystem whilst the dyno is running only last until that dyno is
shut down or restarted. Each dyno boots with a clean copy of the
filesystem from the most recent deploy. This is similar to how many
container based systems, such as Docker, operate.
In addition, under normal operations dynos will restart every day in a
process known as "Cycling".
These two facts mean that the filesystem on Heroku is not suitable for
persistent storage of data. In cases where you need to store data we
recommend using a database addon such as Postgres (for data) or a
dedicated file storage service such as AWS S3 (for static files). If
you don't want to set up an account with AWS to create an S3 bucket we
also have addons here that handle storage and processing of static
assets https://elements.heroku.com/addons
Source: https://help.heroku.com/K1PPS2WM/why-are-my-file-uploads-missing-deleted
First of all I am a beginner with node.js.
In node.js when I use functions such as fs.writeFile(); the file is created and is visible in my repository. But when this same process is done on a cloud such as heroku no file is visible in the repository(cloned via git). I know the file is being made because I am able to read it but I cannot view it. Why is this??? Plus how can I view the file?
I had the same issue, and found out that Heroku and other cloud services generally prefer that you don't write in their file system; everything you write/save will be store in "ephemeral filesystem", it's like a ghost file system really.
Usually you would want to use Amazon S3 or reddis for json files etc, and other bigger ones like mp3.
I think it will work if you rent a remote server, like ECS, with a linux system, and a mounted storage space, then this might work.