my Node.js app keeps losing static images(.jpg, .png...) after a while. It doesn't lose any images on my local win10 desktop and even on heroku, my webpack bundle.js is served from the same static route(/pub or /dist) and they work just fine. somehow only the static images, they are served alright for first few minutes when I first uploaded then after a while, it disappears. I am using express.static for static route declaration and multer for file upload. The files used for test were all lowercase .jpg(since I've heard heroku arbitrarily changes all uppercase extensions) so I don't know what's causing the problem.
server code:
const storage = multer.diskStorage({
destination: (req,file,cb)=>{
cb(null,'pub/')
},
filename: (req,file,cb)=>{
cb(null,Date.now() + file.originalname)
}
})
const upload = multer({storage:storage})
//access to static files
app.use('/pub', express.static(pubDir))
app.use('/dist', express.static(dstDir))
app.post('/modwimg',upload.any(),(req,res,next)=>{
//here I connect filename from files array to db
})
then if there's a client request, the server fetches filename from the db and put '/pub/' in front of it. It works just fine on both my local machine and heroku. it's only that images on heroku disappear after a while.
The heroku file system is transient. If you want to allow users to upload files to your app, you'll need to use external storage like S3, database blobs, or a hosted service like cloudinary. See this thread for more information: https://www.reddit.com/r/rails/comments/2k9sq4/heroku_any_files_you_upload_will_not_be_saved/
Related
i just discovered that Heroku uses an Ephemeral File System, so i opted to use AWS S3 to store images and upload them. the problem i encountered is creating the file to upload it.
my app is built with NextJS and my app is hosted on a basic dyno in Heroku. this app is an api with no front end code.
here is the code i use to create the file in my app using 'fs' library:
if (
!fs.existsSync(
path +
new Date().toISOString().split("T")[0] +
".csv"
)
) {
fs.writeFileSync(
path +
new Date().toISOString().split("T")[0] +
".csv",
await csv.toString(true)
);
} else {
fs.appendFileSync(
path +
new Date().toISOString().split("T")[0] +
".csv",
await csv.toString(false),
"utf-8"
);
}
this code creates the file normally in my computer.
the 'path' variable is created using either :
let folder = await fs.promises.mkdir(path, {
recursive:true,
mode: '0o755'
});
or i pass it to my request body.
i tried different paths like NextJS public folder and the "/tmp" folder, i also created folders manually in both paths but no result.
after a bit of research i found that you can create files it just gets erased when the dyno restarts and that usually happens every 24Hrs or so.
so my request is if you know how i can create a file even temporally just enough time to upload it to AWS S3? or a way to upload a file (png, csv) that is not created in the file system?
thanks in advance for your help.
Sorry in advance for my lengthy post.
I am having difficulty changing from MERN localhost to an online hosting. I've been hosting my frontend/ client side website on heroku, with the file structure as follows:
Client Structure
- build
- src
- App.js
- Index.js
- Other Components, Images, Fonts, Pages
-package.json
-package-lock.json
This file structure works fine with heroku.
However, I've been following a Youtube tutorial that uses a MERN stack to build a blog: https://www.youtube.com/watch?v=OML9f6LXUUs&t=3s --> it is a 3 part series
In the video, he has a file structure as such:
API and Client Structure
- api
- images
- models
- public
- routes
- index.js
- package.json
- package-lock.json
- client (frontend code)
- build
- src
- App.js
- Index.js
- Other Components, Images, Fonts, Pages
-package.json
-package-lock.json
2 Instances of LocalHost
He runs api on localhost:3001 and client on localhost:3000.
Methodology
1. Storing Images
He does not store images in MongoDB cloud, but rather store the image name only, and stores the images directly on the api/images directory.
In the api folder, he uses multer to store the images.
app.use("/images", express.static(path.join(__dirname, "/images")))
const storage = multer.diskStorage({
destination: (request, file, callback) => {
callback(null, 'images');
},
filename: (request, file ,callback) => {
callback(null, request.body.name);
},
})
const upload = multer({storage: storage});
app.post('/api/upload', upload.single('file'), (request, response) => {
response.status(200).json('File has been uploaded');
});
Then in the client side, he sends the data via axios:
const response = await axios.post('/upload', data);
2. Retrieving Images
When retrieving, he specifies a public folder PF, uses axios to get the image name, and then concatenates them to get the images.
const PF = "http://localhost:3001/images/";
<Image src={PF + post.image} />
My question is:
How can I change from a localhost to a cloud based hosting (preferably Heroku)? What paths do I specify? I would think that this image retrieval works because I am running both locally on the same computer. I wouldn't be able to use it if they are online as the image src directory will not be searchable
Is it possible to host both simultaneously on 1 Heroku Dyno, and how can I do it?
This is my first time learning backend so thanks for the patience!
Firstly, you'll have to remove all hardcoded 'http://localhost:3000`. This would interfere with the cloud site since it uses its own.
Instead, you could use an environment file to store the API site and in production use a different one.
You will need to make a mongodb cluster (its a cloud database and its free) https://www.mongodb.com/docs/atlas/tutorial/create-new-cluster/. You'll have to manually change your database to the cluster link. I would use an environment variable for this as well, so the db link would be something like this in your db file
const mongoURI = process.env.MONGODB_URI || 'mongodb://localhost:27017/ProjectAPI'
where process.env.mongodb_uri contains cluster uri
In the server file where you host your routes add this to the end of your file and change to match your app:
app.use(express.static(__dirname + "/../client/build"));
app.get("*", (req, res) => {
// check for page routes that we expect in the frontend to provide correct status code.
const goodPageRoutes = ["/", "/login", "/home"];
if (!goodPageRoutes.includes(req.url)) {
// if url not in expected page routes, set status to 404.
res.status(404);
}
// send index.html
res.sendFile(__dirname + "/../client/build/index.html");
});
After that just follow the guide: https://devcenter.heroku.com/articles/git and https://devcenter.heroku.com/articles/deploying-nodejs
I have a code that allows me to send files to server folder uploads using multer from react client side the process of sending data to server works perfectly and also getting back the file from server works perfectly.
The client side is running under 3000 and the server is running under 4000 .
the problem that I'm facing right now is displaying the file on the front part for ex an img
<img
src={`http:\\localhost:4000\\server\\${text}`}
className="messageText colorWhite"
alt="img"
/>
the image of the error is below
the text contains uploads/image1.jpg
when I inspect the content I found this
which mean that the image is well called from server side
Could it be possible to help me on this?
Best Regards,
The upload folder which contains the files is not being served up by the server at localhost:4000.
You must server the files in the uploads folder first. For eg in an express server you can access the files by specifying this route
app.get('/uploads/:filename', (req, res) => {
res.sendFile(req.params.filename, {root: path.join(__dirname, '/uploads')});
})
Now setting src=http://localhost:4000\uploads\img.png would get the image img.png stored in the uploads folder
As an alternative solution, you can server up the entire uploads folder by
app.use(express.static("uploads"));
Make sure you specify the full path in "uploads"
I am working on my first ever MERN stack web app. I finished development and trying to host in VPS.
Everything working fine. But, I am not able to view the uploaded images. I share you the few details about the project structure.
React app is build for production and kept in public folder.
Upload folder is created where user uploaded images are kept and its location is stored in DB. (ex: /uploads/user1/img1.jpg). Using Multer to handle image upload.
This is the folder structure of my Express server which also servers static files.
-MAIN_EXPRESS_APP_FOLDER
------Routes
------Public
------Uploads
------package.json
To make public and upload static, following codes are added in Server.js file:
app.use(express.static(path.join(__dirname,'/public')));
app.get('/*',(req,res)=>{
res.sendfile(path.join(__dirname='public/index.html'));
})
app.use('/uploads', express.static('uploads'))
I am able to access my web app and also i can insert data in DB. Image upload is working as expected. But, Express not serving uploaded images.
What could be problem?
As for static content you set the folder to be public now what i think is you should move your uploads folder inside public and try to access the images hope they will be served currently your application is not allowing to server data else then public directory hope it will help you
I have an issue while uploading file. I am trying to upload file into folder using Node.js and my app is deployed in Heroku. I am providing my code below.
var multer = require('multer')
var storage =multer.diskStorage({
destination: function (req, file, callback) {
callback(null, './uploads');
},
filename: function (req, file, callback) {
callback(null, Date.now()+'-'+file.originalname);
}
});
var upload = multer({ storage : storage });
app.post('/api/users/save-card-file',upload.single('file'), function (req, res, next) {
var data={'filename':res.req.file.filename};
res.send(data);
})
Here I am trying to upload file into uploads folder and it's working fine while running in localhost but I am uploading my App into Heroku and there the response is coming but no file uploaded into uploads folder. I am getting the below response after uploading the file on Heroku.
{
"filename": "1503419364442-btechmarksheet.jpg"
}
But no file is present inside uploads folder. It's working file in localhost but in Heroku I am getting this type of issue.
From https://devcenter.heroku.com/articles/dynos#ephemeral-filesystem:
Each dyno gets its own ephemeral filesystem, with a fresh copy of the most recently deployed code. During the dyno’s lifetime its running processes can use the filesystem as a temporary scratchpad, but no files that are written are visible to processes in any other dyno and any files written will be discarded the moment the dyno is stopped or restarted. For example, this occurs any time a dyno is replaced due to application deployment and approximately once a day as part of normal dyno management.
Heroku's filesystem CAN NOT be used as a persistent filesystem. Dyno restarts after some idle time between subsequent requests, re-initializing the filesystem and this is the reason even when the upload succeeds, you're not able to find the file later.