I have an issue while uploading file. I am trying to upload file into folder using Node.js and my app is deployed in Heroku. I am providing my code below.
var multer = require('multer')
var storage =multer.diskStorage({
destination: function (req, file, callback) {
callback(null, './uploads');
},
filename: function (req, file, callback) {
callback(null, Date.now()+'-'+file.originalname);
}
});
var upload = multer({ storage : storage });
app.post('/api/users/save-card-file',upload.single('file'), function (req, res, next) {
var data={'filename':res.req.file.filename};
res.send(data);
})
Here I am trying to upload file into uploads folder and it's working fine while running in localhost but I am uploading my App into Heroku and there the response is coming but no file uploaded into uploads folder. I am getting the below response after uploading the file on Heroku.
{
"filename": "1503419364442-btechmarksheet.jpg"
}
But no file is present inside uploads folder. It's working file in localhost but in Heroku I am getting this type of issue.
From https://devcenter.heroku.com/articles/dynos#ephemeral-filesystem:
Each dyno gets its own ephemeral filesystem, with a fresh copy of the most recently deployed code. During the dyno’s lifetime its running processes can use the filesystem as a temporary scratchpad, but no files that are written are visible to processes in any other dyno and any files written will be discarded the moment the dyno is stopped or restarted. For example, this occurs any time a dyno is replaced due to application deployment and approximately once a day as part of normal dyno management.
Heroku's filesystem CAN NOT be used as a persistent filesystem. Dyno restarts after some idle time between subsequent requests, re-initializing the filesystem and this is the reason even when the upload succeeds, you're not able to find the file later.
Related
Sorry in advance for my lengthy post.
I am having difficulty changing from MERN localhost to an online hosting. I've been hosting my frontend/ client side website on heroku, with the file structure as follows:
Client Structure
- build
- src
- App.js
- Index.js
- Other Components, Images, Fonts, Pages
-package.json
-package-lock.json
This file structure works fine with heroku.
However, I've been following a Youtube tutorial that uses a MERN stack to build a blog: https://www.youtube.com/watch?v=OML9f6LXUUs&t=3s --> it is a 3 part series
In the video, he has a file structure as such:
API and Client Structure
- api
- images
- models
- public
- routes
- index.js
- package.json
- package-lock.json
- client (frontend code)
- build
- src
- App.js
- Index.js
- Other Components, Images, Fonts, Pages
-package.json
-package-lock.json
2 Instances of LocalHost
He runs api on localhost:3001 and client on localhost:3000.
Methodology
1. Storing Images
He does not store images in MongoDB cloud, but rather store the image name only, and stores the images directly on the api/images directory.
In the api folder, he uses multer to store the images.
app.use("/images", express.static(path.join(__dirname, "/images")))
const storage = multer.diskStorage({
destination: (request, file, callback) => {
callback(null, 'images');
},
filename: (request, file ,callback) => {
callback(null, request.body.name);
},
})
const upload = multer({storage: storage});
app.post('/api/upload', upload.single('file'), (request, response) => {
response.status(200).json('File has been uploaded');
});
Then in the client side, he sends the data via axios:
const response = await axios.post('/upload', data);
2. Retrieving Images
When retrieving, he specifies a public folder PF, uses axios to get the image name, and then concatenates them to get the images.
const PF = "http://localhost:3001/images/";
<Image src={PF + post.image} />
My question is:
How can I change from a localhost to a cloud based hosting (preferably Heroku)? What paths do I specify? I would think that this image retrieval works because I am running both locally on the same computer. I wouldn't be able to use it if they are online as the image src directory will not be searchable
Is it possible to host both simultaneously on 1 Heroku Dyno, and how can I do it?
This is my first time learning backend so thanks for the patience!
Firstly, you'll have to remove all hardcoded 'http://localhost:3000`. This would interfere with the cloud site since it uses its own.
Instead, you could use an environment file to store the API site and in production use a different one.
You will need to make a mongodb cluster (its a cloud database and its free) https://www.mongodb.com/docs/atlas/tutorial/create-new-cluster/. You'll have to manually change your database to the cluster link. I would use an environment variable for this as well, so the db link would be something like this in your db file
const mongoURI = process.env.MONGODB_URI || 'mongodb://localhost:27017/ProjectAPI'
where process.env.mongodb_uri contains cluster uri
In the server file where you host your routes add this to the end of your file and change to match your app:
app.use(express.static(__dirname + "/../client/build"));
app.get("*", (req, res) => {
// check for page routes that we expect in the frontend to provide correct status code.
const goodPageRoutes = ["/", "/login", "/home"];
if (!goodPageRoutes.includes(req.url)) {
// if url not in expected page routes, set status to 404.
res.status(404);
}
// send index.html
res.sendFile(__dirname + "/../client/build/index.html");
});
After that just follow the guide: https://devcenter.heroku.com/articles/git and https://devcenter.heroku.com/articles/deploying-nodejs
I'm writing a small Node express app. Currently writing a post method to capture those requests with a file and trying to save locally. When I tested it with my own machine (localhost:8888), everything works fine. Then I used another machine which connects to same router and posting a request with a file, I couldn't capture the file at all. Below is my code,
var storage = multer.diskStorage({
destination: function (req, file, cb) {
cb(null, './uploads/');
},
filename: function (req, file, cb) {
console.log(req.file);
cb(null, Date.now() + path.extname(file.originalname));
},
}
);
var save = multer({ storage: storage });
router.post('/save', save.single('somefile'), (req, res) => {
console.log(req.file);
res.send();
});
When I test locally, I could see the file successfully uploaded to my "uploads" folder. But when I use another machine under same router, and calling this localhost by IP, although it shows status code 200, the req.file shows undefined, thus the file will not be saved to my localhost machine. What did I miss here?
Below's how I tested with Postman on both machines.
It turns out working fine after I switch to another testing env (from my friend's home to my own house). So it shouldn't be anything wrong with coding or testing. It's more likely to be the env, I'm suspecting the router. So that this question is no more valid under given categories. I don't know how to close this question. But folks could discuss the cause of this weird behavior if interested.
I am uploading a one megabyte file that is just 1mb of ones. It appears that any post page that does not handle file uploads will cause the server to save a temporary instance of that file that has an unknown time of existance. This is an issue that can be experienced in any post request. How can I fix this so that no one could use this as a vector of attack?
POST page http://cryptsy.tv/speedtest
speedtest: function(req,res){
res.end("Done");
}
https://github.com/justin7674/PeerJS-SwarmConnection/tree/master/Server
When using express-fileupload, any file upload to any post page (regardless of whether or not the post page is intended to handle file uploads) the temporary file will upload without permissions and the created file will stay in the server directory with no expiry time known/provided. This could allow for a server to be overfilled and memory maxed by simply sending a file to any post page available. I fixed it by passing this to every page utilizing POST or PUT.
,handlefiles: function(req,res,next){
const files = req.files || {};
function cleanupFiles () {
res.removeListener('finish', cleanupFiles);
res.removeListener('close', cleanupFiles);
const files = req.files || {};
for(var fn in files){
fs.unlinkSync(files[fn].path);
}
}
res.on('finish', cleanupFiles);
res.on('close', cleanupFiles);
next();
},
my Node.js app keeps losing static images(.jpg, .png...) after a while. It doesn't lose any images on my local win10 desktop and even on heroku, my webpack bundle.js is served from the same static route(/pub or /dist) and they work just fine. somehow only the static images, they are served alright for first few minutes when I first uploaded then after a while, it disappears. I am using express.static for static route declaration and multer for file upload. The files used for test were all lowercase .jpg(since I've heard heroku arbitrarily changes all uppercase extensions) so I don't know what's causing the problem.
server code:
const storage = multer.diskStorage({
destination: (req,file,cb)=>{
cb(null,'pub/')
},
filename: (req,file,cb)=>{
cb(null,Date.now() + file.originalname)
}
})
const upload = multer({storage:storage})
//access to static files
app.use('/pub', express.static(pubDir))
app.use('/dist', express.static(dstDir))
app.post('/modwimg',upload.any(),(req,res,next)=>{
//here I connect filename from files array to db
})
then if there's a client request, the server fetches filename from the db and put '/pub/' in front of it. It works just fine on both my local machine and heroku. it's only that images on heroku disappear after a while.
The heroku file system is transient. If you want to allow users to upload files to your app, you'll need to use external storage like S3, database blobs, or a hosted service like cloudinary. See this thread for more information: https://www.reddit.com/r/rails/comments/2k9sq4/heroku_any_files_you_upload_will_not_be_saved/
My application's need is as follows:
I upload the image to Cloudinary and store the url of image in my mongodb database.
To upload the image to cloudinary, I needed to give the file path, and for that, I am using multer.I use the statement:
app.use(multer({ dest: './uploads/'}));
The problem I face is that, everytime I upload an image from my local system to the database on Cloudinary, a local copy gets created in './uploads/'.I want this to not happen, since my images are there on Cloudinary.
I read the documentation of multer where it says:
Multer accepts an options object, the most basic of which is the dest property, which tells Multer where to upload the files. In case you omit the options object, the file will be renamed and uploaded to the temporary directory of the system.
I am unable to make out if that temporary upload space needs cleaning or if it is taken care of.My images are uploaded on Cloudinary.I used multer only for getting :
req.files.photo.path
to work.Is there any other way to do the same or if multer can be configured in a way to not store images to my local system?
July 2018:
So if you want to store an image into some sort of database, you probably want to convert it into a buffer, and then insert buffer into the database. If you are using multer to process the multipart form (in this case the image file you want to upload), you can use multer's memoryStorage to get the buffer of the image directly. In other words:
var storage = multer.memoryStorage();
var upload = multer({ storage: storage });
to get the buffer:
When using memory storage, the file info will contain a field called buffer that contains the entire file.
that means you can get the buffer from command like req.files[0].buffer
I updated a simple repo demonstrating upload images to database without making a copy to local in this repo
From Multer info page at
https://www.npmjs.com/package/multer
If the inMemory option is true - no data is written to
disk but data is kept in a buffer accessible in the file object.
Prior to 14-07-2016 you could not configure multer to store files locally.
But multer is just a wrapper of busboy. So, we can try use busboy directly if we want to avoid hitting the disk.
Connect-bubsboy will help us:
app.use(busboy());
// request handler
req.busboy.on('file', function(fieldname, file, filename, encoding, mimetype) {
file.on('end', function() {
// do stuff with req.files[fieldname]
});
});
Update 14-07-2016
Now multer has MemoryStorage to store files in memory as Buffer.
var storage = multer.memoryStorage()
var upload = multer({ storage: storage })
You can also delete the file once it's uploaded to Cloudinary.
// used to delete images from local directory
const fs = require('fs'); // gain access to file system
const util = require('util');
const deleteFile = util.promisify(fs.unlink); // unlink will delete the file
// in your post request
app.post('/images', upload.single('myFile'), async function(req, res) {
const file = req.file; // multer gives access to the file object in the request
// code to upload the file to Cloudinary
await deleteFile(file.path); // remove locally stored image by passing the file's path
});
If you're storing images with Cloudinary then try using multer-storage-cloudinary