I am able to upload very large files (12-15GB) when running my NodeJS server locally using multer. However, I'm having trouble getting large files to upload when deploying the node server on AWS.
When the node server is deployed, I can upload small files, but when uploading large files it hangs and never returns until I kill the node server. This seems like an environment issue inherent to being on AWS because I never get this problem locally.
In AWS I have my node server deployed on a t2.micro Windows Server 2016 Datacenter.
Is there something I'm missing fundamentally about AWS setup when transferring large files over the network? Some file limit somewhere? Something with Windows Firewall or a setting in my AWS profile?
Since I have been able to FTP any size file to my AWS server, I didn't think there was a file limit size imposed. So I'm inclined to believe it's probably some other issue.
Let me know if any further information is needed to help out with this if it's not obvious immediately.
Here is the multer part of my nodejs code in case that helps.
import { Router } from 'express';
import multer from 'multer';
var storage = multer.diskStorage({
destination: function (req, file, cb) {
cb(null, 'C:/inetpub/wwwroot/Archive/' + req.body.destination)
},
filename: function (req, file, cb) {
cb(null, file.originalname)
}
});
let upload = multer({ storage });
export default ({ config, db }) => {
let api = Router();
var cpUpload = upload.fields([
{ name: 'destination' },
{ name: 'files' }])
api.post('/upload', cpUpload, function (req, res, next) {
res.json(req.files);
});
return api;
}
Related
I am upload images using multer and Nodejs. The upload works perfectly. The problem is I want to upload those images directly to a folder inside wwwroot folder of the IIS Webserver.
var Storage = multer.diskStorage({
destination: function (req, file, callback) {
// callback(null, "../grit-server/uploads/");
callback(null, "https://example.com/grit/images/photogallery/");
},
filename: function (req, file, callback) {
const sqc = sqlConn;
var fileNames = (file.originalname.split('_'))
let query = 'INSERT INTO dbo.MS_PhotoGallery (MinorId,Photo_Name) VALUES (#MinorId,#PhotoName)'
let params = [
sqc.paramize('MinorId', fileNames[1], sql.Int()),
sqc.paramize('PhotoName',fileNames[2], sql.VarChar(255))
]
sqc.SqlExecuteStatementCallback(
query,params,(result)=>{
res.send(result)
}
)
callback(null, file.originalname);
}
});
The above code is of the Multer storage where I am setting the URL where to upload the images. But it is throwing error
'Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client'
What modifications shall I do to upload the images in the URL mentioned under Storage to save any photos in the location.
I have solved this problem hardcoding the path like '../../../../wwwroot/abc/def/'. This is not the efficient way but it works for my specific problem as the images folder is not going to change in future.
I deployed an API to heroku. One of its functions is file uploading. I made a static folder called "uploads" and upload files using multer package.
app.use('/api/uploads', uploadRouter);
app.use('/uploads', express.static(path.join(__dirname, '/uploads')));
Above code snippet comes from server.js. Below is the router and multer setup.
const storage = multer.diskStorage({
destination(req, file, cb) {
cb(null, 'uploads/');
},
filename(req, file, cb) {
cb(null, `${Date.now()}.jpg`);
},
});
const upload = multer({ storage });
uploadRouter.post('/', upload.single('image'), (req, res) => {
res.send(`https://app.herokuapp.com/${req.file.path}`);
});
uploadRouter.post('/imgs', upload.array('images'), (req, res) => {
try{
let filesArray = [];
req.files.forEach(file => {
filesArray.push(`https://app.herokuapp.com/${file.path}`)
})
res.status(201).send(filesArray);
}catch(error) {
res.status(400).send(error.message);
}
});
Something tells me that my approach is far from an ideal one. However, it used to work just perfect until I introduced multiple file upload. Now, images after being uploaded stay there for a little bit (I can access them via links like https://app.herokuapp.com/uploads/1623782012131.jpg) and disappear over few minutes. Then, these URLs throw the following:
Cannot GET /uploads/1623782012131.jpg
The thing is that my previous images that I uploaded before introducing multiple file upload are still accessible. I thought it had to do with heroku file size limitations but it was not the case since I've only used about 50mb out of 500mb. What might be the reason?
Heroku's filesystem is ephemeral/short lived. This means that any images you save on the disk will disappear after you restart or deploy your app. You can't rely on Heroku's filesystem to store images that need to persist for a longer time. Read more here: https://help.heroku.com/K1PPS2WM/why-are-my-file-uploads-missing-deleted
For a reliable image storage, have a look at cloud storage solutions such as Cloudinary or AWS S3.
I have a nodejs backend with Multer to upload images to a local folder named /uploads and after that, I store the image name in my MySQL database. When I run this on localhost the file gets uploaded to the folder and everything seems to work, but after I push the code to Heroku, the image doesn't get uploaded to the folder anymore. I can still access the file when I fetch the URL of where the file should be, but I can't see it. and when I push new code to Heroku then since the image is not in the folder. I'm guessing it gets overridden.
Anyone can explain to me why or how? thanks!
app.post('/api/register', upload.single('avatar'), function (req, res, next) {
pool.getConnection(async (err, connection) => {
if (err) throw err
console.log(req.file);
const username = req.body.username;
var password = req.body.password;
const email = req.body.email;
const picture = req.file.filename;
const salt = await bcrypt.genSalt();
password = await bcrypt.hash(password, salt);
connection.query(`INSERT INTO users (username, password, email, picture) VALUES (?,?,?,?)`,
[username, password, email, picture], (err, rows) => {
connection.release() // return the connection to pool
if (!err) {
console.log("insert succes");
res.send(`user with the record ID has been added.`)
} else {
console.log(err)
}
})
})
})
const storage = multer.diskStorage({
destination(req, file, callback) {
callback(null, './uploads');
},
filename(req, file, callback) {
callback(null, `${file.fieldname}_${Date.now()}_${file.originalname}`);
},
});
const upload = multer({ storage })
const app = express()
app.use('/uploads', express.static(process.cwd() + '/uploads'));
app.use(express.json());
app.use(cors());
As mentioned in the comment via the link, because Heroku is more of a 'container' all storage is ephemeral (meaning it on the running container and is not persistent) So each time a container spins up its just loading your base code into the running image nothing more nothing less. When the image restarts or rebuilds it's a fresh instance and any items added while it was running are basically lost..
So, unless your application is 'stateless' then will need to seperate things like
Sessions
Persistent storage
Fortunately, this is not too difficult to setup. For sessions you would want to use something like Redis to store session information and for images / assets you can use S3 or other object store. Multer has a good s3 connector ( works with any s3 compatible object store), then in your DB you just store the file name and to return it, use the s3 url ( media.example.com/images/[image]) in your application..
Now, to your question.. You should still be able to upload files to the running container (unless permissions prevent you ) there are use-cases where it does make sense, for example its just a temp file that will be read right away in the same thread as the request etc.
From your code it looks like you are trying to upload to the 'uploads' folder off the root context of your app, so I would first ensure you have an 'uploads' folder and it has permissions to read / write, also the correct owner as well. Node should give you an error message in the console if there is an error so you will need to SSH to the running heroku instance to see what is going on from that point.
I am building an simple files uploader application in Nodejs. The files get successfully saved on the local storage in a directory 'upload' but now i have deployed my app to herukou. It is showing an error as. This is probably that there is no upload directory on herukou.
'use strict';
const express = require('express');
const multer=require('multer');
const uuid = require('uuid').v4;
const app = express();
let port=process.env.PORT || 8080;
var storage= multer.diskStorage({
destination: function(req,file,cb){
cb(null,'upload')
},
filename: function(req,file,cb){
cb(null,`${uuid()}-${file.originalname}`)
}
})
const upload = multer({ storage : storage});
app.use(express.static('Frontend'))
app.post('/upload', upload.array('file'),function(req, res){
res.send({
status: 200,
NumberofFileUploaded:req.files.length
})
})
app.listen(port, ()=> console.log(`Server Listening ${port}`))
The deployed link of my app is
https://fileuploader1997.herokuapp.com/
Anyone who can help me with this. Thanking you in advance
Heroku filesystem is ephemeral, so the files that you try to save are not going to be stored. I would suggest using something like AWS S3 or similar in order to persist the uploaded files.
I'll leave the link to the Heroku help page where they explain this in detail.
I am developing one web application in which server is stored in different Azure server and front-end is also stored in different one.
When I try to use multer in node js to store URL to front-end server it always returns that path is not found.when I saw the error log it I found this
Error: ENOENT: no such file or directory, open
'D:\home\site\wwwroot\http:\nomadiccare-portal.azurewebsites.net\images\undefined.png'
at Error (native)
Here is my source code for node js which is working perfectly on localhost.
var uploadImageURL="http://nomadiccare-portal.azurewebsites.net/images/";
var currentClientId;
var storage = multer.diskStorage({
destination: function (request, file, callback) {
callback(null, uploadImageURL);
},
filename: function (request, file, callback){
console.log("It Is In"+file);
callback(null,currentClientId+".png");
}
});
var upload = multer({ storage: storage });
How can I remove D:\home\site\wwwroot\ from URL?
destination of DiskStorage is used to determine within which folder the uploaded files should be stored. It must be the path of the disk rather than a URL of the website.
If the other server has a post route to upload a file, you can make a post request with the file data from your server after you saved the file successfully. Refer to this you can learn how to send HTTP post request from a server to another server with data in Node.js.
var storage = multer.diskStorage({
destination: uploadImageURL, // change here
filename: function (request, file, callback) {
console.log('It Is In' + file)
callback(null, currentClientId + '.png')
}
})