I deployed an API to heroku. One of its functions is file uploading. I made a static folder called "uploads" and upload files using multer package.
app.use('/api/uploads', uploadRouter);
app.use('/uploads', express.static(path.join(__dirname, '/uploads')));
Above code snippet comes from server.js. Below is the router and multer setup.
const storage = multer.diskStorage({
destination(req, file, cb) {
cb(null, 'uploads/');
},
filename(req, file, cb) {
cb(null, `${Date.now()}.jpg`);
},
});
const upload = multer({ storage });
uploadRouter.post('/', upload.single('image'), (req, res) => {
res.send(`https://app.herokuapp.com/${req.file.path}`);
});
uploadRouter.post('/imgs', upload.array('images'), (req, res) => {
try{
let filesArray = [];
req.files.forEach(file => {
filesArray.push(`https://app.herokuapp.com/${file.path}`)
})
res.status(201).send(filesArray);
}catch(error) {
res.status(400).send(error.message);
}
});
Something tells me that my approach is far from an ideal one. However, it used to work just perfect until I introduced multiple file upload. Now, images after being uploaded stay there for a little bit (I can access them via links like https://app.herokuapp.com/uploads/1623782012131.jpg) and disappear over few minutes. Then, these URLs throw the following:
Cannot GET /uploads/1623782012131.jpg
The thing is that my previous images that I uploaded before introducing multiple file upload are still accessible. I thought it had to do with heroku file size limitations but it was not the case since I've only used about 50mb out of 500mb. What might be the reason?
Heroku's filesystem is ephemeral/short lived. This means that any images you save on the disk will disappear after you restart or deploy your app. You can't rely on Heroku's filesystem to store images that need to persist for a longer time. Read more here: https://help.heroku.com/K1PPS2WM/why-are-my-file-uploads-missing-deleted
For a reliable image storage, have a look at cloud storage solutions such as Cloudinary or AWS S3.
Related
I am upload images using multer and Nodejs. The upload works perfectly. The problem is I want to upload those images directly to a folder inside wwwroot folder of the IIS Webserver.
var Storage = multer.diskStorage({
destination: function (req, file, callback) {
// callback(null, "../grit-server/uploads/");
callback(null, "https://example.com/grit/images/photogallery/");
},
filename: function (req, file, callback) {
const sqc = sqlConn;
var fileNames = (file.originalname.split('_'))
let query = 'INSERT INTO dbo.MS_PhotoGallery (MinorId,Photo_Name) VALUES (#MinorId,#PhotoName)'
let params = [
sqc.paramize('MinorId', fileNames[1], sql.Int()),
sqc.paramize('PhotoName',fileNames[2], sql.VarChar(255))
]
sqc.SqlExecuteStatementCallback(
query,params,(result)=>{
res.send(result)
}
)
callback(null, file.originalname);
}
});
The above code is of the Multer storage where I am setting the URL where to upload the images. But it is throwing error
'Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client'
What modifications shall I do to upload the images in the URL mentioned under Storage to save any photos in the location.
I have solved this problem hardcoding the path like '../../../../wwwroot/abc/def/'. This is not the efficient way but it works for my specific problem as the images folder is not going to change in future.
I have a nodejs backend with Multer to upload images to a local folder named /uploads and after that, I store the image name in my MySQL database. When I run this on localhost the file gets uploaded to the folder and everything seems to work, but after I push the code to Heroku, the image doesn't get uploaded to the folder anymore. I can still access the file when I fetch the URL of where the file should be, but I can't see it. and when I push new code to Heroku then since the image is not in the folder. I'm guessing it gets overridden.
Anyone can explain to me why or how? thanks!
app.post('/api/register', upload.single('avatar'), function (req, res, next) {
pool.getConnection(async (err, connection) => {
if (err) throw err
console.log(req.file);
const username = req.body.username;
var password = req.body.password;
const email = req.body.email;
const picture = req.file.filename;
const salt = await bcrypt.genSalt();
password = await bcrypt.hash(password, salt);
connection.query(`INSERT INTO users (username, password, email, picture) VALUES (?,?,?,?)`,
[username, password, email, picture], (err, rows) => {
connection.release() // return the connection to pool
if (!err) {
console.log("insert succes");
res.send(`user with the record ID has been added.`)
} else {
console.log(err)
}
})
})
})
const storage = multer.diskStorage({
destination(req, file, callback) {
callback(null, './uploads');
},
filename(req, file, callback) {
callback(null, `${file.fieldname}_${Date.now()}_${file.originalname}`);
},
});
const upload = multer({ storage })
const app = express()
app.use('/uploads', express.static(process.cwd() + '/uploads'));
app.use(express.json());
app.use(cors());
As mentioned in the comment via the link, because Heroku is more of a 'container' all storage is ephemeral (meaning it on the running container and is not persistent) So each time a container spins up its just loading your base code into the running image nothing more nothing less. When the image restarts or rebuilds it's a fresh instance and any items added while it was running are basically lost..
So, unless your application is 'stateless' then will need to seperate things like
Sessions
Persistent storage
Fortunately, this is not too difficult to setup. For sessions you would want to use something like Redis to store session information and for images / assets you can use S3 or other object store. Multer has a good s3 connector ( works with any s3 compatible object store), then in your DB you just store the file name and to return it, use the s3 url ( media.example.com/images/[image]) in your application..
Now, to your question.. You should still be able to upload files to the running container (unless permissions prevent you ) there are use-cases where it does make sense, for example its just a temp file that will be read right away in the same thread as the request etc.
From your code it looks like you are trying to upload to the 'uploads' folder off the root context of your app, so I would first ensure you have an 'uploads' folder and it has permissions to read / write, also the correct owner as well. Node should give you an error message in the console if there is an error so you will need to SSH to the running heroku instance to see what is going on from that point.
I am able to upload very large files (12-15GB) when running my NodeJS server locally using multer. However, I'm having trouble getting large files to upload when deploying the node server on AWS.
When the node server is deployed, I can upload small files, but when uploading large files it hangs and never returns until I kill the node server. This seems like an environment issue inherent to being on AWS because I never get this problem locally.
In AWS I have my node server deployed on a t2.micro Windows Server 2016 Datacenter.
Is there something I'm missing fundamentally about AWS setup when transferring large files over the network? Some file limit somewhere? Something with Windows Firewall or a setting in my AWS profile?
Since I have been able to FTP any size file to my AWS server, I didn't think there was a file limit size imposed. So I'm inclined to believe it's probably some other issue.
Let me know if any further information is needed to help out with this if it's not obvious immediately.
Here is the multer part of my nodejs code in case that helps.
import { Router } from 'express';
import multer from 'multer';
var storage = multer.diskStorage({
destination: function (req, file, cb) {
cb(null, 'C:/inetpub/wwwroot/Archive/' + req.body.destination)
},
filename: function (req, file, cb) {
cb(null, file.originalname)
}
});
let upload = multer({ storage });
export default ({ config, db }) => {
let api = Router();
var cpUpload = upload.fields([
{ name: 'destination' },
{ name: 'files' }])
api.post('/upload', cpUpload, function (req, res, next) {
res.json(req.files);
});
return api;
}
I have configured multer as;
var storage = multer.diskStorage({
destination: function(req, file, cb) {
cb(null, '../images/profile');
},
filename: function(req, file, cb) {
cb(null, req.body.username + '.jpeg'); // file does not get renamed
}
});
var upload = multer({storage: storage});
// Route that uses multer
router.post('/auth/signup/upload', upload.single('image'), function(req, res) {
console.log(req.body.username); // contains value
res.send();
});
Although req.body.username has a value, the file does not get renamed.
Wht am i missing here ?
From the multer manual:
Note that req.body might not have been fully populated yet. It depends on the order that the client transmits fields and files to the server.
Sadly, I don't believe there's a nice way to solve this. You could try switching the order of the fields in your HTML form, but this probably won't lead to consistent behaviours across browsers. You could also send the username on the query string instead (i.e. POST the file to http://foo.bar?username=me). You could also manually move the file afterwards, or store the mappings between usernames and files elsewhere.
I'm using multer to upload images with express and node from a form, however all the files names come out like "8f92a1388f70c6c88eb32489f6bcfcc9". There isn't even an extension attached. How to I display this on the client side?
try:
var storage = multer.diskStorage({
destination: function (req, file, cb) {
cb(null, '/where/ever/the/upload/dir/is')
},
filename: function (req, file, cb) {
cb(null, file.orignalname)
}
})
var upload = multer({ storage: storage })
Instead of:
var upload = multer({ dest: 'uploads/' })
Requesting the file:
With the proper permissions set on the file/or directory your server should be able to request it fine, remember to explicitly write the file name with an extension if you aren't doing anything fancy after the file is written ;)
If you want more control over your uploads, you'll want to use the
storage option instead of dest. Multer ships with storage engines
DiskStorage and MemoryStorage; More engines are available from third
parties.
--The Horse
(ref: github: expressjs/multer)
Note: Multer will not append any file extension for you, your function should return a filename complete with an file extension