They way Multer works on top of express is wired!, why Multer should precede the controller in the chain of Middlewares, which which by design causes the server to upload stuff before the DB operation is even checked?
For instance if there was a post operation to articles, and it contains a bunch of fields one of them is a file.
articleModel{title:String,image:String};
router.post('/', multer, articleController.createArticle);
now at the time the request hits, first thing in the chain is to upload the file in the request, but what if an error happened at the execution of the record to the DB like validation or even duplicates, what if I am going to update the article title only? the old files will be uploaded again?
How to make multer upload the files in the response of the http operation callback?
You can indeed make all kind of stuff before Multer actually process the image:
var upload = multer({
dest: 'uploads/',
fileFilter: function (req, file, cb) {
// only images are allowed
var filetypes = /jpeg|jpg|png/;
var mimetype = filetypes.test(file.mimetype);
var extname = filetypes.test(path.extname(file.originalname).toLowerCase());
if (mimetype && extname) {
return cb(null, true);
}
cb("Error");
}
}).single('localImg');
app.post('/api/file', checkBody, auth, uploadFile, controller.aController);
Take this code for example, you can make all kind of middleware actions BEFORE multer process your file, but multer is a library to process multipart/form-data, not files only, people use multipart for sending files mainly but you can send all kind of data too and it will append them to the body (req.body)
Your question is: "Why multer should upload files before any operation?"
You can execute multer when ever you want, but multer will process the request and get your data into the body. Unless you don't need the body data first hand, you need multer to be in the first middleware.
Your other question is: "what if I am going to update the article title only? the old files will be uploaded again?"
No, it will be uploaded once, if there is any problem with the database, any error or reject, you can always use the filesystem (fs) to remove the file from your server, if you already upload it to a third party system, you can delete it.
Hope it helps
Related
I am using Multer for upload files using Nodejs. Problem is that I cannot verify exist file on server before it deliver to server.
I will check original name, mime type and file size to make sure it is same.
fileFilter: async function (req, file, callback) {
//but file here return only `original name` and `mime type`.
}
The files are sent to server successfully, multer return in req.files but files uploaded. It is not middleware anymore.
I'm new to NodeJS, I'm developing app where I have created registeration form. To upload images, I'm using multer. And for form validation I'm using express-validator. The issue with me is that, before form validation, file has upload. So, if user hasn't created account because of some validation errors, but file goes upload. The problem is that I get file upload for each form submition no matter, account has successfully created or not.
What I want to do?
I want save file only when user has successfully created account. So, is there anyway to use multer inside callback, so when my form validations are done, then files goes upload. I can't use express-validation middleware between multer midleware and final callback, because express-validation passes it's errors to next call back.
Here is my some code
router.post('/registeration', multerMiddleware, validationMiddleware, finalCallback)
validationMiddleware is an array.
var multer = require('multer');
var multerMiddleware= multer({dest: './profile-images', fileFilter: userRegisteration.fileFilter}).single('profileImage');
var validationMiddleware = [check(...), check(...),...]
var finalCallback = function (req, res, next) {
// code is here
}
Express router executes the middlewares in the same sequence you mount
them
router.post('/registeration', multerMiddleware, validationMiddleware, finalCallback)
So, your multerMiddleware is executed ( and hence saved the file ) before your validationMiddleware has done the validation.
Change the code to:
router.post('/registeration', validationMiddleware,multerMiddleware, finalCallback)
So, multerMiddleware will only save file once the request is validated
My application's need is as follows:
I upload the image to Cloudinary and store the url of image in my mongodb database.
To upload the image to cloudinary, I needed to give the file path, and for that, I am using multer.I use the statement:
app.use(multer({ dest: './uploads/'}));
The problem I face is that, everytime I upload an image from my local system to the database on Cloudinary, a local copy gets created in './uploads/'.I want this to not happen, since my images are there on Cloudinary.
I read the documentation of multer where it says:
Multer accepts an options object, the most basic of which is the dest property, which tells Multer where to upload the files. In case you omit the options object, the file will be renamed and uploaded to the temporary directory of the system.
I am unable to make out if that temporary upload space needs cleaning or if it is taken care of.My images are uploaded on Cloudinary.I used multer only for getting :
req.files.photo.path
to work.Is there any other way to do the same or if multer can be configured in a way to not store images to my local system?
July 2018:
So if you want to store an image into some sort of database, you probably want to convert it into a buffer, and then insert buffer into the database. If you are using multer to process the multipart form (in this case the image file you want to upload), you can use multer's memoryStorage to get the buffer of the image directly. In other words:
var storage = multer.memoryStorage();
var upload = multer({ storage: storage });
to get the buffer:
When using memory storage, the file info will contain a field called buffer that contains the entire file.
that means you can get the buffer from command like req.files[0].buffer
I updated a simple repo demonstrating upload images to database without making a copy to local in this repo
From Multer info page at
https://www.npmjs.com/package/multer
If the inMemory option is true - no data is written to
disk but data is kept in a buffer accessible in the file object.
Prior to 14-07-2016 you could not configure multer to store files locally.
But multer is just a wrapper of busboy. So, we can try use busboy directly if we want to avoid hitting the disk.
Connect-bubsboy will help us:
app.use(busboy());
// request handler
req.busboy.on('file', function(fieldname, file, filename, encoding, mimetype) {
file.on('end', function() {
// do stuff with req.files[fieldname]
});
});
Update 14-07-2016
Now multer has MemoryStorage to store files in memory as Buffer.
var storage = multer.memoryStorage()
var upload = multer({ storage: storage })
You can also delete the file once it's uploaded to Cloudinary.
// used to delete images from local directory
const fs = require('fs'); // gain access to file system
const util = require('util');
const deleteFile = util.promisify(fs.unlink); // unlink will delete the file
// in your post request
app.post('/images', upload.single('myFile'), async function(req, res) {
const file = req.file; // multer gives access to the file object in the request
// code to upload the file to Cloudinary
await deleteFile(file.path); // remove locally stored image by passing the file's path
});
If you're storing images with Cloudinary then try using multer-storage-cloudinary
I should implement an upload form
I thought of using bodyparser but I read
http://andrewkelley.me/post/do-not-use-bodyparser-with-express-js.html
so what's the way to upload a file with express using the mean stack ?
may be formidable or other modules ?
That warning is specifically against adding the express.bodyparser middleware to your entire stack as it adds express.multipart to all POST endpoints and therefore file uploads are automatically accepted at all POST endpoints. By default the framework automatically saves any uploaded files to /tmp and so unless you are cleaning them up an attacker could flood your disk with uploaded files.
If you want to avoid using additional modules, what you should do is implement express.multipart on the endpoint(s) where you want to allow file uploads. Here's what I'm talking about:
var express = require("express")
, app = express();
// middleware (no bodyparser here)
app.use(express.json());
app.use(express.urlencoded());
// average GET endpoint
app.get("/", function(req,res) {
res.send('ok');
});
// average POST endpont
app.post("/login", function(req,res) {
res.send('ok');
});
// File upload POST endpoint
app.post('/upload', express.multipart, function(req, res) {
//File upload logic here
//Make sure to delete or move the file accordingly here, otherwise files will pile up in `/tmp`
});
Note the inclusion of express.multipart in the file upload endpoint. This endpoint will now process multipart file uploads, and assuming you handle them correctly they won't be a threat.
Now, having told you all of this, Connect is moving to deprecate multipart due to this exact issue, but there don't seem to be any plans to add a stream based file upload replacement. What they instead recommend is that you use node-multiparty which uses streams to avoid ever placing a file on disk. However, there don't seem to be any good references I can find for using multiparty as a middleware without saving files though, so you'll have to contact the author of multiparty or take a closer look at the API for implementing it with Express.
I created an example that uses Express & Multer - very simple, avoids all Connect warnings
https://github.com/jonjenkins/express-upload
I'm trying to build a quick and simple image uploading service with Node, that takes the received images and saves them to Mongo's GridFS.
GridFS get requires a Buffer object NodeJS Mongo Driver GridFS put
The question is pretty simple: how do I exactly cast/transform the received request body into a proper buffer.
My code so far (only the important pieces):
api.js
var express = require('express');
var connect = require('connect');
var app = module.exports = express.createServer();
app.configure(function(){
app.use(express.bodyParser());
app.use(express.methodOverride());
app.use(app.router);
});
var upload = require('./upload.js');
app.post('/upload', upload.upload);
upload.js
exports.upload = (function(req, res, next){
console.log("Uploading image...");
// Create buffer
// Rest of the code
}
I've tried:
var buffer = new Buffer(util.inspect(req.body),'binary');
Creates the buffer, but it has a wrong size and probably not the correct content since util.inspect is obviously not the right way to go.
And:
var buffer = new Buffer(req.body);
Result:
[Decode error - output not utf-8][Decode error - output not utf-8]
Buffer length = 0
I'm quite new to both Node and JavaScript developing in general, so probably I'm missing something quite simple, don't hesitate to point the obvious :)
Thanks!
First, remember that Express is built on top of Connect, which is the library that handles a large amount of the lower-level HTTP work, and it's where bodyParser() comes from.
The body parser middleware internally uses Formidable to parse file uploads.
Formidable's default behavior is to write uploaded files directly to disk – in other words, you don't actually have access to the uploaded file stream within your route handler. You get the values of any regular form fields (<input>s) sent along in req.body, and you get uploaded file paths via req.files, but you don't get file content.
The easy answer here is to simply read the file from disk and use that to save into Mongo, remembering to delete the temporary file when done. Of course, this introduces the unnecessary intermediate step of writing the file upload to a temporary folder and then loading to Mongo.
If you want to stream file data directly into Mongo, you have more of a challenge in front of you. You'll have to write your own middleware to parse the upload stream.
This is actually relatively easy. You can just start with the internal Connect body parser implementation—using Formidable to do the heavy lifting—and use the onPart API to pass the stream back to your route handler so that you can pass it off to the Mongo driver.