I'm trying to build a quick and simple image uploading service with Node, that takes the received images and saves them to Mongo's GridFS.
GridFS get requires a Buffer object NodeJS Mongo Driver GridFS put
The question is pretty simple: how do I exactly cast/transform the received request body into a proper buffer.
My code so far (only the important pieces):
api.js
var express = require('express');
var connect = require('connect');
var app = module.exports = express.createServer();
app.configure(function(){
app.use(express.bodyParser());
app.use(express.methodOverride());
app.use(app.router);
});
var upload = require('./upload.js');
app.post('/upload', upload.upload);
upload.js
exports.upload = (function(req, res, next){
console.log("Uploading image...");
// Create buffer
// Rest of the code
}
I've tried:
var buffer = new Buffer(util.inspect(req.body),'binary');
Creates the buffer, but it has a wrong size and probably not the correct content since util.inspect is obviously not the right way to go.
And:
var buffer = new Buffer(req.body);
Result:
[Decode error - output not utf-8][Decode error - output not utf-8]
Buffer length = 0
I'm quite new to both Node and JavaScript developing in general, so probably I'm missing something quite simple, don't hesitate to point the obvious :)
Thanks!
First, remember that Express is built on top of Connect, which is the library that handles a large amount of the lower-level HTTP work, and it's where bodyParser() comes from.
The body parser middleware internally uses Formidable to parse file uploads.
Formidable's default behavior is to write uploaded files directly to disk – in other words, you don't actually have access to the uploaded file stream within your route handler. You get the values of any regular form fields (<input>s) sent along in req.body, and you get uploaded file paths via req.files, but you don't get file content.
The easy answer here is to simply read the file from disk and use that to save into Mongo, remembering to delete the temporary file when done. Of course, this introduces the unnecessary intermediate step of writing the file upload to a temporary folder and then loading to Mongo.
If you want to stream file data directly into Mongo, you have more of a challenge in front of you. You'll have to write your own middleware to parse the upload stream.
This is actually relatively easy. You can just start with the internal Connect body parser implementation—using Formidable to do the heavy lifting—and use the onPart API to pass the stream back to your route handler so that you can pass it off to the Mongo driver.
Related
I have a client in React that sends a form data with a file. When that file arrives to the server, the body is parsed by body parser and its result is a buffer. The idea is that the file keep saved in some place of my server, because I want to use it later from my client. So I'd like to know how should I handle this problem.
I tried to write directly this buffer as a file with fs, but the file created has an error of format, so I can't access it.
You can do stuff like this
var fs = require('fs');
fs.writeFile('newImage', req.files.image, function (err) {
if (err) throw err;
console.log("It's saved");
});
correct parameter order of fs.writeFile is the filename first then the content.
If you're using express.bodyParser() you'll have the uploaded files in the req.files field.
And of course you should write the callback:
POV: Your image file should be in req.files not the body.
I think you need a package for storing a file on your backend service. I had used morgan package for that and I was satisfied with using it. I have just searched other packages for storing a file, i found express-fileupload package. Maybe you want to look at how to use those. If you want to store a file, using the third package would be better for you and for your effort.
I want to build a NodeJS server that accepts a .wav file (1Mb) sent to its single endpoint, then changes the file through AudioContext API and then sends back the response with the result?
The server shouldn't store anything, so, no database required.
How can I achieve this? (or, please correct me if don't understand how things work)
I would do this with express: https://expressjs.com/
and as middleware add express-fileuplaod: https://www.npmjs.com/package/express-fileupload
app.post('/upload', function(req, res) {
console.log(req.files.foo); // the uploaded file object
});
instead of the console.log(); you'd make a readable stream / buffer and then use it in the
AudioContext API
here is also a interesting Article explaining to use this:
https://www.russellgood.com/process-uploaded-file-web-audio-api/
So im new on using expressjs, usually i choose Laravel as my backend. but because some certain consideration, i choose expressjs.
On Laravel, when we handling file upload, we can write upload logic everywhere, its your freedom to do that. You can encapsulate it inside your model function, or put it on service, or anywhere you want.
But when i use expressjs, so many articles on internet that recommend us to use multer for upload your file. As my background is using Laravel previously, i found its weird to use multer. Its because multer is a middleware. Why on earth we use middleware to upload our images/files.
With this i cant encapsulate my business logic into one service and its make the code separated and with this thats mean i need to maintain one business logic from multiple place.
Could you explain me why everyone choose multer ?
why dont just upload it to our local storage manually ?( actually for now i dont know how to do this ).
What is pros on mins from using this library ?
multer is a body parsing middleware that handles content type multipart/form-data
That means it parses the raw http request data which are primarily used for file upload, and makes it more accessible (storing on disk / in memory /...) for further processing.
Without multer, you would have to parse the raw data yourself if you want to access the file.
With this i cant encapsulate my business logic into one service and
its make the code separated and with this thats mean i need to
maintain one business logic from multiple place. Could you explain me
why everyone choose multer ?
multer, just like other middlewares, can be used at the root for all routes, but can also be put on only specific routes too.
More on express middleware
First of all, Express/body-parser does not handle file uploads natively, so that is why you see other libraries being loaded to handle them. They are all going to be loaded as middleware so they can be injected into the request and handle that a file was actually uploaded.
Coming from a Symfony background, I understand where you are coming from with wanting to handle things more manually, as I do the same. There are other alternatives to multer; for example I use express-fileupload which allows you to load the the uploading middleware for your entire app, and then you can use req.files to handle your uploads. For example:
// load the file upload library as app middleware
const fileUpload = require('express-fileupload');
app.use(fileUpload({
limits: { fileSize: 50 * 1024 * 1024 },
}));
Then let's say you have a file input named 'foo':
<input name="foo" type="file" />
In your route you would handle it like so:
// now handle a file upload
app.post('/upload', function(req, res) {
console.log(req.files.foo); // the uploaded file object
});
The file-upload documentation has examples for the req.files object as well as options you can pass to the middleware itself.
They way Multer works on top of express is wired!, why Multer should precede the controller in the chain of Middlewares, which which by design causes the server to upload stuff before the DB operation is even checked?
For instance if there was a post operation to articles, and it contains a bunch of fields one of them is a file.
articleModel{title:String,image:String};
router.post('/', multer, articleController.createArticle);
now at the time the request hits, first thing in the chain is to upload the file in the request, but what if an error happened at the execution of the record to the DB like validation or even duplicates, what if I am going to update the article title only? the old files will be uploaded again?
How to make multer upload the files in the response of the http operation callback?
You can indeed make all kind of stuff before Multer actually process the image:
var upload = multer({
dest: 'uploads/',
fileFilter: function (req, file, cb) {
// only images are allowed
var filetypes = /jpeg|jpg|png/;
var mimetype = filetypes.test(file.mimetype);
var extname = filetypes.test(path.extname(file.originalname).toLowerCase());
if (mimetype && extname) {
return cb(null, true);
}
cb("Error");
}
}).single('localImg');
app.post('/api/file', checkBody, auth, uploadFile, controller.aController);
Take this code for example, you can make all kind of middleware actions BEFORE multer process your file, but multer is a library to process multipart/form-data, not files only, people use multipart for sending files mainly but you can send all kind of data too and it will append them to the body (req.body)
Your question is: "Why multer should upload files before any operation?"
You can execute multer when ever you want, but multer will process the request and get your data into the body. Unless you don't need the body data first hand, you need multer to be in the first middleware.
Your other question is: "what if I am going to update the article title only? the old files will be uploaded again?"
No, it will be uploaded once, if there is any problem with the database, any error or reject, you can always use the filesystem (fs) to remove the file from your server, if you already upload it to a third party system, you can delete it.
Hope it helps
I should implement an upload form
I thought of using bodyparser but I read
http://andrewkelley.me/post/do-not-use-bodyparser-with-express-js.html
so what's the way to upload a file with express using the mean stack ?
may be formidable or other modules ?
That warning is specifically against adding the express.bodyparser middleware to your entire stack as it adds express.multipart to all POST endpoints and therefore file uploads are automatically accepted at all POST endpoints. By default the framework automatically saves any uploaded files to /tmp and so unless you are cleaning them up an attacker could flood your disk with uploaded files.
If you want to avoid using additional modules, what you should do is implement express.multipart on the endpoint(s) where you want to allow file uploads. Here's what I'm talking about:
var express = require("express")
, app = express();
// middleware (no bodyparser here)
app.use(express.json());
app.use(express.urlencoded());
// average GET endpoint
app.get("/", function(req,res) {
res.send('ok');
});
// average POST endpont
app.post("/login", function(req,res) {
res.send('ok');
});
// File upload POST endpoint
app.post('/upload', express.multipart, function(req, res) {
//File upload logic here
//Make sure to delete or move the file accordingly here, otherwise files will pile up in `/tmp`
});
Note the inclusion of express.multipart in the file upload endpoint. This endpoint will now process multipart file uploads, and assuming you handle them correctly they won't be a threat.
Now, having told you all of this, Connect is moving to deprecate multipart due to this exact issue, but there don't seem to be any plans to add a stream based file upload replacement. What they instead recommend is that you use node-multiparty which uses streams to avoid ever placing a file on disk. However, there don't seem to be any good references I can find for using multiparty as a middleware without saving files though, so you'll have to contact the author of multiparty or take a closer look at the API for implementing it with Express.
I created an example that uses Express & Multer - very simple, avoids all Connect warnings
https://github.com/jonjenkins/express-upload