I need to receive, in a single request, either JSON data and files. So I've been using body-parser which works perfect. However I'm having problems finding a module working nice with express.
This is my router setup:
router.post('/',
// controllers.requireAuthorization,
controllers.multipartMiddleware,
function (req, res) {
console.log(req.body);
return res.json({ body: req.body });
},
controllers.sitters.validate,
controllers.sitters.create,
controllers.sitters.serialize
);
This is what my multipart middleare function looks like, as you can see I'm using multiparty:
function multipartMiddleware(req, res, next) {
if (req.get('Content-Type').indexOf('multipart/form-data') + 1) {
new multiparty.Form().parse(req, function (err, fields, files) {
console.log(JSON.stringify(files));
req.body = fields;
return next(err);
});
} else {
console.log(req.get('Content-Type'));
return next();
}
}
Of course I've added that premature response return for debug purposes. So I need:
Receives files which will be streamed right away to S3.
Parse the rest of data as normal data.
Correctly get the data parsed
The issues are seeing right now:
It takes long sometimes even for small files (less than 512k, up to 5 seconds, this could be becase I'm using vagrant while developing but I think it's odd).
Fields are not parsed properly:
See how the value of loca is inside a wrapping array.
I would checkout multer, its a popular Express middleware and has a number of different ways to handle files (one or many). It also still allows for fields to be set which will come through on your req.body.
var multer = require('multer');
var upload = multer.dest({ 'temp/' });
// looking for single file, named 'file'
app.put('/file', upload.single('file'), function(req, res) {
// access the file
var file = req.file
// any form fields sent in addition to the file are here
var body = req.body;
});
Another popular package thats an alternative to multer is busboy. Its also worth noting that multer is written on top of busboy.
Related
I am using postman to test a rest API I'm building for a project. I'm trying to send some data to a post method, but the data is getting lost somewhere between Postman and the endpoint.
I've tried tracing the data with console logs, but nothing comes out (req.body is undefined). I'm pretty sure the issue isn't with the endpoint or router, as the same error comes up in postman as in the console of my IDE, which means there's some sort of communication.
// json I'm putting into postman. validated with Jsonlint.com
{
"Name": "testN",
"file": "file1",
"Path": "/home/userf",
"userName": "user1"
}
// profileWrite.js
const dbProfileWrite = require('../...db-ProfileWrite');
const bodyParser = require('body-parser');
var app = express();
app.use(bodyParser.json());
// my post method
async function post(req, res, next) {
try {
console.log("attempting to parse data: " + req.body);
let profile = req.body;
console.log("data parsed, writing profiles");
profile= await dbProfileWrite.writeProfile(profile);
res.status(201).json(profile);
} catch (err) {
next(err);
}
}
module.exports.post = post;
UPDATE 7/15/19:I have recreated a microversion of my project that is having this issue on stackblitz. there's some sort of package error that I'm working on, but here's the link. I've recreated the methodology I'm using in my project with the router and all and looked at the example in the express docs. hopefully this helps isolate the issue.The data still comes in undefined when I post to this api through postman, so helpfully this much smaller view of the whole project helps.
Assuming you are using Express framework, by the look of the post function. You need to use a middlewear function to process request body using body-parser. Make sure you are using the correct parser in this case
app.use(bodyParser.json())
You don't need body-parser anymore it was put back in to the core of express in the form of express.json, simply use app.use(express.json()).
To access the body of your request use req.body, it should come with a object with the keys representing the json sent;
var app = express();
app.use(express.json());
async function post(req, res, next) {
try {
console.log("attempting to parse data: " + req.body);
let profile = req.body; // no need to use any functions to parse it
console.log("data parsed, writing profiles");
profile= await dbProfileWrite.writeProfile(profile);
res.status(201).json(profile);
console.log("profilecreated");
} catch (err) {
next(err);
}
}
See the express documentation
Solved the issue myself with a little help from John Schmitz. The issue was that I was defining the router and the server before actually telling it how to handle json bodies/ objects, so the body came through as the default undefined. In my index.js, the following is what fixed the code:
app.use(express.urlencoded({ extended: true }));
app.use(express.json());
app.use('/api/v1', router);
the key to this is that the app is told to use json and express.urlencoded before the router is declared. these actions have to happen in this order, and all before app.listen is called. once the app is listening, all of its params are set and you can't change them. Tl;dr: node is VERY picky, and you HAVE to define these things in the right place. thanks all for the help.
I am trying to get the image when user submits the form and inserting it into mongoDB server.For image I am using Multer plugin but its showing me the error.Here is my code of NodeJS
const multer = require('multer');
mongoose.connect('mongodb://localhost:27017/mytable',{useNewUrlParser:true} )
.then(()=>
console.log("Mongodb connected"))
.catch(err => console.error("could not connected",err));
const Schema =new mongoose.Schema({
name:String,
email:String,
lastname:String,
pass:String,
phonenumber:String,
zipcode:String,
birthdate:String,
img: {contentType:String,data:Buffer }
});
Schema.plugin(mongoosePaginate)
var user = mongoose.model('mytable', Schema);
//Multer for include image into directory
app.use(multer({ dest: '/public/'}).single('files'));
app.post('/save',(req,res)=>{
console.log("image is" +req.body.img);
var model = new user();
model.name = req.body.name,
model.email=req.body.email,
model.lastname=req.body.lastname,
model.pass=req.body.pass,
model.phonenumber=req.body.phonenumber,
model.zipcode=req.body.zipcode,
model.birthdate=req.body.birthdate,
/* model.img.data = req.body.img, */
model.img.data = fs.readFileSync(req.files.userPhoto.path);
newPic.image.contentType = 'image/png';
model.save(function(err,doc){
});
res.json({result:'sucess'});
res.end();
});
I just uploaded the required code. I am getting the error of Cannot read property 'userPhoto' of undefined .I don't know what should I write in fs.readFilesync.Please help me to insert image into a server .
You ask Multer to handle a .single() file that is expected to be referred to by the input name "files". According to the doc:
The single file will be stored in req.file
But you try to access req.files instead. (And it seems you're expecting this file to be referred to as "userPhoto", maybe?).
See also what information Multer exposes to retrieve the uploaded file's path.
Finally, you might want to restrict your middleware usage to the routes that need it.
EDIT: a few comments
// This tells the WHOLE app that:
// when a request comes in, execute this
app.use(
// Multer will do:
// when I'm given an incoming request (here it's every request)
// then I'm looking for *one* file being uploaded
// this file should be named "files" (i.e. client has <input type="file" name="files">)
// if I find it, then I store it in /public
multer({ dest: '/public/'}).single('files')
);
// This tells the app that:
// if a request comes in for endpoint /save with method POST, this is the code to execute
app.post('/save', (req, res) => {
// in here, Multer will have been executed already
});
So:
does your form really names its file to be uploaded "files"? I'd guess your form names the file "userPhoto"... just a guess!
if such a file exists in the request, Multer documentation says that your route handler can access it in req.file (not req.files)
if not, Multer will just let the request pass (that's what middlewares do), so you won't have a req.file
if req.file is mounted on the request by Multer, it exposes several data fields, such as req.file.path
I also hinted that you may not want to enable Multer for the whole app, but just for the routes that require it. Instead of a "global" app.use, you can define several times a route (or you could explicitly use the router, I don't see much of a difference), like:
app.post('/save', multer(...));
app.post('/save', (req, res) => {...});
// which can also be written as
app.post('/save', multer(...), (req, res) => {...});
This way, all other routes do not consider file uploading, I'm sure I don't need to highlight how better this is.
the problem is not with mongoose! as it says in your error message, req.files is undefined. it's a problem with multer documentation! when you're using single your file will be available in req.file
so this would fix your problem:
app.post('/save',(req,res)=>{
console.log("image is" +req.body.img);
var model = new user();
model.name = req.body.name,
model.email=req.body.email,
model.lastname=req.body.lastname,
model.pass=req.body.pass,
model.phonenumber=req.body.phonenumber,
model.zipcode=req.body.zipcode,
model.birthdate=req.body.birthdate,
/* model.img.data = req.body.img, */
model.img.data = fs.readFileSync(req.file.path); // here's the fix
newPic.image.contentType = 'image/png';
model.save(function(err,doc){
});
res.json({result:'sucess'});
res.end();
});
date: { type: Date, default: Date.now } // stick that into the Schema
I have a Node server using express.
I was originally using body-parser, but that doesn't allow for file uploads. So, I switched to multer (the easiest integration with express). However, in order to get any of the req (specifically req.body), this is my code:
var multer = require('multer');
var upload = multer({ dest : 'uploads/' });
server.all('/example', function (req, res, next) {
var up = upload.single('photo')
up(req, res, function(err) {
console.log(req.body); // I can finally access req.body
});
}
The problem with this, is that not all of my routes need to upload a file. Do I need to waste the CPU on calling upload.single() for each route in order to get access to the body? upload.single('') ends up not uploading any file, but it's still precious time spent on the main thread.
It appears that upload.single() waits for the callback, so it may not be as big of a deal as I'm making it, but I don't like calling functions when I don't have to.
Is there a way around calling upload.single(), or am I just making a bigger deal out of this than it really is?
For text-only multipart forms, you could use any of the multer methods, which are .single(), .array(), fields()
For instance using .array()
var multer = require('multer');
var upload = multer({ dest : 'uploads/' });
server.all('/example', upload.array(), function (req, res, next) {
console.log(req.body);
});
It doesn't really matter which you use, as long as it's invoked without arguments Multer will only parse the text-fields of the form for you, no files
Using Express with Node, I can upload a file successfully and pass it to Azure storage in the following block of code.
app.get('/upload', function (req, res) {
res.send(
'<form action="/upload" method="post" enctype="multipart/form-data">' +
'<input type="file" name="snapshot" />' +
'<input type="submit" value="Upload" />' +
'</form>'
);
});
app.post('/upload', function (req, res) {
var path = req.files.snapshot.path;
var bs= azure.createBlobService();
bs.createBlockBlobFromFile('c', 'test.png', path, function (error) { });
res.send("OK");
});
This works just fine, but Express creates a temporary file and stores the image first, then I upload it to Azure from the file. This seems like an inefficient and unnecessary step in the process and I end up having to manage cleanup of the temp file directory.
I should be able to stream the file directly to Azure storage using the blobService.createBlockBlobFromStream method in the Azure SDK, but I am not familiar enough with Node or Express to understand how to access the stream data.
app.post('/upload', function (req, res) {
var stream = /// WHAT GOES HERE ?? ///
var bs= azure.createBlobService();
bs.createBlockBlobFromStream('c', 'test.png', stream, function (error) { });
res.send("OK");
});
I have found the following blog which indicates that there may be a way to do so, and certainly Express is grabbing the stream data and parsing and saving it to the file system as well. http://blog.valeryjacobs.com/index.php/streaming-media-from-url-to-blob-storage/
vjacobs code is actually downloading a file from another site and passing that stream to Azure, so I'm not sure if it can be adapted to work in my situation.
How can I access and pass the uploaded files stream directly to Azure using Node?
SOLUTION (based on discussion with #danielepolencic)
Using Multiparty(npm install multiparty), a fork of Formidable, we can access the multipart data if we disable the bodyparser() middleware from Express (see their notes on doing this for more information). Unlike Formidable, Multiparty will not stream the file to disk unless you tell it to.
app.post('/upload', function (req, res) {
var blobService = azure.createBlobService();
var form = new multiparty.Form();
form.on('part', function(part) {
if (part.filename) {
var size = part.byteCount - part.byteOffset;
var name = part.filename;
blobService.createBlockBlobFromStream('c', name, part, size, function(error) {
if (error) {
res.send({ Grrr: error });
}
});
} else {
form.handlePart(part);
}
});
form.parse(req);
res.send('OK');
});
Props to #danielepolencic for helping to find the solution to this.
As you can read from the connect middleware documentation, bodyparser automagically handles the form for you. In your particular case, it parses the incoming multipart data and store it somewhere else then exposes the saved file in a nice format (i.e. req.files).
Unfortunately, we do not need (and necessary like) black magic primarily because we want to be able to stream the incoming data to azure directly without hitting the disk (i.e. req.pipe(res)). Therefore, we can turn off bodyparser middleware and handle the incoming request ourselves. Under the hood, bodyparser uses node-formidable, so it may be a good idea to reuse it in our implementation.
var express = require('express');
var formidable = require('formidable');
var app = express();
// app.use(express.bodyParser({ uploadDir: 'temp' }));
app.get('/', function(req, res){
res.send('hello world');
});
app.get('/upload', function (req, res) {
res.send(
'<form action="/upload" method="post" enctype="multipart/form-data">' +
'<input type="file" name="snapshot" />' +
'<input type="submit" value="Upload" />' +
'</form>'
);
});
app.post('/upload', function (req, res) {
var bs = azure.createBlobService();
var form = new formidable.IncomingForm();
form.onPart = function(part){
bs.createBlockBlobFromStream('taskcontainer', 'task1', part, 11, function(error){
if(!error){
// Blob uploaded
}
});
};
form.parse(req);
res.send('OK');
});
app.listen(3000);
The core idea is that we can leverage node streams so that we don't need to load in memory the full file before we can send it to azure, but we can transfer it as it comes along. The node-formidable module supports streams, hence piping the stream to azure will achieve our objective.
You can easily test the code locally without hitting azure by replacing the post route with:
app.post('/upload', function (req, res) {
var form = new formidable.IncomingForm();
form.onPart = function(part){
part.pipe(res);
};
form.parse(req);
});
Here, we're simply piping the request from the input to the output. You can read more about bodyParser here.
There are different options for uploading binary data (e.g. images) via Azure Storage SDK for Node, not using multipart.
Based on the Buffer and Stream definitions in Node and manipulating them, these could be handled using almost all the methods for BLOB upload: createWriteStreamToBlockBlob, createBlockBlobFromStream, createBlockBlobFromText.
References could be found here: Upload a binary data from request body to Azure BLOB storage in Node.js [restify]
People having trouble with .createBlockBlobFromStream trying to implement the solutions, note that this method has been changed slightly in newer versions
Old version:
createBlockBlobFromStream(containerName, blobName, part, size, callback)
New version
createBlockBlobFromStream(containerName, blobName, part, size, options, callback)
(if you don't care about options, try an empty array) for the parameter.
Oddly enough, "options" is supposed to be optional, but for whatever reason, mine fails if I leave it out.
I am trying to upload files using express and formidable (eventualy forwarding to MongoDB and GridFS). I am starting by creating a form with a field of type file. On the action of that field I use the following route....
exports.addItem = function(req, res, next){
var form = new formidable.IncomingForm(),
files = [],
fields = [];
form
.on('file', function(field, file) {
console.log(field, file);
})
.on('end', function() {
console.log('-> upload done');
});
}
Everything runs fine but when I post I don't see anything in the console and it hangs.
The route looks like the following...
app.post('/item/add', routes.addItem, routes.getPlaylist, routes.index)
Any ideas?
UPDATE
Here is an example of grabbing the file, however, this still doesn't include formidable...
https://gist.github.com/2963261
The reason it is hanging is because you need to call next() to tell Express to continue.
Also use the bodyParser() middleware in express (included by default) to get the files. Something like this:
exports.addItem = function(req, res, next){
if(req.files.length > 0)
{
// process upload
console.log(req.files);
}
next();
}