I am wanting to post an image in the form of binary to my Express app.
I'm assuming it should come through in the req.body object but will need some form of middleware to be able to handle binary data?
When I send an image as binary from postman and try log req.body, the object is empty.
I am using express-generator as a boilder plate which comes with body-parser like so:
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: false }));
I had a look at Multer but think that is just for multipart data
Also looked at busboy but couldn't figure out if that will handle binary data.
Am I correct that the post data will still come through in req.body?
And what middleware do I need to handle binary data?
Thanks
The method I ended up using:
const multer = require('multer')
const storage = multer.memoryStorage()
const upload = multer({ storage: storage })
router.post('/upload', upload.single('image'), function(req, res, next) {
const image = req.file.buffer
});
Unfortunately, you can't use the body-parser to handle the binary data like files and stuff like that. But wut you can do is use a module call formidable to handle this
Example snipper
app.post('/', (req, res) => {
const form = new formidable.IncomingForm();
form.parse(req, (error, fields, files) => {
if(error){
console.log(error)
}
console.log(fields.name)
const cuteCat = files.cat_image;
console.log(cuteCat.name) // The origin file name
console.log(cuteCat.path) // The temporary file name something like /tmp/<random string>
})
});
<input name="cat_image" type="file" />
<input name="name" type="text" />
Related
I found other similar questions regarding Multer, but with no answers. I'm trying to upload a file with next.js (front-end) and node.js (back-end). The data is being posted via the network tab when using dev tools.
Below is my setup:
app.js
const express = require('express');
const bodyParser = require('body-parser');
// Get routes
const routeUser = require('./routes/user');
// Create an express server
var app = express();
// Add necessary middleware
app.use(bodyParser.urlencoded({ extended: true })); // Support encoded bodies
app.use(bodyParser.json({
type: ["application/x-www-form-urlencoded", "application/json"], // Support json encoded bodies
}));
// Custom routes
routeUser(app);
// Start server on port 1234
app.listen(1234, () => {
console.log("API is running.");
});
route/user.js
const multer = require('multer');
module.exports = function(app) {
const upload = multer({
storage: multer.diskStorage({
destination: (req, file, cb) => {
cb(null, "./user_photos");
},
filename: (req, file, cb) => {
cb(null, file.originalname)
}
})
});
app.post('/user/update', upload.single('user_photo'), (req, res) => {
console.log(req.body, req.file);
});
}
Form
submit(event) {
event.preventDefault();
let form_values = new FormData(event.target);
fetch(this.props.env.api + "/user/update", {
method: 'post',
headers: {
'Content-Type': 'multipart/form-data; boundary=MyBoundary',
},
body: form_values
}).then((response) => {
return response.json();
}).then((response) => {
console.log(response);
});
}
------
<form onSubmit={this.submit}>
<input type="file" name="user_photo"/>
<button type="submit">Upload</button>
</form>
I've tried several tutorials, I'm setting it up according to the docs, yet I keep getting the following error:
Error: Unexpected end of form
at Multipart._final (...\node_modules\busboy\lib\types\multipart.js:588:17)
If I remove multipart/form-data as Content-Type, the form submits with no problem, but with no file. My guess it has something to do with the way the formData object is being received.
Hey #SReca, just faced this issue today and hope I can help you out here and anybody else reading this.
Resolving Unexpected end of form
Regarding the original issue about Unexpected end of form, you are correct in removing the Content-Type as a solution. This is because sending FormData() with regular Fetch or XMLHttpRequest will automatically have the header set by the Browser. The Browser will also attach the boundary needed in all multipart/form-data requests in order indicate when a form starts and ends. More details can be found on MDN's docs about Using FormData Objects... there's a warning about explicitly setting Content-Type.
Potential fix for missing file
Now, about the missing file, it's possible that it has an incorrect reference to the path you're expecting. I am not using diskStorage, but I am using regular dest option to save my file and had the same problem (wanted to save images to ./assets/images). What worked for me was to supply the complete directory path. Maybe changing your callback function in the destination property to
// Assuming the the path module is 'required' or 'imported' beforehand
cb(null, path.resolve(__dirname, '<path-to-desired-folder>'));
will solve the issue. Hope this helps!
What caused this too me was because I was using multiple middleware on the same route, I was using global middleware and then applied another middleware in sub route. so this caused conflits.
In my case the problem magically went away by downgrading Multer to 1.4.3.
See https://github.com/expressjs/multer/issues/1144
I've tried many StackOverflow answers, and this method normally works using body-parser, however I've been having issues with getting any output from req.body with either AJAX or form data.
In server.js:
app.use(helmet()); // Helmet middleware
app.use('/assets', express.static('resources/web/assets')); // Makes /assets public
app.use(require('./resources/modules/session.js')); // Custom session middleware
app.set('view engine', 'ejs'); // Sets EJS to the view engine
app.set('views', `${__dirname}/resources/web/pages`); // Sets the views folder
app.use(cookieParser()); // cookie-parser middleware
app.use(bodyParser.urlencoded({ extended: true })); // body-parser's middleware to handle encoded data
app.use(bodyParser.json()); // body-parser's middleware to handle JSON
app.use(fileUpload({ limits: { fileSize: 100 * 1024 * 1024 } })); // express-fileupload middleware (bushboy wrapper)
app.use('/api', require('./resources/routes/api.js')); // External API router
// ...
app.post('/login', (req, res) => {
console.log(req.body);
res.render('login', {
config,
page: {
name: 'Login'
},
error: ''
});
res.end();
});
My login.ejs code:
<form method="POST">
<div class="input-group">
<i class="las la-user"></i>
<input placeholder="Username" name="username" type="text" required>
</div>
<div class="input-group">
<i class="las la-lock"></i>
<input placeholder="Password" name="password" type="password" required>
</div>
<button type="submit">
<i class="las la-paper-plane"></i> Login
</button>
</form>
No matter what I try, I always get an empty {} in the console with no avail. I've tried debugging; I need a fresh pair of eyes to see what I've done wrong.
Here's the form data:
And I've tried using jQuery's AJAX ($.get) too:
$.post('', {username:'test', password:'test'})
.fail(console.error)
.done(() => console.log('Success'));
Edit: After trying multer's app.use(require('multer')().array()); and app.use(require('multer')().none()); middleware, I'm still at the same old issue, except with multer req.body is now undefined instead of {}. This is due to the data being sent as application/x-www-form-urlencoded instead of what I previously thought was application/form-data. As that is the case, the body-parser middleware method should work. If contributing, please do not contribute an answer relating to parsing application/form-data!
Edit 2: For those asking for the session.js code, here it is:
const enmap = require('enmap'),
sessions = new enmap('sessions');
module.exports = (req, res, next) => {
if (!req.cookies) next();
const { cookies: { session: sessionID } } = req;
if (sessionID) {
const session = sessions.get(sessionID);
if (session) {
req.session = session;
} else {
req.session = undefined;
};
} else {
req.session = undefined;
};
next();
};
I'm attaching the whole source code as you people claim to be able to reproduce it somehow. Download it at https://dropfile.nl/get/F7KF (DM me on Discord if it isn't working - PiggyPlex#9993).
For the specific case you're talking about, you usually need only 'body-parser' module to be able to access the form input fields. The minimum example that I advice you to build above it is the following:
var express = require('express');
var bodyParser = require('body-parser');
var app = express();
app.use(bodyParser.urlencoded({extended : true}));
app.use(bodyParser.json());
app.get('/login', (req, res) => { /* ... */ });
app.post('/login', (req, res) => {
console.log(req.body);
// ...
});
app.listen(3000);
So my advice is to narrow on the cause of the problem by removing any other middleware except for the bodyParser. Try to comment them one-by-one and then you will be able to find the guilty!
Also note that no need to bother yourself trying to make it work with Ajax as it will make no difference. Keep it simple and just try the normal browser submission.
When you found the problematic middleware, debug it. If it's made by you, make sure that you don't make any changes to the req.body. If it a thirdparty middleware, so please consult their installation steps very carefully and I'm happy for further explanation and support
Edit: Some other hints
Make sure that the request is being submitted with the header Content-Type: application/x-www-form-urlencoded
Check if there any CORS problems
File upload middleware to be on a separate path just like the assets
Removing enctype="multipart/form-data" from the form elements works for me, also after registering these middlewares:
app.use(express.json());
app.use(express.urlencoded({ extended: true }));
Sometimes the main point this doesn't work is - when your data passed in the body is in text format, or as in my case in the headers I had 'Content-Type': 'application/x-www-form-urlencoded', but your req.body is expecting JSON data - so please make sure to double-check the 'Content-Type':'application/json' is set on the request headers.
You need to use other module to deal with multipart/form-data
https://github.com/pillarjs/multiparty
app.post('/login', function (req, res) {
const form = new multiparty.Form();
form.parse(req, function(err, fields, files) {
console.log(fields);
});
})
Please use body parser to get the input from the form
var express = require('express');
var bodyParser = require('body-parser');
var app = express();
app.use(bodyParser.urlencoded({extended : true}));
app.use(bodyParser.json());
app.post('/anyRoute', (req, res) => {
console.log(req.body); //your input value
});
app.listen(3000);
I had this same error and what fixed it was by removing enctype="multipart/form-data" from the form element. If you are not sending a file alongside the form with the enctype attribute, then you get such errors
I have a Node server using express.
I was originally using body-parser, but that doesn't allow for file uploads. So, I switched to multer (the easiest integration with express). However, in order to get any of the req (specifically req.body), this is my code:
var multer = require('multer');
var upload = multer({ dest : 'uploads/' });
server.all('/example', function (req, res, next) {
var up = upload.single('photo')
up(req, res, function(err) {
console.log(req.body); // I can finally access req.body
});
}
The problem with this, is that not all of my routes need to upload a file. Do I need to waste the CPU on calling upload.single() for each route in order to get access to the body? upload.single('') ends up not uploading any file, but it's still precious time spent on the main thread.
It appears that upload.single() waits for the callback, so it may not be as big of a deal as I'm making it, but I don't like calling functions when I don't have to.
Is there a way around calling upload.single(), or am I just making a bigger deal out of this than it really is?
For text-only multipart forms, you could use any of the multer methods, which are .single(), .array(), fields()
For instance using .array()
var multer = require('multer');
var upload = multer({ dest : 'uploads/' });
server.all('/example', upload.array(), function (req, res, next) {
console.log(req.body);
});
It doesn't really matter which you use, as long as it's invoked without arguments Multer will only parse the text-fields of the form for you, no files
Problem
I have a server that needs to upload files, I have tried multiparty, connect-multiparty and multer.
But every case, has the same problem: the file only uploads some times, i mean, i could send a file and there is a chance that the libraries don't parse the files, and never continue the code, resulting on not uploading the files.
In a While, the request send an error "Request Aborted", but its the normal response when the request time out
This is the problematic node.js file:
var multiparty = require('multiparty');
var multer = require('multer');
var upload = multer({
dest: "/uploads/"
});
///----rest of code----
//1. Multiparty
app.post("/upload",[function(req, res){
var form = new multiparty.Form({uploadDir:'/uploads/'});
console.log("to upload")
form.parse(req, function (err, fields, files) {
console.log("uploaded");
res.json({uploaded: true});
})
}]
//2. multer
app.post("/upload2",[
function(req, res, next){
console.log("to upload");
next();
},
upload.fields([
{name: "file"},
{name: "thumbnail"}
]),
function(req, res){
console.log("uploaded");
res.json({uploaded: true});
}]
Make sure your form looks like this
<form enctype="multipart/form-data" action="..." method="...">
...
</form>
And to be honest you will be better off using node-formidable. It is the most used multipart/form-data package on npm.
The example works straight out of the box.
Cheers
https://stackoverflow.com/a/23975955/4920678
I was using the setup from this answer, to use http and https over the same Port.
Turns out, the setup with that proxy damaged the packages that where too large or something, and then the files never get parsed
Using Express with Node, I can upload a file successfully and pass it to Azure storage in the following block of code.
app.get('/upload', function (req, res) {
res.send(
'<form action="/upload" method="post" enctype="multipart/form-data">' +
'<input type="file" name="snapshot" />' +
'<input type="submit" value="Upload" />' +
'</form>'
);
});
app.post('/upload', function (req, res) {
var path = req.files.snapshot.path;
var bs= azure.createBlobService();
bs.createBlockBlobFromFile('c', 'test.png', path, function (error) { });
res.send("OK");
});
This works just fine, but Express creates a temporary file and stores the image first, then I upload it to Azure from the file. This seems like an inefficient and unnecessary step in the process and I end up having to manage cleanup of the temp file directory.
I should be able to stream the file directly to Azure storage using the blobService.createBlockBlobFromStream method in the Azure SDK, but I am not familiar enough with Node or Express to understand how to access the stream data.
app.post('/upload', function (req, res) {
var stream = /// WHAT GOES HERE ?? ///
var bs= azure.createBlobService();
bs.createBlockBlobFromStream('c', 'test.png', stream, function (error) { });
res.send("OK");
});
I have found the following blog which indicates that there may be a way to do so, and certainly Express is grabbing the stream data and parsing and saving it to the file system as well. http://blog.valeryjacobs.com/index.php/streaming-media-from-url-to-blob-storage/
vjacobs code is actually downloading a file from another site and passing that stream to Azure, so I'm not sure if it can be adapted to work in my situation.
How can I access and pass the uploaded files stream directly to Azure using Node?
SOLUTION (based on discussion with #danielepolencic)
Using Multiparty(npm install multiparty), a fork of Formidable, we can access the multipart data if we disable the bodyparser() middleware from Express (see their notes on doing this for more information). Unlike Formidable, Multiparty will not stream the file to disk unless you tell it to.
app.post('/upload', function (req, res) {
var blobService = azure.createBlobService();
var form = new multiparty.Form();
form.on('part', function(part) {
if (part.filename) {
var size = part.byteCount - part.byteOffset;
var name = part.filename;
blobService.createBlockBlobFromStream('c', name, part, size, function(error) {
if (error) {
res.send({ Grrr: error });
}
});
} else {
form.handlePart(part);
}
});
form.parse(req);
res.send('OK');
});
Props to #danielepolencic for helping to find the solution to this.
As you can read from the connect middleware documentation, bodyparser automagically handles the form for you. In your particular case, it parses the incoming multipart data and store it somewhere else then exposes the saved file in a nice format (i.e. req.files).
Unfortunately, we do not need (and necessary like) black magic primarily because we want to be able to stream the incoming data to azure directly without hitting the disk (i.e. req.pipe(res)). Therefore, we can turn off bodyparser middleware and handle the incoming request ourselves. Under the hood, bodyparser uses node-formidable, so it may be a good idea to reuse it in our implementation.
var express = require('express');
var formidable = require('formidable');
var app = express();
// app.use(express.bodyParser({ uploadDir: 'temp' }));
app.get('/', function(req, res){
res.send('hello world');
});
app.get('/upload', function (req, res) {
res.send(
'<form action="/upload" method="post" enctype="multipart/form-data">' +
'<input type="file" name="snapshot" />' +
'<input type="submit" value="Upload" />' +
'</form>'
);
});
app.post('/upload', function (req, res) {
var bs = azure.createBlobService();
var form = new formidable.IncomingForm();
form.onPart = function(part){
bs.createBlockBlobFromStream('taskcontainer', 'task1', part, 11, function(error){
if(!error){
// Blob uploaded
}
});
};
form.parse(req);
res.send('OK');
});
app.listen(3000);
The core idea is that we can leverage node streams so that we don't need to load in memory the full file before we can send it to azure, but we can transfer it as it comes along. The node-formidable module supports streams, hence piping the stream to azure will achieve our objective.
You can easily test the code locally without hitting azure by replacing the post route with:
app.post('/upload', function (req, res) {
var form = new formidable.IncomingForm();
form.onPart = function(part){
part.pipe(res);
};
form.parse(req);
});
Here, we're simply piping the request from the input to the output. You can read more about bodyParser here.
There are different options for uploading binary data (e.g. images) via Azure Storage SDK for Node, not using multipart.
Based on the Buffer and Stream definitions in Node and manipulating them, these could be handled using almost all the methods for BLOB upload: createWriteStreamToBlockBlob, createBlockBlobFromStream, createBlockBlobFromText.
References could be found here: Upload a binary data from request body to Azure BLOB storage in Node.js [restify]
People having trouble with .createBlockBlobFromStream trying to implement the solutions, note that this method has been changed slightly in newer versions
Old version:
createBlockBlobFromStream(containerName, blobName, part, size, callback)
New version
createBlockBlobFromStream(containerName, blobName, part, size, options, callback)
(if you don't care about options, try an empty array) for the parameter.
Oddly enough, "options" is supposed to be optional, but for whatever reason, mine fails if I leave it out.