I'm using Node.js/Express and submitting an image to the server from a webpage via AJAX. The image has been processed and cropped, so I'm using the resultant dataURL and sending it to my server by a post request like this. The dataURL is completely good and accurate and contains the image as it should. So I simply send it to the server.
var fd = new FormData();
fd.append('dataURL',dataURL);
console.log('dataURL length:',dataURL.length);
$.ajax({
type: "POST",
url: `/pwa/update-attachment`,
data: fd,
contentType: false,
processData: false,
success: function(result)
{
}
});
On the server side there are no errors but the dataURL parameters is truncated to 1mb. For reference, my server is setup to accept multipart form data using using bodyparser as follows. I tried using size limits with bodyParser (either high or low) and neither seem to take effect in this case.
const bodyParser = require('body-parser');
app.use(bodyParser.urlencoded({extended: false })); // support encoded bodies
I receive my post like this:
router.post('/update-attachment',(req,res) => {
console.log('dataurl:',req.body.dataURL.length);
...
});
The dataURL is truncated to 1048576 bytes. Works perfectly fine for smaller images.
The reason that there is not an error is that it is being truncated on the client side by the browser. To send the image, it should not be sent as a dataURL, but as a blob. The following answer explains this:
Convert Data URI to File then append to FormData
Related
I have a file in memory (buffer) - there is no file on the file system.
I want to send that buffer to another server that talks HTTP.
For example, some API A creates a file in memory, SignServer manipulates such files, and responds with a new buffer. My API takes the file from A and feeds it to SignServer.
I tried sending the file to SignServer in multiple ways, but it keeps responding with status 400 (missing field 'data' in request).
What I tried:
var http = require('http');
var querystring = require('querystring');
var data = querystring.stringify({
workerName: 'PDFSigner',
data: file_buffer
});
var request = new http.ClientRequest({
hostname: 'localhost',
port: 8080,
path: '/signserver/process',
method: 'GET',
headers: {
'Content-Type': 'application/x-www-form-urlencoded',
// I also tried 'multipart/form-data'
'Content-Length': Buffer.byteLength(data)
}
});
request.end(data);
I tried printing data, and it showed:
workerName=PDFSigner&data=
Which is bad because data wasn't set to file_buffer.
I tried printing file_buffer, and it does have content (not null, not undefined, actually has bytes inside).
So stringifying the buffer gave an empty string.
I tried doing the same thing with the request module and it didn't work either.
Note that SignServer isn't written in Node nor JavaScript. It's a Java application, so it probably doesn't work with json (which is why I'm trying to do it with querystring). Yes, I tried sending json.
The reason why data is set to an empty string is described in this issue and the solution is given in this issue.
escape and stringify the buffer like so:
var data = querystring.stringify({
workerName: 'PDFSigner',
data: escape(file_buffer).toString('binary')
});
As #robertklep mentioned, your other problem is that you can't send a big file using application/x-www-form-urlencoded. You'd need to do it with multipart/form-data.
I have a simple uploading code by node.js.
var http = require('http')
var fs = require('fs')
var server = http.createServer(function(req, res){
if(req.url == '/upload') {
var a = fs.createWriteStream('a.jpg', { defaultEncoding: 'binary'})
req.on('data', function(chunk){
a.write(chunk)
})
.on('end', function()){
a.end()
res.end('okay')
})
}
else {
fs.createReadStream('./index.html').pipe(res);
// just show <form>
}
})
server.listen(5000)
when I upload some image, I cannot get exact same file.
Always files are broken.
When I try to do this using formidable, I can get a fine file.
So I studied formidable but I cannot understand how did it catch data and save.
I could find formidable use parser to calculate something about chunk from request but I did not get it all.
(It is definitely my brain issue :( ).
Anyway, what is the difference between my code and formidable?
What am I missing?
Is it a wrong way to just add all chunks from http request and save it by
fs.createWriteStream or fs.writeFile ?
What concepts am I missing?
First, req is a Readable stream. You can simply do:
req.pipe(fs.createWriteStream('a.jpg'))
for the upload part. This is copying all byte data from request stream to file.
This will work when you send raw file data as the request body:
curl --data-binary #"/home/user/Desktop/a.jpg" http://localhost:8080/upload
Because this sends request body exactly as image binary data, that gets streamed to a file on server.
But, there is another request format called multipart/form-data. This is what web browsers use with <form> to upload files.
curl -form "image=#/home/user1/Desktop/a.jpg" http://localhost:8080/upload
Here the request body contains multiple "parts", one for each file attachment or form field, separated by special "boundary" characters:
--------------------------e3f25f5319cd6624
Content-Disposition: form-data; name="image"; filename="a.jpg"
Content-Type: application/octet-stream
JPG IHDRH-ÑtEXtSoftwareAdobe.....raw file data
--------------------------e3f25f5319cd6624
Hence you will need much more complicated code to extract the file part data from it. NPM Modules like busboy and formidable do exactly that.
I'm using the request.js node module to make a GET request for an image. The body I get back looks like this:
body: 'u0014�����T���8�\u00029�\u001fZ\u0000m(\u0007�\u001d�A\u0014�9E9Oz#E8s��`8d�x�j`�<rq... etc'
How do I read that as a JPEG?
What I'm doing is just forwarding that content as a PUT request to another endpoint. This is working, except that the image data is no readable on the new URL (which is a CouchDB document attachment).
My PUT request looks like this:
request({
url: newDocUrl + '/' + aName + "?rev=" + resRev,
method: 'PUT',
headers: headersAttachment, //{'Content-Type': 'image/jpeg'}
body: attachment
}, function(e, r, b) {
console.log('body', b);
});
Questions: How do I read JPEG data from an HTTP res? What format should JPEG data be to forward an image? (i.e. base64, hex, something else?)
I found the answer here:
Node.js get image from web and encode with base64
I didn't know that request.js encodes URL responses. By turning it off the image was posted fine.
I have an Express application that uses multer to upload images to an S3 bucket. I'm not doing anything special, just a straight upload, but when they are displayed in the browser some of the iPhone images are sideways.
I know this is technically a browser bug and Firefox now supports the image-orientation rule, but Chrome still displays the images on their side.
Is there a way I can have Express read the EXIF data and just rotate them before uploading?
Right, I figured it out. I used a combination of JavaScript Load Image and the FormData API.
First I'm using Load Image to get the orientation of the image from the exif data and rotating it. I'm then converting the canvas output that Load Image provides and converting that to a blob (you may also need the .toBlob() polyfill for iOS as it does not support this yet.
That blob is then attached to the FormData object and I'm also putting it back in the DOM for a file preview.
// We need a new FormData object for submission.
var formData = new FormData();
// Load the image.
loadImage.parseMetaData(event.target.files[0], function (data) {
var options = {};
// Get the orientation of the image.
if (data.exif) {
options.orientation = data.exif.get('Orientation');
}
// Load the image.
loadImage(event.target.files[0], function(canvas) {
canvas.toBlob(function(blob) {
// Set the blob to the image form data.
formData.append('image', blob, 'thanksapple.jpg');
// Read it out and stop loading.
reader.readAsDataURL(blob);
event.target.labels[0].innerHTML = labelContent;
}, 'image/jpeg');
}, options);
reader.onload = function(loadEvent) {
// Show a little image preview.
$(IMAGE_PREVIEW).attr('src', loadEvent.target.result).fadeIn();
// Now deal with the form submission.
$(event.target.form).submit(function(event) {
// Do it over ajax.
uploadImage(event, formData);
return false;
});
};
});
Now for the uploadImage function which I'm using jQuery's AJAX method for. Note the processData and contentType flags, they are important.
function uploadImage(event, formData) {
var form = event.target;
$.ajax({
url: form.action,
method: form.method,
processData: false,
contentType: false,
data: formData
}).done(function(response) {
// And we're done!
});
// Remove the event listener.
$(event.target).off('submit');
}
All the info is out there but it's spread across multiple resources, hopefully this will save someone a lot of time and guessing.
I have a Node.js app running on a Google Compute VM instance that receives file uploads directly from POST requests (not via the browser) and streams the incoming data to Google Cloud Storage (GCS).
I'm using Restify b/c I don't need the extra functionality of Express and because it makes it easy to stream the incoming data.
I create a random filename for the file, take the incoming req and toss it to a neat little Node wrapper for GCS (found here: https://github.com/bsphere/node-gcs) which makes a PUT request to GCS. The documentation for GCS using PUT can be found here: https://developers.google.com/storage/docs/reference-methods#putobject ... it says Content-Length is not necessary if using chunked transfer encoding.
Good news: the file is being created inside the appropriate GCS storage "bucket"!
Bad News:
I haven't figured out how to get the incoming file's extension from Restify (notice I'm manually setting '.jpg' and the content-type manually).
The file is experiencing slight corruption (almost certainly do to something I'm doing wrong with the PUT request). If I download the POSTed file from Google, OSX tells me its damaged ... BUT, if I use PhotoShop, it opens and looks just fine.
Update / Solution
As pointed out by vkurchatkin, I needed to parse the request object instead of just piping the whole thing to GCS. After trying out the lighter busboy module, I decided it was just a lot easier to use multiparty. For dynamically setting the Content-Type, I simply used Mimer (https://github.com/heldr/mimer), referencing the file extension of the incoming file. It's important to note that since we're piping the part object, the part.headers must be cleared out. Otherwise, unintended info, specifically content-type, will be passed along and can/will conflict with the content-type we're trying to set explicitly.
Here's the applicable, modified code:
var restify = require('restify'),
server = restify.createServer(),
GAPI = require('node-gcs').gapitoken,
GCS = require('node-gcs'),
multiparty = require('multiparty'),
Mimer = require('mimer');
server.post('/upload', function(req, res) {
var form = new multiparty.Form();
form.on('part', function(part){
var fileType = '.' + part.filename.split('.').pop().toLowerCase();
var fileName = Math.random().toString(36).slice(2) + fileType;
// clear out the part's headers to prevent conflicting data being passed to GCS
part.headers = null;
var gapi = new GAPI({
iss: '-- your -- #developer.gserviceaccount.com',
scope: 'https://www.googleapis.com/auth/devstorage.full_control',
keyFile: './key.pem'
},
function(err) {
if (err) { console.log('google cloud authorization error: ' + err); }
var headers = {
'Content-Type': Mimer(fileType),
'Transfer-Encoding': 'Chunked',
'x-goog-acl': 'public-read'
};
var gcs = new GCS(gapi);
gcs.putStream(part, myBucket, '/' + fileName, headers, function(gerr, gres){
console.log('file should be there!');
});
});
});
};
You can't use the raw req stream since it yields whole request body, which is multipart. You need to parse the request with something like multiparty give you a readable steam and all metadata you need.