I'm using the request.js node module to make a GET request for an image. The body I get back looks like this:
body: 'u0014�����T���8�\u00029�\u001fZ\u0000m(\u0007�\u001d�A\u0014�9E9Oz#E8s��`8d�x�j`�<rq... etc'
How do I read that as a JPEG?
What I'm doing is just forwarding that content as a PUT request to another endpoint. This is working, except that the image data is no readable on the new URL (which is a CouchDB document attachment).
My PUT request looks like this:
request({
url: newDocUrl + '/' + aName + "?rev=" + resRev,
method: 'PUT',
headers: headersAttachment, //{'Content-Type': 'image/jpeg'}
body: attachment
}, function(e, r, b) {
console.log('body', b);
});
Questions: How do I read JPEG data from an HTTP res? What format should JPEG data be to forward an image? (i.e. base64, hex, something else?)
I found the answer here:
Node.js get image from web and encode with base64
I didn't know that request.js encodes URL responses. By turning it off the image was posted fine.
Related
I'm using Node.js/Express and submitting an image to the server from a webpage via AJAX. The image has been processed and cropped, so I'm using the resultant dataURL and sending it to my server by a post request like this. The dataURL is completely good and accurate and contains the image as it should. So I simply send it to the server.
var fd = new FormData();
fd.append('dataURL',dataURL);
console.log('dataURL length:',dataURL.length);
$.ajax({
type: "POST",
url: `/pwa/update-attachment`,
data: fd,
contentType: false,
processData: false,
success: function(result)
{
}
});
On the server side there are no errors but the dataURL parameters is truncated to 1mb. For reference, my server is setup to accept multipart form data using using bodyparser as follows. I tried using size limits with bodyParser (either high or low) and neither seem to take effect in this case.
const bodyParser = require('body-parser');
app.use(bodyParser.urlencoded({extended: false })); // support encoded bodies
I receive my post like this:
router.post('/update-attachment',(req,res) => {
console.log('dataurl:',req.body.dataURL.length);
...
});
The dataURL is truncated to 1048576 bytes. Works perfectly fine for smaller images.
The reason that there is not an error is that it is being truncated on the client side by the browser. To send the image, it should not be sent as a dataURL, but as a blob. The following answer explains this:
Convert Data URI to File then append to FormData
From a node.js back end, I need to send an HTTP message to a REST endpoint. The endpoint requires some parameters that it will expect to find in the HTTP message. Some of the parameters are simple enough, just requiring a number or a string as an argument. But one of the parameters is to be "the raw binary file content being uploaded" and this has puzzled me. As far as I understand, the parameters need to be gathered together into a string to put in the body of the HTTP request; How do I add raw binary data to a string? Obviously, for it to be in the string, it cannot be raw binary data; it needs to be encoded into characters.
The endpoint in question is the Twitter media upload API. The "raw binary data" parameter is called media. Below is an incomplete code snippet showing the basic gist of what I've tried. Specifically, the line where I build the requestBody string. I don't believe it is anywhere near correct, because the endpoint is returning a "bad request" message.
var https = require("https");
var base64ImageData = /* (some base 64 string here) */;
var options = {
host: "api.twitter.com",
path: "/1.1/media/upload.json",
method: "POST",
headers: {
"Content-Type": "multipart/form-data"
}
};
var request = https.request(options, function(response) {});
var requestBody = "media_id=18283918294&media=" + Buffer.from(base64ImageData, "base64").toString("binary");
request.write(requestBody);
request.end();
Also worth noting, Twitter themselves note the following extremely confusing statement:
"When posting base64 encoded images, be sure to set the “Content-Transfer-Encoding: base64” on the image part of the message."
Source: https://developer.twitter.com/en/docs/media/upload-media/uploading-media/media-best-practices
That might be part of the answer to my question, but what I don't understand is: How do I apply different headers to different parts of the HTTP message? Because apparently, the image data needs to have a Content-Transfer-Encoding header of "base64" while the rest of the HTTP message does not...
How do I apply different headers to different parts of the HTTP message?
This is the point of the multipart/form-data content type. A multi-part message looks like this:
Content-Type: multipart/form-data; boundary=---foo---
---foo---
Content-Disposition: form-data; name="datafile1"; filename="r.gif"
Content-Transfer-Encoding: base64
Content-Type: image/gif
// data goes here
---foo---
Content-Disposition: form-data; name="datafile2"; filename="g.png"
Content-Transfer-Encoding: base64
Content-Type: image/png
// another file's data goes here
---foo---
You probably don't want to put all this together yourself. There are a bunch of good libraries for putting together complex POSTs. For example: https://www.npmjs.com/package/form-data
I am making a POST request to retrieve a pdf. The request works fine if I do it in postman, but I get an empty pdf if I do it through node.js using the request package. Here's my request using the request package:
let body = {
attr1: "attr1",
attr2: "attr2"
}
let opts = {
url: "some_url",
method: "post",
headers: {
"Content-Type": "application/x-www-form-urlencoded",
},
body
}
request(requestOpts).then(pdf => {
console.log(pdf) // prints out the binary version of the pdf file
fs.writeFileSync("testing.pdf", pdf);
});
I use the exact same request parameters when I use postman but it returns the pdf w/ the correct content.
Can someone help? Or is the way I am saving my pdf incorrect?
Thanks in advance!
Solution - i had to set encoding: false in the request options.
Try
fs.writeFileSync("testing.pdf", pdf, 'binary');
The third argument here tells fs to write binary rather than trying to UTF-8 encode it.
According to the docs the third paramter should be a string that represents the encoding.
For pdf files the encoding is 'application/pdf'
So this should work for you : fs.writeFileSync("testing.pdf", pdf, 'application/psf');
I have a file in memory (buffer) - there is no file on the file system.
I want to send that buffer to another server that talks HTTP.
For example, some API A creates a file in memory, SignServer manipulates such files, and responds with a new buffer. My API takes the file from A and feeds it to SignServer.
I tried sending the file to SignServer in multiple ways, but it keeps responding with status 400 (missing field 'data' in request).
What I tried:
var http = require('http');
var querystring = require('querystring');
var data = querystring.stringify({
workerName: 'PDFSigner',
data: file_buffer
});
var request = new http.ClientRequest({
hostname: 'localhost',
port: 8080,
path: '/signserver/process',
method: 'GET',
headers: {
'Content-Type': 'application/x-www-form-urlencoded',
// I also tried 'multipart/form-data'
'Content-Length': Buffer.byteLength(data)
}
});
request.end(data);
I tried printing data, and it showed:
workerName=PDFSigner&data=
Which is bad because data wasn't set to file_buffer.
I tried printing file_buffer, and it does have content (not null, not undefined, actually has bytes inside).
So stringifying the buffer gave an empty string.
I tried doing the same thing with the request module and it didn't work either.
Note that SignServer isn't written in Node nor JavaScript. It's a Java application, so it probably doesn't work with json (which is why I'm trying to do it with querystring). Yes, I tried sending json.
The reason why data is set to an empty string is described in this issue and the solution is given in this issue.
escape and stringify the buffer like so:
var data = querystring.stringify({
workerName: 'PDFSigner',
data: escape(file_buffer).toString('binary')
});
As #robertklep mentioned, your other problem is that you can't send a big file using application/x-www-form-urlencoded. You'd need to do it with multipart/form-data.
I have a simple uploading code by node.js.
var http = require('http')
var fs = require('fs')
var server = http.createServer(function(req, res){
if(req.url == '/upload') {
var a = fs.createWriteStream('a.jpg', { defaultEncoding: 'binary'})
req.on('data', function(chunk){
a.write(chunk)
})
.on('end', function()){
a.end()
res.end('okay')
})
}
else {
fs.createReadStream('./index.html').pipe(res);
// just show <form>
}
})
server.listen(5000)
when I upload some image, I cannot get exact same file.
Always files are broken.
When I try to do this using formidable, I can get a fine file.
So I studied formidable but I cannot understand how did it catch data and save.
I could find formidable use parser to calculate something about chunk from request but I did not get it all.
(It is definitely my brain issue :( ).
Anyway, what is the difference between my code and formidable?
What am I missing?
Is it a wrong way to just add all chunks from http request and save it by
fs.createWriteStream or fs.writeFile ?
What concepts am I missing?
First, req is a Readable stream. You can simply do:
req.pipe(fs.createWriteStream('a.jpg'))
for the upload part. This is copying all byte data from request stream to file.
This will work when you send raw file data as the request body:
curl --data-binary #"/home/user/Desktop/a.jpg" http://localhost:8080/upload
Because this sends request body exactly as image binary data, that gets streamed to a file on server.
But, there is another request format called multipart/form-data. This is what web browsers use with <form> to upload files.
curl -form "image=#/home/user1/Desktop/a.jpg" http://localhost:8080/upload
Here the request body contains multiple "parts", one for each file attachment or form field, separated by special "boundary" characters:
--------------------------e3f25f5319cd6624
Content-Disposition: form-data; name="image"; filename="a.jpg"
Content-Type: application/octet-stream
JPG IHDRH-ÑtEXtSoftwareAdobe.....raw file data
--------------------------e3f25f5319cd6624
Hence you will need much more complicated code to extract the file part data from it. NPM Modules like busboy and formidable do exactly that.