How can I POST multiple binary files to a server with Content-Type: "form-data", using only the http module?
For example, my keys would look like this:
{
Image1: <binary-data>,
Image2: <binary-data>
}
TL:DR; See the full example code at the bottom of this answer.
I was trying to figure out how to POST multiple binary image files to a server in NodeJS using only core NodeJs modules (without using anything like npm install request, etc). After about 3 hours typing the wrong search terms into DuckDuckGo and finding no examples online, I was about to switch careers. But after banging my head against the desk for almost half the day, my sense of urgency was dulled and I managed to hobble together a working solution. This would not have been possible without the good people who wrote the answer to this Stackoverflow post, and this Github Gist. I used PostMan and Charles Proxy to analyze successful HTTP POSTS, as I dug around in the NodeJS docs.
There are a few things to understand to POST two binary images and a text field as multipart/form-data, relying only on core NodeJS modules:
1) Boundary Identifiers
What is the boundary in multipart/form-data?
The first part of the solution is to create a "boundary identifier", which is a string of dashes -, appended with a random number. You could probably use whatever you wish, foorbar and such.
------------0123456789
Then place that boundary between each blob of data; whether that is binary data or just text data. When you have listed all your data, you add the boundary identifier at the end, and append two dashes:
------------0123456789--
You also need to add the boundary to the headers so that server receiving the post understands which lines of your post data form the boundary between fields.
const headers = {
// Inform the server what the boundary identifier looks like
'Content-Type': `multipart/form-data; boundary=${partBoundary}`,
'Content-Length': binaryPostData.length
}
2) Form Field Meta Descriptors
(This probably is not what they are called)
You will also need a way to write the meta data for each form-field you send, whether that form field contains a binary or a text object. Here are the descriptors for an image file, which contain the mime type:
Content-Disposition: form-data; name="Image1"; filename="image-1.jpg"
Content-Type: image/jpeg
And here is the descriptor for a text field:
Content-Disposition: form-data; name="comment"
The Post Data Output
So the entire post data that is sent to the server will look like this:
----------------------------9110957588266537
Content-Disposition: form-data; name="Image1"; filename="image-1.jpg"
Content-Type: image/jpeg
ÿØÿàJFIFHHÿáLExifMMi
ÿí8Photoshop 3.08BIM8BIM%ÔÙ²é ìøB~ÿÀ... <<<truncated for sanity>>>
----------------------------9110957588266537
Content-Disposition: form-data; name="Image2"; filename="image-2.jpg"
Content-Type: image/jpeg
ÿØÿàJFIFHHÿáLExifMMi
ÿí8Photoshop 3.08BIM8BIM%ÔÙ²é ìøB~ÿÀ... <<<truncated for sanity>>>
----------------------------9110957588266537
Content-Disposition: form-data; name="comment"
This is a comment.
----------------------------9110957588266537--
Once this post data is generated, it can be converted to binary and written to the HTTP POST request: request.write(binaryPostData).
Example Code
Here is the full example code that will allow you to POST binary file and text data without having to include other NodeJS libraries and packages in your code.
// This form data lists 2 binary image fields and text field
const form = [
{
name: 'Image1',
type: 'file',
value: 'image-1.jpg'
},
{
name: 'Image2',
type: 'file',
value: 'image-2.jpg'
},
{
name: 'comment',
type: 'text',
value: 'This is a comment.'
}
]
// Types of binary files I may want to upload
const types = {
'.json': 'application/json',
'.jpg': 'image/jpeg'
}
const config = {
host: 'ec2-192.168.0.1.compute-1.amazonaws.com',
port: '80',
path: '/api/route'
}
// Create an identifier to show where the boundary is between each
// part of the form-data in the POST
const makePartBoundary = () => {
const randomNumbers = (Math.random() + '').split('.')[1]
return '--------------------------' + randomNumbers
}
// Create meta for file part
const encodeFilePart = (boundary, type, name, filename) => {
let returnPart = `--${boundary}\r\n`
returnPart += `Content-Disposition: form-data; name="${name}"; filename="${filename}"\r\n`
returnPart += `Content-Type: ${type}\r\n\r\n`
return returnPart
}
// Create meta for field part
const encodeFieldPart = (boundary, name, value) => {
let returnPart = `--${boundary}\r\n`
returnPart += `Content-Disposition: form-data; name="${name}"\r\n\r\n`
returnPart += value + '\r\n'
return returnPart
}
const makePostData = {
// Generate the post data for a file
file: (item, partBoundary) => {
let filePostData = ''
// Generate the meta
const filepath = path.join(__dirname, item.value)
const extention = path.parse(filepath).ext
const mimetype = types[extention]
filePostData += encodeFilePart(partBoundary, mimetype, item.name, item.value)
// Add the binary file data
const fileData = fs.readFileSync(filepath, 'binary')
filePostData += fileData
filePostData += '\r\n'
return filePostData
},
// Generate post data for the text field part of the form
text: (item, partBoundary) => {
let textPostData = ''
textPostData += encodeFieldPart(partBoundary, item.name, item.value)
return textPostData
}
}
const post = () => new Promise((resolve, reject) => {
let allPostData = ''
// Create a boundary identifier (a random string w/ `--...` prefixed)
const partBoundary = makePartBoundary()
// Loop through form object generating post data according to type
form.forEach(item => {
if (Reflect.has(makePostData, item.type)) {
const nextPostData = makePostData[item.type](item, partBoundary)
allPostData += nextPostData
}
})
// Create the `final-boundary` (the normal boundary + the `--`)
allPostData += `--${partBoundary}--`
// Convert the post data to binary (latin1)
const binaryPostData = Buffer.from(allPostData, 'binary')
// Generate the http request options
const options = {
host: config.host,
port: config.port,
path: config.path,
method: 'POST',
headers: {
// Inform the server what the boundary identifier looks like
'Content-Type': `multipart/form-data; boundary=${partBoundary}`,
'Content-Length': binaryPostData.length
}
}
// Initiate the HTTP request
const req = http.request(options, res => {
res.setEncoding('utf8')
let body = ''
// Accumulate the response data
res.on('data', chunk => {
body += chunk
})
// Resolve when done
res.on('end', () => {
resolve(body)
})
res.on('close', () => {
resolve(body)
})
res.on('error', err => {
reject(err)
})
})
// Send the binary post data to the server
req.write(binaryPostData)
// Close the HTTP request object
req.end()
})
// Post and log response
post().then(data => {
console.log(data)
})
.catch(err => {
console.error(err)
})
Related
I am trying to upload a text file to google drive via API call. I am working low level, so I don't use any external libraries. I send HTTPS Request, the file gets uploaded on Google Drive along with the content, but the document name is "Untitled". I am able to change the name in a separate HTTPS request using PATCH method, but I want to know if it is possible to set the name before sending the initial HTTPS request. Below is my the header file and HTTPS code I am sending, I would appreciate it if you can help me set the name anything other than "Untitled" .
const options = {
method: "POST",
headers: {
"Authorization": `${token_credential.token_type} ${token_credential.access_token}`,
"Content-Type":'text/plain' ,
},
body: {
name:"MyTextFile.txt"
}
}
let makeFileRequest=https.request(
sendDriveTask,
options,
(res, err) => {
if(err) {
console.log(err)
} else {
console.log("success", res.statusCode)
}
}
)
makeFileRequest.end(filecontent);
}
Below is the respond I get back from google drive:
{"kind":"drive#file","id":"1XBcb9q8Q__b6gIP0sH5oOhZOKQMezlIK","name":"Untitled","mimeType":"text/plain"}
From your showing script, I understood that you want to achieve your goal using https.request. In your situation, how about the following modification?
Modification points:
About I am trying to upload a text file to google drive via API call., when I saw your script, the text data is not included. If you want to create a text file with empty content, please modify the endpoint.
If you want to create a text file with the text content, please include the text data.
Modified script 1:
In this modification, a text file with empty content is created.
const https = require("https");
const token_credential = {
token_type: "Bearer",
access_token: "###", // Please set your access token.
};
const sendDriveTask = "https://www.googleapis.com/drive/v3/files";
const body = JSON.stringify({ name: "MyTextFile.txt" });
const options = {
method: "POST",
headers: {
Authorization: `${token_credential.token_type} ${token_credential.access_token}`,
"Content-Type": "application/json",
},
};
let makeFileRequest = https.request(sendDriveTask, options, (res) => {
res.on("data", (r) => {
console.log(r.toString());
});
res.on("end", () => {
console.log("Done.");
});
});
makeFileRequest.on("error", (e) => {
console.log(e.message);
});
makeFileRequest.write(body);
makeFileRequest.end();
When this script is run, a text file of "MyTextFile.txt" with empty content is created in the root folder.
Modified script 2:
In this modification, a text file with sample text content is created. In this case, the request body is sent as multipart/related.
const https = require("https");
const metadata = JSON.stringify({ name: "MyTextFile.txt" }); // Please set the file metadata.
const textData = "sample text data"; // Please set the sample text content.
const token_credential = {
token_type: "Bearer",
access_token: "###", // Please set your access token.
};
const sendDriveTask = "https://www.googleapis.com/upload/drive/v3/files?uploadType=multipart";
const boundary = "sample123";
const body = [`--${boundary}\r\n`, "Content-Type: application/json; charset=utf-8\r\n\r\n", metadata + "\r\n", `--${boundary}\r\n`, "Content-Type: text/plain\r\n\r\n", textData + "\r\n", `--${boundary}--`].join("");
const options = {
method: "POST",
headers: {
Authorization: `${token_credential.token_type} ${token_credential.access_token}`,
"Content-Type": "multipart/related; boundary=" + boundary,
},
};
let makeFileRequest = https.request(sendDriveTask, options, (res) => {
res.on("data", (r) => {
console.log(r.toString());
});
res.on("end", () => {
console.log("Done.");
});
});
makeFileRequest.on("error", (e) => {
console.log(e.message);
});
makeFileRequest.write(body);
makeFileRequest.end();
When this script is run, a text file of "MyTextFile.txt" with the sample text of sample text data is created in the root folder.
Note:
This sample script supposes that your access token can be used for creating a file on Google Drive. Please be careful about this.
About the 2nd pattern, the maximum file size is 5 MB. When you want to upload more sizes, please use the resumable upload. Ref
References:
Files: create
Upload file data
I'm not knowledgeable enough to know all the ins and outs of formatting data for http requests. I'm trying to send FormData from a vue.js app into a netlify serverless function (lambda) and then pass that FormData along to my Contact Form 7 WordPress plugin REST API.
I managed to get my FormData passed to my lambda using JSON.stringify, and when I JSON.parse the data seems to be intact. I then used form-data in node to build a new FormData object to pass.
I noticed however that I'm unable to console.log it's contents using the client-side method of:
// I get values is not a function
for (var value of formData.values()) {
console.log('>> VALUE = ',value);
}
// I get entries is not a function
for (var pair of formData.entries()) {
console.log(pair[0]+ ', ' + pair[1]);
This is a red flag to me, telling me that FormData in node might not be handled the same as FormData in my vue.js code..
When I try to hit my Contact Form 7 endpoint with the data, I get an error in my response saying that one or more of my fields are in error, even though it seems to look ok to me, so something is up, but I've been banging my head against the wall trying to determine what the solution is.
My gut is telling me that there's something I need to do still, to format the data, or send the data in a way that Contact Form 7 is expecting..
Earlier in my project history, when I ran an axios.post in vue.js (not using netlify lambda) it worked and my contact form emails were sending, so I know I'm hitting the right endpoint with the right codes/data.
Here is all the relevant code I'm using for this project:
// --------------------------------------------
// in my vue.js component:
// --------------------------------------------
this.bodyFormData = new FormData()
this.bodyFormData.append( 'your-name', this.value_name )
this.bodyFormData.append( 'tel-725', this.value_phone )
this.bodyFormData.append( 'your-email', this.value_email )
this.bodyFormData.append( 'your-subject', this.value_subject )
this.bodyFormData.append( 'your-message', this.value_message )
// (...)
let theFormData = JSON.stringify(Object.fromEntries(this.bodyFormData))
Vue.prototype.$http.post('/.netlify/functions/myfunction',{token:token, formData:theFormData})
// --------------------------------------------
// in my netlify serverless lambda function myfunction.js :
// --------------------------------------------
const axios = require('axios');
const FormData = require('form-data');
const AUTH_API_ENDPOINT = 'https://www.####.com/wp-json/jwt-auth/v1/token/'
const FORM_API_ENDPOINT = 'https://www.####.com/wp-json/contact-form-7/v1/contact-forms/1217/feedback'
const captchaThreshhold = 0.5
exports.handler = async function(event, context) {
const eventBody = JSON.parse(event.body)
const captchaToken = eventBody.token
const stringFormData = eventBody.formData
let parsedFormData = JSON.parse(stringFormData);
console.log('>> parsedFOrmData ', parsedFormData) //logs a JSON object with correct key/value pairs
// logs:
// >> parsedFOrmData {
// 'your-name': 'Jon Doe',
// 'tel-725': '(555) 555-5555',
// 'your-email': 'jon#doe.com',
// 'your-subject': 'Suuuuubject',
// 'your-message': 'Meeeeesage!'
// }
let formData = new FormData();
for ( var key in parsedFormData ) {
formData.append(key, parsedFormData[key])
}
// I get values is not a function
for (var value of formData.values()) {
console.log('>> VALUE = ',value);
}
// I get entries is not a function
for (var pair of formData.entries()) {
console.log(pair[0]+ ', ' + pair[1]);
}
// (...)
axios.post(FORM_API_ENDPOINT, {formData}, {
headers: {
'Authorization': `Bearer ${res.data.token}`,
// 'Content-Type': 'multipart/form-data; charset="utf-8"', //do I need this?
}
})
.then( res => {
console.log('>> response came back from the Form endpoint : ',res)
})
// the res.data I get back form WordPress Contact Form 7 Plugin Endpoint:
data: {
into: '#',
status: 'validation_failed',
message: 'One or more fields have an error. Please check and try again.',
posted_data_hash: '',
invalid_fields: [ [Object], [Object], [Object], [Object] ]
}
//res.config data logs as this:
{"formData":{"_overheadLength":545,"_valueLength":54,"_valuesToMeasure":[],"writable":false,"readable":true,"dataSize":0,"maxDataSize":2097152,"pauseStreams":true,"_released":false,"_streams":["----------------------------611729353459041078880042\\r\\nContent-Disposition: form-data; name=\\"your-name\\"\\r\\n\\r\\n","Jon Doe",null,"----------------------------611729353459041078880042\\r\\nContent-Disposition: form-data; name=\\"tel-725\\"\\r\\n\\r\\n","(555) 555-5555",null,"----------------------------611729353459041078880042\\r\\nContent-Disposition: form-data; name=\\"your-email\\"\\r\\n\\r\\n","jon#doe.com",null,"----------------------------611729353459041078880042\\r\\nContent-Disposition: form-data; name=\\"your-subject\\"\\r\\n\\r\\n","Suuuuubject",null,"----------------------------611729353459041078880042\\r\\nContent-Disposition: form-data; name=\\"your-message\\"\\r\\n\\r\\n","Meeeeesage!",null],"_currentStream":null,"_insideLoop":false,"_pendingNext":false,"_boundary":"--------------------------611729353459041078880042"}}
If you know what the problem is.. Please tell me what I'm doing wrong! Thank you! :)
I solved the issue... it seems that the FormData headers need to be passed along with the data.. I randomly stumbled across the answer while messing around with Postman and found the answer buried in the Node.js code view.
For those of you who have the same issue.. see below:
axios.post(FORM_API_ENDPOINT, formData, {
headers: {
'Authorization': `Bearer ${res.data.token}`,
'Content-Type': 'multipart/form-data; charset="utf-8"',
...formData.getHeaders() // <--- THIS LINE HERE
}
})
It's been days now that I am stuck on what I guess should be a simple problem to solve...
I have a nodeJS express app that, upon request to a specific route, makes an AXIOS post request with a payload to an external service.
This service does respond with a multipart/mixed content-type (content type is mentioned in headers).
The respond contains a PDF and a JSON with info.
I am successfully saving response.data entirely as a PDF on AWS S3.
But I cannot find at least one way to parse the response.data to extract the JSON information that I need.
Here is what I get in my Postman response panel,
(As you can notice, it s a method to generate a french postal service label on Colissimo)
May I aslo add that I tried, unsuccessfully many npm packages.
Thanks a lot if you know how to manipulate this !
--uuid:f0bf15c8-03ba-41a4-8c9a-aac9d8e0d9ca
Content-Type: application/json;charset=UTF-8
Content-Transfer-Encoding: binary
Content-ID: <jsonInfos>
{"messages":[{"id":"0","type":"INFOS","messageContent":"La requête a été traitée avec succès","replacementValues":[]}],"labelXmlV2Reponse":null,"labelV2Response":{"parcelNumber":"8R42943164203","parcelNumberPartner":null,"pdfUrl":null,"fields":null}}
--uuid:f0bf15c8-03ba-41a4-8c9a-aac9d8e0d9ca
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary
Content-ID: <label>
%PDF-1.4
%����
3 0 obj <</Filter/FlateDecode/Length 1567>>stream
x��X�n7��W�L��;�`��)��!��I)�#��Ts�����������]
.... and more
My simple AXIOS request inside my route
let url = "https://ws.colissimo.fr/sls-ws/SlsServiceWSRest/2.0/generateLabel";
const response = await axios({
url,
method: 'POST',
data,
responseType: "stream"
});
console.log("data", response.data);
I had the same problem has you for requesting a label to the Colissimo web service.
Bellow, the solution I found when doing tests:
function streamToString (stream) {
const chunks = [];
return new Promise((resolve, reject) => {
stream.on('data', (chunk) => chunks.push(Buffer.from(chunk)));
stream.on('error', (err) => reject(err));
stream.on('end', () => resolve(Buffer.concat(chunks).toString('utf8')));
})
}
axios.post(urlApiRest, generateLabelData, {responseType: 'stream'}).then(res => {
res.data.pipe(fs.createWriteStream('../public/test.pdf')); // write label to pdf
streamToString(res.data).then(strData => {
const jsonInfo = extractInfoFromColissimoResponse(res, strData);
console.log(jsonInfo.labelV2Response.parcelNumber); // retrieve parcel number
});
}).catch(err => {
console.log(err);
});
function extractInfoFromColissimoResponse(res, data) {
let header = res.headers['content-type'];
let boundary = header.split(' ')[1];
boundary = boundary.split('=')[1];
boundary = boundary.split('"')[1];
const infoPart = data.split('--' + boundary)[1];
const txtInfo = infoPart.split('<jsonInfos>')[1];
return JSON.parse(txtInfo);
}
i got a problem here.
Lets start from we have API, this API returns data with next headers:
Content-Type image/png
Transfer-Encoding chunked
Connection keep-alive
And response body where only file is.
When im try to write this body data into file - it's always broken. I mean i've binary data, his mime, body.length is match original filesize, but this image could not be opened in any viewer after i save it.
What i'm do:
public userFile(req, res: Response) {
const data = {
fileId: parseInt(req.body.fileId),
};
let params = {
headers: {
'Authorization': keys.token,
},
};
axios.post('/api/getfile/', data, params,)
.then((response: AxiosResponse) => {
const fs = require('fs');
const dir = require('path').resolve(__dirname + '../../../files/storage');
const ext = {
'image/png': '.png'
};
fs.writeFile(dir + '/' + img + ext[response.headers['content-type']], response.data, (er) => {
res.send(response.data);
});
})
.catch((err) => {
logger.error("AXIOS ERROR: ", err)
})
}
BUT! When i get this file with postman... here it is!
So, i need your help - what i do wrong?
You must specifically declare which response type it is in params as
responseType: 'stream'
then save stream to a file using pipe
response.data.pipe(fs.createWriteStream("/dir/xyz.png"))
I am receiving a PDF file from a node server (it is running jsreport in this server) and i need to download this PDF in the client (i am using react in the client) but the problem is that when i download the file, it comes all blank and the title some strange symbols. After a lot of tests and researchs, i found that the problem may be that the file is coming enconded as chunked (i can see that in the headers of the response) and i need to decode do become a file again.
So, how to decode this chunked string to a file again?
In the client i am just downloading the file that comes in the responde:
handleGerarRelatorioButtonClick(){
axios.post(`${REQUEST_URL}/relatorios`, this.state.selectedExam).then((response) => {
fileDownload(response.data, this.state.selectedExam.cliente.nome.replace(' ', '_') + ".pdf");
});
}
In my server, i am making a request to my jsreport that is other node server and it returns the report as a PDF:
app.post('/relatorios', (request, response) => {
var exame = new Exame(request.body);
var pdf = '';
var body = {
"template": {
"shortid": "S1C9birB-",
"data": exame
}
};
var options = {
hostname: 'localhost',
port: 5488,
path: '/api/report',
method: 'POST',
headers: {
'Content-Type': 'application/json'
}
};
var bodyparts = [];
var bodylength = 0;
var post = http.request(options, (res) => {
res.on('data', (chunk) => {
bodyparts.push(chunk);
bodylength += chunk.length;
});
res.on('end', () => {
var pdf = new Buffer(bodylength);
var pdfPos = 0;
for(var i=0;i<bodyparts.length;i++){
bodyparts[i].copy(pdf, pdfPos, 0, bodyparts[i].length);
pdfPos += bodyparts[i].length;
}
response.setHeader('Content-Type', 'application/pdf');
response.setHeader('Content-disposition', exame._id + '.pdf');
response.setHeader('Content-Length', bodylength);
response.end(Buffer.from(pdf));
});
});
post.write(JSON.stringify(body));
post.end();
});
I am sure that my report is being rendered as expected because if i make a request from postman, it returns the PDF just fine.
Your solution is simply relaying data chunks but you are not telling your front end what to expect of these chunks or how to assemble them. At a minimum you should be setting the the Content-Type response header to application/pdf and to be complete should also be sending the Content-disposition as well as Content-Length. You may need to collect the PDF from your 3rd party source into a buffer and then send that buffer to your client if you are not able to set headers and pipe to response successfully.
[edit] - I'm not familiar with jsreport but it is possible (and likely) that the response they send is a buffer. If that is the case you could use something like this in place of your response to the client:
myGetPDFFunction(params, (err, res) => {
if (err) {
//handle it
} else {
response.writeHead(200, {
'Content-Type': 'application/pdf',
'Content-Length': [your buffer's content length]
});
response.end(Buffer.from([the res PDF buffer]));
}
}
What you haven't shown is the request made to obtain that PDF, so I couldn't be more specific at this time. You should look into the documentation of jsreport to see what it sends in its response, and you can also read up on buffers here
This is rough pseudo code but the point is to respond with the PDF buffer after setting the headers to their proper values.