Sending Form Data with the native node module - node.js

for my current project I have to send form-data from my lambda function to an api endpoint. The api endpoint essentially expects two images (that it compares with one another) and a key. As mentioned before, I somehow seem unable to send the correct form-data to the api endpoint. I checked out postman, and it seems to have worked alright, but something doesn't seem to work in my function. I presume it must be related the form-data string that I'm sending. Below you can find a shortened version of the function (I excluded the two image files), but somehow I'm getting an error back telling me that the api cannot read the key property:
const http = require('http');
const https = require('https');
const httpPromise = (protocol, params, postData) => {
return new Promise((resolve, reject) => {
const requestModule = protocol === 'http' ? http : https;
const req = requestModule.request(params, res => {
// grab request status
const statusCode = res.statusCode;
if(statusCode < 200 || statusCode > 299) {
throw new Error('Request Failed with Status Code:', statusCode);
}
let body = '';
// continuosly update data with incoming data
res.setEncoding('utf8');
res.on('data', data => body += data);
// once all data was received
res.on('end', () => resolve(body));
})
// write data to a post request
if(typeof(params.method) === 'string' && params.method === 'POST' && postData) {
req.write(postData)
}
// bind to the error event
req.on('error', err => reject(err));
// end the request
req.end();
})
}
const controller = async () => {
const apiKey = "00000000";
const options = {
hostname: '***"
port: 80,
path: '***'
method: 'POST',
headers: {"content-type": "multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW"}
}
const postData = "------WebKitFormBoundary7MA4YWxkTrZu0gW\r\nContent-Disposition: form-data; name=\"key\"\r\n\r\00000000\r\n------WebKitFormBoundary7MA4YWxkTrZu0gW--"
let result = await httpPromise('http', options, postData)
console.log(result);
}
yeah, so somehow it just doesn't seem to recognise the key in the postData string. I have tried various different combinations but just can't seem to get this to work.

The default http and https libraries are kind of wordy and annoying.
Would recommend using the request library instead. Read more here
In which case, to make the request, you can simply write it as :
var request = require('request');
var formData = {
// Pass a simple key-value pair
my_field: 'my_value',
}
request.post({url:'http://service.com/upload', formData: formData}, (err, response, body) => {
// Handle response here
});

Alright, so for anyone who might also face the same issue, it took me a little but figured out what the issue was. I didn't set the Content-Length header, which then in turn meant that node automatically added the Transfer-Encoding Header and set its value to chunk. This broke the receiving api and resulted in the issue. Setting the Content-Length header to the correct length and setting the Transfer-Encoding Header to an empty string solved my issue here (but I think one could also simply omit the transfer-encoding header once you defined the Content-Length Header).

Related

Nodejs, express http request - response not unzipping

I am trying to write a function that creates an HTTPS request.
This is all part of an expressjs project with Typescript.
I can get the HTTPS request working and getting a response - but the response is encoded with GZIP. I am trying my best to follow the documentation. But no luck, the response stays zipped.
This is my code
private getData = (host, pathname): Promise<string> => {
return new Promise((resolve, reject) => {
const options = {
hostname: host,
path: pathname,
gzip: true,
method: 'GET',
headers: {'x-apikey': 'XXXX'}
}
const req = https.request(options, (res) => {
if (res.statusCode < 200 || res.statusCode >= 300) {
console.log('error!')
return reject(new Error('statusCode=' + res.statusCode));
}
let body = '';
res.on('data', function (chunk) {
body += chunk;
});
res.on('end', function () {
try {
console.log('res.headers', res.headers)
console.log('res.headers', body)
body = JSON.parse.toString();
resolve(body);
} catch (e) {
reject(e);
}
resolve(body);
});
req.on('error', (e) => {
reject(e.message);
});
// send the request
});
req.end();
});
}
The console log statement shows this
So obviously the JSONparsing is failing. What am I missing?
Thanks heaps for the help
https.request() does not natively support gzip compression. So, you will have to either add support for it yourself as shown here or use an https request library such as got() that already supports gzip (and also already supports promises too).
import got from 'got';
private getData = (host, pathname): Promise<string> => {
const options = {
method: 'GET',
headers: {'x-apikey': 'XXXX'}
}
return got(`https://${host}${pathname}`, options).json();
}
FYI, it's not clear in your question what type of response you are expecting. You attempt to use:
body = JSON.parse.toString();
which doesn't make any sense because JSON.parse.toString() is trying to get string version of the JSON.parse function without actually even calling that function - very odd. Perhaps you meant JSON.parse(body)? But, your typescript makes it look like you're expecting a promise that resolves to a string so that wouldn't usually be the result of JSON parsing.
If you are expecting a gzipped JSON response, then you would use the:
return got(`https://${host}${pathname}`, options).json();
I show above. If you are expecting just a string, not JSON, then you would change that line to:
return got(`https://${host}${pathname}`, options).text();

aws elasticsearch getting signature error on post request

Got a 403 signature error , when using the below fetch function:
function elasticsearchFetch(AWS, elasticsearchDomain, endpointPath, options = {}, region = process.env.AWS_REGION) {
return new Promise((resolve, reject) => {
const { body, method = 'GET' } = options;
const endpoint = new AWS.Endpoint(elasticsearchDomain);
const request = new AWS.HttpRequest(endpoint, region);
request.method = method;
request.path += endpointPath;
request.headers.host = elasticsearchDomain;
if (body) {
request.body = body;
request.headers['Content-Type'] = 'application/json';
request.headers['Content-Length'] = request.body.length;
}
const credentials = new AWS.EnvironmentCredentials('AWS');
const signer = new AWS.Signers.V4(request, 'es');
signer.addAuthorization(credentials, new Date());
const client = new AWS.HttpClient();
client.handleRequest(request, null, (res) => {
let chunks = '';
res.on('data', (chunk) => {
chunks += chunk;
});
res.on('end', () => {
if (res.statusCode !== 201) console.log('Got these options STATUSCODE', JSON.stringify(options, false, 2));
return resolve({ statusCode: res.statusCode, body: chunks });
});
}, (error) => {
console.log('Got these options ERROR', JSON.stringify(options, false, 2));
return reject(error);
});
});
}
This is the options used for the request in above function :
{
"method": "POST",
"body": "{\"prefix\":\"image_233/ArtService/articles-0/GB/ART-60297885/\",\"id\":\"ART-60297885\",\"retailUnit\":\"GB\",\"commercial\":{\"name\":{\"en-GB\":\"FÖRBÄTTRA\"}},\"schemaType\":\"product\",\"productType\":\"ART\"}"
}
and got this error :
{
"statusCode": 403,
"body": "{\"message\":\"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\"}"
}
This is the endpoint : 233/_doc/
I believe your Content-Length header is incorrect, causing the signature mismatch.
Your payload includes the string FÖRBÄTTRA, which has two double-byte characters.
You're setting the Content-Length to request.body.length, which comes to 186.
While this is the number of characters in the body, it is not the number of bytes in the body (188).
To calculate the Content-Length, use Buffer.byteLength(request.body). For a POST request like this, you can even remove that line of code altogether, and the request will succeed.
// Content-Length is only needed for DELETE requests that include a request
// body, but including it for all requests doesn't seem to hurt anything.
request.headers['Content-Length'] = Buffer.byteLength(request.body);
Source: https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-request-signing.html#es-request-signing-node
By the way, why not use elasticsearch client for nodejs to communicate with elasticsearch rather than writing your own logic. You can consider using http-aws-es which does the request signing part for you. The code will look like
const { Client } = require("elasticsearch");
const esConnectionClass = require("http-aws-es");
const elasticsearchConfig = {
host: "somePath",
connectionClass: esConnectionClass
};
const nativeClient = new Client(elasticsearchConfig);
const result = await nativeClient.search({});

Unable to get response of http request to a live website URL using node http module

Basically I want to test 500+ live website URLs and get the statusCode of them using protractor and getting help from node's http module.
But I'm not getting anything in response and all console statements related to the http request are ignored in my terminal.
I've tried using some external libraries such as axios and protractor-http-client but all are giving me same result.
I'm using TypeScript.
import request from 'request' // tried this one earlier instead of http
import https from 'https';
async testFunction () {
const req = await https.get('https://www.google.com');
let body = '';
req.on('data',(d) => {
body += d;
});
req.on('end',(resp) => {
console.log(resp.statusCode); // Not printing in terminal at all
body = JSON.parse(body);
});
req.on('response', (resp) => { // also tried information instead of response
console.log(resp.statusCode); // Not printing in terminal at all
});
console.log(body); // printing as empty
}
}
I would expect the response status code be 200.
I'm new to both protractor and node. So any kind of help and suggestion would be appreciated.
const http = require('http');
const res = http.request({
host: 'google.de',
method: 'get',
path: '/'
}, (response) => {
let dataStr = [];
const receiveData = chunk => {
const b = Buffer.from(chunk);
dataStr.push(b);
};
response.on('end', () => {
console.log('status Code:', response.statusCode);
console.log('response:', Buffer.concat(dataStr).toString());
});
response.on('error', (e) => console.error(e));
response.on('data', receiveData);
});
res.end();

How to make an https version of a Unirest example

I would like to use the https library in node.js to send a request to this api:
https://rapidapi.com/dimas/api/NasaAPI?endpoint=apiendpoint_b4e69440-f966-11e7-809f-87f99bda0814getPictureOfTheDay
The given example on the RapidAPI website uses Unirest, and I would like to only use the https library. I've tried to write it like this:
const https = require('https');
var link = "https://NasaAPIdimasV1.p.rapidapi.com/getPictureOfTheDay";
var options = {host: "https://NasaAPIdimasV1.p.rapidapi.com/getPictureOfTheDay",
path: "/", headers: {"X-RapidAPI-Key": "---MY KEY(Yes, I've replaced it)---", "Content-Type": "application/x-www-form-urlencoded"}}
https.get(link, options, (resp) => {
let data = '';
resp.on('data', (chunk) => {
data += chunk;
});
resp.on('end', () => {
console.log(data);
});
}).on("error", (err) => {
console.log("https error 4: " + err.message);
});
But that returns the following response:
{"message":"Endpoint\/ does not exist"}
Thanks for any help
There are several mistakes.
First, you essentially pass URL in https twice - first as link param, second as combination of host and path properties for options param.
Second, your host is actually the full path - but it shouldn't be. In the end, looks like the library got confused and sent request to https://NasaAPIdimasV1.p.rapidapi.com/ instead.
Finally, this particular API requires using 'POST', not 'GET' method. That's actually mentioned in the documentation. That's why you have 'endpoint does not exist' error even on correctly formed request.
One possible approach is dropping link altogether, sending URL as part of options:
var options = {
host: 'NasaAPIdimasV1.p.rapidapi.com',
method: 'POST',
path: '/getPictureOfTheDay',
headers: {/* the same */}
};
https.request(options, (resp) => { /* the same */ }).end();

Get the response "ContentType" in a nodejs http request

I want to donwload a file via http and check the "ContentType" response header. My Download looks like this:
var fileUrl = "<url>";
var request = https.get(fileUrl, function (res) {
res.on('data', function (data) {
//...
});
res.on('error', function (error) {
//...;
});
I get the data, but is there any way to acces the content type resonse header?
The res variable is an instance of http.IncomingMessage, which has a headers property that contains the headers:
var request = https.get(fileUrl, function (res) {
var contentType = res.headers['content-type'];
...
});
If you want to get only mime-type, be aware that Content-Type header can include other information such as charset or boundary.
Use parser such as content-type-parser instead of reading header directly.
const contentTypeParser = require("content-type-parser");
const contentType = contentTypeParser(req.headers['content-type']);
const mimeType = contentType.type+'/'+contentType.subtype;

Resources