nodejs http request using async/await syntax [duplicate] - node.js

In my program I make async call for my function from another API module:
var info = await api.MyRequest(value);
Module code:
var request = require("request")
module.exports.MyRequest = async function MyRequest(value) {
var options = {
uri: "http://some_url",
method: "GET",
qs: { // Query string like ?key=value&...
key : value
},
json: true
}
try {
var result = await request(options);
return result;
} catch (err) {
console.error(err);
}
}
Execution returns immediately, however result and therefore info contains request object and request body - info.body like key=value&..., not required response body.
What I'm doing wrong? How to fix? What is proper request usage with async, or it only works correctly with promises like mentioned here: Why await is not working for node request module? Following article mentioned it is possible: Mastering Async Await in Node.js.

You need to use the request-promise module, not the request module or http.request().
await works on functions that return a promise, not on functions that return the request object and expect you to use callbacks or event listeners to know when things are done.
The request-promise module supports the same features as the request module, but asynchronous functions in it return promises so you can use either .then() or await with them rather than the callbacks that the request module expects.
So, install the request-promise module and then change this:
var request = require("request");
to this:
const request = require("request-promise");
Then, you can do:
var result = await request(options);
EDIT Jan, 2020 - request() module in maintenance mode
FYI, the request module and its derivatives like request-promise are now in maintenance mode and will not be actively developed to add new features. You can read more about the reasoning here. There is a list of alternatives in this table with some discussion of each one.
I have been using got() myself and it's built from the beginning to use promises, supports many of the same options as the request() module and is simple to program.

Pretty sure you can also do the following. If what you need does not return Promise by default you can provide it via new Promise method. Above answer is less verbose though.
async function getBody(url) {
const options = {
url: url,
method: 'GET',
};
// Return new promise
return new Promise(function(resolve, reject) {
// Do async job
request.get(options, function(err, resp, body) {
if (err) {
reject(err);
} else {
resolve(body);
}
})
})
}

I just managed to get it to work with async/await. I wrapped it inside a function promisifiedRequest to return a promise that runs the request callback and resolves or rejects it based on error and response.
const request = require('request');
const promisifiedRequest = function(options) {
return new Promise((resolve,reject) => {
request(options, (error, response, body) => {
if (response) {
return resolve(response);
}
if (error) {
return reject(error);
}
});
});
};
(async function() {
const options = {
url: 'https://www.google.com',
method: 'GET',
gzip: true,
headers: {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.96 Safari/537.36'
}
};
let response = await promisifiedRequest(options);
console.log(response.headers);
console.log(response.body);
})();

Since request-promise has been deprecated, here are other options that don't depend on the NPM request package. got has been mentioned already, but it depends on 11 other packages. axios, in contrast, only has 1 dependency (for redirects). Everything else is natively implemented and built on top of the native NodeJS packages.
Here is the same example using axios:
const axios = require('axios')
const response = await axios.get(url)
const result = response.data
or, as a one-liner in JavaScript
const result = (await axios.get(url)).data
One-liner in TypeScript:
const {data} = await axios.get(url)

For simple cases where you don't need advanced features like cookies, following redirects and retrying, you can use native http/https module to make requests:
const https = require('https')
async function fetch(url) {
return new Promise((resolve, reject) => {
const request = https.get(url, { timeout: 1000 }, (res) => {
if (res.statusCode < 200 || res.statusCode > 299) {
return reject(new Error(`HTTP status code ${res.statusCode}`))
}
const body = []
res.on('data', (chunk) => body.push(chunk))
res.on('end', () => {
const resString = Buffer.concat(body).toString()
resolve(resString)
})
})
request.on('error', (err) => {
reject(err)
})
request.on('timeout', () => {
request.destroy()
reject(new Error('timed out'))
})
})
}
const res = await fetch('https://...')

Related

Nodejs, express http request - response not unzipping

I am trying to write a function that creates an HTTPS request.
This is all part of an expressjs project with Typescript.
I can get the HTTPS request working and getting a response - but the response is encoded with GZIP. I am trying my best to follow the documentation. But no luck, the response stays zipped.
This is my code
private getData = (host, pathname): Promise<string> => {
return new Promise((resolve, reject) => {
const options = {
hostname: host,
path: pathname,
gzip: true,
method: 'GET',
headers: {'x-apikey': 'XXXX'}
}
const req = https.request(options, (res) => {
if (res.statusCode < 200 || res.statusCode >= 300) {
console.log('error!')
return reject(new Error('statusCode=' + res.statusCode));
}
let body = '';
res.on('data', function (chunk) {
body += chunk;
});
res.on('end', function () {
try {
console.log('res.headers', res.headers)
console.log('res.headers', body)
body = JSON.parse.toString();
resolve(body);
} catch (e) {
reject(e);
}
resolve(body);
});
req.on('error', (e) => {
reject(e.message);
});
// send the request
});
req.end();
});
}
The console log statement shows this
So obviously the JSONparsing is failing. What am I missing?
Thanks heaps for the help
https.request() does not natively support gzip compression. So, you will have to either add support for it yourself as shown here or use an https request library such as got() that already supports gzip (and also already supports promises too).
import got from 'got';
private getData = (host, pathname): Promise<string> => {
const options = {
method: 'GET',
headers: {'x-apikey': 'XXXX'}
}
return got(`https://${host}${pathname}`, options).json();
}
FYI, it's not clear in your question what type of response you are expecting. You attempt to use:
body = JSON.parse.toString();
which doesn't make any sense because JSON.parse.toString() is trying to get string version of the JSON.parse function without actually even calling that function - very odd. Perhaps you meant JSON.parse(body)? But, your typescript makes it look like you're expecting a promise that resolves to a string so that wouldn't usually be the result of JSON parsing.
If you are expecting a gzipped JSON response, then you would use the:
return got(`https://${host}${pathname}`, options).json();
I show above. If you are expecting just a string, not JSON, then you would change that line to:
return got(`https://${host}${pathname}`, options).text();

AWS Lambda async function requires multiple runs to work properly

Essentially the issue is that I have multiple chained promised in an async function. So obviously the code should be waiting for the response before proceeding onto the next part. However with my test case it only works properly after multiple runs. Otherwise it shows the console.log statements sequentially after each run until around 5 runs or so when the final data appears. Sorry if this is an obvious answer or my code is atrocious. I'm rather new to both NodeJS and Async functions. It should also be noted I'm using NodeJS v8.10.
The order is essentially: Call to s3 to get ID in text file -> async function to call api to verify group details -> async function to call api to get other information based on the last function etc.
I've tried messing with the order of the async functions, and searching online for anything like this phenomenon but I was unable to find any examples.
const https = require('https');
var AWS = require('aws-sdk');
var s3 = new AWS.S3();
exports.handler = async (event, context) => {
var groupData;
var groupID;
await fetchGroupID().then(function(fetchedGroupID){
groupID = fetchedGroupID;
return fetchedGroupID;
}).then(function(data){
fetchGroupInfo().then(function(groupInfo){
groupData = groupInfo;
var fetchedGroupID = groupInfo.id;
fetchGroupEvents(fetchedGroupID).then(function(data){
console.log("Group Events: ", data);
});
});
});
console.log(groupData);
};
// Example of one of the async functions
async function fetchGroupEvents(fetchedGroupID)
{
console.log("Fetched GI: ", fetchedGroupID);
console.log("GI: ", groupID);
return new Promise((resolve, reject) => {
const options = {
protocol: 'https:',
hostname: 'api.boomset.com',
port: 443,
path: '/events?group_id=' + groupID,
method: 'GET',
headers: {
'Content-Type' : 'application/json',
'Authorization' : 'Token ****',
'Accept-Encoding': 'gzip, deflate'
}
};
const req = https.request(options, (res) => {
//console.log(`statusCode: ${res.statusCode}`);
res.setEncoding('utf8');
res.on('data', function(data) {
console.log(data);
//var jsonData = JSON.parse(data);
//console.log(jsonData);
resolve(data);
});
});
req.on('error', (error) => {
//console.error(error);
reject(error.message);
});
req.end();
});
};
Ideally this would all be completed in one run.

Awaiting http request in AWS Lambda / Pulumi

I have an AWS Lambda function which triggers https request to Google API. I want the function to be awaitable, so that it does not end immediately, but only after getting response from Google API.
Yes, I know I pay for the execution, but this will not be called often, so it is fine.
The problem is that the http request does not seem to fire correctly. The callback is never executed.
I have made sure that the async/await works as expected by using setTimeout in a Promise. So the issue is somewhere in the https.request.
Also note that I am using Pulumi to deploy to AWS, so there might be some hidden problem in there. I just can't figure out where.
The relevant code:
AWS Lambda which calls the Google API
import config from '../../config';
import { IUserInfo } from '../../interfaces';
const https = require('https');
function sendHttpsRequest(options: any): Promise<any> {
console.log(`sending request to ${options.host}`);
console.log(`Options are ${JSON.stringify(options)}`);
return new Promise(function (resolve, reject) {
console.log(` request to ${options.host} has been sent A`);
let body = new Array<Buffer>();
const request = https.request(options, function (res: any) {
console.log('statusCode:', res.statusCode);
console.log('headers:', res.headers);
if (res.statusCode != 200) {
reject(res.statusCode);
}
res.on('data', (data: any) => {
console.log(`body length is ${body.length}`);
console.log('data arrived', data);
body.push(data);
console.log('pushed to array');
console.log(data.toString());
});
});
request.on('end', () => {
console.error('Request ended');
// at this point, `body` has the entire request body stored in it as a string
let result = Buffer.concat(body).toString();
resolve(result);
});
request.on('error', async (err: Error) => {
console.error('Errooooorrrr', err.stack);
console.error('Errooooorrrr request failed');
reject(err);
});
request.end();
console.log(` request to ${options.host} has been sent B`);
});
}
/**
* AWS Lambda to create new Google account in TopMonks domain
*/
export default async function googleLambdaImplementation(userInfo: IUserInfo) {
const payload = JSON.stringify({
"primaryEmail": userInfo.topmonksEmail,
"name": {
"givenName": userInfo.firstName,
"familyName": userInfo.lastName
},
"password": config.defaultPassword,
"changePasswordAtNextLogin": true
});
const resultResponse: Response = {
statusCode: 200,
body: 'Default response. This should not come back to users'
}
console.log('Calling google api via post request');
try {
const options = {
host: 'www.googleapis.com',
path: '/admin/directory/v1/users',
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Content-Length': payload.length.toString()
},
form: payload
}
const responseFromGoogle = await sendHttpsRequest(options);
console.log('responseFromGoogle', JSON.stringify(responseFromGoogle));
}
catch (err) {
console.log('Calling google api failed with error', err);
resultResponse.statusCode = 503;
resultResponse.body = `Error creating new Google Account for ${userInfo.topmonksEmail}.`;
return resultResponse;
}
console.log('request to google sent');
return resultResponse;
}
The problem is that the http request does not seem to fire correctly. The callback is never executed.
I believe this part of the issue is related to some combination of (a) potentially not actually sending the https request and (b) not using the correct callback signature for https.request. See the documentation at https://nodejs.org/api/https.html#https_https_request_options_callback for details on both of these.
Use node-fetch package
The following example works for me using node-fetch:
import * as aws from "#pulumi/aws";
import fetch from "node-fetch";
const api = new aws.apigateway.x.API("api", {
routes: [{
method: "GET", path: "/", eventHandler: async (ev) => {
const resp = await fetch("https://www.google.com");
const body = await resp.text();
return {
statusCode: resp.status,
body: body,
}
},
}],
})
export const url = api.url;
Pulumi complains, it something like "Can not serialize native function" or something like that. The problematic part is that node-fetch relies on Symbol.iterator
As noted in the comments, some of the conditions that can lead to this are documented at https://pulumi.io/reference/serializing-functions.html. However, I don't see any clear reason why this code would hit any of those limitations. There may be details of how this is used outside the context of the snippet shared above which lead to this.

Proper request with async/await in Node.JS

In my program I make async call for my function from another API module:
var info = await api.MyRequest(value);
Module code:
var request = require("request")
module.exports.MyRequest = async function MyRequest(value) {
var options = {
uri: "http://some_url",
method: "GET",
qs: { // Query string like ?key=value&...
key : value
},
json: true
}
try {
var result = await request(options);
return result;
} catch (err) {
console.error(err);
}
}
Execution returns immediately, however result and therefore info contains request object and request body - info.body like key=value&..., not required response body.
What I'm doing wrong? How to fix? What is proper request usage with async, or it only works correctly with promises like mentioned here: Why await is not working for node request module? Following article mentioned it is possible: Mastering Async Await in Node.js.
You need to use the request-promise module, not the request module or http.request().
await works on functions that return a promise, not on functions that return the request object and expect you to use callbacks or event listeners to know when things are done.
The request-promise module supports the same features as the request module, but asynchronous functions in it return promises so you can use either .then() or await with them rather than the callbacks that the request module expects.
So, install the request-promise module and then change this:
var request = require("request");
to this:
const request = require("request-promise");
Then, you can do:
var result = await request(options);
EDIT Jan, 2020 - request() module in maintenance mode
FYI, the request module and its derivatives like request-promise are now in maintenance mode and will not be actively developed to add new features. You can read more about the reasoning here. There is a list of alternatives in this table with some discussion of each one.
I have been using got() myself and it's built from the beginning to use promises, supports many of the same options as the request() module and is simple to program.
Pretty sure you can also do the following. If what you need does not return Promise by default you can provide it via new Promise method. Above answer is less verbose though.
async function getBody(url) {
const options = {
url: url,
method: 'GET',
};
// Return new promise
return new Promise(function(resolve, reject) {
// Do async job
request.get(options, function(err, resp, body) {
if (err) {
reject(err);
} else {
resolve(body);
}
})
})
}
I just managed to get it to work with async/await. I wrapped it inside a function promisifiedRequest to return a promise that runs the request callback and resolves or rejects it based on error and response.
const request = require('request');
const promisifiedRequest = function(options) {
return new Promise((resolve,reject) => {
request(options, (error, response, body) => {
if (response) {
return resolve(response);
}
if (error) {
return reject(error);
}
});
});
};
(async function() {
const options = {
url: 'https://www.google.com',
method: 'GET',
gzip: true,
headers: {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.96 Safari/537.36'
}
};
let response = await promisifiedRequest(options);
console.log(response.headers);
console.log(response.body);
})();
Since request-promise has been deprecated, here are other options that don't depend on the NPM request package. got has been mentioned already, but it depends on 11 other packages. axios, in contrast, only has 1 dependency (for redirects). Everything else is natively implemented and built on top of the native NodeJS packages.
Here is the same example using axios:
const axios = require('axios')
const response = await axios.get(url)
const result = response.data
or, as a one-liner in JavaScript
const result = (await axios.get(url)).data
One-liner in TypeScript:
const {data} = await axios.get(url)
For simple cases where you don't need advanced features like cookies, following redirects and retrying, you can use native http/https module to make requests:
const https = require('https')
async function fetch(url) {
return new Promise((resolve, reject) => {
const request = https.get(url, { timeout: 1000 }, (res) => {
if (res.statusCode < 200 || res.statusCode > 299) {
return reject(new Error(`HTTP status code ${res.statusCode}`))
}
const body = []
res.on('data', (chunk) => body.push(chunk))
res.on('end', () => {
const resString = Buffer.concat(body).toString()
resolve(resString)
})
})
request.on('error', (err) => {
reject(err)
})
request.on('timeout', () => {
request.destroy()
reject(new Error('timed out'))
})
})
}
const res = await fetch('https://...')

How to make a external API call inside express server?

Hello I ve been trying to implement OneSignal API on my dashboard and I wonder if it is possible to make a API external call inside express server.
Here is an example:
var sendNotification = function(data) {
var headers = {
"Content-Type": "application/json; charset=utf-8",
"Authorization": "Basic NGEwMGZmMjItY2NkNy0xMWUzLTk5ZDUtMDAwYzI5NDBlNjJj"
};
var options = {
host: "onesignal.com",
port: 443,
path: "/api/v1/notifications",
method: "POST",
headers: headers
};
var https = require('https');
var req = https.request(options, function(res) {
res.on('data', function(data) {
console.log("Response:");
console.log(JSON.parse(data));
});
});
req.on('error', function(e) {
console.log("ERROR:");
console.log(e);
});
req.write(JSON.stringify(data));
req.end();
};
Here it is the app route
app.post('/path', function(req, res){
var message = {
app_id: "5eb5a37e-b458-11e3-ac11-000c2940e62c",
contents: {"en": "English Message"},
included_segments: ["All"]
};
sendNotification(message);
});
Thank you!
I wonder if it is possible to make a API external call inside express
server.
Sure, you can contact any external server from a node.js app with http.request() like you are showing or one of the higher level modules built on top of that like the request module.
Here's a simple example from the request module home page:
const request = require('request');
request('http://www.google.com', function (error, response, body) {
if (!error && response.statusCode == 200) {
console.log(body) // Show the HTML for the Google homepage.
}
});
Or, using promises:
const rp = require('request-promise');
rp('http://www.google.com').then(body => {
console.log(body);
}).catch(err => {
console.log(err);
});
EDIT Jan, 2020 - request() module in maintenance mode
FYI, the request module and its derivatives like request-promise are now in maintenance mode and will not be actively developed to add new features. You can read more about the reasoning here. There is a list of alternatives in this table with some discussion of each one. I have been using got() myself and it's built from the beginning to use promises and is simple to use.
you can use Axios client as Axios is a Promise based HTTP client for the browser as well as node.js.
Using Promises is a great advantage when dealing with code that requires a more complicated chain of events. Writing asynchronous code can get confusing, and Promises are one of several solutions to this problem.
First install Axios in your application using npm install axios --save
and then you can use this code
const axios = require('axios');
axios.get('api-url')
.then(response => {
console.log(response.data.status);
// console.log(response.data);
res.send(response.data.status);
})
.catch(error => {
console.log(error);
});
Kindly try out this solution. i used it and it worked for me.
var Request = require("request");
Request.get("http://httpbin.org/ip", (error, response, body) => {
if(error) {
return console.dir(error);
}
console.dir(JSON.parse(body));
});
You can use request-promise-native that uses native ES6 promises.
Install the request-promise-native package
npm install --save request
npm install --save request-promise-native
Use it as follows :
const request = require('request-promise-native');
const options = {
method: 'GET',
uri: 'https://www.google.com'
}
request(options).then(response => {
console.log(response);
}, error => {
console.log(error);
});
2022 update
Built in fetch api is available in NodeJS 18+. Third party npm modules are no longer needed.
(async () => {
const res = await fetch('https://dummyjson.com/products/1')
if (res.ok) {
const data = await res.json()
console.log(data)
}
})()
More details in the official blog

Resources