I have an array which contains some data and for each data item an api request has to be made. The api will remain same but the array index will increment every time the api request is made.
Also the api request has to be called with a gap of 5 minutes. Hence I can't call the api for the entire array all at once. One api call is made with Array[0] in the request body and after 5 minutes api call is made with Array[1] in the request body.
I tried to implement a cron job with these requirements but there are no proper examples for a cron job within a for loop with api calls.
Any help would be appreciated.
`
const array = ['http://linkedin.com/charles123', 'http://linkedin.com/darwin123' ... ]
//API needs to be called every 5 minutes
const sendConnectionRequest = () => {
var i = 0;
for(i; i< array.length, i++) {
fetch("serverurl:123", {
headers: {
'Content-Type': 'application/json'
},
method: "POST",
body: JSON.stringify(array[i])
})
.then((res) => if(res) { console.log('Connection Request Send') } )
}
}`
May I suggest using an Async Generator this will allow you to manage sequential promises.
const fetch = require("node-fetch");
const sleep = require("util").promisify(setTimeout);
async function* responseGenerator(urls) {
let iterations = 0;
while (urls.length) {
const [url, ...rest] = urls;
urls = rest;
if (iterations > 0) {
await sleep(50000);
}
yield fetch("serverurl:123", {
headers: {
"Content-Type": "application/json"
},
method: "POST",
body: JSON.stringify(url)
});
iterations += 1
}
}
const array = ['http://linkedin.com/charles123', 'http://linkedin.com/darwin123' ]
for await (const response of responseGenerator(array)) {
// response.status
// response.statusText
// response.contentType
}
There are multiple way of doing timers in node.js. Check this link.
setInterval is a infinite loop and between each iteration it wait a certain amount of time.
const array = ['http://linkedin.com/charles123', 'http://linkedin.com/darwin123' ... ]
//API needs to be called every 5 minutes
const sendConnectionRequest = (data) => {
fetch("serverurl:123", {
headers: {
'Content-Type': 'application/json'
},
method: "POST",
body: JSON.stringify(data)
})
.then((res) => if(res) { console.log('Connection Request Send') } )
}
const callApi = setInterval(()=> {
sendConnectionRequest(array[0])
array.shift()
}, 30000);
Related
I am trying to send a lot of https requests and processing that causes my code to crash. I know I can increase my memory but that won't scale. Uncommenting the code below causes OOM crash at some point. The solution should probably be to flush the buffer or something, but I am learning nodejs so not sure what to do.
var https = require('https');
// url anonymized for example
var urlArray = ["https://example.com/blah", ....] // 5000 urls here
var options = {
headers: { "x-api-key": "mykey" }
};
for (let dest of urlArray) {
https.request(dest, options, (res) => {
if (res.statusCode != 200) {
console.log(res.statusCode+" "+res.statusMessage+" at "+dest)
}
})
// uncommenting below causes a crash during runtime
// .on("error", (err) =>
// console.log(err.stack))
.end();
}
NodeJs being non-blocking, is not waiting for the ith http.request to finish before it moves to the i+1th. So the request keeps on accumulating in the memory, and as the memory is not big enough, it crashes. So what we can do here is execute the requests in batches and wait for that batch to finish before starting with the next batch. With this, at any instant, there will be at most n requests present in the memory (n is the batch size).
The code will look something like this:
const request = require('request-promise');
const urlArray = ["https://example.com/blah", ....];
async function batchProcess (urlArray){
const batchSize = 100;
// ^^^^^^^^^^
let i=0;
while(i<urlArray.length) {
let batch = [];
for(let j=0; j<batchSize && i<urlArray.length; j++, i++){
batch.push(request({
uri: urlArray[i],
headers: {
'User-Agent': 'Request-Promise',
"x-api-key": "mykey"
},
json: true // Automatically parses the JSON string in the response
}));
}
let batchResult = await Promise.all(batch);
console.log(batchResult);
}
}
batchProcess(urlArray);
One way would be to turn them into an async iterable where you can run them after another and process them as they return (Apologies for the TypeScript, just pass them through playground to transpile if you don't know TS):
import fetch from "node-fetch";
class MyParallelCalls implements AsyncIterable<any> {
constructor(private readonly urls: string[]) {}
[Symbol.asyncIterator](): AsyncIterator<any> {
return this.iterator();
}
async *iterator(): AsyncGenerator<any> {
for (const url of this.urls) {
yield (await fetch(url, {headers: { "x-api-key": "mykey" }})).json();
}
}
}
async function processAll() {
const calls = new MyParallelCalls(urls);
for await (const call of calls) {
// deal with them as the happen: e.g. pipe them into a process or a destination
console.log(call);
}
}
processAll();
If you want I can modify the above to batch your calls too. It's easy just add an option to the constructor for batch size and you can set how many calls you want to do in your batch and use promise.all for doing the yield.
It will look something like below (Refactored a little so its more generic):
import fetch from "node-fetch";
interface MyParallelCallsOptions {
urls: string[];
batchSize: number;
requestInit: RequestInit;
}
class MyParallelCalls<T> implements AsyncIterable<T[]> {
private batchSize = this.options.batchSize;
private currentIndex = 0;
constructor(private readonly options: MyParallelCallsOptions) {}
[Symbol.asyncIterator](): AsyncIterator<T[]> {
return this.iterator();
}
private getBatch(): string[] {
const batch = this.options.urls.slice(this.currentIndex, this.currentIndex + this.batchSize);
this.setNextBatch();
return batch;
}
private setNextBatch(): void {
this.currentIndex = this.currentIndex + this.batchSize;
}
private isLastBatch(): boolean {
return this.currentIndex === this.options.urls.length;
}
async *iterator(): AsyncGenerator<T[]> {
while (!this.isLastBatch()) {
const batch = this.getBatch();
const requests = batch.map(async (url) => (await fetch(url, this.options.requestInit)).json());
yield Promise.all(requests);
}
}
}
async function processAll() {
const batches = new MyParallelCalls<any>({
urls, // string array of your urls
batchSize: 5,
requestInit: { headers: { "x-api-key": "mykey" } }
});
for await (const batch of batches) {
console.log(batch);
}
}
processAll();
I'm working with some API server that communicates by XML.
I need to send, let's say: 20 identical POST requests.
I'm writing this in Node JS.
Easy.
BUT - since I'm going to multiply the process, and I want to avoid flooding the server (and getting kicked), I need to break the sending loop IF the (XML) response contains a specific text (a success signal): <code>555</code>, or actually just '555' (the text is wrapped with other XML phrases).
I tried to break the loop based on the success signal AND also tried "exporting" it outside the loop (Thinking it could be nice to address it in the loop's condition).
Guess it's easy but being a newbie, I had to call for some help :)
Attaching the relevant code (simplified).
Many thanks !
const fetch = require("node-fetch");
const url = "https://www.apitest12345.com/API/";
const headers = {
"LOGIN": "abcd",
"PASSWD": "12345"
}
const data = '<xml></xml>'
let i = 0;
do { // the loop
fetch(url, { method: 'POST', headers: headers, body: data})
.then((res) => {
return res.text()
})
.then((text) => {
console.log(text);
if(text.indexOf('555') > 0) { // if the response includes '555' it means SUCCESS, and we can stop the loop
~STOP!~ //help me stop the loop :)
}
});
i += 1;
} while (i < 20);
Use simple for loop with async await.
const fetch = require("node-fetch");
const url = "https://www.apitest12345.com/API/";
const headers = {
"LOGIN": "abcd",
"PASSWD": "12345"
}
const data = '<xml></xml>'
for (let i = 0; i < 20; i++) {
const res = await fetch(url, { method: 'POST', headers: headers, body: data});
if (res.text().indexOf('555') !== -1)
break;
}
var request = require('request');
var options = {
'method': 'GET',
'url': 'https://api.github.com/orgs/organizationName/repos?per_page=100&page=1',//To get all the users data from the repos
'url': 'https://api.github.com/orgs/organizationName/repos?per_page=100&page=2',
'url': 'https://api.github.com/orgs/organizationName/repos?per_page=100&page=3',
'url': 'https://api.github.com/orgs/organizationName/repos?per_page=100&page=4',
'url': 'https://api.github.com/orgs/organizationName/repos?per_page=100&page=5',
'url': 'https://api.github.com/orgs/organizationName/repos?per_page=100&page=6',
'url': 'https://api.github.com/orgs/organizationName/repos?per_page=100&page=7',
'url': 'https://api.github.com/orgs/organizationName/repos?per_page=100&page=8',
'url': 'https://api.github.com/orgs/organizationName/repos?per_page=100&page=9',
'url': 'https://api.github.com/orgs/organizationName/repos?per_page=100&page=10',
'url': 'https://api.github.com/orgs/organizationName/repos?per_page=100&page=11',
'headers': {
'Accept': 'application/vnd.github.mercy-preview+json',//to get topics of the repos
'Authorization': 'Bxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
'User-Agent' : 'sxxxxxxxxxxxxx'
}
};
request(options, function (error, response) {
if (error) throw new Error(error);
console.log(response.body);
});
In this above code i want to loop the urls continuously until the end of the page
if not anyone have the idea of using pagination in this help me out
cYou cannot have multiple attributes for one object key. You have to call every url individually. I tried to solve this using asyncronous code, because looping with callback functions is confusing and dangerous with regard to the call stack.
const request = require('request');
// configuration for the url generation
const perPages = 100;
const startPage = 1;
const endPage = 11;
const url = 'https://api.github.com/orgs/organizationName/repos?per_page=%perPages%&page=%page%';
// define a asyncronous call for one url
async function callOneUrl(url) {
// local options for each url
const options = {
method: 'GET',
url: url,
headers: {
Accept: 'application/vnd.github.mercy-preview+json',
Authorization: 'Bxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
'User-Agent': 'sxxxxxxxxxxxxx'
}
}
return new Promise((resolve, reject) => {
request(options, function (error, response) {
if (error) return reject(error);
resolve(response);
});
});
}
// call each url with a for loop
(async () => {
for (let i = startPage; i <= endPage; i++) {
// using the await statement to get the resolved value of the Promise instance or catch the error
try {
var response = await callOneUrl(url.replace('%perPages%', perPages).replace('%page%', i));
// handle response here
console.log(response.body);
} catch (error) {
// handle errors here
throw new Error(error);
}
}
})()
const request = require('request-promise');
const urls = ["http://www.google.com", "http://www.example.com"];
const promises = urls.map(url => request(url));
Promise.all(promises).then((data) => {
// data = [promise1,promise2]
});
Apart from above you can also use async.eachseries or async.parallel etc..
You can download a list of repos with a do...while loop. We'll set a maximum number of pages to download and exit when we reach either this or the last page.
I would suggest using the request-promise-native package to allow us to use the very nice async-await syntax.
Now, I've given the example of downloading repos for the mongodb org. You can easily replace with whatever one you wish.
I would also note that the request library is now deprecated, we can use it of course, but we must consider replacing in the future.
We now also log the repo information and save it to the output file.
const rp = require("request-promise-native");
const fs = require("fs");
async function downloadRepoInformation(org, outputFile) {
let repoList = [];
let page = 0;
const resultsPerPage = 20;
const maxPages = 10;
const uri = `https://api.github.com/orgs/${org}/repos`;
do {
try {
let response = await rp.get({ uri, json: true, qs: { per_page: resultsPerPage, page: ++page }, headers: {"User-Agent" : "request"} });
console.log(`downloadRepoInformation: Downloaded page: ${page}, repos: ${response.length}...`);
repoList = repoList.concat(response);
console.log("downloadRepoInformation: response", JSON.stringify(response, null, 2));
console.log("downloadRepoInformation: repoList.length:", repoList.length);
if (response.length < resultsPerPage) {
console.log(`downloadRepoInformation: Last page reached: exiting loop...`);
break;
}
} catch (error) {
console.error(`downloadRepoInformation: An error occurred:`, error);
break;
}
} while (page <= maxPages)
console.log("downloadRepoInformation: download complete: repoList.length:", repoList.length)
console.log("downloadRepoInformation: Saving to file:", outputFile);
fs.writeFileSync(outputFile, JSON.stringify(repoList, null, 4));
}
downloadRepoInformation("mongodb", "./repolist.json");
I am invoking a Lambda function that retrieves data from ServiceNow via an API call within the Lambda. I have tested the code using call flows withing Amazon Connect, but when trying to utilise the Lambda Test Functionality, it succeeds, but the response returned is null and expect at least a name to be returned.
The input from Amazon Connect to the Lambda Function is a telephone number and I have tried adding this to the parameters section and the customerEndpointAddress section.
const https = require('https');
//Get Phone Details of Customer via Typed in Phone Number or Actual Phone Number
const getPhone = contact => {
const phone = contact.Details.ContactData.CustomerEndpoint.Address;
console.log(`Customer Phone is : ${phone}`);
return phone.length === 13 ? `0${phone.substring(3)}` : phone;
}
//Set API config, passing in the Phone Parameter as query and return both firstname and SysId
const getPhoneRequestOptions = phone => {
const path = `/api/now/table/sys_user?sysparm_query=phone%3D${phone}^ORmobile_phone%3D${phone}&sysparm_fields=first_name,sys_id`;
return {
host: process.env.SERVICENOW_HOST,
port: '443',
path: path,
method: 'get',
headers: {
"Content-Type": 'application/json',
Accept: 'application/json',
Authorization: 'Basic ' + Buffer.from(`${process.env.SERVICENOW_USERNAME}:${process.env.SERVICENOW_PASSWORD}`).toString('base64'),
}
};
};
//Retrieve data, in this case firstname and SysId
const requestUser = (phone, callback) => {
let get_request = https.request(getPhoneRequestOptions(phone), res => {
let body = '';
res.on('data', chunk => {body += chunk});
res.on('end', () => {callback(JSON.parse(body))});
res.on('error', e => {callback(e.message)});
})
get_request.end();
}
//Return data
exports.handler = (contact, context, callback) => {
if (!contact.Details || !contact.Details.Parameters) return;
requestUser(getPhone(contact), response => {
if (response.result && response.result[0] && response.result[0].first_name) {
callback(null, {
"first_name": response.result[0].first_name
});
} else {
callback(null, {
"Error": "No user found"
});
}
});
};
and the test code I have used is:
{
"Details": {
"ContactData" :{
"CustomerEndPoint" : {
"Address" : "01234567890"
}
}
}
}
When the code has been Invoked, the name "Abel" is returned within Amazon Connect, but its' not the case when running the test case against it.
It's because of this line:
if (!contact.Details || !contact.Details.Parameters) return;
In the test event you're using Details does not have the property Parameters (only ContactData). Which causes you to return without calling back with a value.
This is a function in Node.js, which reads data from Analytics:
function getDataFromGA(Dimension, Metric, StartDate, EndDate, MaxResults) {
var fs = require('fs'),
crypto = require('crypto'),
request = require('request'); // This is an external module
var authHeader = {
'alg': 'RS256',
'typ': 'JWT'
},
authClaimSet = {
'iss': '***t#developer.gserviceaccount.com', // Service account email
'scope': 'https://www.googleapis.com/auth/analytics.readonly',
// We MUST tell them we just want to read data
'aud': 'https://accounts.google.com/o/oauth2/token'
},
SIGNATURE_ALGORITHM = '**',
SIGNATURE_ENCODE_METHOD = '**',
GA_KEY_PATH = '**',
//finds current directory then appends private key to the directory
gaKey;
function urlEscape(source) {
return source.replace(/\+/g, '-').replace(/\//g, '_').replace(/\=+$/, '');
}
function base64Encode(obj) {
var encoded = new Buffer(JSON.stringify(obj), 'utf8').toString('base64');
return urlEscape(encoded);
}
function readPrivateKey() {
if (!gaKey) {
gaKey = fs.readFileSync(GA_KEY_PATH, 'utf8');
}
return gaKey;
}
var authorize = function (callback) {
var self = this,
now = parseInt(Date.now() / 1000, 10), // Google wants us to use seconds
cipher,
signatureInput,
signatureKey = readPrivateKey(),
signature,
jwt;
// Setup time values
authClaimSet.iat = now;
authClaimSet.exp = now + 60; // Token valid for one minute
// Setup JWT source
signatureInput = base64Encode(authHeader) + '.' + base64Encode(authClaimSet);
// Generate JWT
cipher = crypto.createSign('RSA-SHA256');
cipher.update(signatureInput);
signature = cipher.sign(signatureKey, 'base64');
jwt = signatureInput + '.' + urlEscape(signature);
// Send request to authorize this application
request({
method: 'POST',
headers: {
'Content-Type': 'application/x-www-form-urlencoded'
},
uri: 'https://accounts.google.com/o/oauth2/token',
body: 'grant_type=' + escape('urn:ietf:params:oauth:grant-type:jwt-bearer') +
'&assertion=' + jwt
}, function (error, response, body) {
if (error) {
console.log(error);
callback(new Error(error));
} else {
var gaResult = JSON.parse(body);
if (gaResult.error) {
callback(new Error(gaResult.error));
} else {
callback(null, gaResult.access_token);
// console.log(gaResult);
console.log("Authorized");
}
}
});
};
var request = require('request'),
qs = require('querystring');
authorize(function (err, token) {
if (!err) {
// Query the number of total visits for a month
var requestConfig = {
'ids': 'ga:72333024',
'dimensions': Dimension,
'metrics': Metric,
// 'sort': '-ga:users',
'start-date': StartDate,
'end-date': EndDate,
'max-results': MaxResults
};
request({
method: 'GET',
headers: {
'Authorization': 'Bearer ' + token // Here is where we use the auth token
},
uri: 'https://www.googleapis.com/analytics/v3/data/ga?' + qs.stringify(requestConfig)
}, function (error, resp, body) {
console.log(body);
var data = JSON.parse(body);
console.log(data.totalsForAllResults);
console.log(data.rows);
});
}
});
}
Here I try to access it from outside:
var gaJSON = utils.getDataFromGA("ga:country", "ga:pageviews", "2011-08-04", "2014-09-12", "50");
res.send(gaJSON);
My question is how I can access the variable data in the end of the first method? How can I call it from outside of the function?
You can assign data to a variable declared in the first function. But since the authorize method is asynchronous the variable data will still be undefined at the end of the first function. The best way to do this is handle with callbacks.
I think you wanna return something related to this variable, right? Try to put a callback parameter to the first function and then call this function passing the result.
callback(variable)
Why do you want to access if from outside. ??
Even though you want to desperately then you need to create a function pass the "data" as argument and then invoke the function .
console.log(body);
var data = JSON.parse(body);
myNewFunction(data);
Write all ur logic inside "myNewFunction" that uses data .