I have a fairly straightforward application which 1) accepts an array of urls 2) iterates over those urls 3) makes a get request to stream media (video / audio). Below is a code snippet illustrating what I'm doing
import request from 'request';
const tasks = urlsArray.map(async (url: string) => {
const startTime = process.hrtime();
let body = '';
let headers = {};
try {
const response = await promisify(request).call(request, {url, method: 'GET',
encoding: null, timeout: 10000});
} catch (e) {
logger.warn(failed to make request)
}
const [seconds] = process.hrtime(startTime);
logger.info(`Took ${seconds} seconds to make request`)
return body;
}
(await Promise.all(tasks)).forEach((body) => {
// processing body...
}
What I'm currently experiencing is that the time to make the request keeps rising as I make more requests and I'm struggling to understand why that is the case.
Related
I am trying to call an existing API Endpoint with a post request multiple times in nodejs. For that I am using a json file with multiple hundred entries, a loop and axios post request. The content of the file in itself changes but not the setup as it is based on a template.
I am now running into the issue that it has to be in a synchronous manner (thats why the await) as the used file has an inherent order which would have to be manually change in the application of the api endpoint. So an array of [Object 1, Object 2, Object 3] schould be created in that order even if it is not the fastest way possible.
So for my riskPostRequest function I am unable to set the title in data in a way that it will be pulled from the array of objects during my async importRisk function.
const jsonString = fs.readFileSync(filepath, 'utf-8')
const myReadStreamRisks = JSON.parse(jsonString)
console.log(myReadStreamRisks)
const riskPostRequest = risk => {
axios({
method: 'post',
port: 443,
url: baseURL + '/Risks',
data: {
parentFolderId: 11633,
title: myReadStreamRisks.title
}
})
.then ((response) => {
console.log(response.data.title,';', response.data.id)
})
.catch(function (error) {
console.log(error.message);
}, myReadStreamRisks)
}
// Import the TeamStoreRisks
const importRisks = async () => {
console.log('Start')
for (var i = 0; i < myReadStreamRisks.length; i++) {
const risk = myReadStreamRisks[i]
const riskPost = await riskPostRequest(risk);
}
}
importRisks()
Here is what I am trying to do.
I am introducing functionality to enable users to search for local restaurants.
I created a HTTP cloud function, so that when the client delivers a keyword, the function will call an external API to search for the keyword, fetch the responses, and deliver the results.
In doing #2, I need to make two separate url requests and merge the results.
When I checked, the function does call the API, fetch the results and merge them without any issue. However, for some reason, it only returns null to the client.
Below is the code: could someone take a look and advise me on where I went wrong?
exports.restaurantSearch = functions.https.onCall((data,context)=>{
const request = data.request;
const k = encodeURIComponent(request);
const url1 = "an_url_to_call_the_external_API"+k;
const url2 = "another_url_to_call_the_external_API"+k;
const url_array = [ url1, url2 ];
const result_array = [];
const info_array = [];
url_array.forEach(url=>{
return fetch(url, {headers: {"Authorization": "API_KEY"}})
.then(response=>{
return response.json()
})
.then(res=>{
result_array.push(res.documents);
if (result_array.length===2) {
const new_result_array_2 = [...new Set((result_array))];
new_result_array_2.forEach(nra=>{
info_array.push([nra.place_name,nra.address_name])
})
//info_array is not null at this point, but the below code only return null when checked from the client
return info_array;
}
})
.catch(error=>{
console.log(error)
return 'error';
})
})
});
Thanks a lot in advance!
You should use Promise.all() instead of running each promise (fetch request) separately in a forEach loop. Also I don't see the function returning anything if result_array.length is not 2. I can see there are only 2 requests that you are making but it's good to handle all possible cases so try adding a return statement if the condition is not satisfied. Try refactoring your code to this (I've used an async function):
exports.restaurantSearch = functions.https.onCall(async (data, context) => {
// Do note the async ^^^^^
const request = data.request;
const k = encodeURIComponent(request);
const url1 = "an_url_to_call_the_external_API" + k;
const url2 = "another_url_to_call_the_external_API" + k;
const url_array = [url1, url2];
const responses = await Promise.all(url_array.map((url) => fetch(url, { headers: { "Authorization": "API_KEY" } })))
const responses_array = await Promise.all(responses.map((response) => response.json()))
console.log(responses_array)
const result_array: any[] = responses_array.map((res) => res.documents)
// Although this if statement is redundant if you will be running exactly 2 promises
if (result_array.length === 2) {
const new_result_array_2 = [...new Set((result_array))];
const info_array = new_result_array_2.map(({place_name, address_name}) => ({place_name, address_name}))
return {data: info_array}
}
return {error: "Array length incorrect"}
});
If you'll be running 2 promises only, other option would be:
// Directly adding promises in Promise.all() instead of using map
const [res1, res2] = await Promise.all([fetch("url1"), fetch("url2")])
const [data1, data2] = await Promise.all([res1.json(), res2.json()])
Also check Fetch multiple links inside of forEach loop
I am trying to send a lot of https requests and processing that causes my code to crash. I know I can increase my memory but that won't scale. Uncommenting the code below causes OOM crash at some point. The solution should probably be to flush the buffer or something, but I am learning nodejs so not sure what to do.
var https = require('https');
// url anonymized for example
var urlArray = ["https://example.com/blah", ....] // 5000 urls here
var options = {
headers: { "x-api-key": "mykey" }
};
for (let dest of urlArray) {
https.request(dest, options, (res) => {
if (res.statusCode != 200) {
console.log(res.statusCode+" "+res.statusMessage+" at "+dest)
}
})
// uncommenting below causes a crash during runtime
// .on("error", (err) =>
// console.log(err.stack))
.end();
}
NodeJs being non-blocking, is not waiting for the ith http.request to finish before it moves to the i+1th. So the request keeps on accumulating in the memory, and as the memory is not big enough, it crashes. So what we can do here is execute the requests in batches and wait for that batch to finish before starting with the next batch. With this, at any instant, there will be at most n requests present in the memory (n is the batch size).
The code will look something like this:
const request = require('request-promise');
const urlArray = ["https://example.com/blah", ....];
async function batchProcess (urlArray){
const batchSize = 100;
// ^^^^^^^^^^
let i=0;
while(i<urlArray.length) {
let batch = [];
for(let j=0; j<batchSize && i<urlArray.length; j++, i++){
batch.push(request({
uri: urlArray[i],
headers: {
'User-Agent': 'Request-Promise',
"x-api-key": "mykey"
},
json: true // Automatically parses the JSON string in the response
}));
}
let batchResult = await Promise.all(batch);
console.log(batchResult);
}
}
batchProcess(urlArray);
One way would be to turn them into an async iterable where you can run them after another and process them as they return (Apologies for the TypeScript, just pass them through playground to transpile if you don't know TS):
import fetch from "node-fetch";
class MyParallelCalls implements AsyncIterable<any> {
constructor(private readonly urls: string[]) {}
[Symbol.asyncIterator](): AsyncIterator<any> {
return this.iterator();
}
async *iterator(): AsyncGenerator<any> {
for (const url of this.urls) {
yield (await fetch(url, {headers: { "x-api-key": "mykey" }})).json();
}
}
}
async function processAll() {
const calls = new MyParallelCalls(urls);
for await (const call of calls) {
// deal with them as the happen: e.g. pipe them into a process or a destination
console.log(call);
}
}
processAll();
If you want I can modify the above to batch your calls too. It's easy just add an option to the constructor for batch size and you can set how many calls you want to do in your batch and use promise.all for doing the yield.
It will look something like below (Refactored a little so its more generic):
import fetch from "node-fetch";
interface MyParallelCallsOptions {
urls: string[];
batchSize: number;
requestInit: RequestInit;
}
class MyParallelCalls<T> implements AsyncIterable<T[]> {
private batchSize = this.options.batchSize;
private currentIndex = 0;
constructor(private readonly options: MyParallelCallsOptions) {}
[Symbol.asyncIterator](): AsyncIterator<T[]> {
return this.iterator();
}
private getBatch(): string[] {
const batch = this.options.urls.slice(this.currentIndex, this.currentIndex + this.batchSize);
this.setNextBatch();
return batch;
}
private setNextBatch(): void {
this.currentIndex = this.currentIndex + this.batchSize;
}
private isLastBatch(): boolean {
return this.currentIndex === this.options.urls.length;
}
async *iterator(): AsyncGenerator<T[]> {
while (!this.isLastBatch()) {
const batch = this.getBatch();
const requests = batch.map(async (url) => (await fetch(url, this.options.requestInit)).json());
yield Promise.all(requests);
}
}
}
async function processAll() {
const batches = new MyParallelCalls<any>({
urls, // string array of your urls
batchSize: 5,
requestInit: { headers: { "x-api-key": "mykey" } }
});
for await (const batch of batches) {
console.log(batch);
}
}
processAll();
I'm working with some API server that communicates by XML.
I need to send, let's say: 20 identical POST requests.
I'm writing this in Node JS.
Easy.
BUT - since I'm going to multiply the process, and I want to avoid flooding the server (and getting kicked), I need to break the sending loop IF the (XML) response contains a specific text (a success signal): <code>555</code>, or actually just '555' (the text is wrapped with other XML phrases).
I tried to break the loop based on the success signal AND also tried "exporting" it outside the loop (Thinking it could be nice to address it in the loop's condition).
Guess it's easy but being a newbie, I had to call for some help :)
Attaching the relevant code (simplified).
Many thanks !
const fetch = require("node-fetch");
const url = "https://www.apitest12345.com/API/";
const headers = {
"LOGIN": "abcd",
"PASSWD": "12345"
}
const data = '<xml></xml>'
let i = 0;
do { // the loop
fetch(url, { method: 'POST', headers: headers, body: data})
.then((res) => {
return res.text()
})
.then((text) => {
console.log(text);
if(text.indexOf('555') > 0) { // if the response includes '555' it means SUCCESS, and we can stop the loop
~STOP!~ //help me stop the loop :)
}
});
i += 1;
} while (i < 20);
Use simple for loop with async await.
const fetch = require("node-fetch");
const url = "https://www.apitest12345.com/API/";
const headers = {
"LOGIN": "abcd",
"PASSWD": "12345"
}
const data = '<xml></xml>'
for (let i = 0; i < 20; i++) {
const res = await fetch(url, { method: 'POST', headers: headers, body: data});
if (res.text().indexOf('555') !== -1)
break;
}
I have an API that searches for the user-provided term, returns an array of results, then fires off async requests for each of the results and gets results for each of these second batch of requests. I'd like the API to report progress as it happens rather than just the final result. So, if I do the following request, I should get updates like so
$ curl 'http://server/?q=foobar'
searching for ${q}…
found 76… now getting images…
found 30 images… done
{
result
}
Most of relevant code is shown below. Fwiw, I am using hapijs for my application.
let imagesOfRecords = {};
const getImages = async function (q) {
console.log(`searching for ${q}…`);
const uri = `http://remoteserver/?q=${q}`;
const {res, payload} = await Wreck.get(uri);
const result = JSON.parse(payload.toString()).hits;
const numOfFoundRecords = result.total;
if (result.total) {
console.log(`found ${result.total}… now getting images…`);
const foundRecords = result.hits.map(getBuckets);
Promise.all(foundRecords).then(function() {
console.log(`found ${Object.keys(imagesOfRecords).length} images… done`);
reply(imagesOfRecords).headers = res.headers;
}).catch(error => {
console.log(error)
});
}
else {
console.log('nothing found');
reply(0).headers = res.headers;
}
};
const getBuckets = async function(record) {
const { res, payload } = await Wreck.get(record.links.self);
const bucket = JSON.parse(payload.toString()).links.bucket;
await getImageFiles(bucket, record.links.self);
};
const getImageFiles = async function(uri, record) {
const { res, payload } = await Wreck.get(uri);
const contents = JSON.parse(payload.toString()).contents;
imagesOfRecords[record] = contents.map(function(el) {
return el.links.self;
});
};
Once I can implement this, my next task would be to implement this progressive update in a web app that uses the above API.
To show result with each step of your requests for backend you can use EventEmitter, which will emit event on each progress step. You can read about events here.
Simple implementation:
const events = require('events');
const eventEmitter = new events.EventEmitter();
//your request code
Promise.all(foundRecords).then(function() {
console.log(`found ${Object.keys(imagesOfRecords).length} images… done`);
eventEmitter.emit('progress');
reply(imagesOfRecords).headers = res.headers;
})
const eventReaction = (e) => {
// do something with event, console log for example.
}
eventEmitter.on('progress', eventReaction);
More examples you can find here and here.
To show events to client you can use library socket.io. I think you can find pretty straightforward explanations how socket.io works in documentation.
If you want to send events between servers or processes and want to go little further, you can read more about 0MQ (zero mq) and it's node implementation