Processing a lot of requests without crashing - node.js

Question

I am trying to send a lot of https requests and processing that causes my code to crash. I know I can increase my memory but that won't scale. Uncommenting the code below causes OOM crash at some point. The solution should probably be to flush the buffer or something, but I am learning nodejs so not sure what to do.
var https = require('https');
// url anonymized for example
var urlArray = ["https://example.com/blah", ....] // 5000 urls here
var options = {
headers: { "x-api-key": "mykey" }
};
for (let dest of urlArray) {
https.request(dest, options, (res) => {
if (res.statusCode != 200) {
console.log(res.statusCode+" "+res.statusMessage+" at "+dest)
}
})
// uncommenting below causes a crash during runtime
// .on("error", (err) =>
// console.log(err.stack))
.end();
}

Answer 1

NodeJs being non-blocking, is not waiting for the ith http.request to finish before it moves to the i+1th. So the request keeps on accumulating in the memory, and as the memory is not big enough, it crashes. So what we can do here is execute the requests in batches and wait for that batch to finish before starting with the next batch. With this, at any instant, there will be at most n requests present in the memory (n is the batch size).
The code will look something like this:
const request = require('request-promise');
const urlArray = ["https://example.com/blah", ....];
async function batchProcess (urlArray){
const batchSize = 100;
// ^^^^^^^^^^
let i=0;
while(i<urlArray.length) {
let batch = [];
for(let j=0; j<batchSize && i<urlArray.length; j++, i++){
batch.push(request({
uri: urlArray[i],
headers: {
'User-Agent': 'Request-Promise',
"x-api-key": "mykey"
},
json: true // Automatically parses the JSON string in the response
}));
}
let batchResult = await Promise.all(batch);
console.log(batchResult);
}
}
batchProcess(urlArray);

Answer 2

One way would be to turn them into an async iterable where you can run them after another and process them as they return (Apologies for the TypeScript, just pass them through playground to transpile if you don't know TS):
import fetch from "node-fetch";
class MyParallelCalls implements AsyncIterable<any> {
constructor(private readonly urls: string[]) {}
[Symbol.asyncIterator](): AsyncIterator<any> {
return this.iterator();
}
async *iterator(): AsyncGenerator<any> {
for (const url of this.urls) {
yield (await fetch(url, {headers: { "x-api-key": "mykey" }})).json();
}
}
}
async function processAll() {
const calls = new MyParallelCalls(urls);
for await (const call of calls) {
// deal with them as the happen: e.g. pipe them into a process or a destination
console.log(call);
}
}
processAll();
If you want I can modify the above to batch your calls too. It's easy just add an option to the constructor for batch size and you can set how many calls you want to do in your batch and use promise.all for doing the yield.
It will look something like below (Refactored a little so its more generic):
import fetch from "node-fetch";
interface MyParallelCallsOptions {
urls: string[];
batchSize: number;
requestInit: RequestInit;
}
class MyParallelCalls<T> implements AsyncIterable<T[]> {
private batchSize = this.options.batchSize;
private currentIndex = 0;
constructor(private readonly options: MyParallelCallsOptions) {}
[Symbol.asyncIterator](): AsyncIterator<T[]> {
return this.iterator();
}
private getBatch(): string[] {
const batch = this.options.urls.slice(this.currentIndex, this.currentIndex + this.batchSize);
this.setNextBatch();
return batch;
}
private setNextBatch(): void {
this.currentIndex = this.currentIndex + this.batchSize;
}
private isLastBatch(): boolean {
return this.currentIndex === this.options.urls.length;
}
async *iterator(): AsyncGenerator<T[]> {
while (!this.isLastBatch()) {
const batch = this.getBatch();
const requests = batch.map(async (url) => (await fetch(url, this.options.requestInit)).json());
yield Promise.all(requests);
}
}
}
async function processAll() {
const batches = new MyParallelCalls<any>({
urls, // string array of your urls
batchSize: 5,
requestInit: { headers: { "x-api-key": "mykey" } }
});
for await (const batch of batches) {
console.log(batch);
}
}
processAll();