Multiple api calls in loop fails - node.js

Question

Having some issues when calling an external api to fetch information inside a loop in my node/express backend. I need information I get from the loop to get the correct data back from the endpoint. When I loop through the data and make the call I get this error.
{ message: 'Too Many Requests Limit 30. Reset time 1594218437315',
status: 429 }
I get the correct data back sometimes and sometimes only this message. There will be about 10 000 or so calls I need to make. I've tried a multiple of throttling libs and a lot of the code here on SO but it's pretty much always the same result which is the error message or it doesnt work at all.
I think I need a way send about 20-30 requests at a time and then wait a second or so and then continue. Or is there another better way? How would I achieve this?
const product = await NewProduct.find({});
product.map((item, index) => {
item.stores.map(async inner => {
inner.trackingUrl = await fetchRetry(inner.programId, inner.productUrl)
})
})
async function fetchRetry(id, urlToTrack) {
url1 = 'https://api.adtraction.com/v2/affiliate/tracking/link/?token=token';
const data = {
channelId,
programId: id,
shortLink: true,
url: urlToTrack
};
const options = {
method: 'POST',
headers: {
'Content-type': 'application/json',
Accept: 'application/json',
'Accept-Charset': 'utf-8',
},
body: JSON.stringify(data),
};
const res = await fetch(url, {
method: 'POST',
headers: {
'Content-type': 'application/json',
Accept: 'application/json',
'Accept-Charset': 'utf-8',
},
body: JSON.stringify(data),
});
const json = await res.json();
console.log(json) // Error
return json.trackingUrl;
}

Answer 1

Well, I'll share with you something that I've done in one of my projects. For all of your data that you call with your api. Keep making requests and when you encounter any error or response with throttling then wait for 5 minutes (or required in your api) and move to previous index (previous data to call the api with)
async function callAPI(data) {
for (let index = 0; index < data.length; index++) {
try {
// api call with your data
yourApiCall(data[index]);
} catch (e) {
if (e.type == "RequestThrottled") {
let oneMinute = 60000;
// wait for the time your request will be availabl
await sleep(5 * oneMinute);
// because of error in this request get back to
//previous request (or data)
index--;
}
}
}
}
Function using setTimeout to wait synchronously.
async function sleep(ms) {
await new Promise((resolve) => setTimeout(resolve, ms));
}
Also, you may want to run it on some background process as it might block your main event loop.