So I have an axios request to a rapid API, my function looks like this...
//Initialize the lookup API that utalizes rapidAPI to get breach data
app.get("/lookup/:email/:function", (req, res) => {
var options = {
method: "GET",
url: "https://breachdirectory.p.rapidapi.com/",
params: { func: `${req.params.function}`, term: `${req.params.email}` },
headers: {
"x-rapidapi-host": "breachdirectory.p.rapidapi.com",
"x-rapidapi-key": `${config.RAPID_API_KEY}`,
},
};
axios
.request(options)
.then(function (response) {
res.json(response.data);
})
.catch(function (error) {
console.error(error);
});
}
});
The res.json(response.data); will show on the page a result like this:
{
"disclaimer": "This data is aggregated from BreachDirectory, HaveIBeenPwned, and Vigilante.pw.",
"info": "For full source info, request e.g. https://breachdirectory.tk/api/source?name=Animoto",
"sources": [
"123RF",
"500px",
"Adobe",
"AntiPublic",
"Apollo",
"Bitly",
"Dave",
"Disqus",
"Dropbox",
"ExploitIn",
"ShareThis",
"Straffic",
"Ticketfly",
"Tumblr",
"VerificationsIO"
]
}
I want to loop through everything in the "sources" array, and call upon the following:
https://haveibeenpwned.com/api/v3/breach/[ITEM]
So, the first one will call upon https://haveibeenpwned.com/api/v3/breach/123RF
So each result from that call will look like this:
{
"Name": "123RF",
"Title": "123RF",
"Domain": "123rf.com",
"BreachDate": "2020-03-22",
"AddedDate": "2020-11-15T00:59:50Z",
"ModifiedDate": "2020-11-15T01:07:10Z",
"PwnCount": 8661578,
"Description": "In March 2020, the stock photo site 123RF suffered a data breach which impacted over 8 million subscribers and was subsequently sold online. The breach included email, IP and physical addresses, names, phone numbers and passwords stored as MD5 hashes. The data was provided to HIBP by dehashed.com.",
"LogoPath": "https://haveibeenpwned.com/Content/Images/PwnedLogos/123RF.png",
"DataClasses": [
"Email addresses",
"IP addresses",
"Names",
"Passwords",
"Phone numbers",
"Physical addresses",
"Usernames"
],
"IsVerified": true,
"IsFabricated": false,
"IsSensitive": false,
"IsRetired": false,
"IsSpamList": false
}
I want to make my res.json send over a JSON string that will have all the sources still there, along with the "Title","Description", and "LogoPath" from the API calls that it pulled for each one of the sources. So I will have a JSON string with the sources along with the title of each source, description of each source, and LogoPath of each source.
You have two options:
Create an array of promises and run with Promise.all
app.get('/lookup/:email/:function', async (req, res) => {
var options = {
method: 'GET',
url: 'https://breachdirectory.p.rapidapi.com/',
params: { func: `${req.params.function}`, term: `${req.params.email}` },
headers: {
'x-rapidapi-host': 'breachdirectory.p.rapidapi.com',
'x-rapidapi-key': `${config.RAPID_API_KEY}`,
},
};
axios.request(options)
.then((response) => {
const requestTasks = [];
for (let item of response.data.sources) {
const itemOption = {
method: 'GET',
url: `https://haveibeenpwned.com/api/v3/breach/${item}`,
headers: {
'content-type': 'application/json; charset=utf-8'
}
};
requestTasks.push(axios.request(itemOption));
}
return Promise.all(requestTasks);
})
.then((responseList) => {
for (let response of responseList) {
console.log(response.data);
}
})
.catch((error) => {
console.error(error);
});
});
Use async/await (promise) and for await for get data from for loop
app.get('/lookup/:email/:function', async (req, res) => {
try {
var options = {
method: 'GET',
url: 'https://breachdirectory.p.rapidapi.com/',
params: { func: `${req.params.function}`, term: `${req.params.email}` },
headers: {
'x-rapidapi-host': 'breachdirectory.p.rapidapi.com',
'x-rapidapi-key': `${config.RAPID_API_KEY}`,
},
};
const response = await axios.request(options);
for await (let item of response.data.sources) {
const itemOption = {
method: 'GET',
url: `https://haveibeenpwned.com/api/v3/breach/${item}`,
headers: {
'content-type': 'application/json; charset=utf-8'
}
};
const itemResponse = await axios.request(itemOption);
console.log(itemResponse.data);
}
} catch (error) {
console.error(error);
}
});
This how I managed to make it works.
first: I didn't had any APi key (and didn't want to register to get one) So i used a dummy Api.
although the logic stay the same as i have tested the result.
second i kept all your initial url just next to the one i used.so you can easily switch back to your original url.
finally i put comment to any critical part, and i named variable in a
way that they almost describe what they do.
so you can copy past test it to understand my logic then adapt it to your use case.
here the code
// make sure to replace /lookup by /lookup/:email/:function after testing my logic
app.get('/lookup', async (req, res) => {
try {
// in this options no change just switch back to your url
var options = {
method: 'GET',
url: 'https://jsonplaceholder.typicode.com/albums',
// url: "https://breachdirectory.p.rapidapi.com/",
// params: { func: `${req.params.function}`, term: `${req.params.email}` },
// headers: {
// 'x-rapidapi-host': 'breachdirectory.p.rapidapi.com',
// 'x-rapidapi-key': `${config.RAPID_API_KEY}`,
// },
};
// here you get all your sources list (in my case it an array of object check picture 1 bellow)
const allSources = await axios.request(options)
console.log(allSources.data);
// because my dummy api response is a huge array i slice to limited number
const reduceAllsource = allSources.data.slice(0,5);
console.log(reduceAllsource);
// note here you need to replace reduceAllsource.map by allSources.data.map
// because you don't need a sliced array
const allSourcesWithDetails = reduceAllsource.map(async (_1sourceEachtime)=>{
// here you can switch back to your original url
// make sure to replace [ITEM] by ${_1sourceEachtime}
const itemOption = await axios({
method: 'GET',
url: `https://jsonplaceholder.typicode.com/albums/${_1sourceEachtime.id}/photos`,
// url:`https://haveibeenpwned.com/api/v3/breach/[ITEM]`
headers: {
'content-type': 'application/json; charset=utf-8'
}
});
// this the place you can mix the 2 result.
const mixRes1AndRes2 ={
sources:_1sourceEachtime.title,
details:itemOption.data.slice(0,1)
}
return mixRes1AndRes2;
})
// final result look like the picture 2 below
finalRes= await Promise.all(allSourcesWithDetails);
return res.status(200).json({response: finalRes});
}
catch (error) {
console.log(error);
}
});
Picture 1
Picture 2
Having some issues when calling an external api to fetch information inside a loop in my node/express backend. I need information I get from the loop to get the correct data back from the endpoint. When I loop through the data and make the call I get this error.
{ message: 'Too Many Requests Limit 30. Reset time 1594218437315',
status: 429 }
I get the correct data back sometimes and sometimes only this message. There will be about 10 000 or so calls I need to make. I've tried a multiple of throttling libs and a lot of the code here on SO but it's pretty much always the same result which is the error message or it doesnt work at all.
I think I need a way send about 20-30 requests at a time and then wait a second or so and then continue. Or is there another better way? How would I achieve this?
const product = await NewProduct.find({});
product.map((item, index) => {
item.stores.map(async inner => {
inner.trackingUrl = await fetchRetry(inner.programId, inner.productUrl)
})
})
async function fetchRetry(id, urlToTrack) {
url1 = 'https://api.adtraction.com/v2/affiliate/tracking/link/?token=token';
const data = {
channelId,
programId: id,
shortLink: true,
url: urlToTrack
};
const options = {
method: 'POST',
headers: {
'Content-type': 'application/json',
Accept: 'application/json',
'Accept-Charset': 'utf-8',
},
body: JSON.stringify(data),
};
const res = await fetch(url, {
method: 'POST',
headers: {
'Content-type': 'application/json',
Accept: 'application/json',
'Accept-Charset': 'utf-8',
},
body: JSON.stringify(data),
});
const json = await res.json();
console.log(json) // Error
return json.trackingUrl;
}
Well, I'll share with you something that I've done in one of my projects. For all of your data that you call with your api. Keep making requests and when you encounter any error or response with throttling then wait for 5 minutes (or required in your api) and move to previous index (previous data to call the api with)
async function callAPI(data) {
for (let index = 0; index < data.length; index++) {
try {
// api call with your data
yourApiCall(data[index]);
} catch (e) {
if (e.type == "RequestThrottled") {
let oneMinute = 60000;
// wait for the time your request will be availabl
await sleep(5 * oneMinute);
// because of error in this request get back to
//previous request (or data)
index--;
}
}
}
}
Function using setTimeout to wait synchronously.
async function sleep(ms) {
await new Promise((resolve) => setTimeout(resolve, ms));
}
Also, you may want to run it on some background process as it might block your main event loop.
I'm trying to do a remote call in my node.js application to an external URL, then parse that request and perform an action based on the return. I'm using express and mysql.
I was able to get the remote URL content, however, I'm having some kind of race condition where my output is always changing and is not reliable. I tried to use async/await but wasn't able to.
This is the function called to run the app:
function lista(servidores) {
return new Promise(function(resolve, reject) {
var sql = ' SELECT sv.id as svid, sv.ip as svip'+
' FROM servidores sv'
dbconfig.conexao.query(sql, function (err, result, fields) {
Promise.all(
result.map(row => {
var ipsv = row.svip;
var urlprobe = 'http://201.182.96.14:8000/PING/' + ipsv;
fetch(urlprobe, {
method: 'get',
headers: {
'Accept': 'application/json, application/xml, text/plain, text/html, *.*',
'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8'
},})
.then(
response => response.json(),
error => console.log('Ocorreu um erro', error)
)
.then(
json => console.log(json),
)
})
).then(result => result);
return resolve();
});
})
}
When all these functions are called, everything works ok until the monitora() function. The output is random based on which fetch answers faster, and therefore the result is not reliable. The ideal is that
monitora() performs each fetch separetely and then process the if's based on each one of the results.
#edit: I edited the code and made the fetch directly on the main function, however I'm still receiving inconsistent results, like if there was some sort of caching somewhere.
I'm not sure if I understand correctly but you can try using Promise.all() function to wait for all fetch requests to finish, before taking an action.
Promise.all(
result.map(row => {
var urlMon = 'http://' + row.pip + ':8000/PING/' + ipsv;
monitora(urlMon, idsv);
})
).then(result => result /*Do something */);
Looking to find a clear and complete example of a case where async is used to handle chained function steps. I have psuedo code below showing intent, but not certain if there is an example out there that would show; clearly, the actual code needed to call multiple steps from within the async function.
The function is using a basic async wrapper.
getUserById: asyncHandler ( (req, res, next) => {
validateUser();
SavetoDB();
res.json({"message": "TBD...success"});
})
Well, your question isn't very well-formed.
Your SaveToDB() method will probably do some database task, which will be async. So you'll want to return a promise. So SaveToDB() could be something like :
exports.submitPost = async () => {
const data = await fetch('/post', {
method: 'POST',
headers: {
'Accept': 'application/json',
'Content-Type': 'application/json'
},
body: JSON.stringify({
field: {
txt: this.state.newfield
}
})
})
const jsonData = await data.json();
return jsonData;
}
Then you can call is with .then:
submitPost().then(data => console.log(data))
i'm gonna test REST API using Cypress.io , but using chaining request, it wants to be work like this, JSON response body on the first API will be used on the next API Headers for Authorization
I'm already try doing by using cypress commands and printing on console.log, but seems like it does not captured on the log, or is there any clue for this, or i just using another command like cy.route ?
Cypress.Commands.add("session", () => {
return cy.request({
method: 'POST',
url: '/auth/',
headers: {
'Content-Type': 'application/json',
},
body: {
"client_secret" : ""+config.clientSecret_staging,
"username": ""+config.email_staging,
"password": ""+config.password_staging
}
}).then(response => {
const target = (response.body)
})
})
it('GET /capture', () => {
cy.session().then(abc =>{
cy.request({
method: 'GET',
url: '/capture/'+target
})
})
})
the goal is to capture parse of JSON array from target = (response.body)
You have two options:
leave the code as is, be aware that the
.then(response => {
const target = (response.body)
})
code isn't returning anything so the cy.session().then(abc =>{ ... code is getting the whole response (abc is the response of the first .then)
change the code into
.then(response => {
const target = (response.body)
return target // I added this return
})
and then your abc param will be equal to response.body and not to response
That's because if you don't return a subject from your chainable calls the default one will be passed to the next .then function.
Let me know if it satisfies your issue.
p.s. Welcome 👋