I am trying to process response data and do not make next request before current data didn't processed. I tried use async/await and generators.
Generator:
private *readData() {
const currentUrl = this.getUrl();
const requestSettings = this.getRequestSettings();
yield axios.get( currentUrl, requestSettings).then( (response: any) => {
console.log('Make request');
return this.getData(response);
});
}
*readItem() {
let responseData: any;
if (!responseData) {
const response = this.readData();
responseData = response.next();
}
console.log('res data:', responseData['value']);
yield responseData['value'].then((res: any) => {
return res;
});
}
and then in the main code I do next:
for (let i=0;i<10;i++) {
item = transport.readItem().next();
console.log("R:", item);
}
Another idea was using async/await
async readItems() {
const headers = this.settings.headers;
const response = await axios.get( url, {
headers: headers
});
return response.data;
}
But in this case I get a promise in the response, if I just try to call this method 10 times.
I read 10 items, but I still make 10 requests to the server. Is it possible make one request, processed 10 items, and then make the second request? Maybe I have to use another pattern or whatever.
Async/await is right approach, just put await in front of readItem() and the Promise you get will be awaited that will give you desired. If your loop is in top level use readItem().then(). The latest NodeJS version allows await in top level.
for (let i=0;i<10;i++) {
item = await transport.readItem();
console.log("R:", item);
}
I found next solution
(async () => {
for (let i = 0; i < 10; i++) {
item = await transport.readItem();
console.log("R:", item);
}
})();
because I ran this part of code inside script without any functions/methods
Related
I created this function to get list all my drives from GDrive.
async getAllDrives(token) {
let nextPageToken = ""
let resultArray = []
const config= {
headers: {
Authorization: `Bearer ${token}`
}
};
const bodyParams = {
pageSize: 2,
fields: 'nextPageToken, drives(id, name)',
q:`hidden=false`,
};
do {
axios.get(
`https://www.googleapis.com/drive/v3/drives`,
config,
bodyParams,
).then(result => {
nextPageToken = result.data.nextPageToken;
resultArray.push(result.data.drives);
resultArray = resultArray.flat();
console.log("result", resultArray);
}).catch(error => {
console.log(error);
//res.send(error);
});
}while(nextPageToken);
resultArray = resultArray.flat();
resultArray.map(drive => {
drive.isSharedDrive = true;
return drive;
});
return JSON.stringify(resultArray);
}
When I look in console.log
then(result => {
nextPageToken = result.data.nextPageToken;
resultArray.push(result.data.drives);
resultArray = resultArray.flat();
console.log("result", resultArray);
})
I have the expected result,
result [
{
kind: 'drive#drive',
id: '**',
name: ' ★ 🌩'
},
]
but return JSON.stringify(resultArray); is empty.
I found a similar question here, How do I return the response from an asynchronous call? but the answer is not satisfying.
You used the async call slightly incorrectly. You calling axios.get without await keyword, but with .then chaining. Since you don't wait for result to return, you getting empty array first, returning you nothing. And only then your callback function inside .then is getting called. To simplify, you doing this in your example:
function getAllDrives() {
// Local variable where you want your result
let result = [];
// You calling the axios.get method, but don't wait for result
axios.get().then(result => {})
// Empty result is getting returned immediately
return result;
}
And when response is returned from the remote server, function inside .then trying to save result to local variable. But function is already completed, so you don't get anything.
What you actually should do is call axios.get with await keyword:
// You should always cover your asynchronous code with a try/catch block
try {
// Instead of `then` callback use `await` keyword. Promise returned from
// this method will contain result. If error occurs, it will be thrown,
// and you can catch it inside `catch`.
const result = await axios.get(
`https://www.googleapis.com/drive/v3/drives`,
config,
bodyParams
);
// Here is your code as you wrote it inside `then` callback
nextPageToken = result.data.nextPageToken;
resultArray.push(result.data.drives);
resultArray = resultArray.flat();
console.log("result", resultArray);
} catch (error) {
// And here is your error handling code as you wrote it inside `catch`
console.log(error);
}
This way your method will not complete until your request is not executed.
You can read more about async/await functions here.
I believe your goal is as follows.
You want to retrieve the drive list using axios.
Your access token can be used for retrieving the drive list using Drive API.
Modification points:
In order to use nextPageToken in the request, in this case, it is required to run the script with a synchronous process. So, async/await is used. This has already been mentioned in the existing answers.
When I saw your script, I thought that the query parameter might be required to be included in the 2nd argument of axios.get().
In order to use nextPageToken, it is required to include the property of pageToken. In your script, pageToken is not used. By this, the infinite loop occurs because nextPageToken is continued to be returned.
When these points are reflected in your script, how about the following modification?
Modified script:
let resultArray = [];
const config = {
headers: {
Authorization: `Bearer ${token}`,
},
params: {
pageSize: 2,
fields: "nextPageToken, drives(id, name)",
q: `hidden=false`,
pageToken: "",
},
};
do {
const { data } = await axios
.get(`https://www.googleapis.com/drive/v3/drives`, config)
.catch((error) => {
if (error.response) {
console.log(error.response.status);
console.log(error.response.data);
}
});
if (data.drives.length > 0) {
resultArray = [...resultArray, ...data.drives];
}
nextPageToken = data.nextPageToken;
config.params.pageToken = nextPageToken;
} while (nextPageToken);
resultArray.map((drive) => {
drive.isSharedDrive = true;
return drive;
});
return JSON.stringify(resultArray);
Testing:
When this script is run, the following result is obtained.
[
{"id":"###","name":"###","isSharedDrive":true},
{"id":"###","name":"###","isSharedDrive":true},
,
,
,
]
Note:
From the official document of "Drives: list",
pageSize: Maximum number of shared drives to return per page. Acceptable values are 1 to 100, inclusive. (Default: 10)
So, when pageSize is 100, the number of loops can be reduced. If you want to test the loop using nextPageToken, please reduce the value.
References:
axios
Drives: list
I recommend you study a little more about async/await.
It makes no sense for you to use async and put a .then().catch(), the purpose of async to get these encapsulated syntaxes.
async getAllDrives(token) {
try {
const getDrives = await this.request(token)
console.log(getDrives)
const results = this.resultArray(getDrives)
return results
} catch (e) {
console.log(e)
}
}
I didn't quite understand your while or your objective, adapt it to your code or remove it
async request(token) {
let nextPageToken = 1 // ????????
const config = {
headers: {
Authorization: `Bearer ${token}`
}
};
const bodyParams = {
pageSize: 2,
fields: 'nextPageToken, drives(id, name)',
q: `hidden=false`,
};
let getDrives = [];
// loop for each request and create a request array
for (let x = 0; x < fields.nextPageToken; x++) {
const request = axios.get(
`https://www.googleapis.com/drive/v3/drives`,
config,
bodyParams
);
getDrives.push(request)
}
const drives = await Promise.all(getDrives)
return drives
}
async resultArray(drivers) {
// result treatment here
}
The return of promise all will be an array of the driver's responses
Note: The response in request.data
const request = await axios.get()
const resposta = request.data
Read about
https://developer.mozilla.org/pt-BR/docs/Web/JavaScript/Reference/Global_Objects/Promise/all
I have a script using axios that hits an API with a limit of 5 requests per second. At present my request array length is 72 and will grow over time. I receive an abundance of 429 errors. The responses per endpoint change with each run of the script; ex: url1 on iteration1 returns 429, then url1 on iteration2 returns 200, url1 on iteration3 returns 200, url1 on iteration4 returns 429.
Admittedly my understanding of async/await and promises are spotty at best.
What I understand:
I can have multiple axios.get running because of async. The variable I set in my main that uses the async function can include the await to ensure all requests have processed before continuing the script.
Promise.all can run multiple axios.gets but, if a single request fails the chain breaks and no more requests will run.
Because the API will only accept 5 requests per second I have to chunk my axios.get requests to 5 endpoints, wait for those to finish processing before sending the next chunk of 5.
setTimeout will assign a time limit to a single request, once the time is up the request is done and will not be sent again no matter the return being other than 200.
setInterval will assign a time limit but it will send the request again after time's up and keep requesting until it receives a 200.
async function main() {
var endpoints = makeEndpoints(boards, whiteList); //returns an array of string API endpoints ['www.url1.com', 'www.url2.com', ...]
var events = await getData(endpoints);
...
}
The getData() has seen many iterations in attempt to correct the 429's. Here are a few:
// will return the 200's sometimes and not others, I believe it's the timeout but that won't attempt the hit a failed url (as I understand it)
async function getData(endpoints) {
let events = [];
for (x = 0; x < endpoints.length; x++) {
try {
let response = await axios.get(endpoints[x], {timeout: 2000});
if ( response.status == 200 &&
response.data.hasOwnProperty('_embedded') &&
response.data._embedded.hasOwnProperty('events')
) {
let eventsArr = response.data._embedded.events;
eventsArr.forEach(event => {
events.push(event)
});
}
} catch (error) {
console.log(error);
}
}
return events;
}
// returns a great many 429 errors via the setInterval, as I understand this function sets a delay of N seconds before attempting the next call
async function getData(endpoints) {
let data = [];
let promises = [];
endpoints.forEach((url) => {
promises.push(
axios.get(url)
)
})
setInterval(function() {
for (i = 0; i < promises.length; i += 5) {
let requestArr = promises.slice(i, i + 5);
axios.all(requestArr)
.then(axios.spread((...res) => {
console.log(res);
}))
.catch(err => {
console.log(err);
})
}
}, 2000)
}
// Here I hoped Promise.all would allow each request to do its thing and return the data, but after further reading I found that if a single request fails the rest will fail in the Promise.all
async getData(endpoints) {
try {
const res = await Promise.all(endpoints.map(url => axios.get(url))).catch(err => {});
} catch {
throw Error("Promise failed");
}
return res;
}
// Returns so many 429 and only 3/4 data I know to expect
async function getData(endpoints) {
const someFunction = () => {
return new Promise(resolve => {
setTimeout(() => resolve('222'), 100)
})
}
const requestArr = endpoints.map(async data => {
let waitForThisData = await someFunction(data);
return axios.get(data)
.then(response => { console.log(response.data)})
.catch(error => console.log(error.toString()))
});
Promise.all(requestArr).then(() => {
console.log('resolved promise.all')
})
}
// Seems to get close to solving but once an error is it that Promise.all stops processing endpoint
async function getData(endpoints) {
(async () => {
try {
const allResponses = await Promise.all(
endpoints.map(url => axios.get(url).then(res => console.log(res.data)))
);
console.log(allResponses[0]);
} catch(e) {
console.log(e);
// handle errors
}
})();
}
It seems like I have so many relevant pieces but I cannot connect them in an efficient and working model. Perhaps axios has something completely unknown to me? I've also tried using blurbird concurrent to limit the request to 5 per attempt but that still returned the 429 from axios.
I've been starring at this for days and with so much new information swirling in my head I'm at a loss as to how to send 5 requests per second, await the response, then send another set of 5 requests to the API.
Guidance/links/ways to improve upon the question would be much appreciated.
I'm trying to download a bunch of files. Let's say 1.jpg, 2.jpg, 3.jpg and so on. If 1.jpg exist, then I want to try and download 2.jpg. And if that exist I will try the next, and so on.
But the current "getFile" returns a promise, so I can't loop through it. I thought I had solved it by adding await in front of the http.get method. But it looks like it doesn't wait for the callback method to finish. Is there a more elegant way to solve this than to wrap the whole thing in a new async method?
// this returns a promise
var result = getFile(url, fileToDownload);
const getFile = async (url, saveName) => {
try {
const file = fs.createWriteStream(saveName);
const request = await http.get(url, function(response) {
const { statusCode } = response;
if (statusCode === 200) {
response.pipe(file);
return true;
}
else
return false;
});
} catch (e) {
console.log(e);
return false;
}
}
I don't think your getFile method is returning promise and also there is no point of awaiting a callback. You should split functionality in to two parts
- get file - which gets the file
- saving file which saves the file if get file returns something.
try the code like this
const getFile = url => {
return new Promise((resolve, reject) => {
http.get(url, response => {
const {statusCode} = response;
if (statusCode === 200) {
resolve(response);
}
reject(null);
});
});
};
async function save(saveName) {
const result = await getFile(url);
if (result) {
const file = fs.createWriteStream(saveName);
response.pipe(file);
}
}
What you are trying to do is getting / requesting images in some sync fashion.
Possible solutions :
You know the exact number of images you want to get, then go ahead with "request" or "http" module and use promoise chain.
You do not how the exact number of images, but will stop at image no. N-1 if N not found. then go ahed with sync-request module.
your getFile does return a promise, but only because it has async keyword before it, and it's not a kind of promise you want. http.get uses old callback style handling, luckily it's easy to convert it to Promise to suit your needs
const tryToGetFile = (url, saveName) => {
return new Promise((resolve) => {
http.get(url, response => {
if (response.statusCode === 200) {
const stream = fs.createWriteStream(saveName)
response.pipe(stream)
resolve(true);
} else {
// usually it is better to reject promise and propagate errors further
// but the function is called tryToGetFile as it expects that some file will not be available
// and this is not an error. Simply resolve to false
resolve(false);
}
})
})
}
const fileUrls = [
'somesite.file1.jpg',
'somesite.file2.jpg',
'somesite.file3.jpg',
'somesite.file4.jpg',
]
const downloadInSequence = async () => {
// using for..of instead of forEach to be able to pause
// downloadInSequence function execution while getting file
// can also use classic for
for (const fileUrl of fileUrls) {
const success = await tryToGetFile('http://' + fileUrl, fileUrl)
if (!success) {
// file with this name wasn't found
return;
}
}
}
This is a basic setup to show how to wrap http.get in a Promise and run it in sequence. Add error handling wherever you want. Also it's worth noting that it will proceed to the next file as soon as it has received a 200 status code and started downloading it rather than waiting for a full download before proceeding
How can I add a setTimeout to my async await function call?
I have
request = await getProduct(productids[i]);
where
const getProduct = async productid => {
return requestPromise(url + productid);
};
I've tried
request = await setTimeout((getProduct(productids[i])), 5000);
and got the error TypeError: "callback" argument must be a function which makes sense. The request is inside of a loop which is making me hit the rate limit on an api call.
exports.getProducts = async (req, res) => {
let request;
for (let i = 0; i <= productids.length - 1; i++) {
request = await getProduct(productids[i]);
//I want to wait 5 seconds before making another call in this loop!
}
};
You can use a simple little function that returns a promise that resolves after a delay:
function delay(t, val) {
return new Promise(function(resolve) {
setTimeout(function() {
resolve(val);
}, t);
});
}
// or a more condensed version
const delay = (t, val) => new Promise(resolve => setTimeout(resolve, t, val));
And, then await that inside your loop:
exports.getProducts = async (req, res) => {
let request;
for (let id of productids) {
request = await getProduct(id);
await delay(5000);
}
};
Note: I also switched your for loop to use for/of which is not required, but is a bit cleaner than what you had.
Or, in modern versions of nodejs, you can use timersPromises.setTimeout() which is a built-in timer that returns a promise (as of nodejs v15):
const setTimeoutP = require('timers/promises').setTimeout;
exports.getProducts = async (req, res) => {
let request;
for (let id of productids) {
request = await getProduct(id);
await setTimeoutP(5000);
}
};
Actually, I have a pretty standard chunk of code that I use to do that:
function PromiseTimeout(delayms) {
return new Promise(function (resolve, reject) {
setTimeout(resolve, delayms);
});
}
Usage:
await PromiseTimeout(1000);
If you're using Bluebird promises, then it's built in as Promise.timeout.
More to your problem: Have you checked API docs? Some APIs tell you how much you have to wait before next request. Or allow downloading data in larger bulk.
As of node v15 you can use the Timers Promises API:
const timersPromises = require('timers/promises');
async function test() {
await timersPromises.setTimeout(1000);
}
test();
Note that this feature is experimental and may change in future versions.
Since Node 15 and above, there is the new Timers Promises API that let you to avoid to build the wrapping:
import {
setTimeout,
setImmediate,
setInterval,
} from 'timers/promises';
console.log('before')
await setTimeout(1000)
console.log('after 1 sec')
So your issues you could write it with async iterator:
import {
setTimeout
} from 'timers/promises'
async function getProducts (req, res) {
const productids = [1, 2, 3]
for await (const product of processData(productids)) {
console.log(product)
}
}
async function * processData (productids) {
while (productids.length > 0) {
const id = productids.pop()
const product = { id }
yield product
await setTimeout(5000)
}
}
getProducts()
I have done api delay test as below.
It is possible to delay it as if by hanging setTimeout.
sleep(ms) {
const wakeUpTime = Date.now() + ms;
while (Date.now() < wakeUpTime) {}
}
callAPI = async() => {
... // Execute api logic
await this.sleep(2147483647);
... // Execute api logic
}
await callAPI();
My application uses an internal webservice for fetching data, i have a job which creates approx 500 requests which getsfired async to complete the fetch operation.
I make use of Axios, by creating an array of axios promises and then resolving them using using Axios.all();
It works fine until some 200 requests but post that i get socket hung up, however on the server side i see the requests are being processed.
How to configure axios to set custom time out, or is it a better idea to splice my promises array and then run them as multiple batches ?
Source code
let getAxiosPromiseArray = (urlList) => {
var axiosArrayofPromise = [];
return new Promise ( (resolve, reject) => {
try {
urlList.forEach ( (URL) => {
axiosArrayofPromise.push(axios.get(URL));
});
resolve(axiosArrayofPromise);
}
catch (err) {
reject("There is a problem getting Axios array of promises " + err);
}
})
}
async function processAxiosPromises (PromiseArray) {
try {
var results = []
results = await axios.all(PromiseArray);
return results;
}
catch(err) {
throw("There was a problem resolving promises array (Axios) " + err);
}
}
getallID().then ( (urlList) => {
return getAxiosPromiseArray(urlList);
}).then( (AxiosPromises) => {
return processAxiosPromises(AxiosPromises);
}).then ((resultData) => {
console.log(resultData);
});
Error
There was a problem resolving promises array (Axios) Error: socket hang up
First, that pair of functions getAxiosPromiseArray() and processAxiosPromises() needs fixing.
Your new Promise() construction is unnecessary. You can simply return Promise.all(arrayofPromise) (or axios.all(...) if you must) and do away with the other function.
Renaming the remaining function to something meaningful, you would end up with eg :
let getData = (urlList) => {
return Promise.all(urlList.map(URL => axios.get(URL)))
.catch(error => {
error.message = "There is a problem getting Axios array of promises " + error.message; // augment the error message ...
throw error; // ... and re-throw the errror.
});
};
And call as follows :
getallID().then(getData)
.then(resultData => {
console.log(resultData);
}).catch(error => {
console.error(error);
});
That will put you on solid ground but, on its own, is unlikely to fix a concurrency problem (if that's what it is), for which the simplest approach is to use Bluebird's Promise.map with the concurrency option.
The caller code can remain the same, just change getData(), as follows:
let getData = (urlList) => {
let concurrency = 10; // play with this value to find a reliable concurrency limit
return Promise.map(urlList, URL => axios.get(URL), {'concurrency': concurrency})
.catch(error => {
error.message = "There is a problem getting Axios array of promises " + error.message;
throw error;
});
};
// where `Promise` is Bluebird.
const axios = require('axios');
const axiosThrottle = require('axios-throttle');
//pass axios object and value of the delay between requests in ms
axiosThrottle.init(axios,200)
const options = {
method: 'GET',
};
const urlList = [
'https://jsonplaceholder.typicode.com/todos/1',
'https://jsonplaceholder.typicode.com/todos/2',
'https://jsonplaceholder.typicode.com/todos/3',
'https://jsonplaceholder.typicode.com/todos/4',
'https://jsonplaceholder.typicode.com/todos/5',
'https://jsonplaceholder.typicode.com/todos/6',
'https://jsonplaceholder.typicode.com/todos/7',
'https://jsonplaceholder.typicode.com/todos/8',
'https://jsonplaceholder.typicode.com/todos/9',
'https://jsonplaceholder.typicode.com/todos/10'
];
const promises = [];
const responseInterceptor = response => {
console.log(response.data);
return response;
};
//add interceptor to work with each response seperately when it is resolved
axios.interceptors.response.use(responseInterceptor, error => {
return Promise.reject(error);
});
for (let index = 0; index < urlList.length; index++) {
options.url = urlList[index];
promises.push(axiosThrottle.getRequestPromise(options, index));
}
//run when all promises are resolved
axios.all(promises).then(responses => {
console.log(responses.length);
});
https://github.com/arekgotfryd/axios-throttle