I have a node api that returns an object but the object returned are not always at the same order. This is how I did it:
const somelist = await Model.find({condition blah blah})
.sort({created_at: -1})
.select('some fields');
if(!somelist){
res.status(500).json({ success: false });
}else{
let arr = [];
await Promise.all(somelist.map(async (item) => {
let user = await User.findById(item.some);
if(user) {
arr.push(item);
}
}));
console.log(arr)
res.status(200).json(arr);
}
The original display would be 1, 2, 3 but sometimes, it will become 2, 1, 3 but very rarely. Since I added a sort function in my somelist, maybe the issue is in the promise. How should I deal with this?
No, neither Promise.all nor map does change the oder of an array. Promise.all is guaranteed to return the results in the same order as the promisearray, you put in.
BUT: the order, in which the promises are resolved isn't guaranteed, thus the arr.push() can happen in any order, so you may receive different results. So you can do it somehow like this
let arr = (await Promise.all(somelist.map(i => User.findById(i.some)))
.filter(r => !!r)
Assuming that User.findById() returns a promise, somelist.map(...) returns an array of promises, which you can await with Promise.all. Once Promise.all has resolved, you can filter out all elements, which returned a undefined or null result.
Another possiblity would be the following
for (let item of somelist) {
let u = await User.findById(item.some);
if (u) arr.push(u);
}
But that is serializing your requests and executing one after the other. So this would probably waste a lot of time waiting for some external results (network, database, disk, ...)
I believe you should use this code:
let arr = await Promise.all(somelist.map(async (item) => {
return await User.findById(item.some);
}));
In your code you push the async function result into an array and this statement not necessarily would work in the specified order. But Promise.all would return resolved results in your original array order.
Related
I have a function with multiple forEach loops:
async insertKpbDocument(jsonFile) {
jsonFile.doc.annotations.forEach((annotation) => {
annotation.entities.forEach(async (entity) => {
await this.addVertex(entity);
});
annotation.relations.forEach(async (relation) => {
await this.addRelation(relation);
});
});
return jsonFile;
}
I need to make sure that the async code in the forEach loop calling the this.addVertex function is really done before executing the next one.
But when I log variables, It seems that the this.addRelation function is called before the first loop is really over.
So I tried adding await terms before every loops like so :
await jsonFile.doc.annotations.forEach(async (annotation) => {
await annotation.entities.forEach(async (entity) => {
await this.addVertex(entity);
});
await annotation.relations.forEach(async (relation) => {
await this.addRelation(relation);
});
});
But same behavior.
Maybe it is the log function that have a latency? Any ideas?
As we've discussed, await does not pause a .forEach() loop and does not make the 2nd item of the iteration wait for the first item to be processed. So, if you're really trying to do asynchronous sequencing of items, you can't really accomplish it with a .forEach() loop.
For this type of problem, async/await works really well with a plain for loop because they do pause the execution of the actual for statement to give you sequencing of asynchronous operations which it appears is what you want. Plus, it even works with nested for loops because they are all in the same function scope:
To show you how much simpler this can be using for/of and await, it could be done like this:
async insertKpbDocument(jsonFile) {
for (let annotation of jsonFile.doc.annotations) {
for (let entity of annotation.entities) {
await this.addVertex(entity);
}
for (let relation of annotation.relations) {
await this.addRelation(relation);
}
}
return jsonFile;
}
You get to write synchronous-like code that is actually sequencing asynchronous operations.
If you are really avoiding any for loop, and your real requirement is only that all calls to addVertex() come before any calls to addRelation(), then you can do this where you use .map() instead of .forEach() and you collect an array of promises that you then use Promise.all() to wait on the whole array of promises:
insertKpbDocument(jsonFile) {
return Promise.all(jsonFile.doc.annotations.map(async annotation => {
await Promise.all(annotation.entities.map(entity => this.addVertex(entity)));
await Promise.all(annotation.relations.map(relation => this.addRelation(relation)));
})).then(() => jsonFile);
}
To fully understand how this works, this runs all addVertex() calls in parallel for one annotation, waits for them all to finish, then runs all the addRelation() calls in parallel for one annotation, then waits for them all to finish. It runs all the annotations themselves in parallel. So, this isn't very much actual sequencing except within an annotation, but you accepted an answer that has this same sequencing and said it works so I show a little simpler version of this for completeness.
If you really need to sequence each individual addVertex() call so you don't call the next one until the previous one is done and you're still not going to use a for loop, then you can use the .reduce() promise pattern put into a helper function to manually sequence asynchronous access to an array:
// helper function to sequence asynchronous iteration of an array
// fn returns a promise and is passed an array item as an argument
function sequence(array, fn) {
return array.reduce((p, item) => {
return p.then(() => {
return fn(item);
});
}, Promise.resolve());
}
insertKpbDocument(jsonFile) {
return sequence(jsonFile.doc.annotations, async (annotation) => {
await sequence(annotation.entities, entity => this.addVertex(entity));
await sequence(annotation.relations, relation => this.addRelation(relation));
}).then(() => jsonFile);
}
This will completely sequence everything. It will do this type of order:
addVertex(annotation1)
addRelation(relation1);
addVertex(annotation2)
addRelation(relation2);
....
addVertex(annotationN);
addRelation(relationN);
where it waits for each operation to finish before going onto the next one.
foreach will return void so awaiting it will not do much. You can use map to return all the promises you create now in the forEach, and use Promise.all to await all:
async insertKpbDocument(jsonFile: { doc: { annotations: Array<{ entities: Array<{}>, relations: Array<{}> }> } }) {
await Promise.all(jsonFile.doc.annotations.map(async(annotation) => {
await Promise.all(annotation.entities.map(async (entity) => {
await this.addVertex(entity);
}));
await Promise.all(annotation.relations.map(async (relation) => {
await this.addRelation(relation);
}));
}));
return jsonFile;
}
I understand you can run all the addVertex concurrently. Combining reduce with map splitted into two different set of promises you can do it. My idea:
const first = jsonFile.doc.annotations.reduce((acc, annotation) => {
acc = acc.concat(annotation.entities.map(this.addVertex));
return acc;
}, []);
await Promise.all(first);
const second = jsonFile.doc.annotations.reduce((acc, annotation) => {
acc = acc.concat(annotation.relations.map(this.addRelation));
return acc;
}, []);
await Promise.all(second);
You have more loops, but it does what you need I think
forEach executes the callback against each element in the array and does not wait for anything. Using await is basically sugar for writing promise.then() and nesting everything that follows in the then() callback. But forEach doesn't return a promise, so await arr.forEach() is meaningless. The only reason it isn't a compile error is because the async/await spec says you can await anything, and if it isn't a promise you just get its value... forEach just gives you void.
If you want something to happen in sequence you can await in a for loop:
for (let i = 0; i < jsonFile.doc.annotations.length; i++) {
const annotation = jsonFile.doc.annotations[i];
for (let j = 0; j < annotation.entities.length; j++) {
const entity = annotation.entities[j];
await this.addVertex(entity);
});
// code here executes after all vertix have been added in order
Edit: While typing this a couple other answers and comments happened... you don't want to use a for loop, you can use Promise.all but there's still maybe some confusion, so I'll leave the above explanation in case it helps.
async/await does not within forEach.
A simple solution: Replace .forEach() with for(.. of ..) instead.
Details in this similar question.
If no-iterator linting rule is enabled, you will get a linting warning/error for using for(.. of ..). There are lots of discussion/opinions on this topic.
IMHO, this is a scenario where we can suppress the warning with eslint-disable-next-line or for the method/class.
Example:
const insertKpbDocument = async (jsonFile) => {
// eslint-disable-next-line no-iterator
for (let entity of annotation.entities) {
await this.addVertex(entity)
}
// eslint-disable-next-line no-iterator
for (let relation of annotation.relations) {
await this.addRelation(relation)
}
return jsonFile
}
The code is very readable and works as expected. To get similar functionality with .forEach(), we need some promises/observables acrobatics that i think is a waste of effort.
We are integrating with Outlook api and we need to group the attachments in the recovered emails:
We re trying this way:
const result = await client
.api('/me/messages')
.filter(searchMailFrom)
.select('subject, from, receivedDateTime, sentDateTime, isRead, toRecipients, hasAttachments')
.get()
let dadosAnexo = result.value.map(async item => {
if (item.hasAttachments) {
const resultAtt = await client
.api('/me/messages/' + item.id + '/attachments')
.get()
item.anexos = resultAtt.value
}
})
await Promise.all(dadosAnexo)
return res.status(200).send(result.value)
But when we put Promise.all (), the system simply returns nothing
You are not returning anything from inside your .map functions. So, dadosAnexo becomes an array of Promises that will each resolve to undefined.
Check the MDN docs for more details about how .map works: Map | MDN.
Then, you are passing dadosAnexo to your Promise.all call.
But when we put Promise.all (), the system simply returns nothing
Here, your assumption is wrong.
await Promise.all(dadosAnexo)
The await Promise.all call above will actually return an array of undefined. Because you are passing it dadosAnexo (an array of Promises, each of which resolves to undefined). Also, you are not assigning the return value to any variable (so, you don't actually know if it's returning something or not).
Check the MDN docs for more details about how Promise.all works: Promise.all() | MDN
Now to solve your problem, here's a solution:
const result = await client
.api('/me/messages')
.filter(searchMailFrom)
.select('subject, from, receivedDateTime, sentDateTime, isRead, toRecipients, hasAttachments')
.get()
// promisesToAttach will be an array containing some Promises and some item values
const promisesToAttach = result.value.map(item => {
if (item.hasAttachments) {
// returning a promise
return client
.api('/me/messages/' + item.id + '/attachments')
.get()
.then(resultAtt => {
item.anexos = resultAtt.value
return item
})
}
// returning an item value
return item
})
// resultWithAttachments will be an array of resolved item values
const resultWithAttachments = await Promise.all(promisesToAttach)
return res.status(200).send(resultWithAttachments)
I would like some guidance on a good way to pass data along with an array of promises so that the data can be used later after calling await Promise.all().
I'm pretty new to node and promises. The code below, in summary, does what I want, but it seems messy to have the promises array separate from the corresponding data array. I would think this is a common occurrence and would have a simpler solution.
I was able to accomplish the same result by attaching a then() function to the asyncFunc return, but this makes the code messier and for some reason takes twice as long to execute.
async function foo(inputs) {
var promises = [];
var datas = [];
for (var input of inputs.values()) {
promises.push(asyncFunc(input.a));
datas.push(input.b);
}
var outputs = [];
for (let result of (await Promise.all(promises)).values()) {
outputs.push({
data: datas.shift(),
result: result
});
}
return outputs;
}
You will often face this kind of problem in nodeJs, the answer is how you want your async function react.
If you want it run sequentially, you don't need promise all. The code will look like this.
async function foo(inputs) {
const results = [];
const datas = [];
for (const input of inputs.values()) {
const result = await asyncFunc(input.a);
results.push(result);
datas.push(input.b);
}
const outputs = [];
for (const result of results) {
outputs.push({
data: datas.shift(),
result: result
});
}
return outputs;
}
In case you have to use the promise all, you can use another variable to hold the values, this will make the code less messy.
const results = await Promise.all(promises)).values();
for (const result of results) {
outputs.push({
data: datas.shift(),
result: result
});
}
I'm trying to make requests to Twitter with Twitter for Node and store responses in an array for handling later. I'm pushing returned tweets to an array with each push() happening in a callback, which seems to be working fine. My problem is that I'm having trouble accessing the full array with all pushed tweets.
The cause, of course, is that any attempt to work with that array is getting called before the results from the Twitter API have arrived, so I'm getting an empty array.
How can I (and should I) get my function working with the full array into another callback? I'm asking this from the perspective of someone still trying to get a firm grasp on asynchronous programming - especially multiple callbacks or functions that have to run asynchronously.
Again, current result is tweetHold = [], and I'd like tweetHold to contain all matched tweets for all users in the searchArray.
let searchArray = {
users: ['ByBuddha', 'thetweetofgod']
}
let tweetHold = [];
let T = new Twitter(config);
for (user of searchArray.users) {
let params = {
q: 'from:'+ user,
count: 1,
tweet_mode: 'extended',
result_type: 'recent',
lang: 'en'
}
T.get('search/tweets', params, returnedTweets);
}
function returnedTweets(err, tweets, response) {
tweetHold.push(tweets);
}
// obviously, doesn't work as the array is logged to console before T.get() is done
console.log(tweetHold);
T.get accepts a callback function that gets called once the asynchronous operation is done. But since you want to get multiple responses, and not just one, using just the callbacks by themselves would be a bit messy. For example, inside returnedTweets, you could increment a persistent counter variable, and call the next function once counter === searchArray.users.length, but it would be more elegant to use Promises instead.
Map each T.get call to a Promise that resolves with the tweets variable you're interested in, and then call Promise.all on an array of those Promises. Promise.all takes an array of Promises and returns a Promise that resolves once every Promise in the passed array has resolved.
Note that it looks like you're currently ignoring the err that might come back from T.get - that's probably not a good idea, it would be better to be able to check when errors occur, and then handle the error somehow (otherwise, the tweetHold array may sometimes contain broken data). Luckily, if you use Promises, implementing this is easy - just reject if there's an err, and catch after the Promise.all:
const T = new Twitter(config);
const searchObject = {
users: ['ByBuddha', 'thetweetofgod']
};
const searchPromises = searchArray.users.map((user) => {
return new Promise((resolve, reject) => {
const params = {
q: 'from:'+ user,
count: 1,
tweet_mode: 'extended',
result_type: 'recent',
lang: 'en'
};
T.get('search/tweets', params, (err, tweets) => {
if (err) reject(err);
else resolve(tweets);
});
});
});
Promise.all(searchPromises)
.then((tweetHold) => {
// tweetHold will be an array containing the `tweets` variable for each user searched
console.log(tweetHold);
})
.catch((err) => {
// there was an error, best to handle it somehow
// the `.then` above will not be entered
});
I would like to have a function called on the return of the last promise made during a forEach() loop.
Sample code:
var mainList = getArrayData()
mainList.forEach((item,i)=>{
mongooseSchema.find(_id:item.id).exec((err,doc)=>{
doStuff(doc)
})
})
Any code after the second block will run immediately after all the mongo queries are sent off. If you wrap the whole thing in a promise (i.e. the array data is from a separate mongo query) you still get the same effect.
Is there any way I can have a special function/callback for the last query, or even better, a promise that returns after all the queries have returned?
Thanks
You could use Promise.all. It will fire off once all of the promises in an array are resolved.
The Promise.all(iterable) method returns a promise that resolves when all of the promises in the iterable argument have resolved, or rejects with the reason of the first passed promise that rejects.
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/all
var mainList = getArrayData()
let promises = [];
mainList.forEach((item,i)=>{
let p = new Promise((resolve, reject) => {
mongooseSchema.find(_id:item.id).exec((err,doc)=>{
doStuff(doc)
resolve();
});
});
promises.push(p);
});
Promise.all(promises).then(() => {
// do something when all are resolved
});