I have an array called noNCreatedResources. I want do some operation on each item of array and push item in createdResources array and remove the item from noNCreatedResources array and continue to do that until noNCreatedResources be empty. For this I've written CreateResources function including nested while and for loop. It works fine but i realize that it don't work synchronously. For example: it must iterate twice in while loop but iterates 4 times and I don't know why.
I think I don't understand concept of async/await/non-blocking concept of node.js. can any body help me to realize what the problem is?
CreateResources = async () => {
while (this.noNCreatedResources.length > 0) {
for (let index = 0; index < this.noNCreatedResources.length; index++) {
if (this.resourceHasCreatedDependencies(this.noNCreatedResources[index])) {
const resourceModel = this.someOperation(this.noNCreatedResources[index]);
this.createdResources.push(resourceModel);
this.noNCreatedResources.splice(index, 1);
}
}
}
}
First of all you are not doing anything asynchronous in you function so you can remove the async keyword from your function. Since you are not doing anything asynchronous so, your problem is not related to it. It is more of an implementation problem IMO.
Your while loop is useless for what you are trying to achieve. Also, your logic is broken!
Example: The following code will output 1, 3, and 5.
let x = [1,2,3,4,5];
for(let i = 0; i < x.length; i++) {
console.log(x[i]);
x.splice(i, 1);
}
I do not think you need to remove item from array to achieve your expected result. If you need to reset the array then at the end you can just do this x = [] to reset the array.
The problem you have is not due to async calls. Actually, your code is entirely synchronous. Try to take a look at where the "noNCreatedResources" is been created/updated. Async calls happens when you're sending a http request, reading a file etc, in other words, operations that doesn't happens inside your code. It allows the code to go on, not blocking the next function calls, and when the promise is fulfilled, the callback function is invoked.
Related
I have a for loop that queries a database. I want the db calls to be async.
This is the structure of the code:
for(var idx=0; idx<arr.rows.length; idx++)
{
db.query(`SELECT ...`)
.then((result) => {
console.log("Value is: " + result.rows[idx].val);
});
}
As you can see, I want the parameter idx printed in the resolve (db.query returns a Promise). But this way the wrong idx is printed because the idx value is incremented when the promise is resolved.
What is the proper way to pass the variable idx?
Thank you.
Use let instead of var.
Change from this:
for(var idx=0; idx<arr.rows.length; idx++)
to this:
for(let idx=0; idx<arr.rows.length; idx++)
let is block-scoped to the block of the for loop and will maintain a separate copy of the variable for each iteration of the loop so that each one has its own and thus your asynchronous .then() handler that gets called after the for loop is completely done will still have the appropriate value in its own copy of idx.
I have to check if foreignKey exist, but I can't make a loop with my asynchronous query function
function checkAllFK(tables, foreignKeys) {
let index = -1;
for (var key in foreignKeys) {
index++;
QueryFunction(tables[index], key, foreignKeys[key])
.then(result => {
if(result == null) {
//here, if result is null, that's mean the foreignKey doesn't exist, then, we have to stop the loop and return false;
return false;
}
else if(index == (tables.length - 1)) {
//here, that's mean we are at the end of the loop, and we doesn't break it with the previous if, that's mean all foreignKey exist, then, we return true;
return true;
}
}
the problem is that at the end of the first iteration, you exit the function and the result of the return depends only on the first iteration: false if the if condition is met at the first iteration, null if not
even having looked at many similar topics here, I haven't found a solution to my problem.
Your operation "check all foreignKeys against all tables" can be written in one line.
function checkAllFK(tables, foreignKeys) {
return Promise.all(tables.map(t => Promise.all(foreignKeys.map(k => QueryFunction(t, k))));
}
This function returns a promise that resolves when all queries are done, so you call it like
checkAllFK(tables, foreignKeys)
.then(/* success */)
.catch(/* error */);
However, depending on how many foreignKeys and tables there are and how complex QueryFunction is, this can put enormous stress on the database server. If there are 10 tables and 1000 foreign keys, this would attempt to run 10,000 queries in parallel against the database server. This is not a smart thing to do.
SQL is made to handle these situations. Instead of running 10,000 queries for one thing each, you can decide to run one query for 10,000 things. Or 10 queries for 1000 things each. Both are obviously better than hammering the database server with 10,000 requests.
For example, this returns all foreign keys that do not exist in table_1 in one step.
SELECT
k.key_column
FROM
foreign_keys k
LEFT JOIN table_1 t ON t.key_column = k.key_column
WHERE
t.key_column IS NULL
It depends on what you do in your QueryFunction how the actual SQL needs to look like.
The fact that you have more than one table to check the same foreign keys against is worrying as well, this usually is an indication of poor database design.
there is few common begginer mistakes. Lets start with the tricky one and it is using var keyword in a for-loop in asynchronous context. As you can see, this will return you only 10s, not 1, 2, 3.
for (var i=0; i < 10; i++) {
setTimeout(() => console.log(i), 100);
}
Fix is easy in this case - just use let, which has different scope than var and works as you would expect.
for (let i=0; i < 10; i++) {
setTimeout(() => console.log(i), 100);
}
Second is asynchronous context - the for-cycle ends before you execute the async context inside the promise that is returned by QueryFunction. If you can use newer version of Node.js then async/await is your saviour, just mark your function as async and have
const result = await QueryFunction(tables[index], key, foreignKeys[key])
However be awared - once you have something in Promise/Asynchronous context, you basically cant get back to the synchronous context. So all your logic needs be aware that you are in asynchronous part. That basically means that all the results will be promises and you will need to then them or await them. Its not bug or something, its behaviour and you need to count on that.
you can do same like print a message in the console or write in a file, can get the visual result.
if you want to get a result, use 'wait', please.
I'm trying to map a large array(around 11k items). The actual mapping function is super simple, but the amount of items in the array is just too much and it blocks everything.
What's the best approach to avoid this? I tried using Async map, but I'm getting the same problem.
You can somehow change the sync (map) operation to an async operation using Promise or setTimeout. Recursive function can be used to progressively process the items in large array.
For example:
const largeArrays = [];
const resultArrays = [];
function process(source, target, index) {
if (index === target.length) {
// Now the result Arrays should have all processed data
return
}
// Dummy map action here for example, please change to your own one
target.push(source[index] + 1);
setTimeout(() => { process(source, target, index + 1) }, 0);
}
process(largeArrays, resultArrays, 0)
You can wrap about code into a Promise and resolve it instead of using the return statement above.
You don't need any fancy library, just native javascript function. You can check on two of my blogs illustrating ideas for these kinds of problems.
How to avoid Stack overflow error on recursion
How to make long running loop breakable?
i did not try this but using an async function that handles the mapping part, then call that function in every iteration with necessary information (index, array item etc.) wont help?
I made loop and now I want to use protractor expectation for every i.
Loop works ok, but expectations doesn't. If count is 4, there should be 4 expectations. If I run a test, I get pass, without any expectations(which should be false).
I found articles about that but I couldn't make it happen. I tried with Push, but there is only empty value.
Thanks for help.
myelement.count().then(function(count){
console.log("whatever", count);
for (var i=0; i<count; i++){
var o = location.get(i);
expect(o.getText()).toEqual("something");
};
});
Be aware. Almost all commands are promises. If you don't handle them correct it will never work. You will first need to resolve them before you can go to the next one.
Something like this can do the trick
let promise;
const promises = [];
myelement.count()
.then(function(count) {
console.log("whatever", count);
for (var i = 0; i < count; i++) {
var o = location.get(i);
promises.push(expect(o.getText()).toEqual("something"));
};
return Promise.all(promises);
});
Full test-case code would help, but I assume that you've forgot to either return a promise from 'it' or define and call 'done' callback, since what you're doing - is an async operation.
Your problem descrption is not clear. I'm assuming that location is list of elements. So i'm applying each() on set elements called 'location' . It will work perfectly.
//replace location with myelement if it the correct one
location.each(function(ele,index){
expect(ele.getText()).toEqual("something");
});
I've got a problem with redis and nodejs. I have to loop through a list of phone numbers, and check if this number is present in my redis database. Here is my code :
function getContactList(contacts, callback) {
var contactList = {};
for(var i = 0; i < contacts.length; i++) {
var phoneNumber = contacts[i];
if(utils.isValidNumber(phoneNumber)) {
db.client().get(phoneNumber).then(function(reply) {
console.log("before");
contactList[phoneNumber] = reply;
});
}
}
console.log("after");
callback(contactList);
};
The "after" console log appears before the "before" console log, and the callback always return an empty contactList. This is because requests to redis are asynchronous if I understood well. But the thing is I don't know how to make it works.
How can I do ?
You have two main issues.
Your phoneNumber variable will not be what you want it to be. That can be fixed by changing to a .forEach() or .map() iteration of your array because that will create a local function scope for the current variable.
You have create a way to know when all the async operations are done. There are lots of duplicate questions/answers that show how to do that. You probably want to use Promise.all().
I'd suggest this solution that leverages the promises you already have:
function getContactList(contacts) {
var contactList = {};
return Promise.all(contacts.filter(utils.isValidNumber).map(function(phoneNumber) {
return db.client().get(phoneNumber).then(function(reply) {
// build custom object
constactList[phoneNumber] = reply;
});
})).then(function() {
// make contactList be the resolve value
return contactList;
});
}
getContactList.then(function(contactList) {
// use the contactList here
}, funtion(err) {
// process errors here
});
Here's how this works:
Call contacts.filter(utils.isValidNumber) to filter the array to only valid numbers.
Call .map() to iterate through that filtered array
return db.client().get(phoneNumber) from the .map() callback to create an array of promises.
After getting the data for the phone number, add that data to your custom contactList object (this is essentially a side effect of the .map() loop.
Use Promise.all() on the returned array of promises to know when they are all done.
Make the contactList object we built up be the resolve value of the returned promise.
Then, to call it just use the returned promise with .then() to get the final result. No need to add a callback argument when you already have a promise that you can just return.
The simplest solution may be to use MGET with a list of phone numbers and put the callback in the 'then' section.
You could also put the promises in an array and use Promise.all().
At some point you might want your function to return a promise rather than with callback, just to stay consistent.
Consider refactoring your NodeJS code to use Promises.
Bluebird is an excellent choice: http://bluebirdjs.com/docs/working-with-callbacks.html
you put async code into a for loop (sync operations). So, each iteration of the for loop is not waiting for the db.client(...) function to end.
Take a look at this stackoverflow answer, it explains how to make async loops :
Here