I have to check if foreignKey exist, but I can't make a loop with my asynchronous query function
function checkAllFK(tables, foreignKeys) {
let index = -1;
for (var key in foreignKeys) {
index++;
QueryFunction(tables[index], key, foreignKeys[key])
.then(result => {
if(result == null) {
//here, if result is null, that's mean the foreignKey doesn't exist, then, we have to stop the loop and return false;
return false;
}
else if(index == (tables.length - 1)) {
//here, that's mean we are at the end of the loop, and we doesn't break it with the previous if, that's mean all foreignKey exist, then, we return true;
return true;
}
}
the problem is that at the end of the first iteration, you exit the function and the result of the return depends only on the first iteration: false if the if condition is met at the first iteration, null if not
even having looked at many similar topics here, I haven't found a solution to my problem.
Your operation "check all foreignKeys against all tables" can be written in one line.
function checkAllFK(tables, foreignKeys) {
return Promise.all(tables.map(t => Promise.all(foreignKeys.map(k => QueryFunction(t, k))));
}
This function returns a promise that resolves when all queries are done, so you call it like
checkAllFK(tables, foreignKeys)
.then(/* success */)
.catch(/* error */);
However, depending on how many foreignKeys and tables there are and how complex QueryFunction is, this can put enormous stress on the database server. If there are 10 tables and 1000 foreign keys, this would attempt to run 10,000 queries in parallel against the database server. This is not a smart thing to do.
SQL is made to handle these situations. Instead of running 10,000 queries for one thing each, you can decide to run one query for 10,000 things. Or 10 queries for 1000 things each. Both are obviously better than hammering the database server with 10,000 requests.
For example, this returns all foreign keys that do not exist in table_1 in one step.
SELECT
k.key_column
FROM
foreign_keys k
LEFT JOIN table_1 t ON t.key_column = k.key_column
WHERE
t.key_column IS NULL
It depends on what you do in your QueryFunction how the actual SQL needs to look like.
The fact that you have more than one table to check the same foreign keys against is worrying as well, this usually is an indication of poor database design.
there is few common begginer mistakes. Lets start with the tricky one and it is using var keyword in a for-loop in asynchronous context. As you can see, this will return you only 10s, not 1, 2, 3.
for (var i=0; i < 10; i++) {
setTimeout(() => console.log(i), 100);
}
Fix is easy in this case - just use let, which has different scope than var and works as you would expect.
for (let i=0; i < 10; i++) {
setTimeout(() => console.log(i), 100);
}
Second is asynchronous context - the for-cycle ends before you execute the async context inside the promise that is returned by QueryFunction. If you can use newer version of Node.js then async/await is your saviour, just mark your function as async and have
const result = await QueryFunction(tables[index], key, foreignKeys[key])
However be awared - once you have something in Promise/Asynchronous context, you basically cant get back to the synchronous context. So all your logic needs be aware that you are in asynchronous part. That basically means that all the results will be promises and you will need to then them or await them. Its not bug or something, its behaviour and you need to count on that.
you can do same like print a message in the console or write in a file, can get the visual result.
if you want to get a result, use 'wait', please.
Related
I have an array called noNCreatedResources. I want do some operation on each item of array and push item in createdResources array and remove the item from noNCreatedResources array and continue to do that until noNCreatedResources be empty. For this I've written CreateResources function including nested while and for loop. It works fine but i realize that it don't work synchronously. For example: it must iterate twice in while loop but iterates 4 times and I don't know why.
I think I don't understand concept of async/await/non-blocking concept of node.js. can any body help me to realize what the problem is?
CreateResources = async () => {
while (this.noNCreatedResources.length > 0) {
for (let index = 0; index < this.noNCreatedResources.length; index++) {
if (this.resourceHasCreatedDependencies(this.noNCreatedResources[index])) {
const resourceModel = this.someOperation(this.noNCreatedResources[index]);
this.createdResources.push(resourceModel);
this.noNCreatedResources.splice(index, 1);
}
}
}
}
First of all you are not doing anything asynchronous in you function so you can remove the async keyword from your function. Since you are not doing anything asynchronous so, your problem is not related to it. It is more of an implementation problem IMO.
Your while loop is useless for what you are trying to achieve. Also, your logic is broken!
Example: The following code will output 1, 3, and 5.
let x = [1,2,3,4,5];
for(let i = 0; i < x.length; i++) {
console.log(x[i]);
x.splice(i, 1);
}
I do not think you need to remove item from array to achieve your expected result. If you need to reset the array then at the end you can just do this x = [] to reset the array.
The problem you have is not due to async calls. Actually, your code is entirely synchronous. Try to take a look at where the "noNCreatedResources" is been created/updated. Async calls happens when you're sending a http request, reading a file etc, in other words, operations that doesn't happens inside your code. It allows the code to go on, not blocking the next function calls, and when the promise is fulfilled, the callback function is invoked.
I have complex solution and I just need to run knex synchronously, is it possible?
I have scenario when knex query is run inside Promise.mapSeries for array with unknown number of elements. For each element some knex query is called, including insert query.
So, this insert could affect result for the next element of array.
var descriptionSplitByCommas = desc.split(",");
Promise.mapSeries(descriptionSplitByCommas, function (name) {
// knex.select
// knex.insert if select doesn't return results
});
This was not my initial code, so maybe even Promise.mapSeries should be removed. But I need each descriptionSplitByCommas array elements to be processed syncrhonously.
Otherwise often while processing next description in array I get SQL error, because of duplicate elements inserted for column with unique index. This would not happen if query would be synchronous.
I am using native promises, so I do not have experience with mapSeries, therefore I cannot tell you what exactly is going on at current state.
However running several asynchronous commands in series instead of parallel is quite common. There is one important thing, you have to know - once you create Promise, you do not have control about how and when it will be resolved. So if you create 100 Promises, they all start resolving in parallel.
This is the reason, there is no method for native promises like Promise.series - it is not possible.
What are your options? If you need to "create promise at one place, but run it in another", then factory method is your friend:
const runPromiseLater = () => Promise.resolve(25);
// some code
const myRealPromise = runPromiseLater();
myRealPromise.then( //
Of course, you can create array with these methods, then is question - how to run it in series?
If you can use Node with support for async/await, then for cycle is good enough
async function runInSeries(array) {
for (let i=0;i < array.length; i++){
await array[i]();
// or if you have only instructions in array then you get the value and then call some // await myMethod(array[i])
}
}
If you cant use that, then async library is your friend: https://caolan.github.io/async/docs.html#series
If you need to use the value from previous calls, you can use .waterfall
Okay so below is a snippet of my code where I have cut many unnecessary things and unrelated but I have left the part dealing with the question.
I am using callbacks while calling the functions needed to run the necessary queries. Since I have many queries like these below, I was wondering if thats the right way to ensure the wanted order for the queries to be executed. I know I could remove the functions and simply put them inside a serialize but its really ugly to repeat the same code so I put them in functions, to put it more clear here is my question.
Question: If I have many queries inside functions the correct way to ensure the get executed in the wanted order is with callbacks as I have done?, even in cases where you dont want to return anything e.g (when updating a row/table in the DB)
get_data(pel, function(results){
var cntl = results;
get_user(pel, function(results_from_user){
update_data(0, 0, function(cb_result){
//do some stuff
});
});
});
function get_data(dt, callback)
{
db.get(`SELECT * FROM my_table`, function(error, row) {
var data_to_return = [..];
return callback(data_to_return);
});
}
function update_data(vdr,dwe,callback)
{
db.run(`UPDATE my_table SET val1='${..}', val2 = '${..}'`);
//..
return callback("updated");
}
function get_user(ms, callback)
{
db.get(`SELECT id FROM my_table_2 WHERE id=${..};`, function(error, row) {
if(row == undefined) db.run(`INSERT INTO my_table_2 (id) VALUES (?)`,[0]);
//..
var id_to_return = [..];
return callback(id_to_return);
});
}
perhaps I should add my code is working as expected, I am just making sure I am not using a weird way.
I can ensure you that you have made a typical solution. in fact callback are used to wait for the response before moving on to the next statement.Goog job
I have an orientdb database. I want to use nodejs with RESTfull calls to create a large number of records. I need to get the #rid of each for some later processing.
My psuedo code is:
for each record
write.to.db(record)
when the async of write.to.db() finishes
process based on #rid
carryon()
I have landed in serious callback hell from this. The version that was closest used a tail recursion in the .then function to write the next record to the db. However, I couldn't carry on with the rest of the processing.
A final constraint is that I am behind a corporate proxy and cannot use any other packages without going through the network administrator, so using the native nodejs packages is essential.
Any suggestions?
With a completion callback, the general design pattern for this type of problem makes use of a local function for doing each write:
var records = ....; // array of records to write
var index = 0;
function writeNext(r) {
write.to.db(r, function(err) {
if (err) {
// error handling
} else {
++index;
if (index < records.length) {
writeOne(records[index]);
}
}
});
}
writeNext(records[0]);
The key here is that you can't use synchronous iterators like .forEach() because they won't iterate one at a time and wait for completion. Instead, you do your own iteration.
If your write function returns a promise, you can use the .reduce() pattern that is common for iterating an array.
var records = ...; // some array of records to write
records.reduce(function(p, r) {
return p.then(function() {
return write.to.db(r);
});
}, Promsise.resolve()).then(function() {
// all done here
}, function(err) {
// error here
});
This solution chains promises together, waiting for each one to resolve before executing the next save.
It's kinda hard to tell which function would be best for your scenario w/o more detail, but I almost always use asyncjs for this kind of thing.
From what you say, one way to do it would be with async.map:
var recordsToCreate = [...];
function functionThatCallsTheApi(record, cb){
// do the api call, then call cb(null, rid)
}
async.map(recordsToCreate, functionThatCallsTheApi, function(err, results){
// here, err will be if anything failed in any function
// results will be an array of the rids
});
You can also check out other ones to enable throttling, which is probablya good idea.
I'm new to Node.js and am curious what the prescribed methodology is for running a loop on a process (repeatedly) where at the end of the execution some next step is to take place, but ONLY after all the iterations' callbacks have fired.
Specifically I'm making SQL calls and I need to close the sql connection after making a bunch of inserts and updates, but since they're all asynchronous, I have no way of knowing when all of them have in fact completed, so that I can call end() on the session.
Obviously this is a problem that extends far beyond this particular example, so, I'm not looking for the specific solution regarding sql, but more the general practice, which so far, I'm kind of stumped by.
What I'm doing now is actually setting a global counter to the length of the loop object and decrementing from it in each callback to see when it reaches zero, but that feels REALLY klugy, and I'm hoping theres a more elegant (and Javascript-centric) way to achieve this monitoring.
TIA
There are a bunch of flow-control libraries available that apply patterns to help with this kind of thing. My favorite is async. If you wanted to run a bunch of SQL queries one after another in order, for instance, you might use series:
async.series([
function(cb) { sql.exec("SOME SQL", cb) },
function(cb) { sql.exec("SOME MORE SQL", cb) },
function(cb) { sql.exec("SOME OTHER SQL", cb) }
], function(err, results) {
// Here, one of two things are true:
// (1) one of the async functions passed in an error to its callback
// so async immediately calls this callback with a non-null "err" value
// (2) all of the async code is done, and "results" is
// an array of each of the results passed to the callbacks
});
I wrote my own queue library to do this (I'll publish it one of these days), basically push queries onto a queue (an array basically) execute each one as it's removed, have a callback take place when the array is empty.
It doesn't take much to do it.
*edit. I've added this example code. It isn't what I've used before and I haven't tried it in practice, but it should give you a starting point. There's a lot more you can do with the pattern.
One thing to note. Queueing effectively makes your actions synchronous, they happen one after another. I wrote my mysql queue script so I could execute queries on multiple tables asynchronously but on any one table in synch, so that inserts and selects happened in the order they were requested.
var queue = function() {
this.queue = [];
/**
* Allows you to pass a callback to run, which is executed at the end
* This example uses a pattern where errors are returned from the
* functions added to the queue and then are passed to the callback
* for handling.
*/
this.run = function(callback){
var i = 0;
var errors = [];
while (this.queue.length > 0) {
errors[errors.length] = this.queue[i]();
delete this.queue[i];
i++;
}
callback(errors);
}
this.addToQueue = function(callback){
this.queue[this.queue.length] = callback;
}
}
use:
var q = new queue();
q.addToQueue(function(){
setTimeout(function(){alert('1');}, 100);
});
q.addToQueue(function(){
setTimeout(function(){alert('2');}, 50);
});
q.run();