Well the problem is simple, I am using a query within the for loop and and I want to get out of the for loop if I get the count less than 15, otherwise increase the assigned. But I can't be able to use the break statement and the loop will continue to execute even after the first callback.
for (var i = 0; i < test; i++) {
var sql = "SELECT count(*) as count FROM `tb_members` WHERE `assigned`=?";
connection.query(sql, [assigned], function (err, response) {
if (response[0].count < 15) {
callback(assigned);
}
else {
++assigned;
if (i == test - 1) {
callback(0);
}
}
});
}
The way your code is written, all your SQL queries are going to get started at once. Then, sometime later, the queries will start returning with results. So, you can't break out of the for loop because it's already done and all SQL queries have already been sent.
If you want to decide whether to send the next query based on the previous one's results, then you have to only send one at a time and because of the async nature of the results, you can't use a for loop for that.
One way of sending the queries one at a time and then deciding whether to send the next one is this:
function sendQueries() {
var i = 0;
function next() {
if (i < test) {
var sql = "SELECT count(*) as count FROM `tb_members` WHERE `assigned`=?";
connection.query(sql, [assigned], function (err, response) {
i++;
if (response[0].count < 15) {
callback(assigned);
} else {
++assigned;
if (i == test - 1) {
callback(0);
}
}
// here you can decide whether you want to do the next iteration
// or not by either calling next() or not.
next();
});
}
}
next();
}
You should be using async to write such a loop. If you want a serial behavior - you might consider using async.eachSeries, and then you can stop the loop by calling the callback function with error. At that point the loop will not execute anymore. But in the error handler - ignore the err.
Check it out here: https://github.com/caolan/async#collections
Related
I have to check if foreignKey exist, but I can't make a loop with my asynchronous query function
function checkAllFK(tables, foreignKeys) {
let index = -1;
for (var key in foreignKeys) {
index++;
QueryFunction(tables[index], key, foreignKeys[key])
.then(result => {
if(result == null) {
//here, if result is null, that's mean the foreignKey doesn't exist, then, we have to stop the loop and return false;
return false;
}
else if(index == (tables.length - 1)) {
//here, that's mean we are at the end of the loop, and we doesn't break it with the previous if, that's mean all foreignKey exist, then, we return true;
return true;
}
}
the problem is that at the end of the first iteration, you exit the function and the result of the return depends only on the first iteration: false if the if condition is met at the first iteration, null if not
even having looked at many similar topics here, I haven't found a solution to my problem.
Your operation "check all foreignKeys against all tables" can be written in one line.
function checkAllFK(tables, foreignKeys) {
return Promise.all(tables.map(t => Promise.all(foreignKeys.map(k => QueryFunction(t, k))));
}
This function returns a promise that resolves when all queries are done, so you call it like
checkAllFK(tables, foreignKeys)
.then(/* success */)
.catch(/* error */);
However, depending on how many foreignKeys and tables there are and how complex QueryFunction is, this can put enormous stress on the database server. If there are 10 tables and 1000 foreign keys, this would attempt to run 10,000 queries in parallel against the database server. This is not a smart thing to do.
SQL is made to handle these situations. Instead of running 10,000 queries for one thing each, you can decide to run one query for 10,000 things. Or 10 queries for 1000 things each. Both are obviously better than hammering the database server with 10,000 requests.
For example, this returns all foreign keys that do not exist in table_1 in one step.
SELECT
k.key_column
FROM
foreign_keys k
LEFT JOIN table_1 t ON t.key_column = k.key_column
WHERE
t.key_column IS NULL
It depends on what you do in your QueryFunction how the actual SQL needs to look like.
The fact that you have more than one table to check the same foreign keys against is worrying as well, this usually is an indication of poor database design.
there is few common begginer mistakes. Lets start with the tricky one and it is using var keyword in a for-loop in asynchronous context. As you can see, this will return you only 10s, not 1, 2, 3.
for (var i=0; i < 10; i++) {
setTimeout(() => console.log(i), 100);
}
Fix is easy in this case - just use let, which has different scope than var and works as you would expect.
for (let i=0; i < 10; i++) {
setTimeout(() => console.log(i), 100);
}
Second is asynchronous context - the for-cycle ends before you execute the async context inside the promise that is returned by QueryFunction. If you can use newer version of Node.js then async/await is your saviour, just mark your function as async and have
const result = await QueryFunction(tables[index], key, foreignKeys[key])
However be awared - once you have something in Promise/Asynchronous context, you basically cant get back to the synchronous context. So all your logic needs be aware that you are in asynchronous part. That basically means that all the results will be promises and you will need to then them or await them. Its not bug or something, its behaviour and you need to count on that.
you can do same like print a message in the console or write in a file, can get the visual result.
if you want to get a result, use 'wait', please.
I am new in node js / express I am trying to query, foreach id in an array, some data from the sqlite3 database. My for loop looks like as follows
response.patients = results;
for (var i = 0; i < results.length; i++) {
response.patients[i].check = "false";
var patient = response.patients[i];
db.each("SELECT visit_id FROM patient_visits where patient_id='"+patient.id+"' AND visitdate >='"+moment().format('YYYY-MM-DD')+"'", function(err, row) {
if (row) {
response.patients[i].check = "true";
}
});
}
res.send(response);
the problem is that the for loop continues before the query is finished. Is there a way to check if the query has finished?
I only need to set a flag true / false.
The problem is that db.each is a callback function. This function executes and the response takes some time to be retrieved, however the system doesn't stop and the for loop continues. When it ends, res.send is called while the responses from the db.each are still being processed.
You can try
db.all(
SELECT visit_id FROM patient_visits where patient_id IN ?
AND visitdate >='"+moment().format('YYYY-MM-DD')+"'", response.patients,
function(err, rows) {
if (err) {
throw err;
}
rows.forEach((row) => {
response.patients[row.visit_id].check = "true";
});
res.send(response);
});
I'm not sure about the content of row variable to get the visit_id but before print the variable and check if it works.
If you have any doubt let me know. And I strongly advise you to read some content about callbacks and how they work and then promises and async/await because callbacks aren't used now, with async await, your problem would be solved easily.
I use redis client in node.js
var db = require("redis");
var dbclient = db.createClient();
I load the DB in the next way:
dbclient.zrange("cache", -1000000000000000, +1000000000000000, function(err, replies){
logger.info("Go to cache");
for (var i=0; i < replies.length; i++){
(function(i){
// Do some commands with the result
})(i)
}
})
I notice that where my application is started, it takes 30~ sec. for the DB query to execute. In this time, no other request from Express module are served.
How can I slove this issue? Why is no a asynchronous?
If you're worried about the uptime then you should use ZSCAN
Use the COUNT option to get more data on every call but remember:
While SCAN does not provide guarantees about the number of elements returned at every iteration, it is possible to empirically adjust the behavior of SCAN using the COUNT option
And at every result use the setImmediate function to iterate over to the next SCAN.
var cursor = '0';
function doscan(){
dbclient.zscan("cache", cursor, "COUNT", "100", function(err, replies){
logger.info("Go to cache");
// Update the cursor position for the next scan
cursor = res[0];
if (cursor === '0') {
return console.log('Iteration complete');
} else {
for (var i=0; i < replies.length; i++){
(function(i){
// Do some commands with the result
})(i)
}
setImmediate(function() { doscan() });
}
})
}
As stdob said, the only part of your code that is not asynchronous is the for loop part. Personally, I would create a child process and run the DB query as well as whatever you decide to do with the output. If that doesn't work for your application, you may just have to delay starting your program for those 30 seconds or handle the cache differently.
You can use Iced-CoffeeScript for that , like
https://github.com/Terebinth/britvic/blob/master/server/models/hive_iced.iced#L101
I have been working on nodeJS + MongoDB, using the Express and Mongoose frameworks for a few months, and I wanted to ask you guys what is really happening in a situation such as the following:
Model1.find({}, function (err, elems) {
if (err) {
console.log('ERROR');
} else {
elems.forEach(function (el) {
Model2.find({[QUERY RELATED WITH FIELDS IN 'el']}, function (err, elems2) {
if (err) {
console.log('ERROR');
} else {
//DO STAFF.
}
});
});
}
});
My best guess is that there's a main thread looping over elems, and then different threads attending each query over Model2, but I'm not really sure.
Is that correct? And also, is this a good solution? And if not, how would you code in a situation such as this, where you need the information in each of the elements you get from Model1 to get elements from Model2, and perform the actual functionality you are looking for?
I know I could elaborate a more complex query where I could get all the elements each of the 'el' in elems would yield, but I¡d rather not do that, because in that case i would be worried about the memory expense.
Also, I've been thinking about changing the data model, but I've gone over it and I'm confident it is well thought, and I don't think that's the best solution for my aplication.
Thanks!
NodeJS is a single threaded environment and it works asynchronously for blocking function calls such as network requests in your case. So there is only one thread and your query results will be called asynchronously so that nothing will be blocked due to intensive network operation.
In your scenario if the first query returns quite a lot of records such as 100000 thousands you may exhaust your mongo server in your loop as you will query your server as many as the result of first query instantly. This will happen because node won't stop for receiving the results of each query as it works asynchronously.
So usually manually throttling the requests to network operations is a good practice. This is not trivial when working on asynchronous environment. One way to do is to use recursive function call. Basically you split your tasks into groups and do each group in batch, once you are done with one batch you start with your next group.
Here is a simple example on how to do it, I have used promises instead of callback functions, Q is a promise library that is very useful for handling promises:
var rows = [...]; // array of many
function handleRecursively(startIndex, batchSize){
var promises = [];
for(i = 0; i < batchSize && i + batchSize < rows.length; i++){
var theRow = rows[startIndex + i];
promises.push(doAsynchronousJobWithTheRow(theRow));
}
//you wait until you handle all tasks in this iteration
Q.all(promises).then(function(){
startIndex += batchSize;
if(startIndex < rows.length){ // if there is still task to do continue with next batch
handleRecursively(startIndex, batchSize); }
})
}
handleRecursively(0, 1000);
Here is the best solution :
Model1.find({}, function (err, elems) {
if (err) {
console.log('ERROR');
} else {
loopAllElements(0,elems);
}
});
function loopAllElements(startIndex,elems){
if (startIndex==elems.length) {
return "success";
}else{
Model2.find({[QUERY RELATED WITH FIELDS IN elems[startIndex] ]}, function (err, elems2) {
if (err) {
console.log('ERROR');
return "error";
} else {
//DO STAFF.
loopAllElements(startIndex+1, elems);
}
});
}
}
Here is the situation:
I am new to node.js, I have a 40MB file containing multilevel json file like:
[{},{},{}] This is an array of objects (~7000 objects). Each object has properties and a one of those properties is also an array of objects
I wrote a function to read the content of the file and iterate it. I succeeded to get what I wanted in terms of content but not usability. I thought that I wrote an async function that would allow node to serve other web requests while iterating the array but that is not the case. I would be very thankful if anyone can point me to what I've done wrong and how to rewrite it so I can have a non-blocking iteration. Here's the function that handles the situation:
function getContents(callback) {
fs.readFile(file, 'utf8', function (err, data) {
if (err) {
console.log('Error: ' + err);
return;
}
js = JSON.parse(data);
callback();
return;
});
}
getContents(iterateGlobalArr);
var count = 0;
function iterateGlobalArr() {
if (count < js.length) {
innerArr = js.nestedProp;
//iterate nutrients
innerArr.forEach(function(e, index) {
//some simple if condition here
});
var schema = {
//.....get props from forEach iteration
}
Model.create(schema, function(err, post) {
if(err) {
console.log('\ncreation error\n', err);
return;
}
if (!post) {
console.log('\nfailed to create post for schema:\n' + schema);
return;
}
});
count++;
process.nextTick(iterateGlobalArr);
}
else {
console.log("\nIteration finished");
next();
}
Just so it is clear how I've tested the above situation. I open two tabs one loading this iteration which takes some time and second with another node route which does not load until the iteration is over. So essentially I've written a blocking code but not sure how to re-factor it! I suspect that just because everything is happening in the callback I am unable to release the event loop to handle another request...
Your code is almost correct. What you are doing is inadvertently adding ALL the items to the very next tick... which still blocks.
The important piece of code is here:
Model.create(schema, function(err, post) {
if(err) {
console.log('\ncreation error\n', err);
return;
}
if (!post) {
console.log('\nfailed to create post for schema:\n' + schema);
return;
}
});
// add EVERYTHING to the very same next tick!
count++;
process.nextTick(iterateGlobalArr);
Let's say you are in tick A of the event loop when getContents() runs and count is 0. You enter iterateGlobalArr and you call Model.create. Because Model.create is async, it is returning immediately, causing process.nextTick() to add processing of item 1 to the next tick, let's say B. Then it calls iterateGlobalArr, which does the same thing, adding item 2 to the next tick, which is still B. Then item 3, and so on.
What you need to do is move the count increment and process.nextTick() into the callback of Model.create(). This will make sure the current item is processed before nextTick is invoked... which means next item is actually added to the next tick AFTER the model item has been created... which will give your app time to handle other things in between. The fixed version of iterateGlobalArr is here:
function iterateGlobalArr() {
if (count < js.length) {
innerArr = js.nestedProp;
//iterate nutrients
innerArr.forEach(function(e, index) {
//some simple if condition here
});
var schema = {
//.....get props from forEach iteration
}
Model.create(schema, function(err, post) {
// schedule our next item to be processed immediately.
count++;
process.nextTick(iterateGlobalArr);
// then move on to handling this result.
if(err) {
console.log('\ncreation error\n', err);
return;
}
if (!post) {
console.log('\nfailed to create post for schema:\n' + schema);
return;
}
});
}
else {
console.log("\nIteration finished");
next();
}
}
Note also that I would strongly suggest that you pass in your js and counter with each call to iterageGlobalArr, as it will make your iterateGlobalArr alot easier to debug, among other things, but that's another story.
Cheers!
Node is single-threaded so async will only help you if you are relying on another system/subsystem to do the work (a shell script, external database, web service etc). If you have to do the work in Node you are going to block while you do it.
It is possible to create one node process per core. This solution would result in only blocking one of the node processes and leave the rest to service your requests, but this feature is still listed as experimental http://nodejs.org/api/cluster.html.
A single instance of Node runs in a single thread. To take advantage
of multi-core systems the user will sometimes want to launch a cluster
of Node processes to handle the load.
The cluster module allows you to easily create child processes that
all share server ports.