I use redis client in node.js
var db = require("redis");
var dbclient = db.createClient();
I load the DB in the next way:
dbclient.zrange("cache", -1000000000000000, +1000000000000000, function(err, replies){
logger.info("Go to cache");
for (var i=0; i < replies.length; i++){
(function(i){
// Do some commands with the result
})(i)
}
})
I notice that where my application is started, it takes 30~ sec. for the DB query to execute. In this time, no other request from Express module are served.
How can I slove this issue? Why is no a asynchronous?
If you're worried about the uptime then you should use ZSCAN
Use the COUNT option to get more data on every call but remember:
While SCAN does not provide guarantees about the number of elements returned at every iteration, it is possible to empirically adjust the behavior of SCAN using the COUNT option
And at every result use the setImmediate function to iterate over to the next SCAN.
var cursor = '0';
function doscan(){
dbclient.zscan("cache", cursor, "COUNT", "100", function(err, replies){
logger.info("Go to cache");
// Update the cursor position for the next scan
cursor = res[0];
if (cursor === '0') {
return console.log('Iteration complete');
} else {
for (var i=0; i < replies.length; i++){
(function(i){
// Do some commands with the result
})(i)
}
setImmediate(function() { doscan() });
}
})
}
As stdob said, the only part of your code that is not asynchronous is the for loop part. Personally, I would create a child process and run the DB query as well as whatever you decide to do with the output. If that doesn't work for your application, you may just have to delay starting your program for those 30 seconds or handle the cache differently.
You can use Iced-CoffeeScript for that , like
https://github.com/Terebinth/britvic/blob/master/server/models/hive_iced.iced#L101
Related
I am new in node js / express I am trying to query, foreach id in an array, some data from the sqlite3 database. My for loop looks like as follows
response.patients = results;
for (var i = 0; i < results.length; i++) {
response.patients[i].check = "false";
var patient = response.patients[i];
db.each("SELECT visit_id FROM patient_visits where patient_id='"+patient.id+"' AND visitdate >='"+moment().format('YYYY-MM-DD')+"'", function(err, row) {
if (row) {
response.patients[i].check = "true";
}
});
}
res.send(response);
the problem is that the for loop continues before the query is finished. Is there a way to check if the query has finished?
I only need to set a flag true / false.
The problem is that db.each is a callback function. This function executes and the response takes some time to be retrieved, however the system doesn't stop and the for loop continues. When it ends, res.send is called while the responses from the db.each are still being processed.
You can try
db.all(
SELECT visit_id FROM patient_visits where patient_id IN ?
AND visitdate >='"+moment().format('YYYY-MM-DD')+"'", response.patients,
function(err, rows) {
if (err) {
throw err;
}
rows.forEach((row) => {
response.patients[row.visit_id].check = "true";
});
res.send(response);
});
I'm not sure about the content of row variable to get the visit_id but before print the variable and check if it works.
If you have any doubt let me know. And I strongly advise you to read some content about callbacks and how they work and then promises and async/await because callbacks aren't used now, with async await, your problem would be solved easily.
Is it possible to set a Limit to parallel running processes with async.parallelLimit ? I used following very simple code to test how it works.
var async = require("async");
var i = 0;
function write() {
i++;
console.log("Done", i);
}
async.parallelLimit([
function(callback) {
write();
callback();
}
], 10, function() {
console.log("finish");
});
Of course everything what I got back was this:
Done 1
finish
In my case I have a function wich is called very often, but I only want it to run only 5 times simultaneously. The other calls should be queued. (Yes I know about async.queue but, this is not what I want in this question).
So now the question is, is this possible with async.parallelLimit?
//EDIT:
I actually have something similar to this:
for(var i = 0; i < somthing.length; i++) { //i > 100 for example
write();
}
And 'cause of the non synchronicity of node.js this could run 100 times at the same time. But how shell I limit the parallel running processes in this case?
Very short answer; Yes. That's exactly what asyncParallelLimit does.
In your case, you are passing only one function to parallelLimit. That's why it only get's called once. If you were to pass an array with this same function many times, it will get executed as many times as you put it in the array.
Please note that your example function doesn't actually do any work asynchronously. As such, this example function will always get executed in series. If you have a function that does async work, for example a network request or file i/o, it will get executed in parallel.
A better example-function for a async workload would be:
function(callback) {
setTimeout(function(){
callback();
}, 200);
}
For completion, to add to the existing answer, if you want to run the same function multiple times in parallel with a limit, here's how you do it:
// run 'my_task' 100 times, with parallel limit of 10
var my_task = function(callback) { ... };
var when_done = function(err, results) { ... };
// create an array of tasks
var async_queue = Array(100).fill(my_task);
async.parallelLimit(async_queue, 10, when_done);
In a Node.js program that asynchronously handles lines of input constantly coming in from stdin, how can I ensure the asynchronous handlers print their results in the same order the inputs came in?
SSCCE program.js (dependency: npm install split):
var executeCommand = function(line) {
setTimeout(function() { console.log(line); }, 1000 * Math.random());
};
var split = require("split");
process.stdin.pipe(split("\n")).on("data", function(line) {
executeCommand(line);
});
Running printf "A\nB\nC\nD\nE\nF" | node program.js produces
B
E
A
D
C
F
This is because the handler (executeCommand) has an unpredictable delay, modelled here as a random setTimeout. The "processing" (the setTimeouts) should happen concurrently, but their outputs (console.logs) should be in the same order as the constantly incoming inputs.
How can I make that happen?
I'd usually just exclaim "It's Async.js time!", but this time I can't see an appropriate existing helper: Since tasks are constantly coming in, anything that operates on a fixed collection of inputs won't cut it.
I figured it out.
As #Peter and #jfriend pointed out, handler results must be queued to a queue that only allows dequeueing completed tasks. A good time to check for finished tasks is whenever a handler completes.
A picture might clarify how it works:
Turns out a transform stream is a nice way to model that. ("Stuff comes in and eventually stuff related to the incoming stuff comes out" is pretty much the description of a transform stream.) Whenever results finish, completed tasks are pushed.
Here's the question's example, modified to work:
var Transform = require("stream").Transform;
var split = require("split");
var orderedParallel = function(worker) {
var s = new Transform({ objectMode : true });
var resultsQueue = [];
var sendFinishedFromQueue = function() {
while (resultsQueue[0] && resultsQueue[0].done) {
s.push(resultsQueue.shift().data);
}
}
s._transform = function(chunk, encoding, callback) {
var resultObject = { done : false, data : null };
resultsQueue.push(resultObject);
worker(chunk, function(result) {
resultObject.data = result;
resultObject.done = true;
sendFinishedFromQueue();
});
callback();
};
s._flush = function(callback) {
// Do nothing.
//
// We don't have anything to flush, because as workers complete,
// they'll handle sending any and all messages we're allowed to send
// right now.
};
return s;
};
var executeCommand = function(line, cb) {
setTimeout(function() { cb(line); }, 1000 * Math.random());
};
process.stdin.pipe(split("\n")).pipe(orderedParallel(executeCommand))
.on("data", function(x) { console.log(x); });
To convince yourself it works, try a hundred parallel tasks:
for (( i=0; i<=100; i++ ))
do
echo "$i"
done | node program.js
They should complete in parallel (within 1 second at random), but come out of the orderedParallel transform stream in order regardless.
async.queue with limit of 1 and where the worker function both executes the command and prints the results will do it. You won't have optimal concurrency, but it will behave correctly, so I suggest coding that even if it's a stepping stone. Keeping the correct behavior but adding some concurrency will require both queueing the main work function but also some buffering of output in the case of output2 being ready before output1 arrives.
I have a problem in a nodeJS app with mongoDB, i'm trying to do a forum and for each topic i want a button to display every sub topics.
So i need to get everything in the request:
One array with main topics
Another map array with ['topic1'] containing sub topics
Without the mapping (not an actual problem) i have this:
Post.find({'path': path})
.exec(function (err, posts){
if(err)
console.log("Get post list:" + err);
else
{
var sub_posts = new Array; // Second array with sub topics
for (var i = 0; posts[i]; i++) //Here is the loop for each topic
{
var tmp_path = ... // Path and next request works
Post.find({'path': tmp_path}) // Here is the second request
.exec(function(err, bis_posts) {
if (err) console.log('Error loading subforum');
else sub_posts.push(bis_posts); // Affectation fail !!!
})
}
res.render(... 'post_list': posts, 'sub_posts': sub_posts); // Send result
}
})}
So i get it's a scope problem and i should use callback but with the loop i can't resolve this problem.
Sorry for my english and thanks for your answers !
I have no idea what you mean by "affectation fail", but it looks like you're calling res.render too early — callbacks are invoked asynchronously after your current context finishes executing, so when you call res.render(...) after your for loop has finished, your Post.find(...)... operations still haven't finished and their callbacks haven't been invoked, so sub_posts will be still empty.
My node.js and Mongo are rusty, so perhaps this isn't the canonical way to do it, but I'd add a counter to track the state of the pending requests and only call res.render when all subposts have been fetched:
var sub_posts = new Array;
var pending = 0;
for (var i = 0; posts[i]; i++)
{
var tmp_path = ...
Post.find({'path': tmp_path})
.exec(function(err, bis_posts) {
if (err) console.log('Error loading subforum');
else sub_posts.push(bis_posts);
pending -= 1;
if (!pending) {
// all pending subpost lookups finished, render the response:
res.render(... 'post_list': posts, 'sub_posts': sub_posts);
}
});
pending += 1;
}
Well the problem is simple, I am using a query within the for loop and and I want to get out of the for loop if I get the count less than 15, otherwise increase the assigned. But I can't be able to use the break statement and the loop will continue to execute even after the first callback.
for (var i = 0; i < test; i++) {
var sql = "SELECT count(*) as count FROM `tb_members` WHERE `assigned`=?";
connection.query(sql, [assigned], function (err, response) {
if (response[0].count < 15) {
callback(assigned);
}
else {
++assigned;
if (i == test - 1) {
callback(0);
}
}
});
}
The way your code is written, all your SQL queries are going to get started at once. Then, sometime later, the queries will start returning with results. So, you can't break out of the for loop because it's already done and all SQL queries have already been sent.
If you want to decide whether to send the next query based on the previous one's results, then you have to only send one at a time and because of the async nature of the results, you can't use a for loop for that.
One way of sending the queries one at a time and then deciding whether to send the next one is this:
function sendQueries() {
var i = 0;
function next() {
if (i < test) {
var sql = "SELECT count(*) as count FROM `tb_members` WHERE `assigned`=?";
connection.query(sql, [assigned], function (err, response) {
i++;
if (response[0].count < 15) {
callback(assigned);
} else {
++assigned;
if (i == test - 1) {
callback(0);
}
}
// here you can decide whether you want to do the next iteration
// or not by either calling next() or not.
next();
});
}
}
next();
}
You should be using async to write such a loop. If you want a serial behavior - you might consider using async.eachSeries, and then you can stop the loop by calling the callback function with error. At that point the loop will not execute anymore. But in the error handler - ignore the err.
Check it out here: https://github.com/caolan/async#collections