Greeting all,
I want to call a function repeatedly, but wanted each call to run only when the previous call is completed. Does the Async's whilst fit what I need? Or do the calls happen in parallel?
Thanks!
Gary
Whilst will do what you need, it runs each function in series. Before each run it will do the "test" function to make sure it should run again.
Their example:
var count = 0;
async.whilst(
function () { return count < 5; },
function (callback) {
count++;
setTimeout(callback, 1000);
},
function (err) {
// 5 seconds have passed
}
);
As Chad noted, Async's whilst will do the job.
You may want to consider Async's until (inverse of whilst). Both do the same job however the key difference is:
async.whilst will call the function each time the test passes
async.until will call the function each time the test fails
Async's whilst should do the trick for you. Please note the version of async you will be using before referring to the code on the accepted answers. As one of the comments in the accepted answer suggests, the structure of this loop has slightly changed which might not be very easy to spot.
Async v2: Accepted Answer
Async v3: https://caolan.github.io/async/v3/docs.html#whilst
Their Example:
var count = 0;
async.whilst(
function test(cb) { cb(null, count < 5); },
function iter(callback) {
count++;
setTimeout(function() {
callback(null, count);
}, 1000);
},
function (err, n) {
// 5 seconds have passed, n = 5
}
);
Related
I have a function that processes an array of data (first parameter) and, once the procesing is finished, it invokes only one time a callback function (second parameter). I'm using forEach to process data item by item, consisting the processing of each item in some checkings and storing the param in database. The function storeInDB() does the storing work and uses a callback (second parameter) when the item has been stored.
A first approach to the code is the following:
function doWork(data, callback) {
data.forEach(function (item) {
// Do some check on item
...
storeInDB(item, function(err) {
// check error etc.
...
callback();
});
});
}
However, it's wrong, as the the callback function will be invoked several times (as many as element in the data array).
I'd like to know how to refactor my code in order to achieve the desired behaviour, i.e. only one invocation to callback once the storing work is finished. I guess that async could help in this task, but I haven't find the right pattern yet to combine async + forEach.
Any help is appreciated!
You can use a library such as async to do this, although I would recommend using promises if possible. For your immediate problem you can use a counter to determine how many storage calls have completed and call the callback when the total number are completed.
let counter = 0;
data.forEach(function (item) {
// Do some check on item
...
storeInDB(item, function(err) {
// check error etc.
counter++
if (counter == data.length) {
callback();
}
});
});
you can also utilize the three parameters passed to the function to execute on each array method
function doWork(data, callback) {
data.forEach(function (value,idx,arr) {
// Do some check on item
...
storeInDB(arr[idx], function(err) {
// check error etc.
...
if ( (idx + 1) === arr.length ) {
callback();
}
});
});
}
If storeInDB function returns a promise, you can push all async functions to an array and use Promise.all. After all tasks run successfully, It will invokes callback function.
Hope this helps you.
function doWork(data, callback) {
let arr = [];
data.map(function(itm) {
// Do some check on item
...
arr.push(storeInDB(item));
});
Promise.all(arr)
.then(function(res) {
callback();
});
}
I have a function which I need to call a number of time and instead of using a for loop I'm using async.whilst. But what I need is that the next call to function is not made before the previous call completes, which is not what's happening with async.whilst. Is there a way to implement this (I'm using setTimeout to pause between each call but it is not very clean).
Many thanks, C
i'd use the forever construct. Assuming your function's name is myFunction and accepts as parameter a callback:
var count = 0;
var limit = 10; // set as number of the execution of the function
async.forever(
function(next) {
myFunction(function () {
count++;
if(count < limit) {
next();
} else {
next(true);
}
})
},
function(ended) {
// function calling iteration ended
}
);
Is it possible to set a Limit to parallel running processes with async.parallelLimit ? I used following very simple code to test how it works.
var async = require("async");
var i = 0;
function write() {
i++;
console.log("Done", i);
}
async.parallelLimit([
function(callback) {
write();
callback();
}
], 10, function() {
console.log("finish");
});
Of course everything what I got back was this:
Done 1
finish
In my case I have a function wich is called very often, but I only want it to run only 5 times simultaneously. The other calls should be queued. (Yes I know about async.queue but, this is not what I want in this question).
So now the question is, is this possible with async.parallelLimit?
//EDIT:
I actually have something similar to this:
for(var i = 0; i < somthing.length; i++) { //i > 100 for example
write();
}
And 'cause of the non synchronicity of node.js this could run 100 times at the same time. But how shell I limit the parallel running processes in this case?
Very short answer; Yes. That's exactly what asyncParallelLimit does.
In your case, you are passing only one function to parallelLimit. That's why it only get's called once. If you were to pass an array with this same function many times, it will get executed as many times as you put it in the array.
Please note that your example function doesn't actually do any work asynchronously. As such, this example function will always get executed in series. If you have a function that does async work, for example a network request or file i/o, it will get executed in parallel.
A better example-function for a async workload would be:
function(callback) {
setTimeout(function(){
callback();
}, 200);
}
For completion, to add to the existing answer, if you want to run the same function multiple times in parallel with a limit, here's how you do it:
// run 'my_task' 100 times, with parallel limit of 10
var my_task = function(callback) { ... };
var when_done = function(err, results) { ... };
// create an array of tasks
var async_queue = Array(100).fill(my_task);
async.parallelLimit(async_queue, 10, when_done);
I'm sorry if this is a basic question, but I am trying to implement a program in node.js that should wait for the value of a variable available trough a request to a cloud api (photon.variable()) to be 1. This variable should not be requested more than once per second. My first attempt is included in the sample code below. Despite knowing it does not work at all, I think it could be useful to show the functionality I would like to implement.
var photondata = 0;
while (photondata < 1)
{
setTimeout(function () {
photon.variable("witok", function(err, data) {
if (!err) {
console.log("data: ", data.result);
photondata = data.result;
}
else console.log(err);
})}, 1000);
}
Since you couldn't do async stuff in loops before, the traditional approach would be to create a function that adds itself to setTimeout for as long as needed, then calls some other function when it's done. You still need to do this in the browser if not using Babel.
These days, you can stop execution and wait for things to happen when using a generator function (which latest versions of Node now support). There are many libraries that will let you do this and I will advertise ours :)
CL.run(function* () {
var photondata = 0;
while (true) {
yield CL.try(function* () {
var data = yield photon.variable("witok", CL.cb());
console.log("data: ", data.result);
photondata = data.result;
}, function* (err) {
console.log(err.message);
});
if (photondata >= 1) break;
yield CL.sleep(1000);
}
// do whatever you need here
});
I want to stop of executing of my async.queue after first task error was occurred. I need to perform several similar actions in parallel with the concurrency restriction, but stop all the actions after first error. How can I do that or what should I use instead?
Assuming you fired 5 parallel functions, each will take 5 seconds. While in 3rd second, function 1 failed. Then how you can stop the execution of the rest?
It depends of what those functions do, you may poll using setInterval. However if your question is how to stop further tasks to be pushed to the queue. You may do this:
q.push(tasks, function (err) {
if (err && !called) {
//Will prevent async to push more tasks to the queue, however please note that
//whatever pushed to the queue, it will be processed anyway.
q.kill();
//This will not allow double calling for the final callback
called = true;
//This the main process callback, the final callback
main(err, results);
}
});
Here a full working example:
var async = require('async');
/*
This function is the actual work you are trying to do.
Please note for example if you are running child processes
here, by doing q.kill you will not stop the execution
of those processes, so you need actually to keep track the
spawned processed and then kill them when you call q.kill
in 'pushCb' function. In-case of just long running function,
you may poll using setInterval
*/
function worker(task, wcb) {
setTimeout(function workerTimeout() {
if (task === 11 || task === 12 || task === 3) {
return wcb('error in processing ' + task);
}
wcb(null, task + ' got processed');
}, Math.floor(Math.random() * 100));
}
/*
This function that will push the tasks to async.queue,
and then hand them to your worker function
*/
function process(tasks, concurrency, pcb) {
var results = [], called = false;
var q = async.queue(function qWorker(task, qcb) {
worker(task, function wcb(err, data) {
if (err) {
return qcb(err); //Here how we propagate error to qcb
}
results.push(data);
qcb();
});
}, concurrency);
/*
The trick is in this function, note that checking q.tasks.length
does not work q.kill introduced in async 0.7.0, it is just setting
the drain function to null and the tasks length to zero
*/
q.push(tasks, function qcb(err) {
if (err && !called) {
q.kill();
called = true;
pcb(err, results);
}
});
q.drain = function drainCb() {
pcb(null, results);
}
}
var tasks = [];
var concurrency = 10;
for (var i = 1; i <= 20; i += 1) {
tasks.push(i);
}
process(tasks, concurrency, function pcb(err, results) {
console.log(results);
if (err) {
return console.log(err);
}
console.log('done');
});
async documentation on github page is either outdated or incorrect, while inspecting the queue object returned by async.queue() method I do not see the method kill().
Nevertheless there is a way around it. Queue object has property tasks which is an array, simply assigning a reference to an empty array did the trick for me.
queue.push( someTasks, function ( err ) {
if ( err ) queue.tasks = [];
});