For example, I am writing a random generator with crypto.randomBytes(...) along with another async functions. To avoiding fall in callback hell, I though I could use the sync function of crypto.randomBytes. My doubt is if I do that my node program will stop each time I execute the code?. Then I thought if there are a list of async functions which their time to run is very short, these could work as synchronous function, then developing with this list of functions would be easy.
Using the mz module you can make crypto.randomBytes() return a promise. Using await (available in Node 7.x using the --harmony flag) you can use it like this:
let crypto = require('mz/crypto');
async function x() {
let bytes = await crypto.randomBytes(4);
console.log(bytes);
}
x();
The above is nonblocking even though it looks like it's blocking.
For a better demonstration consider this example:
function timeout(time) {
return new Promise(res => setTimeout(res, time));
}
async function x() {
for (let i = 0; i < 10; i++) {
console.log('x', i);
await timeout(2000);
}
}
async function y() {
for (let i = 0; i < 10; i++) {
console.log('y', i);
await timeout(3000);
}
}
x();
y();
And note that those two functions take a lot of time to execute but they don't block each other.
Run it with Node 7.x using:
node --harmony script-name.js
Or with Node 8.x with:
node script-name.js
I show you those examples to demonstrate that it's not a choice of async with callback hell and sync with nice code. You can actually run async code in a very elegant manner using the new async function and await operator available in ES2017 - it's good to read about it because not a lot of people know about those features.
They're asynchronous, learn to deal with it.
Promises now, and in the future ES2017's await and async will make your life a lot easier.
Bluebirds promisifyAll is extremely useful when dealing with any standard Node.js callback API. It adds functions tagged with Async that return a promise instead of requiring a callback.
const Promise = require('bluebird')
const crypto = Promise.promisifyAll(require('crypto'))
function randomString() {
return crypto.randomBytesAsync(4).then(bytes => {
console.log('got bytes', bytes)
return bytes.toString('hex')
})
}
randomString()
.then(string => console.log('string is', string))
.catch(error => console.error(error))
I am trying to do a for loop in a promise but unfortunatly the output that comes out it is not what i am expecting:
My code
var ahaha = function mytestfunction(numb){
return new Promise(function(resolve, reject) {
console.log(numb);
return resolve('test');
})
.then(function(x) {
z='firststep' + x ;
console.log(z);
return z;
})
.then(function(z) {
console.log(z + 'secondstep');
return z
})
.then(function(z) {
console.log(z + 'thirdstep')
});
};
var promises = [];
for (var i = 1; i <= 2; i++) {
promises.push(ahaha(i));
}
Promise.all(promises)
.then(function(data){ console.log("everything is finished")});
What it returns is :
1
2
firststeptest
firststeptest
firststeptestsecondstep
firststeptestsecondstep
firststeptestthirdstep
firststeptestthirdstep
everything is finished
But i want it to return
1
firststeptest
firststeptestsecondstep
firststeptestthirdstep
2
firststeptest
firststeptestsecondstep
firststeptestthirdstep
everything is finished
I don't understand why the promises are not chained one after the other one.
Note that i succeed doing this operation using async.waterfall but i also want to know how to do it with promises.
Thanks a lot
Promise.all() is purposely used when you want things to run in parallel and it will tell you when all are done and they may each finish in any order.
There are lots of different ways to sequence things using promises. If you just have two function calls like your code shows, you can just do them manually:
ahaha(1).then(result => ahaha(2)).then(data => {
console.log("everything finished");
});
Or, a common pattern using .reduce():
[1,2].reduce(p, val => {
return p.then(() => ahaha(val));
}, Promise.resolve()).then(data => {
// everything done here
});
Or, my favorite using the Bluebird promise library:
Promise.mapSeries([1,2], ahaha).then(result => {
// everything done here
});
There are many other schemes, which you can see in these other answers:
How to synchronize a sequence of promises?
JavaScript: Perform a chain of promises synchronously
ES6 Promises - something like async.each?
How can I execute shell commands in sequence?
Can Promise load multi urls in order?
Promise.all is used to run the promises in parallel, not sequentially.
Using the popular BlueBird library, you could have used reduce but there's no equivalent function in standard ES6 but you can do this:
promises.reduce(
(p, next) => p.then(next),
Promise.resolve()
).then(data=>{ console.log("everything is finished")});
I have restructured my code to promises, and built a wonderful long flat promise chain, consisting of multiple .then() callbacks. In the end I want to return some composite value, and need to access multiple intermediate promise results. However the resolution values from the middle of the sequence are not in scope in the last callback, how do I access them?
function getExample() {
return promiseA(…).then(function(resultA) {
// Some processing
return promiseB(…);
}).then(function(resultB) {
// More processing
return // How do I gain access to resultA here?
});
}
Break the chain
When you need to access the intermediate values in your chain, you should split your chain apart in those single pieces that you need. Instead of attaching one callback and somehow trying to use its parameter multiple times, attach multiple callbacks to the same promise - wherever you need the result value. Don't forget, a promise just represents (proxies) a future value! Next to deriving one promise from the other in a linear chain, use the promise combinators that are given to you by your library to build the result value.
This will result in a very straightforward control flow, clear composition of functionalities and therefore easy modularisation.
function getExample() {
var a = promiseA(…);
var b = a.then(function(resultA) {
// some processing
return promiseB(…);
});
return Promise.all([a, b]).then(function([resultA, resultB]) {
// more processing
return // something using both resultA and resultB
});
}
Instead of the parameter destructuring in the callback after Promise.all that only became available with ES6, in ES5 the then call would be replaced by a nifty helper method that was provided by many promise libraries (Q, Bluebird, when, …): .spread(function(resultA, resultB) { ….
Bluebird also features a dedicated join function to replace that Promise.all+spread combination with a simpler (and more efficient) construct:
…
return Promise.join(a, b, function(resultA, resultB) { … });
ECMAScript Harmony
Of course, this problem was recognized by the language designers as well. They did a lot of work and the async functions proposal finally made it into
ECMAScript 8
You don't need a single then invocation or callback function anymore, as in an asynchronous function (that returns a promise when being called) you can simply wait for promises to resolve directly. It also features arbitrary control structures like conditions, loops and try-catch-clauses, but for the sake of convenience we don't need them here:
async function getExample() {
var resultA = await promiseA(…);
// some processing
var resultB = await promiseB(…);
// more processing
return // something using both resultA and resultB
}
ECMAScript 6
While we were waiting for ES8, we already did use a very similar kind of syntax. ES6 came with generator functions, which allow breaking the execution apart in pieces at arbitrarily placed yield keywords. Those slices can be run after each other, independently, even asynchronously - and that's just what we do when we want to wait for a promise resolution before running the next step.
There are dedicated libraries (like co or task.js), but also many promise libraries have helper functions (Q, Bluebird, when, …) that do this async step-by-step execution for you when you give them a generator function that yields promises.
var getExample = Promise.coroutine(function* () {
// ^^^^^^^^^^^^^^^^^ Bluebird syntax
var resultA = yield promiseA(…);
// some processing
var resultB = yield promiseB(…);
// more processing
return // something using both resultA and resultB
});
This did work in Node.js since version 4.0, also a few browsers (or their dev editions) did support generator syntax relatively early.
ECMAScript 5
However, if you want/need to be backward-compatible you cannot use those without a transpiler. Both generator functions and async functions are supported by the current tooling, see for example the documentation of Babel on generators and async functions.
And then, there are also many other compile-to-JS languages
that are dedicated to easing asynchronous programming. They usually use a syntax similar to await, (e.g. Iced CoffeeScript), but there are also others that feature a Haskell-like do-notation (e.g. LatteJs, monadic, PureScript or LispyScript).
Synchronous inspection
Assigning promises-for-later-needed-values to variables and then getting their value via synchronous inspection. The example uses bluebird's .value() method but many libraries provide similar method.
function getExample() {
var a = promiseA(…);
return a.then(function() {
// some processing
return promiseB(…);
}).then(function(resultB) {
// a is guaranteed to be fulfilled here so we can just retrieve its
// value synchronously
var aValue = a.value();
});
}
This can be used for as many values as you like:
function getExample() {
var a = promiseA(…);
var b = a.then(function() {
return promiseB(…)
});
var c = b.then(function() {
return promiseC(…);
});
var d = c.then(function() {
return promiseD(…);
});
return d.then(function() {
return a.value() + b.value() + c.value() + d.value();
});
}
Nesting (and) closures
Using closures for maintaining the scope of variables (in our case, the success callback function parameters) is the natural JavaScript solution. With promises, we can arbitrarily nest and flatten .then() callbacks - they are semantically equivalent, except for the scope of the inner one.
function getExample() {
return promiseA(…).then(function(resultA) {
// some processing
return promiseB(…).then(function(resultB) {
// more processing
return // something using both resultA and resultB;
});
});
}
Of course, this is building an indentation pyramid. If indentation is getting too large, you still can apply the old tools to counter the pyramid of doom: modularize, use extra named functions, and flatten the promise chain as soon as you don't need a variable any more.
In theory, you can always avoid more than two levels of nesting (by making all closures explicit), in practise use as many as are reasonable.
function getExample() {
// preprocessing
return promiseA(…).then(makeAhandler(…));
}
function makeAhandler(…)
return function(resultA) {
// some processing
return promiseB(…).then(makeBhandler(resultA, …));
};
}
function makeBhandler(resultA, …) {
return function(resultB) {
// more processing
return // anything that uses the variables in scope
};
}
You can also use helper functions for this kind of partial application, like _.partial from Underscore/lodash or the native .bind() method, to further decrease indentation:
function getExample() {
// preprocessing
return promiseA(…).then(handlerA);
}
function handlerA(resultA) {
// some processing
return promiseB(…).then(handlerB.bind(null, resultA));
}
function handlerB(resultA, resultB) {
// more processing
return // anything that uses resultA and resultB
}
Explicit pass-through
Similar to nesting the callbacks, this technique relies on closures. Yet, the chain stays flat - instead of passing only the latest result, some state object is passed for every step. These state objects accumulate the results of the previous actions, handing down all values that will be needed later again plus the result of the current task.
function getExample() {
return promiseA(…).then(function(resultA) {
// some processing
return promiseB(…).then(b => [resultA, b]); // function(b) { return [resultA, b] }
}).then(function([resultA, resultB]) {
// more processing
return // something using both resultA and resultB
});
}
Here, that little arrow b => [resultA, b] is the function that closes over resultA, and passes an array of both results to the next step. Which uses parameter destructuring syntax to break it up in single variables again.
Before destructuring became available with ES6, a nifty helper method called .spread() was provided by many promise libraries (Q, Bluebird, when, …). It takes a function with multiple parameters - one for each array element - to be used as .spread(function(resultA, resultB) { ….
Of course, that closure needed here can be further simplified by some helper functions, e.g.
function addTo(x) {
// imagine complex `arguments` fiddling or anything that helps usability
// but you get the idea with this simple one:
return res => [x, res];
}
…
return promiseB(…).then(addTo(resultA));
Alternatively, you can employ Promise.all to produce the promise for the array:
function getExample() {
return promiseA(…).then(function(resultA) {
// some processing
return Promise.all([resultA, promiseB(…)]); // resultA will implicitly be wrapped
// as if passed to Promise.resolve()
}).then(function([resultA, resultB]) {
// more processing
return // something using both resultA and resultB
});
}
And you might not only use arrays, but arbitrarily complex objects. For example, with _.extend or Object.assign in a different helper function:
function augment(obj, name) {
return function (res) { var r = Object.assign({}, obj); r[name] = res; return r; };
}
function getExample() {
return promiseA(…).then(function(resultA) {
// some processing
return promiseB(…).then(augment({resultA}, "resultB"));
}).then(function(obj) {
// more processing
return // something using both obj.resultA and obj.resultB
});
}
While this pattern guarantees a flat chain and explicit state objects can improve clarity, it will become tedious for a long chain. Especially when you need the state only sporadically, you still have to pass it through every step. With this fixed interface, the single callbacks in the chain are rather tightly coupled and inflexible to change. It makes factoring out single steps harder, and callbacks cannot be supplied directly from other modules - they always need to be wrapped in boilerplate code that cares about the state. Abstract helper functions like the above can ease the pain a bit, but it will always be present.
Mutable contextual state
The trivial (but inelegant and rather errorprone) solution is to just use higher-scope variables (to which all callbacks in the chain have access) and write result values to them when you get them:
function getExample() {
var resultA;
return promiseA(…).then(function(_resultA) {
resultA = _resultA;
// some processing
return promiseB(…);
}).then(function(resultB) {
// more processing
return // something using both resultA and resultB
});
}
Instead of many variables one might also use an (initially empty) object, on which the results are stored as dynamically created properties.
This solution has several drawbacks:
Mutable state is ugly, and global variables are evil.
This pattern doesn't work across function boundaries, modularising the functions is harder as their declarations must not leave the shared scope
The scope of the variables does not prevent to access them before they are initialized. This is especially likely for complex promise constructions (loops, branching, excptions) where race conditions might happen. Passing state explicitly, a declarative design that promises encourage, forces a cleaner coding style which can prevent this.
One must choose the scope for those shared variables correctly. It needs to be local to the executed function to prevent race conditions between multiple parallel invocations, as would be the case if, for example, state was stored on an instance.
The Bluebird library encourages the use of an object that is passed along, using their bind() method to assign a context object to a promise chain. It will be accessible from each callback function via the otherwise unusable this keyword. While object properties are more prone to undetected typos than variables, the pattern is quite clever:
function getExample() {
return promiseA(…)
.bind({}) // Bluebird only!
.then(function(resultA) {
this.resultA = resultA;
// some processing
return promiseB(…);
}).then(function(resultB) {
// more processing
return // something using both this.resultA and resultB
}).bind(); // don't forget to unbind the object if you don't want the
// caller to access it
}
This approach can be easily simulated in promise libraries that do not support .bind (although in a somewhat more verbose way and cannot be used in an expression):
function getExample() {
var ctx = {};
return promiseA(…)
.then(function(resultA) {
this.resultA = resultA;
// some processing
return promiseB(…);
}.bind(ctx)).then(function(resultB) {
// more processing
return // something using both this.resultA and resultB
}.bind(ctx));
}
A less harsh spin on "Mutable contextual state"
Using a locally scoped object to collect the intermediate results in a promise chain is a reasonable approach to the question you posed. Consider the following snippet:
function getExample(){
//locally scoped
const results = {};
return promiseA(paramsA).then(function(resultA){
results.a = resultA;
return promiseB(paramsB);
}).then(function(resultB){
results.b = resultB;
return promiseC(paramsC);
}).then(function(resultC){
//Resolve with composite of all promises
return Promise.resolve(results.a + results.b + resultC);
}).catch(function(error){
return Promise.reject(error);
});
}
Global variables are bad, so this solution uses a locally scoped variable which causes no harm. It is only accessible within the function.
Mutable state is ugly, but this does not mutate state in an ugly manner. The ugly mutable state traditionally refers to modifying the state of function arguments or global variables, but this approach simply modifies the state of a locally scoped variable that exists for the sole purpose of aggregating promise results...a variable that will die a simple death once the promise resolves.
Intermediate promises are not prevented from accessing the state of the results object, but this does not introduce some scary scenario where one of the promises in the chain will go rogue and sabotage your results. The responsibility of setting the values in each step of the promise is confined to this function and the overall result will either be correct or incorrect...it will not be some bug that will crop up years later in production (unless you intend it to!)
This does not introduce a race condition scenario that would arise from parallel invocation because a new instance of the results variable is created for every invocation of the getExample function.
Example is available on jsfiddle
Node 7.4 now supports async/await calls with the harmony flag.
Try this:
async function getExample(){
let response = await returnPromise();
let response2 = await returnPromise2();
console.log(response, response2)
}
getExample()
and run the file with:
node --harmony-async-await getExample.js
Simple as can be!
Another answer, using babel-node version <6
Using async - await
npm install -g babel#5.6.14
example.js:
async function getExample(){
let response = await returnPromise();
let response2 = await returnPromise2();
console.log(response, response2)
}
getExample()
Then, run babel-node example.js and voila!
This days, I also hava meet some questions like you. At last, I find a good solution with the quesition, it's simple and good to read. I hope this can help you.
According to how-to-chain-javascript-promises
ok, let's look at the code:
const firstPromise = () => {
return new Promise((resolve, reject) => {
setTimeout(() => {
console.log('first promise is completed');
resolve({data: '123'});
}, 2000);
});
};
const secondPromise = (someStuff) => {
return new Promise((resolve, reject) => {
setTimeout(() => {
console.log('second promise is completed');
resolve({newData: `${someStuff.data} some more data`});
}, 2000);
});
};
const thirdPromise = (someStuff) => {
return new Promise((resolve, reject) => {
setTimeout(() => {
console.log('third promise is completed');
resolve({result: someStuff});
}, 2000);
});
};
firstPromise()
.then(secondPromise)
.then(thirdPromise)
.then(data => {
console.log(data);
});
I am not going to use this pattern in my own code since I'm not a big fan of using global variables. However, in a pinch it will work.
User is a promisified Mongoose model.
var globalVar = '';
User.findAsync({}).then(function(users){
globalVar = users;
}).then(function(){
console.log(globalVar);
});
Another answer, using sequential executor nsynjs:
function getExample(){
var response1 = returnPromise1().data;
// promise1 is resolved at this point, '.data' has the result from resolve(result)
var response2 = returnPromise2().data;
// promise2 is resolved at this point, '.data' has the result from resolve(result)
console.log(response, response2);
}
nynjs.run(getExample,{},function(){
console.log('all done');
})
Update: added working example
function synchronousCode() {
var urls=[
"https://ajax.googleapis.com/ajax/libs/jquery/1.7.0/jquery.min.js",
"https://ajax.googleapis.com/ajax/libs/jquery/1.8.0/jquery.min.js",
"https://ajax.googleapis.com/ajax/libs/jquery/1.9.0/jquery.min.js"
];
for(var i=0; i<urls.length; i++) {
var len=window.fetch(urls[i]).data.text().data.length;
// ^ ^
// | +- 2-nd promise result
// | assigned to 'data'
// |
// +-- 1-st promise result assigned to 'data'
//
console.log('URL #'+i+' : '+urls[i]+", length: "+len);
}
}
nsynjs.run(synchronousCode,{},function(){
console.log('all done');
})
<script src="https://rawgit.com/amaksr/nsynjs/master/nsynjs.js"></script>
When using bluebird, you can use .bind method to share variables in promise chain:
somethingAsync().bind({})
.spread(function (aValue, bValue) {
this.aValue = aValue;
this.bValue = bValue;
return somethingElseAsync(aValue, bValue);
})
.then(function (cValue) {
return this.aValue + this.bValue + cValue;
});
please check this link for further information:
http://bluebirdjs.com/docs/api/promise.bind.html
function getExample() {
var retA, retB;
return promiseA(…).then(function(resultA) {
retA = resultA;
// Some processing
return promiseB(…);
}).then(function(resultB) {
// More processing
//retA is value of promiseA
return // How do I gain access to resultA here?
});
}
easy way :D
I think you can use hash of RSVP.
Something like as below :
const mainPromise = () => {
const promise1 = new Promise((resolve, reject) => {
setTimeout(() => {
console.log('first promise is completed');
resolve({data: '123'});
}, 2000);
});
const promise2 = new Promise((resolve, reject) => {
setTimeout(() => {
console.log('second promise is completed');
resolve({data: '456'});
}, 2000);
});
return new RSVP.hash({
prom1: promise1,
prom2: promise2
});
};
mainPromise()
.then(data => {
console.log(data.prom1);
console.log(data.prom2);
});
Solution:
You can put intermediate values in scope in any later 'then' function explicitly, by using 'bind'. It is a nice solution that doesn't require changing how Promises work, and only requires a line or two of code to propagate the values just like errors are already propagated.
Here is a complete example:
// Get info asynchronously from a server
function pGetServerInfo()
{
// then value: "server info"
} // pGetServerInfo
// Write into a file asynchronously
function pWriteFile(path,string)
{
// no then value
} // pWriteFile
// The heart of the solution: Write formatted info into a log file asynchronously,
// using the pGetServerInfo and pWriteFile operations
function pLogInfo(localInfo)
{
var scope={localInfo:localInfo}; // Create an explicit scope object
var thenFunc=p2.bind(scope); // Create a temporary function with this scope
return (pGetServerInfo().then(thenFunc)); // Do the next 'then' in the chain
} // pLogInfo
// Scope of this 'then' function is {localInfo:localInfo}
function p2(serverInfo)
{
// Do the final 'then' in the chain: Writes "local info, server info"
return pWriteFile('log',this.localInfo+','+serverInfo);
} // p2
This solution can be invoked as follows:
pLogInfo("local info").then().catch(err);
(Note: a more complex and complete version of this solution has been tested, but not this example version, so it could have a bug.)
What I learn about promises is to use it only as return values avoid referencing them if possible. async/await syntax is particularly practical for that. Today all latest browsers and node support it: https://caniuse.com/#feat=async-functions , is a simple behavior and the code is like reading synchronous code, forget about callbacks...
In cases I do need to reference a promises is when creation and resolution happen at independent/not-related places. So instead an artificial association and probably an event listener just to resolve the "distant" promise, I prefer to expose the promise as a Deferred, which the following code implements it in valid es5
/**
* Promise like object that allows to resolve it promise from outside code. Example:
*
```
class Api {
fooReady = new Deferred<Data>()
private knower() {
inOtherMoment(data=>{
this.fooReady.resolve(data)
})
}
}
```
*/
var Deferred = /** #class */ (function () {
function Deferred(callback) {
var instance = this;
this.resolve = null;
this.reject = null;
this.status = 'pending';
this.promise = new Promise(function (resolve, reject) {
instance.resolve = function () { this.status = 'resolved'; resolve.apply(this, arguments); };
instance.reject = function () { this.status = 'rejected'; reject.apply(this, arguments); };
});
if (typeof callback === 'function') {
callback.call(this, this.resolve, this.reject);
}
}
Deferred.prototype.then = function (resolve) {
return this.promise.then(resolve);
};
Deferred.prototype.catch = function (r) {
return this.promise.catch(r);
};
return Deferred;
}());
transpiled form a typescript project of mine:
https://github.com/cancerberoSgx/misc-utils-of-mine/blob/2927c2477839f7b36247d054e7e50abe8a41358b/misc-utils-of-mine-generic/src/promise.ts#L31
For more complex cases I often use these guy small promise utilities without dependencies tested and typed. p-map has been useful several times. I think he covered most use cases:
https://github.com/sindresorhus?utf8=%E2%9C%93&tab=repositories&q=promise&type=source&language=
I am working with zombie.js to scrape one site, I must to use the callback style to connect to each url. The point is that I have got an urls array and I need to process each urls using an async function. This is my first approach:
Array urls = {http..., http...};
function process_url(index)
{
if(index == urls.length)
return;
async_function(url,
function() {
...
//parse the url
...
// Process the next url
process_url(index++);
}
);
}
process_url(0)
Without use someone third party nodejs library to use the asyn funtion as sync function or to wait for the function (wait.for, synchornized, mocha), this is the way that I though to resolve this problem, I don't know what would happen if the array is too big. Is the function released from the memory when the next function is called? or all the functions are in memory until the end?
Any ideas?
Your scheme will work. I call it "manually sequencing async operations".
A general purpose version of what you're doing would look like this:
function processItem(data, callback) {
// do your async function here
// for example, let's suppose it was an http request using the request module
request(data, callback);
}
function processArray(array, fn) {
var index = 0;
function next() {
if (index < array.length) {
fn(array[index++], function(err, result) {
// process error here
if (err) return;
// process result here
next();
});
}
}
next();
}
processArray(arr, processItem);
As to your specific questions:
I don't know what would happen if the array is too big. Is the
function released from the memory when the next function is called? or
all the functions are in memory until the end?
Memory in Javascript is released when it is not longer referenced by any running code and when the garbage collector gets time to run. Since you are running a series of asynchronous operations here, it is likely that the garbage collector gets a chance to run regularly while waiting for the http response from the async operation so memory could get cleaned up then. Functions are just another type of object in Javascript and they get garbage collected just like anything else. When they are no longer reference by running code, they are eligible for garbage collection.
In your specific code, because you are re-calling process_url() only in an async callback, there is no stack build-up (as in normal recursion). The prior instance of process_url() has already completed BEFORE the async callback is called and BEFORE you call the next iteration of process_url().
In general, management and coordination of multiple async operations is much, much easier using promises which are built into the current versions of node.js and are part of the ES6 ECMAScript standard. No external libraries are required to use promises in current versions of node.js.
For a list of a number of different techniques for sequencing your asynchronous operations on your array, both using promises and not using promises, see:
How to synchronize a sequence of promises?.
The first step in using promises is the "promisify" your async function so that it returns a promise instead of takes a callback.
function async_function_promise(url) {
return new Promise(function(resolve, reject) {
async_function(url, function(err, result) {
if (err) {
reject(err);
} else {
resolve(result);
}
});
});
}
Now, you have a version of your function that returns promises.
If you want your async operations to proceed one at a time so the next one doesn't start until the previous one has completed, then a usual design pattern for that is to use .reduce() like this:
function process_urls(array) {
return array.reduce(function(p, url) {
return p.then(function(priorResult) {
return async_function_promise(url);
});
}, Promise.resolve());
}
Then, you can call it like this:
var myArray = ["url1", "url2", ...];
process_urls(myArray).then(function(finalResult) {
// all of them are done here
}, function(err) {
// error here
});
There are also Promise libraries that have some helpful features that make this type of coding simpler. I, myself, use the Bluebird promise library. Here's how your code would look using Bluebird:
var Promise = require('bluebird');
var async_function_promise = Promise.promisify(async_function);
function process_urls(array) {
return Promise.map(array, async_function_promise, {concurrency: 1});
}
process_urls(myArray).then(function(allResults) {
// all of them are done here and allResults is an array of the results
}, function(err) {
// error here
});
Note, you can change the concurrency value to whatever you want here. For example, you would probably get faster end-to-end performance if you increased it to something between 2 and 5 (depends upon the server implementation on how this is best optimized).
I have the following method in Node.js:
var foo = function(items){
for (var i=0; i<items.length; i++){
var item = items[i];
bar(item);
}
return true;
}
I want foo to return true after all of the bar's have finished updating. How do I do that?
EDIT: I am looking for a way to do this without any external libraries if possible.
The definition of the for loop being "completely done" depends on whether or not bar is synchronous or asynchronous. If bar is synchronous--perhaps it is doing some long-running computation--then your code already returns when the loop is "completely done".
But based on the part of your question that says
after all of the bar's have finished updating
it seems to be a reasonable assumption that bar is asynchronous--perhaps it is making a network call. But currently it is missing any mechanism to report when it's done. Therefore, the first job is to redefine bar so it has an asynchronous interface. There are two basic kinds of asynchronous interfaces. The older, classical node approach is callbacks. A new approach is promises.
In either case, therefore, you need to start off by redefining bar in one way or another. You cannot use an interface asynchronously unless it is designed to be used that way.
After we have given bar an asynchronous interface, we can then use that from within foo to correctly report when the bars are "completely done".
Callbacks
In the callback case, we change the calling sequence for bar to bar(item, callback), giving it a way to report back when it is done. We will also need a callback on the foo routine, to be invoked when all the bar calls finish. Following your preference to not use libraries, the logic for this could look something like this:
function foo(items, callback) {
var count = items.length;
var results = [];
items.forEach(function(item, idx) {
bar(item, function(err, data) {
if (err) callback(err);
results[idx] = data;
if (!count--) callback(null, results);
});
});
}
This loops over the items, calling bar for each one. When each bar finishes, we place its result in a results array, and check if this is the final bar, in which case we call the top-level callback.
This would be used as
foo(items, function(err, data) {
if (err) console.log("Error is", err);
else console.log("All succeeded, with resulting array of ", data);
});
The above is essentially equivalent to using async.each, as suggested in another answer, as follows:
function foo(items, callback) {
async.each(items, bar, callback);
}
Waiting for each bar to finish
If you want to wait for each bar to finish before proceeding to the next one, with async it's pretty easy with eachSeries:
function foo(items, callback) {
async.eachSeries(items, bar, callback);
}
Non-library code would be a bit more complicated:
function foo(items, callback) {
var results = [];
function next(i) {
if (i >= items.length) return callback(results));
bar(items[i], function barCallback(err, data) {
if (err) return callback(err);
results[i] = data;
next(++i);
});
}
next(0);
}
Here, the callback reporting that each bar is completed (barCallback) calls the next routine with incremented i to kick off the call to the next bar.
Promises
However, you are likely to be better off using promises. In this case, we will design bar to return a promise, instead of invoking a callback. Assuming bar calls some database routine MYDB, the new version of bar is likely to look like this:
function bar(item) {
return MYDB.findRecordPromise(item.param);
}
If MYDB only provides old-fashioned callback-based interfaces, then the basic approach is
function bar(item) {
return new Promise(function(resolve, reject) {
MYDB.findRecord(item.param, function(err, data) {
if (err) reject(err);
else resolve(data);
});
});
}
Now that we have a promise-based version of bar, we will define foo to return a promise as well, instead of taking an extra callback parameter. Then it's very simple:
function foo(items) {
return Promise.all(items.map(bar));
}
Promise.all takes an array of promises (which we create from items by mapping over bar), and fulfills when all of them fulfill, with a value of an array of fulfilled values, and rejects when any of them reject.
This is used as
foo(items) . then(
function(data) { console.log("All succeeded, with resulting array of ", data); },
function(err) { console.log("Error is", err); }
);
Waiting for each bar to finish
If, with the promises approach, you want to wait for each bar to finish before the next one is kicked off, then the basic approach is:
function foo(items) {
return items.reduce(function(promise, item) {
return promise.then(function() { return bar(item); });
}, Promise.resolve());
}
This starts off with an "empty" (pre-resolved) promise, then each time through the loop, adds a then to the chain to execute the next bar when the preceding one is finished.
If you want to get fancy and use advanced async functions, with the new await keyword:
async function foo(items) {
for (var i = 0; i < items.length; i++) {
await bar(items[i]);
}
}
Aside
As an aside, the presumption in another answer that you might be looking for an asynchronous interface to a routine (foo) which loops over some calls to bar which are either synchronous, or may be asynchronous but can only be kicked off with no way to know when they completed, seems odd. Consider this logic:
var foo = function(items, fooCallback){
for (var i=0; i<items.length; i++){
var item = items[i];
bar(item);
}
fooCallback(true);
}
This does nothing different from the original code, except that instead of returning true, it calls the callback with true. It does not ensure that the bars are actually completed, since, to repeat myself, without giving bar a new asynchronous interface, we have no way to know that they are completed.
The code shown using async.each suffers from a similar flaw:
var async = require('async');
var foo = function(items){
async.each(items, function(item, callback){
var _item = item; // USE OF _item SEEMS UNNECESSARY
bar(_item);
callback(); // BAR IS NOT COMPLETED HERE
},
function(err){
return true; // THIS RETURN VALUE GOES NOWHERE
});
}
This will merely kick off all the bars, but then immediately call the callback before they are finished. The entire async.each part will finish more or less instantaneously, immediately after which the final function(err) will be invoked; the value of true it returns will simply go into outer space. If you wanted to write it this way, it should be
var foo = function(items, fooCallback){
async.each(items, function(item, callback){
bar(item);
callback();
},
function(err){
fooCallback(true); // CHANGE THIS
}
});
}
but even in this case fooCallback will be called pretty much instantaneously, regardless of whether the bars are finished or whether they returned errors.
There are many options to solve what you want, you just need to define the nature of your funcions foo and bar. Here are some options:
Your original code doesn't need it:
Your foo and bar functions doesn't have any callback in their parameters so they are not asynchronous.
Asuming foo function is asynchronous and bar function is synchronous:
You should rewrite your original code using callbacks, something like this:
var foo = function(items, fooCallback){
for (var i=0; i<items.length; i++){
var item = items[i];
bar(item);
}
fooCallback(true);
}
Using a Async:
You should install it at the beggining:
npm install async
You use async, like this.
var async = require('async');
var foo = function(items){
async.each(items, function(item, callback){
var _item = item;
bar(_item);
callback();
},
function(err){
return true;
}
});
}