Javascript recursion, memory leak? - node.js

I am trying to implement a class that will perform an action once every second, it works as expected but I am unsure about memory leaks. This code I'm writing will have an uptime of several months.
Will the code below lead to memory leaks since it's technically recursion which never ends?
class Algorithm{
constructor(){
//there will be many more things in this constructor
//which is why this is a class
const pidTimer = (r,e=0,y=Date.now()-r) => {
this.someFunction();
const now = Date.now();
const dy = now-y;
const err = e+r-dy
const u = err*0.2;
//console.log(dy)
setTimeout(()=>{pidTimer(r,err,now)},r+u);
}
pidTimer(1000);
}
someFunction = () => {}
}

It's not the kind of recursion that has any stack accumulation since the previous pidTimer() function call returns before the setTimeout() fires and calls pidTimer() again. I wouldn't even call this recursion (it's scheduled repeated calling), but that's more a semantic issue.
So, the only place I see there could be some memory leak or excess usage would be inside of this.someFunction(); and that's only because you don't show us the code there to evaluate it and see what it does. The code you show us for pidTimer() itself has no issues on its own.

modern async primitive
There's nothing "wrong" with the current function you have, however I think it could be improved significantly. JavaScript offers a modernized asynchrony atom, Promise and new syntax support async/await. These are preferred over stone-age setTimeout and setInterval as you can easily thread data through asynchronous control flow, stop thinking in terms of "callbacks", and avoid side effects -
class Algorithm {
constructor() {
...
this.runProcess(...)
}
async runProcess(...) { // async
while (true) { // loop instead of recursion
await sleep(...) // sleep some amount of time
this.someFunction() // do work
... // adjust timer variables
}
}
}
sleep is a simple function which resolves a promise after a specified milliseconds value, ms -
function sleep(ms) {
return new Promise(r => setTimeout(r, ms)) // promise
}
async iterables
But see how this.someFunction() doesn't return anything? It would be nice if we could capture the data from someFunction and make it available to our caller. By making runProcess an async generator and implementing Symbol.asyncIterator we can easily handle asynchrony and stop side effects -
class Algorithm {
constructor() {
...
this.data = this.runProcess(...) // assign this.data
}
async *runProcess(...) { // async generator
while (true) {
await sleep(...)
yield this.someFunction() // yield
...
}
}
[Symbol.asyncIterator]() { // iterator
return this.data
}
}
Now the caller has control over what happens as the data comes in from this.someFunction. Below we write to console.log but you could easily swap this for an API call or write to file system -
const foo = new Algorithm(...)
for await (const data of foo)
console.log("process data", data) // or API call, or write to file system, etc
additional control
You can easily add control of the process to by using additional data members. Below we swap out while(true) with a conditional and allow the caller to stop the process -
class Algorithm {
constructor() {
...
}
async *runProcess(...) {
this.running = true // start
while (this.running) { // conditional loop
...
}
}
haltProcess() {
this.running = false // stop
}
...
}
demo
Here's a functioning demo including the concepts above. Note we only implement halt here because run is an infinite generator. Manual halting is not necessary for finite generators. Verify the results in your own browser by running the snippet -
class Algorithm {
async *run() {
this.running = true
while(this.running) {
await sleep(1000)
yield this.someFunction()
}
}
halt() {
this.running = false
}
someFunction() {
return Math.random()
}
[Symbol.asyncIterator] = this.run
}
function sleep(ms) {
return new Promise(r => setTimeout(r, ms))
}
async function main() {
const foo = new Algorithm // init
setTimeout(_ => foo.halt(), 10000) // stop at some point, for demo
for await (const x of foo) // iterate
console.log("data", x) // log, api call, write fs, etc
return "done" // return something when done
}
main().then(console.log, console.error) // "done"
data 0.3953947360028206
data 0.18754462176783115
data 0.23690422070864803
data 0.11237466374294014
data 0.5123244720637253
data 0.39818889343799635
data 0.08627407687877853
data 0.3861902404922477
data 0.8358471443658225
data 0.2770336562516085
done

Related

Reporting during long running async function using set interval in node.js (backend service)

I need to report the status of a long running operation in node.js. The basic use case is outlined in the code below. awaiting the longProcess method I know will act synchronously to the caller, but I must await the method in my code. Should I handle this within the longProcess method? Not sure how to address this issue.
function sleep (ms: number) {
new Promise(resolve => setTimeout(resolve, ms));
}
let processedCount = 0;
async function longProcess() {
// really long operation
while (true) {
processedCount++;
await sleep(1000); // simulate long process
if (processedCount === 10) // just to end the test somehow
break;
}
}
async function report() {
console.log(processedCount);
}
async function main() {
const id = setInterval(report, 500);
await longProcess();
clearInterval(id);
}
main().then(() => console.log("Done"));
The sleep method is just for demonstration purposes to simulate a long running operation. 'longProcess' performs complex and time intensive processing. It calls a callback passed in to report back a processed count the caller. The class that contains the calling method (and the callback), also has a report method that I would like to call at regular intervals. And I need to be able to create a unit test for this
Your sleep function is not returning the promise you are creating. You are calling await on the value returned from the function, which in this case is undefined so it doesn't actually wait at all.
function sleep (ms: number) {
return new Promise(resolve => setTimeout(resolve, ms));
}

How to stop async code from running Node.JS

I'm creating a program where I constantly run and stop async code, but I need a good way to stop the code.
Currently, I have tried to methods:
Method 1:
When a method is running, and another method is called to stop the first method, I start an infinite loop to stop that code from running and then remove the method from the queue(array)
I'm 100% sure that this is the worst way to accomplish it, and it works very buggy.
Code:
class test{
async Start(){
const response = await request(options);
if(stopped){
while(true){
await timeout(10)
}
}
}
}
Code 2:
var tests = [];
Start(){
const test = new test();
tests.push(test)
tests.Start();
}
Stop(){
tests.forEach((t, i) => {t.stopped = true;};
tests = [];
}
Method 2:
I load the different methods into Workers, and when I need to stop the code, I just terminate the Worker.
It always takes a lot of time(1 sec) to create the Worker, and therefore not the best way, since I need the code to run without 1-2 sec pauses.
Code:
const Worker = require("tiny-worker");
const code = new Worker(path.resolve(__dirname, "./Code/Code.js"))
Stopping:
code.terminate()
Is there any other way that I can stop async code?
The program contains Request using nodejs Request-promise module, so program is waiting for requests, it's hard to stop the code without one of the 2 methods.
Is there any other way that I can stop async code?
Keep in mind the basic of how Nodejs works. I think there is some misunderstanding here.
It execute the actual function in the actual context, if encounters an async operation the event loop will schedule it's execetution somewhere in the future. There is no way to remove that scheduled execution.
More info on event loop here.
In general for manage this kind of situations you shuold use flags or semaphores.
The program contains Request using nodejs Request-promise module, so program is waiting for requests, it's hard to stop the code
If you need to hard "stop the code" you can do something like
func stop() {
process.exit()
}
But if i'm getting it right, you're launching requests every x time, at some point you need to stop sending the request without managing the response.
You can't de-schedule the response managemente portion, but you can add some logic in it to (when it will be runned) check if the "request loop" has been stopped.
let loop_is_stopped = false
let sending_loop = null
func sendRequest() {
const response = await request(options) // "wait here"
// following lines are scheduled after the request promise is resolved
if (loop_is_stopped) {
return
}
// do something with the response
}
func start() {
sending_loop = setInterval(sendRequest, 1000)
}
func stop() {
loop_is_stopped = true
clearInterval(sending_loop)
}
module.exports = { start, stop }
We can use Promise.all without killing whole app (process.exit()), here is my example (you can use another trigger for calling controller.abort()):
const controller = new AbortController();
class Workflow {
static async startTask() {
await new Promise((res) => setTimeout(() => {
res(console.log('RESOLVE'))
}, 3000))
}
}
class ScheduleTask {
static async start() {
return await Promise.all([
new Promise((_res, rej) => { if (controller.signal.aborted) return rej('YAY') }),
Workflow.startTask()
])
}
}
setTimeout(() => {
controller.abort()
console.log("ABORTED!!!");
}, 1500)
const run = async () => {
try {
await ScheduleTask.start()
console.log("DONE")
} catch (err) {
console.log("ERROR", err.name)
}
}
run()
// ABORTED!!!
// RESOLVE
"DONE" will never be showen.
res will be complited
Maybe would be better to run your code as script with it's own process.pid and when we need to interrupt this functionality we can kill this process by pid in another place of your code process.kill.

How to monitor number of RXJS subscriptions?

I'm using an Observable to provide event subscription interface for clients from a global resource, and I need to manage that resource according to the number of active subscriptions:
Allocate global resource when the number of subscriptions becomes greater than 0
Release global resource when the number of subscriptions becomes 0
Adjust the resource usage strategy based on the number of subscriptions
What is the proper way in RXJS to monitor the number of active subscriptions?
How to implement the following within RXJS syntax? -
const myEvent: Observable<any> = new Observable();
myEvent.onSubscription((newCount: number, prevCount: number) => {
if(newCount === 0) {
// release global resource
} else {
// allocate global resource, if not yet allocated
}
// for a scalable resource usage / load,
// re-configure it, based on newCount
});
I wouldn't expect a guaranteed notification on each change, hence newCount + prevCount params.
UPDATE-1
This is not a duplicate to this, because I need to be notified when the number of subscriptions changes, and not just to get the counter at some point.
UPDATE-2
Without any answer so far, I quickly came up with a very ugly and limited work-around, through complete incapsulation, and specifically for type Subject. Hoping very much to find a proper solution.
UPDATE-3
After a few answers, I'm still not sure how to implement what I'm trying, which is the following:
class CustomType {
}
class CountedObservable<T> extends Observable<T> {
private message: string; // random property
public onCount; // magical Observable that needs to be implemented
constructor(message: string) {
// super(); ???
this.message = message;
}
// random method
public getMessage() {
return this.message;
}
}
const a = new CountedObservable<CustomType>('hello'); // can create directly
const msg = a.getMessage(); // can call methods
a.subscribe((data: CustomType) => {
// handle subscriptions here;
});
// need that magic onCount implemented, so I can do this:
a.onCount.subscribe((newCount: number, prevCont: number) => {
// manage some external resources
});
How to implement such class CountedObservable above, which would let me subscribe to itself, as well as its onCount property to monitor the number of its clients/subscriptions?
UPDATE-4
All suggested solutions seemed overly complex, and even though I accepted one of the answers, I ended up with a completely custom solution one of my own.
You could achieve it using defer to track subscriptions and finalize to track completions, e.g. as an operator:
// a custom operator that will count number of subscribers
function customOperator(onCountUpdate = noop) {
return function refCountOperatorFunction(source$) {
let counter = 0;
return defer(()=>{
counter++;
onCountUpdate(counter);
return source$;
})
.pipe(
finalize(()=>{
counter--;
onCountUpdate(counter);
})
);
};
}
// just a stub for `onCountUpdate`
function noop(){}
And then use it like:
const source$ = new Subject();
const result$ = source$.pipe(
customOperator( n => console.log('Count updated: ', n) )
);
Heres a code snippet illustrating this:
const { Subject, of, timer, pipe, defer } = rxjs;
const { finalize, takeUntil } = rxjs.operators;
const source$ = new Subject();
const result$ = source$.pipe(
customOperator( n => console.log('Count updated: ', n) )
);
// emit events
setTimeout(()=>{
source$.next('one');
}, 250);
setTimeout(()=>{
source$.next('two');
}, 1000);
setTimeout(()=>{
source$.next('three');
}, 1250);
setTimeout(()=>{
source$.next('four');
}, 1750);
// subscribe and unsubscribe
const subscriptionA = result$
.subscribe(value => console.log('A', value));
setTimeout(()=>{
result$.subscribe(value => console.log('B', value));
}, 500);
setTimeout(()=>{
result$.subscribe(value => console.log('C', value));
}, 1000);
setTimeout(()=>{
subscriptionA.unsubscribe();
}, 1500);
// complete source
setTimeout(()=>{
source$.complete();
}, 2000);
function customOperator(onCountUpdate = noop) {
return function refCountOperatorFunction(source$) {
let counter = 0;
return defer(()=>{
counter++;
onCountUpdate(counter);
return source$;
})
.pipe(
finalize(()=>{
counter--;
onCountUpdate(counter);
})
);
};
}
function noop(){}
<script src="https://unpkg.com/rxjs#6.4.0/bundles/rxjs.umd.min.js"></script>
* NOTE: if your source$ is cold — you might need to share it.
Hope it helps
You are really asking three separate questions here, and I question whether you really need the full capability that you mention. Since most of the resource managment stuff you are asking for is already provided for by the library, doing custom tracking code seems to be redundant. The first two questions:
Allocate global resource when the number of subscriptions becomes greater than 0
Release global resource when the number of subscriptions becomes 0
Can be done with the using + share operators:
class ExpensiveResource {
constructor () {
// Do construction
}
unsubscribe () {
// Do Tear down
}
}
// Creates a resource and ties its lifecycle with that of the created `Observable`
// generated by the second factory function
// Using will accept anything that is "Subscription-like" meaning it has a unsubscribe function.
const sharedStream$ = using(
// Creates an expensive resource
() => new ExpensiveResource(),
// Passes that expensive resource to an Observable factory function
er => timer(1000)
)
// Share the underlying source so that global creation and deletion are only
// processed when the subscriber count changes between 0 and 1 (or visa versa)
.pipe(share())
After that sharedStream$ can be passed around as a base stream which will manage the underlying resource (assuming you implemented your unsubscribe correctly) so that the resource will be created and torn down as the number of subscribers transitions between 0 and 1.
Adjust the resource usage strategy based on the number of subscriptions
The third question I am most dubious on, but I'll answer it for completeness assuming you know your application better than I do (since I can't think of a reason why you would need specific handling at different usage levels other than going between 0 and 1).
Basically I would use a similar approach as above but I would encapuslate the transition logic slightly differently.
// Same as above
class ExpensiveResource {
unsubscribe() { console.log('Tear down this resource!')}
}
const usingReferenceTracking =
(onUp, onDown) => (resourceFactory, streamFactory) => {
let instance, refCount = 0
// Again manage the global resource state with using
const r$ = using(
// Unfortunately the using pattern doesn't let the resource escape the closure
// so we need to cache it for ourselves to use later
() => instance || (instance = resourceFactory()),
// Forward stream creation as normal
streamFactory
)
).pipe(
// Don't forget to clean up the stream after all is said and done
// Because its behind a share this should only happen when all subscribers unsubscribe
finalize(() => instance = null)
share()
)
// Use defer to trigger "onSubscribe" side-effects
// Note as well that these side-effects could be merged with the above for improved performance
// But I prefer them separate for easier maintenance.
return defer(() => onUp(instance, refCount += 1) || r$)
// Use finalize to handle the "onFinish" side-effects
.pipe(finalize(() => onDown(instance, refCount -= 1)))
}
const referenceTracked$ = usingReferenceTracking(
(ref, count) => console.log('Ref count increased to ' + count),
(ref, count) => console.log('Ref count decreased to ' + count)
)(
() => new ExpensiveResource(),
ref => timer(1000)
)
referenceTracked$.take(1).subscribe(x => console.log('Sub1 ' +x))
referenceTracked$.take(1).subscribe(x => console.log('Sub2 ' +x))
// Ref count increased to 1
// Ref count increased to 2
// Sub1 0
// Ref count decreased to 1
// Sub2 0
// Ref count decreased to 0
// Tear down this resource!
Warning: One side effect of this is that by definition the stream will be warm once it leaves the usingReferenceTracking function, and it will go hot on first subscription. Make sure you take this into account during the subscription phase.
What a fun problem! If I am understanding what you are asking, here is my solution to this: create a wrapper class around Observable that tracks the subscriptions by intercepting both subscribe() and unsubscribe(). Here is the wrapper class:
export class CountSubsObservable<T> extends Observable<T>{
private _subCount = 0;
private _subCount$: BehaviorSubject<number> = new BehaviorSubject(0);
public subCount$ = this._subCount$.asObservable();
constructor(public source: Observable<T>) {
super();
}
subscribe(
observerOrNext?: PartialObserver<T> | ((value: T) => void),
error?: (error: any) => void,
complete?: () => void
): Subscription {
this._subCount++;
this._subCount$.next(this._subCount);
let subscription = super.subscribe(observerOrNext as any, error, complete);
const newUnsub: () => void = () => {
if (this._subCount > 0) {
this._subCount--;
this._subCount$.next(this._subCount);
subscription.unsubscribe();
}
}
subscription.unsubscribe = newUnsub;
return subscription;
}
}
This wrapper creates a secondary observable .subCount$ that can be subscribed to which will emit every time the number of subscriptions to the source observable changes. It will emit a number corresponding to the current number of subscribers.
To use it you would create a source observable and then call new with this class to create the wrapper. For example:
const source$ = interval(1000).pipe(take(10));
const myEvent$: CountSubsObservable<number> = new CountSubsObservable(source$);
myEvent$.subCount$.subscribe(numSubs => {
console.log('subCount$ notification! Number of subscriptions is now', numSubs);
if(numSubs === 0) {
// release global resource
} else {
// allocate global resource, if not yet allocated
}
// for a scalable resource usage / load,
// re-configure it, based on numSubs
});
source$.subscribe(result => console.log('result is ', result));
To see it in use, check out this Stackblitz.
UPDATE:
Ok, as mentioned in the comments, I'm struggling a little to understand where the stream of data is coming from. Looking back through your question, I see you are providing an "event subscription interface". If the stream of data is a stream of CustomType as you detail in your third update above, then you may want to use fromEvent() from rxjs to create the source observable with which you would call the wrapper class I provided.
To show this I created a new Stackblitz. From that Stackblitz here is the stream of CustomTypes and how I would use the CountedObservable class to achieve what you are looking for.
class CustomType {
a: string;
}
const dataArray = [
{ a: 'January' },
{ a: 'February' },
{ a: 'March' },
{ a: 'April' },
{ a: 'May' },
{ a: 'June' },
{ a: 'July' },
{ a: 'August' },
{ a: 'September' },
{ a: 'October' },
{ a: 'November' },
{ a: 'December' }
] as CustomType[];
// Set up an arbitrary source that sends a stream of `CustomTypes`, one
// every two seconds by using `interval` and mapping the numbers into
// the associated dataArray.
const source$ = interval(2000).pipe(
map(i => dataArray[i]), // transform the Observable stream into CustomTypes
take(dataArray.length), // limit the Observable to only emit # array elements
share() // turn into a hot Observable.
);
const myEvent$: CountedObservable<CustomType> = new CountedObservable(source$);
myEvent$.onCount.subscribe(newCount => {
console.log('newCount notification! Number of subscriptions is now', newCount);
});
I hope this helps.
First of all, I very much appreciate how much time and effort people have committed trying to answer my question! And I am sure those answers will prove to be a useful guideline to other developers, solving similar scenarios with RXJS.
However, specifically for what I was trying to get out of RXJS, I found in the end that I am better off not using it at all. I specifically wanted the following:
A generic, easy-to-use interface for subscribing to notifications, plus monitoring subscriptions - all in one. With RXJS, the best I would end up is some workarounds that appear to be needlessly convoluted or even cryptic to developers who are not experts in RXJS. That is not what I would consider a friendly interface, more like something that rings over-engineering.
I ended up with a custom, much simpler interface that can do everything I was looking for:
export class Subscription {
private unsub: () => void;
constructor(unsub: () => void) {
this.unsub = unsub;
}
public unsubscribe(): void {
if (this.unsub) {
this.unsub();
this.unsub = null; // to prevent repeated calls
}
}
}
export class Observable<T = any> {
protected subs: ((data: T) => void)[] = [];
public subscribe(cb: (data: T) => void): Subscription {
this.subs.push(cb);
return new Subscription(this.createUnsub(cb));
}
public next(data: T): void {
// we iterate through a safe clone, in case an un-subscribe occurs;
// and since Node.js is the target, we are using process.nextTick:
[...this.subs].forEach(cb => process.nextTick(() => cb(data)));
}
protected createUnsub(cb) {
return () => {
this.subs.splice(this.subs.indexOf(cb), 1);
};
}
}
export interface ISubCounts {
newCount: number;
prevCount: number;
}
export class CountedObservable<T = any> extends Observable<T> {
readonly onCount: Observable<ISubCounts> = new Observable();
protected createUnsub(cb) {
const s = this.subs;
this.onCount.next({newCount: s.length, prevCount: s.length - 1});
return () => {
s.splice(s.indexOf(cb), 1);
this.onCount.next({newCount: s.length, prevCount: s.length + 1});
};
}
}
It is both small and elegant, and lets me do everything I needed to begin with, in a safe and friendly manner. I can do the same subscribe and onCount.subscribe, and get all the same notifications:
const a = new CountedObservable<string>();
const countSub = a.onCount.subscribe(({newCount, prevCount}) => {
console.log('COUNTS:', newCount, prevCount);
});
const sub1 = a.subscribe(data => {
console.log('SUB-1:', data);
});
const sub2 = a.subscribe(data => {
console.log('SUB-2:', data);
});
a.next('hello');
sub1.unsubscribe();
sub2.unsubscribe();
countSub.unsubscribe();
I hope this will help somebody else also.
P.S. I further improved it as an independent module.

How do I achieve a synchronous requirement using asynchronous NodeJS

I am adding user validation an data modification page on a node.js application.
In a synchronous universe, in a single function I would:
Lookup the original record in the database
Lookup the user in LDAP to see if they are the owner or admin
Do the logic and write the record.
In an asynchronous universe that won't work. To solve it I've built a series of hand-off functions:
router.post('/writeRecord', jsonParser, function(req, res) {
post = req.post;
var smdb = new AWS.DynamoDB.DocumentClient();
var params = { ... }
smdb.query(params, function(err,data){
if( err == null ) writeRecordStep2(post,data);
}
});
function writeRecord2( ru, post, data ){
var conn = new LDAP();
conn.search(
'ou=groups,o=amazon.com',
{ ... },
function(err,resp){
if( err == null ){
writeRecordStep3( ru, post, data, ldap1 )
}
}
}
function writeRecord3( ru, post, data ){
var conn = new LDAP();
conn.search(
'ou=groups,o=amazon.com',
{ ... },
function(err,resp){
if( err == null ){
writeRecordStep4( ru, post, data, ldap1, ldap2 )
}
}
}
function writeRecordStep4( ru, post, data, ldap1, ldap2 ){
// Do stuff with collected data
}
Additionally, because the LDAP and Dynamo logic are in their own source documents, these functions are scattered tragically around the code.
This strikes me as inefficient, as well as inelegant. I'm eager to find a more natural asynchronous pattern to achieve the same result.
Any promise library should sort your issue out. My preferred choice is bluebird. In summary they help you in performing blocking operations.
If you haven't heard about bluebird then just use it. It converts all function of a module and return promise which is then-able. Simply put, it promisifies all functions.
Here is the mechanism:
Module1.someFunction() \\do your job and finally pass the return object to next call
.then() \\Use that object which is return from the first call, do your job and return the updated value
.then() \\same goes on
.catch() \\do your job when any error occurs.
Hope you understand. Here is an example:
var readFile = Promise.promisify(require("fs").readFile);
readFile("myfile.js",
"utf8").then(function(contents) {
return eval(contents);
}).then(function(result) {
console.log("The result of evaluating
myfile.js", result);
}).catch(SyntaxError, function(e) {
console.log("File had syntax error", e);
//Catch any other error
}).catch(function(e) {
console.log("Error reading file", e);
});
I could not tell from your pseudo-code exactly which async operations depend upon results from with other ones and knowing that is key to the most efficient way to code a series of asynchronous operations. If two operations do not depend upon one another, they can run in parallel which generally gets to an end result faster. I also can't tell exactly what data needs to be passed on to later parts of the async requests (too much pseudo-code and not enough real code to show us what you're really attempting to do).
So, without that level of detail, I'll show you two ways to approach this. The first runs each operation sequentially. Run the first async operation, when it's done, run the next one and accumulates all the results into an object that is passed along to the next link in the chain. This is general purpose since all async operations have access to all the prior results.
This makes use of promises built into the AWS.DynamboDB interface and makes our own promise for conn.search() (though if I knew more about that interface, it may already have a promise interface).
Here's the sequential version:
// promisify the search method
const util = require('util');
LDAP.prototype.searchAsync = util.promisify(LDAP.prototype.search);
// utility function that does a search and adds the result to the object passed in
// returns a promise that resolves to the object
function ldapSearch(data, key) {
var conn = new LDAP();
return conn.searchAsync('ou=groups,o=amazon.com', { ... }).then(results => {
// put our results onto the passed in object
data[key] = results;
// resolve with the original object (so we can collect data here in a promise chain)
return data;
});
}
router.post('/writeRecord', jsonParser, function(req, res) {
let post = req.post;
let smdb = new AWS.DynamoDB.DocumentClient();
let params = { ... }
// The latest AWS interface gets a promise with the .promise() method
smdb.query(params).promise().then(dbresult => {
return ldapSearch({post, dbresult}, "ldap1");
}).then(result => {
// result.dbresult
// result.ldap1
return ldapSearch(result, "ldap2")
}).then(result => {
// result.dbresult
// result.ldap1
// result.ldap2
// doSomething with all the collected data here
}).catch(err => {
console.log(err);
res.status(500).send("Internal Error");
});
});
And, here's a parallel version that runs all three async operations at once and then waits for all three of the to be done and then has all the results at once:
// if the three async operations you show can be done in parallel
// first promisify things
const util = require('util');
LDAP.prototype.searchAsync = util.promisify(LDAP.prototype.search);
function ldapSearch(params) {
var conn = new LDAP();
return conn.searchAsync('ou=groups,o=amazon.com', { ... });
}
router.post('/writeRecord', jsonParser, function(req, res) {
let post = req.post;
let smdb = new AWS.DynamoDB.DocumentClient();
let params = { ... }
Promise.all([
ldapSearch(...),
ldapSearch(...),
smdb.query(params).promise()
]).then(([ldap1Result, ldap2Result, queryResult]) => {
// process ldap1Result, ldap2Result and queryResult here
}).catch(err => {
console.log(err);
res.status(500).send("Internal Error");
});
});
Keep in mind that due to the pseudo-code nature of the code in your question, this is also pseudo-code where implementation details (exactly what parameters you're searching for, what response you're sending, etc...) have to be filled in. This should be illustrative of promise chaining to serialize operations and the use of Promise.all() for parallelizing operations and promisifying a method that didn't have promises built in.

What are some best practices for handling complex promise tree dependencies? [duplicate]

I have restructured my code to promises, and built a wonderful long flat promise chain, consisting of multiple .then() callbacks. In the end I want to return some composite value, and need to access multiple intermediate promise results. However the resolution values from the middle of the sequence are not in scope in the last callback, how do I access them?
function getExample() {
return promiseA(…).then(function(resultA) {
// Some processing
return promiseB(…);
}).then(function(resultB) {
// More processing
return // How do I gain access to resultA here?
});
}
Break the chain
When you need to access the intermediate values in your chain, you should split your chain apart in those single pieces that you need. Instead of attaching one callback and somehow trying to use its parameter multiple times, attach multiple callbacks to the same promise - wherever you need the result value. Don't forget, a promise just represents (proxies) a future value! Next to deriving one promise from the other in a linear chain, use the promise combinators that are given to you by your library to build the result value.
This will result in a very straightforward control flow, clear composition of functionalities and therefore easy modularisation.
function getExample() {
var a = promiseA(…);
var b = a.then(function(resultA) {
// some processing
return promiseB(…);
});
return Promise.all([a, b]).then(function([resultA, resultB]) {
// more processing
return // something using both resultA and resultB
});
}
Instead of the parameter destructuring in the callback after Promise.all that only became avail­able with ES6, in ES5 the then call would be replaced by a nifty helper method that was provided by many promise libraries (Q, Bluebird, when, …): .spread(function(resultA, resultB) { ….
Bluebird also features a dedicated join function to replace that Promise.all+spread combination with a simpler (and more efficient) construct:
…
return Promise.join(a, b, function(resultA, resultB) { … });
ECMAScript Harmony
Of course, this problem was recognized by the language designers as well. They did a lot of work and the async functions proposal finally made it into
ECMAScript 8
You don't need a single then invocation or callback function anymore, as in an asynchronous function (that returns a promise when being called) you can simply wait for promises to resolve directly. It also features arbitrary control structures like conditions, loops and try-catch-clauses, but for the sake of convenience we don't need them here:
async function getExample() {
var resultA = await promiseA(…);
// some processing
var resultB = await promiseB(…);
// more processing
return // something using both resultA and resultB
}
ECMAScript 6
While we were waiting for ES8, we already did use a very similar kind of syntax. ES6 came with generator functions, which allow breaking the execution apart in pieces at arbitrarily placed yield keywords. Those slices can be run after each other, independently, even asynchronously - and that's just what we do when we want to wait for a promise resolution before running the next step.
There are dedicated libraries (like co or task.js), but also many promise libraries have helper functions (Q, Bluebird, when, …) that do this async step-by-step execution for you when you give them a generator function that yields promises.
var getExample = Promise.coroutine(function* () {
// ^^^^^^^^^^^^^^^^^ Bluebird syntax
var resultA = yield promiseA(…);
// some processing
var resultB = yield promiseB(…);
// more processing
return // something using both resultA and resultB
});
This did work in Node.js since version 4.0, also a few browsers (or their dev editions) did support generator syntax relatively early.
ECMAScript 5
However, if you want/need to be backward-compatible you cannot use those without a transpiler. Both generator functions and async functions are supported by the current tooling, see for example the documentation of Babel on generators and async functions.
And then, there are also many other compile-to-JS languages
that are dedicated to easing asynchronous programming. They usually use a syntax similar to await, (e.g. Iced CoffeeScript), but there are also others that feature a Haskell-like do-notation (e.g. LatteJs, monadic, PureScript or LispyScript).
Synchronous inspection
Assigning promises-for-later-needed-values to variables and then getting their value via synchronous inspection. The example uses bluebird's .value() method but many libraries provide similar method.
function getExample() {
var a = promiseA(…);
return a.then(function() {
// some processing
return promiseB(…);
}).then(function(resultB) {
// a is guaranteed to be fulfilled here so we can just retrieve its
// value synchronously
var aValue = a.value();
});
}
This can be used for as many values as you like:
function getExample() {
var a = promiseA(…);
var b = a.then(function() {
return promiseB(…)
});
var c = b.then(function() {
return promiseC(…);
});
var d = c.then(function() {
return promiseD(…);
});
return d.then(function() {
return a.value() + b.value() + c.value() + d.value();
});
}
Nesting (and) closures
Using closures for maintaining the scope of variables (in our case, the success callback function parameters) is the natural JavaScript solution. With promises, we can arbitrarily nest and flatten .then() callbacks - they are semantically equivalent, except for the scope of the inner one.
function getExample() {
return promiseA(…).then(function(resultA) {
// some processing
return promiseB(…).then(function(resultB) {
// more processing
return // something using both resultA and resultB;
});
});
}
Of course, this is building an indentation pyramid. If indentation is getting too large, you still can apply the old tools to counter the pyramid of doom: modularize, use extra named functions, and flatten the promise chain as soon as you don't need a variable any more.
In theory, you can always avoid more than two levels of nesting (by making all closures explicit), in practise use as many as are reasonable.
function getExample() {
// preprocessing
return promiseA(…).then(makeAhandler(…));
}
function makeAhandler(…)
return function(resultA) {
// some processing
return promiseB(…).then(makeBhandler(resultA, …));
};
}
function makeBhandler(resultA, …) {
return function(resultB) {
// more processing
return // anything that uses the variables in scope
};
}
You can also use helper functions for this kind of partial application, like _.partial from Underscore/lodash or the native .bind() method, to further decrease indentation:
function getExample() {
// preprocessing
return promiseA(…).then(handlerA);
}
function handlerA(resultA) {
// some processing
return promiseB(…).then(handlerB.bind(null, resultA));
}
function handlerB(resultA, resultB) {
// more processing
return // anything that uses resultA and resultB
}
Explicit pass-through
Similar to nesting the callbacks, this technique relies on closures. Yet, the chain stays flat - instead of passing only the latest result, some state object is passed for every step. These state objects accumulate the results of the previous actions, handing down all values that will be needed later again plus the result of the current task.
function getExample() {
return promiseA(…).then(function(resultA) {
// some processing
return promiseB(…).then(b => [resultA, b]); // function(b) { return [resultA, b] }
}).then(function([resultA, resultB]) {
// more processing
return // something using both resultA and resultB
});
}
Here, that little arrow b => [resultA, b] is the function that closes over resultA, and passes an array of both results to the next step. Which uses parameter destructuring syntax to break it up in single variables again.
Before destructuring became available with ES6, a nifty helper method called .spread() was pro­vi­ded by many promise libraries (Q, Bluebird, when, …). It takes a function with multiple parameters - one for each array element - to be used as .spread(function(resultA, resultB) { ….
Of course, that closure needed here can be further simplified by some helper functions, e.g.
function addTo(x) {
// imagine complex `arguments` fiddling or anything that helps usability
// but you get the idea with this simple one:
return res => [x, res];
}
…
return promiseB(…).then(addTo(resultA));
Alternatively, you can employ Promise.all to produce the promise for the array:
function getExample() {
return promiseA(…).then(function(resultA) {
// some processing
return Promise.all([resultA, promiseB(…)]); // resultA will implicitly be wrapped
// as if passed to Promise.resolve()
}).then(function([resultA, resultB]) {
// more processing
return // something using both resultA and resultB
});
}
And you might not only use arrays, but arbitrarily complex objects. For example, with _.extend or Object.assign in a different helper function:
function augment(obj, name) {
return function (res) { var r = Object.assign({}, obj); r[name] = res; return r; };
}
function getExample() {
return promiseA(…).then(function(resultA) {
// some processing
return promiseB(…).then(augment({resultA}, "resultB"));
}).then(function(obj) {
// more processing
return // something using both obj.resultA and obj.resultB
});
}
While this pattern guarantees a flat chain and explicit state objects can improve clarity, it will become tedious for a long chain. Especially when you need the state only sporadically, you still have to pass it through every step. With this fixed interface, the single callbacks in the chain are rather tightly coupled and inflexible to change. It makes factoring out single steps harder, and callbacks cannot be supplied directly from other modules - they always need to be wrapped in boilerplate code that cares about the state. Abstract helper functions like the above can ease the pain a bit, but it will always be present.
Mutable contextual state
The trivial (but inelegant and rather errorprone) solution is to just use higher-scope variables (to which all callbacks in the chain have access) and write result values to them when you get them:
function getExample() {
var resultA;
return promiseA(…).then(function(_resultA) {
resultA = _resultA;
// some processing
return promiseB(…);
}).then(function(resultB) {
// more processing
return // something using both resultA and resultB
});
}
Instead of many variables one might also use an (initially empty) object, on which the results are stored as dynamically created properties.
This solution has several drawbacks:
Mutable state is ugly, and global variables are evil.
This pattern doesn't work across function boundaries, modularising the functions is harder as their declarations must not leave the shared scope
The scope of the variables does not prevent to access them before they are initialized. This is especially likely for complex promise constructions (loops, branching, excptions) where race conditions might happen. Passing state explicitly, a declarative design that promises encourage, forces a cleaner coding style which can prevent this.
One must choose the scope for those shared variables correctly. It needs to be local to the executed function to prevent race conditions between multiple parallel invocations, as would be the case if, for example, state was stored on an instance.
The Bluebird library encourages the use of an object that is passed along, using their bind() method to assign a context object to a promise chain. It will be accessible from each callback function via the otherwise unusable this keyword. While object properties are more prone to undetected typos than variables, the pattern is quite clever:
function getExample() {
return promiseA(…)
.bind({}) // Bluebird only!
.then(function(resultA) {
this.resultA = resultA;
// some processing
return promiseB(…);
}).then(function(resultB) {
// more processing
return // something using both this.resultA and resultB
}).bind(); // don't forget to unbind the object if you don't want the
// caller to access it
}
This approach can be easily simulated in promise libraries that do not support .bind (although in a somewhat more verbose way and cannot be used in an expression):
function getExample() {
var ctx = {};
return promiseA(…)
.then(function(resultA) {
this.resultA = resultA;
// some processing
return promiseB(…);
}.bind(ctx)).then(function(resultB) {
// more processing
return // something using both this.resultA and resultB
}.bind(ctx));
}
A less harsh spin on "Mutable contextual state"
Using a locally scoped object to collect the intermediate results in a promise chain is a reasonable approach to the question you posed. Consider the following snippet:
function getExample(){
//locally scoped
const results = {};
return promiseA(paramsA).then(function(resultA){
results.a = resultA;
return promiseB(paramsB);
}).then(function(resultB){
results.b = resultB;
return promiseC(paramsC);
}).then(function(resultC){
//Resolve with composite of all promises
return Promise.resolve(results.a + results.b + resultC);
}).catch(function(error){
return Promise.reject(error);
});
}
Global variables are bad, so this solution uses a locally scoped variable which causes no harm. It is only accessible within the function.
Mutable state is ugly, but this does not mutate state in an ugly manner. The ugly mutable state traditionally refers to modifying the state of function arguments or global variables, but this approach simply modifies the state of a locally scoped variable that exists for the sole purpose of aggregating promise results...a variable that will die a simple death once the promise resolves.
Intermediate promises are not prevented from accessing the state of the results object, but this does not introduce some scary scenario where one of the promises in the chain will go rogue and sabotage your results. The responsibility of setting the values in each step of the promise is confined to this function and the overall result will either be correct or incorrect...it will not be some bug that will crop up years later in production (unless you intend it to!)
This does not introduce a race condition scenario that would arise from parallel invocation because a new instance of the results variable is created for every invocation of the getExample function.
Example is available on jsfiddle
Node 7.4 now supports async/await calls with the harmony flag.
Try this:
async function getExample(){
let response = await returnPromise();
let response2 = await returnPromise2();
console.log(response, response2)
}
getExample()
and run the file with:
node --harmony-async-await getExample.js
Simple as can be!
Another answer, using babel-node version <6
Using async - await
npm install -g babel#5.6.14
example.js:
async function getExample(){
let response = await returnPromise();
let response2 = await returnPromise2();
console.log(response, response2)
}
getExample()
Then, run babel-node example.js and voila!
This days, I also hava meet some questions like you. At last, I find a good solution with the quesition, it's simple and good to read. I hope this can help you.
According to how-to-chain-javascript-promises
ok, let's look at the code:
const firstPromise = () => {
return new Promise((resolve, reject) => {
setTimeout(() => {
console.log('first promise is completed');
resolve({data: '123'});
}, 2000);
});
};
const secondPromise = (someStuff) => {
return new Promise((resolve, reject) => {
setTimeout(() => {
console.log('second promise is completed');
resolve({newData: `${someStuff.data} some more data`});
}, 2000);
});
};
const thirdPromise = (someStuff) => {
return new Promise((resolve, reject) => {
setTimeout(() => {
console.log('third promise is completed');
resolve({result: someStuff});
}, 2000);
});
};
firstPromise()
.then(secondPromise)
.then(thirdPromise)
.then(data => {
console.log(data);
});
I am not going to use this pattern in my own code since I'm not a big fan of using global variables. However, in a pinch it will work.
User is a promisified Mongoose model.
var globalVar = '';
User.findAsync({}).then(function(users){
globalVar = users;
}).then(function(){
console.log(globalVar);
});
Another answer, using sequential executor nsynjs:
function getExample(){
var response1 = returnPromise1().data;
// promise1 is resolved at this point, '.data' has the result from resolve(result)
var response2 = returnPromise2().data;
// promise2 is resolved at this point, '.data' has the result from resolve(result)
console.log(response, response2);
}
nynjs.run(getExample,{},function(){
console.log('all done');
})
Update: added working example
function synchronousCode() {
var urls=[
"https://ajax.googleapis.com/ajax/libs/jquery/1.7.0/jquery.min.js",
"https://ajax.googleapis.com/ajax/libs/jquery/1.8.0/jquery.min.js",
"https://ajax.googleapis.com/ajax/libs/jquery/1.9.0/jquery.min.js"
];
for(var i=0; i<urls.length; i++) {
var len=window.fetch(urls[i]).data.text().data.length;
// ^ ^
// | +- 2-nd promise result
// | assigned to 'data'
// |
// +-- 1-st promise result assigned to 'data'
//
console.log('URL #'+i+' : '+urls[i]+", length: "+len);
}
}
nsynjs.run(synchronousCode,{},function(){
console.log('all done');
})
<script src="https://rawgit.com/amaksr/nsynjs/master/nsynjs.js"></script>
When using bluebird, you can use .bind method to share variables in promise chain:
somethingAsync().bind({})
.spread(function (aValue, bValue) {
this.aValue = aValue;
this.bValue = bValue;
return somethingElseAsync(aValue, bValue);
})
.then(function (cValue) {
return this.aValue + this.bValue + cValue;
});
please check this link for further information:
http://bluebirdjs.com/docs/api/promise.bind.html
function getExample() {
var retA, retB;
return promiseA(…).then(function(resultA) {
retA = resultA;
// Some processing
return promiseB(…);
}).then(function(resultB) {
// More processing
//retA is value of promiseA
return // How do I gain access to resultA here?
});
}
easy way :D
I think you can use hash of RSVP.
Something like as below :
const mainPromise = () => {
const promise1 = new Promise((resolve, reject) => {
setTimeout(() => {
console.log('first promise is completed');
resolve({data: '123'});
}, 2000);
});
const promise2 = new Promise((resolve, reject) => {
setTimeout(() => {
console.log('second promise is completed');
resolve({data: '456'});
}, 2000);
});
return new RSVP.hash({
prom1: promise1,
prom2: promise2
});
};
mainPromise()
.then(data => {
console.log(data.prom1);
console.log(data.prom2);
});
Solution:
You can put intermediate values in scope in any later 'then' function explicitly, by using 'bind'. It is a nice solution that doesn't require changing how Promises work, and only requires a line or two of code to propagate the values just like errors are already propagated.
Here is a complete example:
// Get info asynchronously from a server
function pGetServerInfo()
{
// then value: "server info"
} // pGetServerInfo
// Write into a file asynchronously
function pWriteFile(path,string)
{
// no then value
} // pWriteFile
// The heart of the solution: Write formatted info into a log file asynchronously,
// using the pGetServerInfo and pWriteFile operations
function pLogInfo(localInfo)
{
var scope={localInfo:localInfo}; // Create an explicit scope object
var thenFunc=p2.bind(scope); // Create a temporary function with this scope
return (pGetServerInfo().then(thenFunc)); // Do the next 'then' in the chain
} // pLogInfo
// Scope of this 'then' function is {localInfo:localInfo}
function p2(serverInfo)
{
// Do the final 'then' in the chain: Writes "local info, server info"
return pWriteFile('log',this.localInfo+','+serverInfo);
} // p2
This solution can be invoked as follows:
pLogInfo("local info").then().catch(err);
(Note: a more complex and complete version of this solution has been tested, but not this example version, so it could have a bug.)
What I learn about promises is to use it only as return values avoid referencing them if possible. async/await syntax is particularly practical for that. Today all latest browsers and node support it: https://caniuse.com/#feat=async-functions , is a simple behavior and the code is like reading synchronous code, forget about callbacks...
In cases I do need to reference a promises is when creation and resolution happen at independent/not-related places. So instead an artificial association and probably an event listener just to resolve the "distant" promise, I prefer to expose the promise as a Deferred, which the following code implements it in valid es5
/**
* Promise like object that allows to resolve it promise from outside code. Example:
*
```
class Api {
fooReady = new Deferred<Data>()
private knower() {
inOtherMoment(data=>{
this.fooReady.resolve(data)
})
}
}
```
*/
var Deferred = /** #class */ (function () {
function Deferred(callback) {
var instance = this;
this.resolve = null;
this.reject = null;
this.status = 'pending';
this.promise = new Promise(function (resolve, reject) {
instance.resolve = function () { this.status = 'resolved'; resolve.apply(this, arguments); };
instance.reject = function () { this.status = 'rejected'; reject.apply(this, arguments); };
});
if (typeof callback === 'function') {
callback.call(this, this.resolve, this.reject);
}
}
Deferred.prototype.then = function (resolve) {
return this.promise.then(resolve);
};
Deferred.prototype.catch = function (r) {
return this.promise.catch(r);
};
return Deferred;
}());
transpiled form a typescript project of mine:
https://github.com/cancerberoSgx/misc-utils-of-mine/blob/2927c2477839f7b36247d054e7e50abe8a41358b/misc-utils-of-mine-generic/src/promise.ts#L31
For more complex cases I often use these guy small promise utilities without dependencies tested and typed. p-map has been useful several times. I think he covered most use cases:
https://github.com/sindresorhus?utf8=%E2%9C%93&tab=repositories&q=promise&type=source&language=

Resources