Async vs Deasync in reality - node.js

The background
I'm writing my next server project in node. The previous version was written in language D, which is majorly an synchronized implementation. We all know, that JS uses a lot of async This is good for performance, but hard to maintain (at least for me). Because node and its node modules might use different async approach (callback, promise and await/async), which makes the code looks not in a consistent style. I'm looking for a balanced approach, that not only can create a high quality and performance node application, but fit the common programming style.
The problem
After first round prototyping, I have my first server. When I start the server, it will first read configuration and then load a set of modules. I expect them to be loaded sequentially. But as I mentioned above, node modules might work in async way. I expect module b to be loaded exactly when module a is loaded, I must use Promise pattern. I have to write code in crazy way, although Promise is already a very programmer friendly approach.
No, I go to the other direction. Currently I choose deasync, which is a gyp module for node. It can really blocks the code execution like this.
const deasync = require('deasync');
let done = false;
loadModule1Async((err, result) => {
done = true;
});
deasync.loopWhile(() => !done);
done = false;
loadModule2Async((err, result) => {
done = true;
});
deasync.loopWhile(() => !done);
OR even more magical like this
const deasync = require('deasync');
const loadModule1Sync = deasync(loadModule1Async);
const loadModule2Sync = deasync(loadModule2Async);
try {
loadModule1Sync();
loadModule2Sync();
} catch(ex) {
console.error(ex);
}
IMO, the deasync way is more straightforward to me, but I have two concerns:
It need to be compiled against the target platform, so that the deployment process will be complicated for production.
I checked its issue list. There are some known serious issue, e.g. hanging in some extream cases.
The question
Can the coming new feature async/await really do the same thing as deasync does? If it is never gonna happen due to the JS nature. I might stop thinking about this.
Please share me some lights on the way to node.

I don't know too much about how deasync works, I took a look at it and looks very convenient, but I don't think it's thought to be used through all your code in a server.
little bit performance downgrade
It depends on your case, but it could be more than a little downgrade. You are wrapping all your asynchronous methods in your code, that does not sound good in general. Doing a benchmark could be a good idea.
Other thing:
Because node and its node modules might use different async approach (callback, promise and await/async)
In general modules follow callback style, Promise style, or both of them. Most of the big ones are compatibles with both of them (in general if you don't pass a callback a Promise is returned).
async/await is different, this works on top of Promises, so a module method that returns a Promise can be used with async/await in your code.
So answering your question, I would stick with Promises in the whole code, it's the tendency now, and in general they are easier to deal with, compared to callbacks.
To me async/await is the easiest way, since it keeps calls asynchronous, but the code is written in a synchronous way. Sadly at the moment async/await is only usable in Node.js through a flag, so it's not recommended to be used in production.

Related

Bluebird instead of Co in Koa?

Seems like Bluebird overlaps Co in generator/coroutine related functionality. Bluebird is espoused to have exceptional speed-performance, so for discussion sake, (assuming the aforementioned overlap premise is true) if one wanted to substitute Bluebird for Co in Koa (Node.js context), could it be easily be done without diminishing Koa's functionality, and if so how?
(My guess is it can't practically be done since it seems Koa is built over Co and doesn't explicitly expose it, but facades it. Such a substitution it seems would be tantamount to replacing jQuery with something else in Bootstrap)
First of all, bluebird and co are not comparable like that. You mean Bluebird.coroutine vs co (short for coroutine).
Now, the difference between Bluebird.coroutine and co is that co only allows you to yield a certain set of hard-coded types. While Bluebird.coroutine can be configured to support yielding arbitrary types, the documentation for example contains examples how you can add support for yielding thunks and callbacks.
Async generators are so trivial that the only differences there can be between implementations is what types you can yield and how it performs. Not much room to be better or worse.
However bluebird.coroutine is only a fraction of bluebird features.
Generators only solve the problem of making a sequence of actions less verbose. There is a lot of useful functionality for more advanced needs like resource management, concurrency coordination, error handling, cancellation+timeouts and long stack traces which are impossible or extremely painful if you only have async generators powered by thunks/callbacks/minimal promises.
You can make a drop-in replacement for co by configuring all the yield types that co supports and then just using bluebird.coroutine:
var co = require("bluebird").coroutine;
// Configure all yield types you need using co.addYieldHandler
// See documentation for examples
module.exports = co;
However this doesn't really make any sense since very little code actually should run directly in your request handler - the functions that the request handler calls however do. And those functions are not helped by koa (hmm so what is the point of koa again? :D), so they can be bluebird coroutines directly.
esailija said this about Bluebird,
a feature is being added that allows not only yielding callbacks, thunks etc but any arbitrary thing that comes to your mind. Bluebird is also the fastest. So after this version koa should be just using bluebird indeed. See https://github.com/petkaantonov/bluebird/issues/131#issuecomment-36975495
That said, I don't believe him. And, I don't believe a bluebird wrapper would be faster than Co -- if such a thing were possible. Co.js works, and there is no way possible to get Bluebird.js to pass the tests currently. If you're using ES6, ignore Bluebird entirely and use Co.

Node.js and Complex Business Logic

I know the following question might be a little bit vague but still I need some answers about it.
I'm about to implement a JSON based WebService that will feed information to a couple of websites and mobile Apps.
Since most of time we're doing CPU intensive MySQL queries and need to cache data, PHP is no longer enough and we decided to move to Node.js.
How can one plan, manage, implement and maintain complex business logic with Node.js since we're talking about fully async programming and not something run-time oriented like PHP, Java etc...?
I've tried to put something very simple together with Node.js and after some minutes all my code starts be full of nasty callbacks, overkill dependency injection and it becomes a mess. How should I convert the typical synchronous workflow into something Node friendly?
I've noticed most of the Node.js community is looking at premisses as a way to solve this issues, but still I don't see it is enough, its still a mess...
Any tips?
Managing complex business logic with node is possible. I can give you some tips :
Code organization point of view
Think async
Easy to say, but when I began node, comming from C programming, it wasn't easy at all.
If you have something like :
read(function(){
//some stuff
write(function(){
//some other stuff
[etc...]
}
}
Your code should probably be :
var onRead = function(){
//some stuff
write(onWrite);
}
var onWrite = function(){
//some other stuff
}
read(onRead);
or even better : Make a onRead module ? wich lead me to second advice :
Make modules
A lot of them. And load them on start.
A 50 line module can't really be that complicated to read, can it?
It takes some time to get accustomed with this, but you'll feel at home really quickly.
Use modules designed to remove "nested callback hells"
async is one of them. There are many out there.
I don't really use those. Often it's just a workaround for bad design, but they can be really useful in some situations.
Software architecture point of view
Doesn't change as much as you can expect. Mostly you'll want something similar to what you'd do on a stateless php app, apply above tips and it should be OK.
If you want more precise advice, maybe you should paste a code sample which is representative of your problems to let us explain how you can improve it.

How to avoid deep nested code with mongoose / node?

I am trying to improve the readability of a biggish nodejs app that uses a lot of mongooose. The problem is that with a lot of dependent queries the callbacks get out of hand.
What are the practices of handling this issue?
There are three common solutions for your problem.
First one is async.js lib.
Second one is using Promises. There are more then one implementation of promises in node.js. I know three implementations:
node promise
vow
Q promise
Third one is to use Fibers. There is fibers promise library that do all the tricky work for you.
There was plenty of similar questions before. For example, check this one.
All this libraries do the same thing - they make node.js asynchronous code pretty and readable. So, just choose one that looks simpler for you.
As for me, I prefer async.js lib.
Update: mongoose.js have its own build-in promise - mpromise. You can access it as mongoose.promise. But whenever you call exec() function on a query in mongoose it returns you a promise. I never actually used mongoose.js promises except for the REPL, but you may give it a try.

node.js modules: Async vs Fibers.promise vs Q_oper8

Just wondering if anyone could give me a comparison of trade-offs between these modules for handling async events. Specifically, I'm interested in knowing about reasons to use Async instead of Fibers.promise, which I am using quite extensively at least in my test code right now. In particular, one of the major pluses I see in Fibers.promise is that I can keep the stack chain front bifurcating, making it possible to use try { } catch { } finally, and also allowing me to ensure that after a request has been handled that the response is ended.
Is anyone using Q_oper8? I found this on another page and was just wondering if that's already dead or if its something I should check out.
I've never heard of Q_oper8, so I can't comment on it, but I'll come at this from the other direction. I heard about async first and Fiber (and its helper libraries) second, and I don't like the latter, actually.
The Downsides of Fiber
Unfamiliarity for other Javascript developers
Fiber introduces the concept of co-routines to Javascript via a compiled Fiber native method that takes over the interpretation of the Javascript code passed to it, intercepting calls to yield to jump back to the waiting co-routine.
This may not matter to you, but if you need to work on a team, you'll have to teach the concept to your members (or hope they have experience with the concept from other languages, like Go).
No Windows Support
So, in order to use Fiber or any of the libraries written on top of it, you'll have to natively compile it for your platform first. I don't use Windows, but note that Fiber is not supported on Windows, so that restricts the utility of your own library off-the-bat. Which means you won't be finding general-purpose Node.js libraries written in Fiber at all (and you probably wouldn't have, anyways, since it adds a costly compilation step that you'd otherwise avoid with async).
Browser Incompatible
This means any code you write using Fiber will not be able to run in the browser, because you can't mix native code with the browser (nor would I as a browser user want you to), even if everything you write is "Javascript" (it's syntatically Javascript, but semantically not).
More Difficult Debugging
While the "callback hell" may be less visually pleasing, Continuation-Passing Style does have one very good thing going for it over Co-Routines -- you know exactly where a problem has occurred from the call stack and can trace backwards. Co-Routines enter the function at more than one point in the program, and can exit from three kinds of calls: return, throw and yield(), where the latter is also a return point.
With co-routines, you have cross-execution between two or more functions running "simultaneously", and you may have more than one set of co-routines running at the same time on the event loop. With traditional callbacks, you're guaranteed that the outer scope of the function is static during the execution of said function, so you only need to check those outer variables once if they're needed. Co-routines need these checks to be run after every yield() (since it's usage with the originating co-routine would be translated into a callback chain in real Javascript).
Basically, I think the co-routine concept is made more difficult to work with because it has to exist inside of the Javascript event loop, rather than being a method to implement one.
What makes Async "better"?
Worse is Better
It's sort of the "worse-is-better" idea, actually. Rather than extend the Javascript language to try and get rid of its warts (and create new ones, in my opinion), Async is a pure-Javascript solution to cover them up, like makeup.
Control flow explicit
The Async functions describe different types of logic flow that needs to cross the event loop barrier, and the library covers up the implementation details of the callback code needed to implement that logic, and you just provide it functions it should run in roughly the linear order they will execute across the event loop.
If you're willing to drop the first indentation level around the async methods' arguments, you have no extra indentation versus Co-Routines and only a minor number of extra lines of function(callback) { declarations, like this:
var async = require('async');
var someArray = [1, 2, 3, 4, 5, 6, 7, 8, 9];
async.forEach(someArray,
function(number, callback) {
//Do something with the number
callback();
}, function(err) {
//Done doing stuff, or one of the calls to the previous function returned an error I need to deal with
});
In this case, you know that all of the variables your code is using could only have been changed before your code is run if they weren't changed by your code, so you can debug easier, and there is only one "return" mechanism: callback(). You either callback with nothing on success or pass the callback an error when something's gone wrong.
Code reuse not difficult
The above example makes code reuse difficult but it doesn't have to be. You can always pass in named functions as the parameters:
var async = require('async');
// Javascript doesn't care about declaration order within a scope,
// so order the declarations in a way that's most readable to you
async.forEach(someArray, frazzleNumber, doneFrazzling);
var someArray = [1, 2, 3, 4, 5, 6, 7, 8, 9];
function frazzleNumber(number, callback) {
// Do something to number
callback();
}
function doneFrazzling(err) {
// Do something or handle error
}
Functional, not imperative
The async module discourages the use of imperative-style flow control and encourages (requires, for the parts that cross the event loop) the use of functions for flow control.
The advantage of the functional style is that you can easily re-use the body of your loop or your conditional, and that you can create new control flow "verbs" that better match the flow of your code (demonstrated by the very existence of the async library), like the async.auto control flow method that implements dependency graph resolution for function call order. (You specify a series of named functions and list the other functions, if any, that it depends on to execute, and auto runs first the "independent" functions then the next function that can run based on when its dependent functions have finished running.)
Rather than writing your code to fit the imperative style dictated by your language, you write your code as the logic of the problem dictates, and implement the "glue" control flow to get it to happen.
In Summary
Fiber, by its very nature of extending the Javascript language, cannot develop a large ecosystem within Node.js, especially when Async gets 80% of the way on the looks department, and has none of the other downsides of co-routines in Javascript.
The short answer:
Async is a pure/classic javascript solution to managing single-thread asynchronousity
Fibers is a node.js extension for creating coroutines. It includes a futures library for managing single-thread asynchronousity.
There are many other futures libraries (listed below) that don't require an extension of javascript.
Q_oper8 is a node.js module for managing multi-process concurrency
Note that none of these offer "threads" and so none can be said to do multithreading (though there is a node.js extension for that too: threads_a_gogo).
Async vs Fiber/futures
Async
Async and Fibers/futures are different ways to solve the same problem: managing asynchronously resolving dependencies. Async seems to have many more "bells and whistles" than many other libraries that try to solve this problem, which in my opinion makes it worse (much more cognitive overhead - ie more crap to learn).
In javascript basic asynchronisity looks like this:
asyncCall(someParam, function(result) {
useThe(result);
});
If you have a situation that requires more than just basic asynchronisity, like where you need the results of two asyncronous calls, you might do something like this:
asyncCall1(someParam, function(result0) {
asyncCall2(someParam, function(result1) {
use(result0, result1);
}
});
Already starts to look like callback hell. Also its inefficient because the second call is waiting for the first call to complete even though it isn't dependent on it, not to mention the code doesn't even do any sort of reasonable error handling. Async provides one solution to writing it a little more efficiently:
async.parallel([
function(callback) {
asyncCall1(someParam, function(result0) {
callback(null,result0);
},
function(callback) {
asyncCall1(someParam, function(result1) {
callback(null,result1);
},
}
],
function(err, results) {
use(results[0], results[1]);
});
So to me, thats rather worse than callback hell, but to each his own I suppose. Despite it being ugly, it allows both calls to happen simultaneously (as long as they make non-blocking IO calls or something like that). Async has many more options for managing asynchronous code, so if you're interested take a look at the documentation.
Enter fiber/futures
The coroutines the Fibers module includes a futures library that uses coroutines to re-inject asynchronous events back into the current continuation (future.wait()).
Fibers is different from most other futures libraries because it allows the current continuation to wait on an asynchronous event - meaning it doesn't require the use of callbacks in order for you to get a value back from an async request - allowing asynchronous code to become synchronous-like. Read about coroutines for more about that.
Node.js has io functions like readFileSync, which lets you wait on the function in-line while it gets the file for you. This is not something that is normally done in javascript, and isn't something that can be written in pure javascript - it requires an extension like Fibers.
Going back to the same asynchronous example above, this is what it would look like with fibers/futures:
var future0 = asyncCall1(someParam);
var future1 = asyncCall2(someParam);
use(future0.wait(), future1.wait());
This is drastically simpler and just as efficient as the Async mess up there. It avoids callback-hell in an elegant efficient way. There are (minor) downsides though. David Ellis overstated many of the downsides, so I'll repeat the only valid one here:
Browser Incompatibility
By virtue of Fibers being a node.js extension, it will not be compatible with browsers. This will make sharing code that uses fibers impossible with both a node.js server and the browser. However, there is a strong argument that most asynchronous code you want on the server (filesystem, database, network calls) is not the same code you want on a browser (ajax calls). Maybe timeouts collide, but that seems like it.
Beyond that, the streamline.js project has the ability to bridge this gap. Seems like it has a compilation process that can transform streamline.js code using synchronization and futures into pure javascript using the callback style, similar to the now unsupported Narrative Javascript. Streamline.js can use a couple different mechanisms behind the scenes, one being node.js Fibers, another being ECMAScript 6 generators, and the last being translation into callback-style javascript which I already mentioned.
More difficult debugging
This one seems like a valid, if minor, gripe. Even if you're just planning on using fibers/futures, and not using coroutines for anything else, there might still be confusing context switches because of unexpected function exit (and entrance) points.
Introduces pre-emptiveness into javascript
This is probably the most major problem with fibers, since it has the possibility (however unlikely) of introducing hard-to-understand bugs. Basically, because a Fiber yield can cause a temporary exit of a set of code to another undetermined function, its possible that some invalid state can be read or introduced. See this article for more info. Personally, I think the incredible cleanness of fibers/futures and similar structures is well worth the rare insidious bugs. Many more bugs are caused by awful concurrency code.
Invalid gripes
Not on windows: this just isn't true anymore
Unfamiliarity with coroutines: A. Unfamiliarity is never a reason to shun something. If its good its good, regardless of how familiar you are with it. B. While coroutines and yields may be unfamiliar, futures are an easy concept to understand.
Other futures libraries
There are many libraries that implement futures, where the concept may be called "futures", "deferred objects", or "promises". This includes libraries like async-future, streamline.js, Q, when.js, promiscuous, jQuery's deferred, coolaj86's futures, kriszyp's promises, and Narrative Javascript.
Most of these use callbacks to resolve the futures, which get around many of the problems Fibers introduces. However, they aren't quite as clean as fibers/futures, tho they are far cleaner than Async. Here's the same example again using my own async-future:
var future0 = asyncCall1(someParam);
var future1 = asyncCall2(someParam);
Future.all([future0, future1]).then(function(results) {
use(results[0], results[1])
}).done()
Q_oper8
Q_oper8 is really a different beast. It runs jobs in a queue using a pool of processes. Since javascript is single-threaded*, and javascript doesn't have native threading available, processes are the usual way to take advantage of more than one processor in node.js. Q_oper8 is intended as an alternative to managing processes using node.js's child_process module.
You should also check out Step.
It handles only a small subset of what async can do, but I think the code is much easier to read. It's great for just handling the normal case of doing a sequence of things, with some of those things happening in parallel.
I tend to use Step for the bulk of my logic, and then use async occasionally when I need to apply methods repeatedly in serial or parallel execution (ie - call this function until, or call this function on each element of this array).
I'm using jQuery's Deferred functionality on the client and jQuery Deferred for nodejs on the server in place of nested callbacks. It has greatly reduced the code and made things so readable.
http://techishard.wordpress.com/2012/05/23/promises-promises-a-concise-pattern-for-getting-and-showing-my-json-array-with-jquery-and-underscore/
http://techishard.wordpress.com/2012/05/29/making-mongoose-keep-its-promises-on-the-server/

Node.js control flow: callbacks or promises? [closed]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
I know there are many control flow libraries for node.js. Some of them let one chain async functions with callbacks (like async, asyncblock, etc), the others use promise concept (Q, deferred, futures, etc). Given a long running script making a series of actions one after another that may fail at any time, which control flow would you prefer and why? What are the pros and cons?
Pros for callbacks:
Simple to understand and create.
Somewhat more efficient, because fewer objects are created and garbage collected.
Node opted for (error,result) callbacks throughout. I recommend following their argument order for consistency. (As opposed to say (result1, result2, result3, error).)
Pros for promises:
Provides a fluent interface, which can sometimes help to mitigate nested callback hell, as shown here. Code appears to flow linearly by chaining .then(foo).then(bar) calls.
A good promises library will let you run many asynchronous operations in parallel, and continue only when they are all completed. The Deferred library does this seamlessly through map, Q has allResolved, and ES6 Promises offer Promise.all(). (This is also possible with callbacks, e.g. using async.parallel(), but not built in.)
A good promises library will let you specify one error handling function which will be called if any of the queued functions fail. To do this with callbacks requires a little boilerplate: if (err) return callback(err); at the start of every callback.
It would make sense to use callbacks near the bottom of the stack, for code which will be run many times per second. Higher up the stack, promises may be preferable as they are easier to read and understand, and can handle errors more elegantly.
It is worth noting that promises can be built from callbacks at runtime. So you can implement your core code in the minimalist callback form, and still expose a promises version of the library if you want to. (As in Q.nfbind().)
I would be interested to hear other pros/cons.
Bonus tip: Always handle errors! With both methods, if you do not handle the error then it will simply disappear, leaving you in the dark about why your code did not work as expected.
Callbacks should always handle if (err) ... and Promises should always have a .catch() if they do not return.
Even if you expect errors sometimes, and don't need to handle those, not handling unexpected errors means you won't hear about errors from developer mistakes such as typos, if the code is changed in future.
An alternative to .catch() for Promises is to listen for unhandled rejections. Personally I use this to issue a warning that .catch() was missing!
I don't think there are many objective pros and cons. Async is very popular (based on npm packages that depend on it).
I like the control flow libraries (specifically async), because it is easier for me to understand. Promises confuse me, while async is easily understandable. I suspect its just a learning curve thing though, and promises would be more readable if I spent the effort in learning them. But should I expect that of people trying to read my code too?
There is a 3rd type too - Fibers. Fibers does not work on Windows yet, but (IMO) offers the clearest syntax for things that should execute in series.
I've been experimenting with a declarative approach with this library: http://chainsjs.org still more work to do on it, but it gives you the ability to define an "execution map" where you can pretty much completely control the flow of execution from a simple mapping.

Resources