While coding in Node.js, I encountered many situations when it is so hard to implement some elaborated logic mixed with database queries (I/O).
Consider an example written in python. We need to iterate over an array of values, for each value we query the database, then, based on the results, we need to compute the average.
def foo:
a = [1, 2, 3, 4, 5]
result = 0
for i in a:
record = find_from_db(i) # I/O operation
if not record:
raise Error('No record exist for %d' % i)
result += record.value
return result / len(a)
The same task in Node.js
function foo(callback) {
var a = [1, 2, 3, 4, 5];
var result = 0;
var itemProcessed = 0;
var error;
function final() {
if (itemProcessed == a.length) {
if (error) {
callback(error);
} else {
callback(null, result / a.length);
}
}
}
a.forEach(function(i) {
// I/O operation
findFromDb(function(err, record) {
itemProcessed++;
if (err) {
error = err;
} else if (!record) {
error = 'No record exist for ' + i;
} else {
result += record.value;
}
final();
});
});
}
You can see that such code much harder to write/read, and it is more prone to errors.
My questions:
Is there a way to make above Node.js code cleaner?
Imagine more sophisticated logic. For example, when we obtained a record from the db, we might need do another db query based on some conditions. In Node.js that becomes a nightmare. What are common patterns for dealing with such tasks?
Based on your experience, does the performance gain deserves the productivity loss when you code with Node.js?
Is there other asynchronous I/O framework/language that is easier to work with?
To answer your questions:
There are libraries such as async which provide a variety of solutions for common scenarios when working with asynchronous tasks. For "callback hell" concerns, there are many ways to avoid that as well, including (but not limited to) naming your functions and pulling them out, modularizing your code, and using promises.
More or less what you currently have is a fairly common pattern: having counter and function index variables with an array of functions to call. Again, async can help here because it reduces this kind of boilerplate that you will probably find yourself repeating often. async currently doesn't have methods that really allow for skipping individual tasks, but you could easily do this yourself if you are writing the boilerplate (just increment the function index variable by 2 for example).
From my own experience, if you properly design your javascript code with asynchronous in mind and use a lot of tools like async, you will find it easier to develop with node. Writing for asynchronous vs synchronous in node is typically always going to be more complicated (although less so with generators, fibers, etc. as compared to callbacks/promises).
I personally think that deciding on a language based upon that single aspect is not worthwhile. You have to consider much much more than just the design of the language, for example the size of the community, availability of third party libraries, performance, technical support options, ease of code debugging, etc.
Just write your code more compactly:
// parallel version
function foo (cb) {
var items = [ 1, 2, 3, 4, 5 ];
var pending = items.length;
var result = 0;
items.forEach(function (item) {
findFromDb(item, function (err, record) {
if (err) return cb(err);
if (!record) return cb(new Error('No record for: ' + item))
result += record.value / items.length;
if (-- pending === 0) cb(null, result);
});
});
}
That clocks in at 13 source lines of code compared to the 9 sloc for python that you posted. However, unlike the python that you posted, this code runs all the jobs in parallel.
To do the same thing in series, a trick I usually do is a next() function defined inline that invokes itself and pops a job off of an array:
// sequential version
function foo (cb) {
var items = [ 1, 2, 3, 4, 5 ];
var len = items.length;
var result = 0;
(function next () {
if (items.length === 0) return cb(null, result);
var item = items.shift();
findFromDb(item, function (err, record) {
if (err) return cb(err);
if (!record) return cb(new Error('No record for: ' + item))
result += record.value / len;
next();
});
})();
}
This time, 15 lines. The nice thing is that you can easily control whether the actions should happen in parallel or sequentially or somewhere in between. That is not so easy in a language like python where everything is synchronous and you've got to do lots of work-arounds like threads or evented libraries to get things back up to asynchronous. Try implementing a parallel version of what you have in python! It would most certainly be longer than the node version.
As for the promise/async route: it's not actually all that hard or bad to use ordinary functions for these relatively simple kinds of tasks. In the future (or in node 0.11+ with --harmony) you can use generators and a library like co, but that feature isn't widely deployed yet.
Everyone here seems to be suggesting async, which is a great library. But to give another suggestion, you should take a look at Promises , which is a new built-in being introduced to the language (and currently has several very good polyfills). It allows you to write asynchronous code in a way that looks much more structured. For example, take a look at this code:
var items = [ 1, 2, 3, 4 ];
var processItem = function(item, callback) {
// do something async ...
};
var values = [ ];
items.forEach(function(item) {
processItem(item, function(err, value) {
if (err) {
// something went wrong
}
values.push(value);
// all of the items have been processed, move on
if (values.length === items.length) {
doSomethingWithValues(values, function(err) {
if (err) {
// something went wrong
}
// and we're done
});
}
});
});
function doSomethingWithValues(values, callback) {
// do something async ...
}
Using promises, it would be written something like this:
var items = [ 1, 2, 3, 4 ];
var processItem = function(item) {
return new Promise(function(resolve, reject) {
// do something async ...
});
};
var doSomethingWithValues = function(values) {
return new Promise(function(resolve, reject) {
// do something async ...
});
};
// promise.all returns a new promise that will resolve when all of the promises passed to it have resolved
Promise.all(items.map(processItem))
.then(doSomethingWithValues)
.then(function() {
// and we're done
})
.catch(function(err) {
// something went wrong
});
The second version is much cleaner and simpler, and that barely even scratches the surface of promises real power. And, like I said, Promises are in es6 as a new language built-in, so (eventually) you won't even need to load in a library, it will just be available.
don't use anonymous (un-named) functions they make the code ugly and they make debugging much harder, so always name your functions and define them outside the function scope not inline.
that is a real issue with Node.js (it is called callback hell or pyramid of doom ,..) you can solve this issue by using promises or using async.js which have so many functions for handling different situations (waterfall, parallel, series, auto, ...)
well the performance gain is absolutely a good thing and it is not that much loss (when you start to master it) and also the Node.js community is great.
Check async.js, q.
The more I work with async the more I love it and I like node more. Let me give you a simple example of what I have for a server initialization.
async.parallel ({
"job1": loadFromCollection1,
"job2": loadFromCollection2,
},
function (initError, results) {
if (initError) {
console.log ("[INIT] Server initialization error occurred: " + JSON.stringify(initError, null, 3));
return callback (initError);
}
// Do more stuff with the results
});
In fact, this very same approach can be followed and one can pass different arguments to the different functions that correspond to the various jobs; see for example Passing arguments to async.parallel in node.js.
To be perfectly honest with you, I prefer the node-way which is also non-blocking. I think node forces someone to have a better design and sometimes you spend time creating more definitions and grouping functions and objects in arrays so that you can write better code. The reason I think is that in the end you want to exploit some variant of async and mix and merge stuff accordingly. In my opinion, spending some extra time and thinking about the code a bit more is well worth it when you also take into account that node is asynchronous.
Other than that, I think it is a habit. The more one writes code for node, the more one improves and writes better asynchronous code. What is good on node is that it really forces someone to write more robust code since one starts respecting all the error codes from all the functions much more. For example, how often do people check, say if malloc or new have succeeded and one does not have an error handler for a NULL pointer after the command has been issued? Writing asynchronous code though forces one to respect the events and the error codes that the events have. I guess one obvious reason is that one respects the code that one writes and in the end we have to write code that returns errors so that caller knows what happened.
I really think that you need to give it more time and start working with async more. That's all.
"If you try to code bussiness db login using pure node.js, you go straight to callback hell"
I've recently created a simple abstraction named WaitFor to call async functions in sync mode (based on Fibers): https://github.com/luciotato/waitfor
check the database example:
Database example (pseudocode)
pure node.js (mild callback hell):
var db = require("some-db-abstraction");
function handleWithdrawal(req,res){
try {
var amount=req.param("amount");
db.select("* from sessions where session_id=?",req.param("session_id"),function(err,sessiondata) {
if (err) throw err;
db.select("* from accounts where user_id=?",sessiondata.user_ID),function(err,accountdata) {
if (err) throw err;
if (accountdata.balance < amount) throw new Error('insufficient funds');
db.execute("withdrawal(?,?),accountdata.ID,req.param("amount"), function(err,data) {
if (err) throw err;
res.write("withdrawal OK, amount: "+ req.param("amount"));
db.select("balance from accounts where account_id=?", accountdata.ID,function(err,balance) {
if (err) throw err;
res.end("your current balance is " + balance.amount);
});
});
});
});
}
catch(err) {
res.end("Withdrawal error: " + err.message);
}
Note: The above code, although it looks like it will catch the exceptions, it will not.
Catching exceptions with callback hell adds a lot of pain, and i'm not sure if you will have the 'res' parameter
to respond to the user. If somebody like to fix this example... be my guest.
using wait.for:
var db = require("some-db-abstraction"), wait=require('wait.for');
function handleWithdrawal(req,res){
try {
var amount=req.param("amount");
sessiondata = wait.forMethod(db,"select","* from session where session_id=?",req.param("session_id"));
accountdata= wait.forMethod(db,"select","* from accounts where user_id=?",sessiondata.user_ID);
if (accountdata.balance < amount) throw new Error('insufficient funds');
wait.forMethod(db,"execute","withdrawal(?,?)",accountdata.ID,req.param("amount"));
res.write("withdrawal OK, amount: "+ req.param("amount"));
balance=wait.forMethod(db,"select","balance from accounts where account_id=?", accountdata.ID);
res.end("your current balance is " + balance.amount);
}
catch(err) {
res.end("Withdrawal error: " + err.message);
}
Note: Exceptions will be catched as expected.
db methods (db.select, db.execute) will be called with this=db
Your Code
In order to use wait.for, you'll have to STANDARDIZE YOUR CALLBACKS to function(err,data)
If you STANDARDIZE YOUR CALLBACKS, your code might look like:
var wait = require('wait.for');
//run in a Fiber
function process() {
var a = [1, 2, 3, 4, 5];
var result = 0;
a.forEach(function(i) {
// I/O operation
var record = wait.for(findFromDb,i); //call & wait for async function findFromDb(i,callback)
if (!record) throw new Error('No record exist for ' + i);
result += record.value;
});
return result/a.length;
}
function inAFiber(){
console.log('result is: ',process());
}
// run the loop in a Fiber (keep node spinning)
wait.launchFiber(inAFiber);
see? closer to python and no callback hell
Here is the situation:
I am new to node.js, I have a 40MB file containing multilevel json file like:
[{},{},{}] This is an array of objects (~7000 objects). Each object has properties and a one of those properties is also an array of objects
I wrote a function to read the content of the file and iterate it. I succeeded to get what I wanted in terms of content but not usability. I thought that I wrote an async function that would allow node to serve other web requests while iterating the array but that is not the case. I would be very thankful if anyone can point me to what I've done wrong and how to rewrite it so I can have a non-blocking iteration. Here's the function that handles the situation:
function getContents(callback) {
fs.readFile(file, 'utf8', function (err, data) {
if (err) {
console.log('Error: ' + err);
return;
}
js = JSON.parse(data);
callback();
return;
});
}
getContents(iterateGlobalArr);
var count = 0;
function iterateGlobalArr() {
if (count < js.length) {
innerArr = js.nestedProp;
//iterate nutrients
innerArr.forEach(function(e, index) {
//some simple if condition here
});
var schema = {
//.....get props from forEach iteration
}
Model.create(schema, function(err, post) {
if(err) {
console.log('\ncreation error\n', err);
return;
}
if (!post) {
console.log('\nfailed to create post for schema:\n' + schema);
return;
}
});
count++;
process.nextTick(iterateGlobalArr);
}
else {
console.log("\nIteration finished");
next();
}
Just so it is clear how I've tested the above situation. I open two tabs one loading this iteration which takes some time and second with another node route which does not load until the iteration is over. So essentially I've written a blocking code but not sure how to re-factor it! I suspect that just because everything is happening in the callback I am unable to release the event loop to handle another request...
Your code is almost correct. What you are doing is inadvertently adding ALL the items to the very next tick... which still blocks.
The important piece of code is here:
Model.create(schema, function(err, post) {
if(err) {
console.log('\ncreation error\n', err);
return;
}
if (!post) {
console.log('\nfailed to create post for schema:\n' + schema);
return;
}
});
// add EVERYTHING to the very same next tick!
count++;
process.nextTick(iterateGlobalArr);
Let's say you are in tick A of the event loop when getContents() runs and count is 0. You enter iterateGlobalArr and you call Model.create. Because Model.create is async, it is returning immediately, causing process.nextTick() to add processing of item 1 to the next tick, let's say B. Then it calls iterateGlobalArr, which does the same thing, adding item 2 to the next tick, which is still B. Then item 3, and so on.
What you need to do is move the count increment and process.nextTick() into the callback of Model.create(). This will make sure the current item is processed before nextTick is invoked... which means next item is actually added to the next tick AFTER the model item has been created... which will give your app time to handle other things in between. The fixed version of iterateGlobalArr is here:
function iterateGlobalArr() {
if (count < js.length) {
innerArr = js.nestedProp;
//iterate nutrients
innerArr.forEach(function(e, index) {
//some simple if condition here
});
var schema = {
//.....get props from forEach iteration
}
Model.create(schema, function(err, post) {
// schedule our next item to be processed immediately.
count++;
process.nextTick(iterateGlobalArr);
// then move on to handling this result.
if(err) {
console.log('\ncreation error\n', err);
return;
}
if (!post) {
console.log('\nfailed to create post for schema:\n' + schema);
return;
}
});
}
else {
console.log("\nIteration finished");
next();
}
}
Note also that I would strongly suggest that you pass in your js and counter with each call to iterageGlobalArr, as it will make your iterateGlobalArr alot easier to debug, among other things, but that's another story.
Cheers!
Node is single-threaded so async will only help you if you are relying on another system/subsystem to do the work (a shell script, external database, web service etc). If you have to do the work in Node you are going to block while you do it.
It is possible to create one node process per core. This solution would result in only blocking one of the node processes and leave the rest to service your requests, but this feature is still listed as experimental http://nodejs.org/api/cluster.html.
A single instance of Node runs in a single thread. To take advantage
of multi-core systems the user will sometimes want to launch a cluster
of Node processes to handle the load.
The cluster module allows you to easily create child processes that
all share server ports.
I'm using the Q library in nodejs and haven't worked too much with promises in the past, but I have semi complex logic that requires lots of nesting and thought Q would be a good solution, however I'm finding that it seems to be almost the same as just "callback hell".
Basically I have say 5 methods, all which require data from the previous or one of the previous. Here's an example:
We start with some binary data that has a sha1 hash generated based on the binary.
var data = {
hash : "XXX"
, binary: ''
}
First we want to see if we already have this, using this method:
findItemByHash(hash)
If we don't have it, we need to save it, using:
saveItem(hash)
Now we need to associate this to a user, but not only the results of the save. There's now a much larger hierarchy that we associate, so we need to get that first, doing:
getItemHierarchy(item_id), we use the item_id returned from our previous saveItem
Now, we can "copy" these results to a user:
saveUserHierarchy(hierarchy)
Now we're done, however, this assumes the item didn't exist yet. So we need to handle a case where the item did exist. This would be:
We need to check if the user may aleady have this:
getUserItemByItemId(item_id) - item_id was returned from findItemByHash
If it exists, we're done.
If it doesn't:
getItemHierarchy(item_id)
Then
saveUserHierarchy(hierarchy)
Ok, so right now we have callbacks that do these checks, which is fine. But we need to handle errors in each case along the way. That's fine too, just adds to the mess. Really, if any part of the flow throws an error or rejects then it can stop and just handle it in a single place.
Now with Q, we could do something like this:
findItemByHash(hash).then(function(res) {
if (!res) {
return saveItem(hash).then(function(item) {
return getItemHierarchy(item.id).then(function(hierarchy) {
return saveUserHierarchy(hierarchy);
});
})
} else {
return getUserItemByItemId(res.id).then(function(user_item) {
if (user_item) {
return user_item;
}
return getItemHierarchy(res.id).then(function(hierarchy) {
return saveUserHierarchy(hierarchy);
});
});
}
})
//I think this will only handle the reject for findItemByHash?
.fail(function(err) {
console.log(err);
})
.done();
So, I guess my question is this. Are there better ways to handle this in Q?
Thanks!
One of the reasons why I love promises is how easy it is to handle errors. In your case, if any one of those promises fail, it will be caught at the fail clause you have defined. You can specify more fail clauses if you want to handle them on the spot, but it isn't required.
As a quick example, sometimes I want to handle errors and return something else instead of passing along the error. I'll do something like this:
function awesomeFunction() {
var fooPromise = getFoo().then(function() {
return 'foo';
}).fail(function(reason) {
// handle the error HERE, return the string 'bar'
return 'bar';
});
return fooPromise;
}
awesomeFunction().then(function(result) {
// `result` will either be "foo" or "bar" depending on if the `getFoo()`
// call was successful or not inside of `awesomeFunction()`
})
.fail(function(reason) {
// This will never be called even if the `getFoo()` function fails
// because we've handled it above.
});
Now as for your question on getting out of "return hell" - as long as the next function doesn't require information about the previous one, you can chain .then clauses instead of nesting them:
doThis().then(function(foo) {
return thenThis(foo.id).then(function(bar) {
// `thenThat()` doesn't need to know anything about the variable
// `foo` - it only cares about `bar` meaning we can unnest it.
return thenThat(bar.id);
});
});
// same as the above
doThis().then(function(foo) {
return thenThis(foo.id);
}).then(function(bar) {
return thenThat(bar.id);
});
To reduce it further, make functions that combine duplicate promise combinations and we're left with:
function getItemHierarchyAndSave(item) {
return getItemHierarchy(item.id).then(function(hierarchy) {
return saveUserHierarchy(hierarchy);
});
}
findItemByHash(hash).then(function(resItem) {
if (!resItem) {
return saveItem(hash).then(function(savedItem) {
return getItemHierarchyAndSave(savedItem);
});
}
return getUserItemByItemId(resItem.id).then(function(userItem) {
return userItem || getItemHierarchyAndSave(resItem);
});
})
.fail(function(err) { console.log(err); })
.done();
Disclaimer: I don't use Q promises, I perfer when promises primarily for the extra goodies it comes with, but the principles are the same.
Whenever I define a Firebase transaction in NodeJS I notice it always runs three times - the first two times with null data, then finally a third time with actually data. Is this normal/intended?
For example this code:
firebaseOOO.child('ref').transaction(function(data) {
console.log(data);
return data;
});
outputs the following:
null
null
i1: { a1: true }
I would have expected that it only print the last item.
To answer a question in the comments, here is the same with a callback:
firebaseOOO.child('ref').transaction(function(data) {
console.log(data);
return data;
}, function(error, committed, snapshot) {
if (error)
console.log('failed');
else if (!committed)
console.log('aborted');
else
console.log('committed');
console.log('fin');
});
Which yields the following output:
null
null
i1: { a1: true }
committed
fin
I had read the details of how transactions work before posting the question, so I had tried setting applyLocally to false like this:
firebaseOOO.child('ref').transaction(function(data) {
console.log('hit');
return data;
}, function(){}, false);
But it still hits 3 times (just double-checked) so I thought it was something different. Getting the 'value' before transacting does "work" as expected, in that it only hits once, and that's regardless of what applyLocally is set to, so I'm not sure what applyLocally does? This is what I mean by getting the value before transacting:
firebaseOOO.child('ref').once('value', function(data) {
console.log('1');
firebaseOOO.child('ref').transaction(function(data) {
console.log('2');
return data;
});
});
Outputs:
1
2
#Michael: How can one make use of this behavior? Transactions are primarily for having data use itself to modify itself - the prototypical increment++ scenario. So if I need to add 1 to the existing value of 10, and continue working with the result of 11, the first two times the function hits I will have an erroneous result of 1 that I need to handle, and finally the correct result of 11 on the third hit. How can I make use of those two initial 1's? Another scenario (and maybe I shouldn't be using transactions for this, but if it worked like I expected it makes for cleaner code) is to insert a value if it does not yet exist. If transactions only hit once, a null value would mean the value does not exist, and so you could, for example, init the counter to 1 in that case, otherwise add 1 to whatever the value is. With the noisy nulls, this is not possible.
It seems the takeaway from all this is to simply use the 'once' pattern more often than not?
ONCE TRANSACTION PATTERN:
firebaseOOO.child('ref').once('value', function(data) {
console.log('1');
firebaseOOO.child('ref').transaction(function(data) {
console.log('2');
return data;
});
});
The behavior you're seeing here is related to how Firebase fires local events and then eventually synchronizes with the Firebase servers. In this specific example, the "running three times" will only happen the very first time you run the code—after that, the state has been completely synchronized and it'll just trigger once from then on out. This behavior is detailed here: https://www.firebase.com/docs/transactions.html (See the "When a Transaction is run, the following occurs" section.)
If, for example, you have an outstanding on() at the same location and then, at some later time, run this same transaction code, you'll see that it'll just run once. This is because everything is in sync prior to the transaction running (in the ideal case; barring any normal conflicts, etc).
transaction() will be called multiple times and must be able to handle null data. Even if there is existing data in your database it may not be locally cached when the transaction function is run.
firebaseOOO.child('ref').transaction(function(data) {
if(data!=null){
console.log(data);
return data;
}
else {
return data;
}
}, function(error, committed, snapshot) {
if (error)
console.log('failed');
else if (!committed)
console.log('aborted');
else
console.log('committed');
console.log('fin');
});
mongoosejs async code .
userSchema.static('alreadyExists',function(name){
var isPresent;
this.count({alias : name },function(err,count){
isPresent = !!count
});
console.log('Value of flag '+isPresent);
return isPresent;
});
I know isPresent is returned before the this.count async function calls the callback , so its value is undefined . But how do i wait for callback to change value of isPresent and then safely return ?
what effect does
(function(){ asynccalls() asynccall() })(); has in the async flow .
What happens if var foo = asynccall() or (function(){})()
Will the above two make return wait ?
can process.nextTick() help?
I know there are lot of questions like these , but nothing explained about problem of returning before async completion
There is no way to do that. You need to change the signature of your function to take a callback rather than returning a value.
Making IO async is one of the main motivation of Node.js, and waiting for an async call to be completed defeats the purpose.
If you give me more context on what you are trying to achieve, I can give you pointers on how to implement it with callbacks.
Edit: You need something like the following:
userSchema.static('alreadyExists',function (name, callback) {
this.count({alias : name}, function (err, count) {
callback(err, err ? null : !!count);
console.log('Value of flag ' + !!count);
});
});
Then, you can use it like:
User.alreadyExists('username', function (err, exists) {
if (err) {
// Handle error
return;
}
if (exists) {
// Pick another username.
} else {
// Continue with this username.
}
}
Had the same problem. I wanted my mocha tests to run very fast (as they originally did), but at the same time to have a anti-DOS layer present and operational in my app. Running those tests just as they originally worked was crazy fast and ddos module I'm using started to response with Too Many Requests error, making the tests fail. I didn't want to disable it just for test purposes (I actually wanted to have automated tests to verify Too Many Requests cases to be there as well).
I had one place used by all the tests that prepared client for HTTPS requests (filled with proper headers, authenticated, with cookies, etc.). It looked like this more or less:
var agent = thiz.getAgent();
thiz.log('preReq for user ' + thiz.username);
thiz.log('preReq for ' + req.url + ' for agent ' + agent.mochaname);
if(thiz.headers) {
Object.keys(thiz.headers).map(function(header) {
thiz.log('preReq header ' + header);
req.set(header, thiz.headers[header]);
});
}
agent.attachCookies(req);
So I wanted to inject there a sleep that would kick in every 5 times this client was requested by a test to perform a request - so the entire suite would run quickly and every 5-th request would wait to make ddos module consider my request unpunishable by Too Many Requests error.
I searched most of the entries here about Async and other libs or practices. All of them required going for callback - which meant I would have to re-write a couple of hundreds of test cases.
Finally I gave up with any elegant solution and fell to the one that worked for me. Which was adding a for loop trying to check status of non-existing file. It caused a operation to be performed long enough I could calibrate it to last for around 6500 ms.
for(var i = 0; i < 200000; ++i) {
try {
fs.statSync('/path' + i);
} catch(err) {
}
};