using async Node.JS to serve HTTP requests - node.js

Question

I've successfully written a few nodejs HTTP handlers to serve data in response to an HTTP request. However, everything I've written has been using *Sync version of functions. I'm now quickly running into limitations of this approach.
I cannot figure out, however, how to properly use asynchronous functions in the HTTP request context. If I try an async call, processing quickly falls through and returns without giving the code a chance to process data.
What's the correct approach? I haven't been able to find any good examples, so any pointers to literature are appreciated. Short of that, what's an example of a handler for a GET request that scans a local directory and, say, returns a json list of file names and corresponding number of lines (or really any stub code of the above that shows the proper technique).

Answer 1

Here's a simple sample:
var http = require('http')
var fs = require('fs')
function dir (req, res) {
fs.readdir('.', function (error, files) {
if (error) {
res.writeHead(500)
res.end(error.message)
return
}
files.forEach(function (file) {
res.write(file + '\n')
})
res.end()
})
}
var server = http.createServer(dir)
server.listen(7000)
Run with node server.js and test it with curl :7000.
Yes the request handler returns before the readdir callback is executed. That is by design. That's how async programming works. It's OK. when the filesystem IO is done, the callback will execute and the response will be sent.

Answer 2

Peter Lyons' answer is great/correct. I'm going to expand on it a bit and suggest a different method of synchronization using promises and co as well as nested/looping asynchronicity.
/* Script to count all lines of a file */
const co = require("co");
// Promisifed fs -- eventually node will support this on its own
const fs = require("mz/fs");
const rootDir = 'files/';
// Recursivey count the lines of all files in the given directory and sum them
function countLines(directory) {
// We can only use `yield` inside a generator block
// `co` allows us to do this and walks through the generator on its own
// `yield` will not move to the next line until the promise resolves
//
// This is still asynchronous code but it is written in a way
// that makes it look synchronized. This entire block is asynchronous, so we
// can `countLines` of multiple directories simultaneously
return co(function* () {
// `files` will be an array of files in the given directory
const files = yield fs.readdir(directory);
// `.map` will create an array of promises. `yield` only completes when
// *all* promises in the array have resolved
const lines = yield files.map(file => countFileLines(file, directory));
// Sum the lines of all files in this directory
return lines.reduce((a, b) => a + b, 0);
});
}
function countFileLines(file, directory) {
// We need the full path to read the file
const fullPath = `${directory}/${file}`;
// `co` returns a promise, so `co` itself can be yielded
// This entire block is asynchronous so we should be able to count lines
// of files without waiting for each file to be read
return co(function* () {
// Used to check whether this file is a directory
const stats = yield fs.stat(fullPath);
if (stats.isDirectory()) {
// If it is, recursively count lines of this directory
return countLines(fullPath);
}
// Otherwise just get the line count of the file
const contents = yield fs.readFile(fullPath, "utf8");
return contents.split("\n").length - 1;
});
}
co(function* () {
console.log(yield countLines(rootDir));
})
// All errors propagate here
.catch(err => console.error(err.stack));
Note that this is just an example. There are probably already libraries to count lines of files in a directory and there are definitely libraries that simplify recursive reading/globbing of files.