How to scan an entire directory tree with node.js? - node.js

Question

Often I would want to scan an entire directory tree (a directory, and everything inside it, including files, subdirs, and those subdir contents too, and their subdir contents too, etc etc).
How would one accomplish this with node? Requirements, is that it should be asynchronous to take advantage of super fast non-blocking IO - and not crash when processing too many files at once.

Answer 1

-- I've updated this answer in 2017 for the progress since 2012 --
Ended up creating these to accomplish it:
https://github.com/bevry/safefs - which now uses https://npmjs.org/package/graceful-fs (which didn't exist before)
https://github.com/bevry/scandirectory - there is also now a vast array of like projects like this
I also created this which is lightweight and super fast:
https://github.com/bevry/readdir-cluster

Answer 2

You can use the module npm dree if you want to achive that. It returns a json that describes the directory tree and it allows you to specify also a fileCallback and a dirCallback, so you can do this:
Here is the code:
const dree = require('dree');
const fileCb = function(file) {
// do what you want
}
const dirCb = function(directory) {
// do what you want
}
dree.scan('path-to-directory', { extensions: [ 'html', 'js' ] }, fileCb, dirCb);

Answer 3

If you want to stick with the 'fs' module, you can do some recursive functions to get them.
Heres a function I made recently to get the tree of a directory.
const fs = require("fs");
// dir is file, depth is how far into a directory it will read.
function treeFiles(dir, depth = 1000) {
if (depth < 1) return;
var sitesList = {};
fs.readdirSync(dir).forEach((file) => {
let base = dir + '/' + file;
// Add file to siteslist object.
sitesList[file] = {"stats": fs.statSync(base), "dir": false};
// Recursive to get directory and tree of files
if (fs.statSync(base).isDirectory()) {
sitesList[file]["dir"] = true;
sitesList[file]["ls"] = treeFiles(base, depth - 1);
}
});
return sitesList;
}
So if I have a file structure which looks like
nodejs_app >
- app.js
- config.js
- images >
- - logo.png
Then the final output of my function reading the nodejs_app directory will look like
{
"app.js": {"stats": {}, "dir": false},
"config.js": {"stats": {}, "dir": false},
"images": {"stats": {}, "dir": true, "ls": {
"logo.png": {"stats": {}, "dir": false}
}
}
Then just call the function with directory and depth into the directory if you want.
let dir = require("path").join(__dirname, "nodejs_app");
let tree = treeFiles(dir);
console.log(tree);
Of course change paths and names to fit your code. I included the depth as to reduce time it takes for it to finish reading a directory.