Add All Routes in ./routes to Middleware Stack - node.js

Right now I am using app.use() and require() for each route in my routes directory to add them to the middleware stack (I am using Express).
app.use('/', require('./routes/index'));
app.use('/users', require('./routes/users'));
app.use('/post', require('./routes/post'));
app.use('/submitPost', require('./routes/submitPost'));
...
Instead of doing this manually for each file, I would like to use a for-loop to iterate through the route files in ./routes and add each file to the middleware stack. This is what I have, but it isn't working:
require('fs').readdir('/routes', function (err, files) {
if (!err) {
for (var i = 0; i < files.length; i++) {
var file = files[i].substr(files[i].lastIndexOf('.'));
app.use('/' + file, require('./routes/' + file));
}
}
});
Could someone help me correct this bit of code. On another note, are there any disadvantages to automatically adding all routes in ./routes to the middleware stack?
Thanks in advance.

The main issue here is probably when you are adding the middleware. You are using readdir - the asynchronous method. You likely have a catch-all 404 handler declared after your code, and as the routes you are requiring are added asynchronously, they will probably be added after the catch-all. When the request propagates through the middleware, this would terminate it before it even got to the route.
One other issue is the path you are using: /routes will attempt to look in the route of your filesystem. ./routes or __dirname + '/routes' is probaby what you want.
The following code sample works for me:
var files = require('fs').readdirSync('./routes')
for (var i = 0; i < files.length; i++) {
var file = files[i].substr(0, files[i].lastIndexOf('.'));
app.use('/' + file, require('./routes/' + file));
}

By the way, you can use file-manifest for this. It was actually created specifically for this use case, although it still expects you to call app.use yourself, since order matters for express routes.
So you can do something like:
var fm = require('file-manifest');
var routes = fm.generate('./routes');
app.use('/', routes.home);
app.use('/foo', routes.foo);
// etc.
If you really want it to all happen magically, you could make that work with a custom reduce function, but this is much more explicit and ensures that routes are set up in the right order (so you don't end up with /foo falling before /foo/bar and preventing it from being reached).
I believe I am supposed to qualify that I wrote this library.

There are a few ways to do this. Here's a clean implentation using the basic fs and path modules.
var fs = require("fs"),
path = require("path");
var root = "./routes/"
fs.readdir(root, function (err, files) {
if (err) {
throw err;
}
files.forEach(function (file) {
var filename = file.slice(0, -3);
var routePath = '/' + ((filename === 'index') ? '' : filename); //filter index to use just '/'
app.use(routepath, require(root + filename));
});
});

Related

fs.createReadStream getting a different path than what's being passed in

I'm using NodeJS on a VM. One part of it serves up pages, and another part is an API. I've run into a problem, where fs.createReadStream attempts to access a different path than what is being passed into the function. I made a small test server to see if it was something else in the server affecting path usage, for whatever reason, but it's happening on my test server as well. First, here's the code:
const fs = require('fs');
const path = require('path');
const csv = require('csv-parser');
const readCSV = (filename) => {
console.log('READ CSV GOT ' + filename); // show me what you got
return new Promise((resolve, reject) => {
const arr = [];
fs.createReadStream(filename)
.pipe(csv())
.on('data', row => {
arr.push(row);
})
.on('error', err => {
console.log(err);
})
.on('end', () => {
resolve(arr);
});
}
}
// tried this:
// const dir = path.relative(
// path.join('path', 'to', 'this', 'file),
// path.join('path', 'to', 'CONTENT.csv')
// );
// tried a literal relative path:
// const dir = '../data/CONTENT.csv';
// tried a literal absolute path:
// const dir = '/repo/directory/server/data/CONTENT.csv';
// tried an absolute path:
const dir = path.join(__dirname, 'data', 'CONTENT.csv');
const content = readCSV(dir)
.then(result => {console.log(result[0]);})
.catch(err => {console.log(err);});
...but any way I slice it, I get the following output:
READCSV GOT /repo/directory/server/data/CONTENT.csv
throw er; // Unhandled 'error' event
^
Error: ENOENT: no such file or directory, open '/repo/directory/data/CONTENT.csv'
i.e., is fs.createReadStream somehow stripping out the directory of the server, for some reason? I suppose I could hard code the directory into the call to createReadStream, maybe? I just want to know why this is happening.
Some extra: I'm stuck on node v8.11, can't go any higher. On the server itself, I believe I'm using older function(param) {...} instead of arrow functions -- but the behavior is exactly the same.
Please help!!
Code is perfect working.
I think you file CONTENT.csv should be in data folder like "/repo/directory/data/CONTENT.csv".
I'm answering my own question, because I found an answer, I'm not entirely sure why it's working, and at least it's interesting. To the best of my estimation, it's got something to do with the call stack, and where NodeJS identifies as the origin of the function call. I've got my server set up in an MVC pattern so my main app.js is in the root dir, and the function that's being called is in /controllers folder, and I've been trying to do relative paths from that folder -- I'm still not sure why absolute paths didn't work.
The call stack goes:
app.js:
app.use('/somepath', endpointRouter);
...then in endpointRouter.js:
router.get('/request/file', endpointController.getFile);
...then finally in endpointController.js:
const readCSV = filename => {
//the code I shared
}
exports.getFile = (req, res, next) => {
// code that calls readCSV(filename)
}
...and I believe that because Node views the chain as originating from app.js, it then treats all relative paths as relative to app.js, in my root folder. Basically when I switched to the super unintuitive single-dot-relative path: './data/CONTENT.csv', it worked with no issue.

Node.JS - fs.exists not working?

I'm a beginner in Node.js, and was having trouble with this piece of code.
var fs = require('fs');
Framework.Router = function() {
this.run = function(req, res) {
fs.exists(global.info.controller_file, function(exists) {
if (exists) {
// Here's the problem
res.writeHead(200, {'Content-Type':'text/html'});
var cname = App.ucfirst(global.info.controller)+'Controller';
var c = require(global.info.controller_file);
var c = new App[cname]();
var action = global.info.action;
c[action].apply(global.info.action, global.info.params);
res.end();
} else {
App.notFound();
return false;
}
});
}
};
The problem lies in the part after checking if the 'global.info.controller_file' exists, I can't seem to get the code to work properly inside the: if (exists) { ... NOT WORKING }
I tried logging out the values for all the variables in that section, and they have their expected values, however the line: c[action].apply(global.info.action, global.info.params);
is not running as expected. It is supposed to call a function in the controller_file and is supposed to do a simple res.write('hello world');. I wasn't having this problem before I started checking for the file using fs.exists. Everything inside the if statement, worked perfectly fine before this check.
Why is the code not running as expected? Why does the request just time out?
Does it have something to do with the whole synchronous vs asynchronous thing? (Sorry, I'm a complete beginner)
Thank you
Like others have commented, I would suggest you rewrite your code to bring it more in-line with the Node.js design patterns, then see if your problem still exists. In the meantime, here's something which may help:
The advice about not using require dynamically at "run time" should be heeded, and calling fs.exists() on every request is tremendously wasteful. However, say you want to load all *.js files in a directory (perhaps a "controllers" directory). This is best accomplished using an index.js file.
For example, save the following as app/controllers/index.js
var fs = require('fs');
var files = fs.readdirSync(__dirname);
var dotJs = /\.js$/;
for (var i in files) {
if (files[i] !== 'index.js' && dotJs.test(files[i]))
exports[files[i].replace(dotJs, '')] = require('./' + files[i]);
}
Then, at the start of app/router.js, add:
var controllers = require('./controllers');
Now you can access the app/controllers/test.js module by using controllers.test. So, instead of:
fs.exists(controllerFile, function (exists) {
if (exists) {
...
}
});
simply:
if (controllers[controllerName]) {
...
}
This way you can retain the dynamic functionality you desire without unnecessary disk IO.

Multiple View paths on Node.js + Express

I'm writing a CMS on Node.js with Express Framework. On my CMS I have several modules for users, pages, etc.
I want that each module will have his files on separate folder, including the view files.
Anyone know how can I achieve that?
I'm using swig as my template engine but I can replace it to something else if it will helps.
Last Update
The multiple view folders feature is supported by the framework since Express 4.10
Just pass an array of locations to the views property, like so.
app.set('views', [__dirname + '/viewsFolder1', __dirname + '/viewsFolder2']);
Express 2.0
As far as I know express doesn't support multiple view paths or namespaces at the moment (like the static middleware do)
But you can modify the lookup logic yourself so that it works the way you want, for example:
function enableMultipleViewFolders(express) {
// proxy function to the default view lookup
var lookupProxy = express.view.lookup;
express.view.lookup = function (view, options) {
if (options.root instanceof Array) {
// clones the options object
var opts = {};
for (var key in options) opts[key] = options[key];
// loops through the paths and tries to match the view
var matchedView = null,
roots = opts.root;
for (var i=0; i<roots.length; i++) {
opts.root = roots[i];
matchedView = lookupProxy.call(this, view, opts);
if (matchedView.exists) break;
}
return matchedView;
}
return lookupProxy.call(express.view, view, options)
};
}
You will enable the new logic by calling the function above and passing express as a parameter, and then you will be able to specify an array of views to the configuration:
var express = require('express');
enableMultipleViewFolders(express);
app.set('views', [__dirname + '/viewsFolder1', __dirname + '/viewsFolder2']);
Or, if you prefer, you can patch the framework directly (updating the view.js file inside it)
This should work in Express 2.x, not sure if it will with the new version (3.x)
UPDATE
Unluckily the above solution won't work in Express 3.x since express.view would be undefined
Another possible solution will be to proxy the response.render function and set the views folder config until it gets a match:
var renderProxy = express.response.render;
express.render = function(){
app.set('views', 'path/to/custom/views');
try {
return renderProxy.apply(this, arguments);
}
catch (e) {}
app.set('views', 'path/to/default/views');
return renderProxy.apply(this, arguments);
};
I've not tested it, it feels very hacky to me anyway, unluckily this feature has been pushed back again:
https://github.com/visionmedia/express/pull/1186
UPDATE 2
This feature has been added in Express 4.10, since the following pull request has been merged:
https://github.com/strongloop/express/pull/2320
In addition to #user85461 answer, the require view part did not work for me.
What i did: removed the path stuff and moved it all to a module i could require,
patch.ViewEnableMultiFolders.js (Works with current express):
function ViewEnableMultiFolders(app) {
// Monkey-patch express to accept multiple paths for looking up views.
// this path may change depending on your setup.
var lookup_proxy = app.get('view').prototype.lookup;
app.get('view').prototype.lookup = function(viewName) {
var context, match;
if (this.root instanceof Array) {
for (var i = 0; i < this.root.length; i++) {
context = {root: this.root[i]};
match = lookup_proxy.call(context, viewName);
if (match) {
return match;
}
}
return null;
}
return lookup_proxy.call(this, viewName);
};
}
module.exports.ViewEnableMultiFolders = ViewEnableMultiFolders;
and used:
var Patch = require('patch.ViewEnableMultiFolders.js');
Patch.ViewEnableMultiFolders(app);
app.set('views', ['./htdocs/views', '/htdocs/tpls']);
Here's a solution for Express 3.x. It monkey-patches express 3.x's "View" object to do the same lookup trick as #ShadowCloud's solution above. Unfortunately, the path lookup for the View object is less clean, since 3.x doesn't expose it to express -- so you have to dig into the bowels of node_modules.
function enable_multiple_view_folders() {
// Monkey-patch express to accept multiple paths for looking up views.
// this path may change depending on your setup.
var View = require("./node_modules/express/lib/view"),
lookup_proxy = View.prototype.lookup;
View.prototype.lookup = function(viewName) {
var context, match;
if (this.root instanceof Array) {
for (var i = 0; i < this.root.length; i++) {
context = {root: this.root[i]};
match = lookup_proxy.call(context, viewName);
if (match) {
return match;
}
}
return null;
}
return lookup_proxy.call(this, viewName);
};
}
enable_multiple_view_folders();
You can however, put all the view files inside the 'view' folder, but separate each module's view into it's own folders inside the 'view' folder. So, the structure is something like this :
views
--moduleA
--moduleB
----submoduleB1
----submoduleB2
--moduleC
Set the view files like usual :
app.set('views', './views');
And when render for each module, include the module's name :
res.render('moduleA/index', ...);
or even submodule's name :
res.render('moduleB/submoduleB1/index', ...);
This solution is also works in express before version 4.x,
Install glob npm install glob
If you have a views directory that looks something like:
views
├── 404.ejs
├── home.ejs
├── includes
│ ├── header.ejs
│ └── footer.ejs
├── post
│ ├── create.ejs
│ └── edit.ejs
└── profile.ejs
You can use this glob function to return an array of subdirectories in the views directory (add the path.substring to remove the trailing /)
let viewPaths = glob.sync('views/**/').map(path => {
return path.substring(0, path.length - 1)
})
console.log(viewPaths)
>> ['views', 'views/post', 'views/includes']
So now you can set
app.set('views', viewPaths)
and now you can use
res.render('404')
res.render('home')
res.render('post/edit')
res.render('post/create')

Node.js leaking path info, how to solve it?

I have a webserver running... and if I curl from another server. Something like that:
curl http://myserver.com/../../../../../etc/rsyslog.conf
then I can see the server info.
Is that a known problem?
UPDATE
here is my server code:
app = express.createServer(
gzip.staticGzip(__dirname + '/public', {maxAge:5000 }),
express.cookieParser(),
express.bodyParser()
);
got a fix like that:
var urlSecurity = function () {
return function (req, res, next) {
if (req.url.indexOf('../') >=0) {
res.send('<div>Server Error</div>' , 500);
} else if (req.url.indexOf('/..') >=0) {
res.send('<div>Server Error</div>' , 500);
} else {
next();
}
}
}
app = express.createServer(
urlSecurity (),
gzip.staticGzip(__dirname + '/public', {maxAge:5000 }),
express.cookieParser(),
express.bodyParser()
);
is this good enough?
You have a serious security flaw in your program. Fix it immediately.
My best guess from the presented symptom is that you're doing something like:
http.createServer(function (request, response) {
var file = path.resolve('/path/to/files', request.url)
fs.createReadStream(file).pipe(response)
})
This is extremely unwise! Always sanitize user input. In this case, it's quite easy:
http.createServer(function (request, response) {
var requestedFile = path.join('/', request.url);
var file = path.join('/path/to/files', requestedFile)
fs.createReadStream(file).pipe(response)
})
So, first we path.join the requested url onto '/'. This will et rid of any .. shenanigans, making it more sanitary. Then, we path.join that onto our url.
Why use path.join rather than path.resolve in this case? Because path.join just joins path parts, rather than resolving them, so a leading / won't have any ill effects.
After the immediate fix, I have done a lot of testing. and I confirm the following:
It is NOT a node problem primarily. It is the gzippo module causing the problem. Gzippo 0.1.3 is causing that problem. 0.1.4 has no problem. NOt sure why is like that. but better not to use the older version of gzippo.
The simplest solution is insecureFileName.split('/').pop() will always returns only fileName.
'index.html'.split('/').pop() => 'index.html'
'../../../index.html'.split('/').pop() => 'index.html'
I sanitize user filenames with:
path.basename(filename);
e.g.:
const path = require('path');
let filename = '../../../../../../../etc/passwd';
filename = path.basename(filename); // 'passwd'
let pathToFile = path.join('/path/from/config/to', filename);
console.log(pathToFile); // 'path/from/config/to/passwd'

node.js require all files in a folder?

How do I require all files in a folder in node.js?
need something like:
files.forEach(function (v,k){
// require routes
require('./routes/'+v);
}};
When require is given the path of a folder, it'll look for an index.js file in that folder; if there is one, it uses that, and if there isn't, it fails.
It would probably make most sense (if you have control over the folder) to create an index.js file and then assign all the "modules" and then simply require that.
yourfile.js
var routes = require("./routes");
index.js
exports.something = require("./routes/something.js");
exports.others = require("./routes/others.js");
If you don't know the filenames you should write some kind of loader.
Working example of a loader:
var normalizedPath = require("path").join(__dirname, "routes");
require("fs").readdirSync(normalizedPath).forEach(function(file) {
require("./routes/" + file);
});
// Continue application logic here
I recommend using glob to accomplish that task.
var glob = require( 'glob' )
, path = require( 'path' );
glob.sync( './routes/**/*.js' ).forEach( function( file ) {
require( path.resolve( file ) );
});
Base on #tbranyen's solution, I create an index.js file that load arbitrary javascripts under current folder as part of the exports.
// Load `*.js` under current directory as properties
// i.e., `User.js` will become `exports['User']` or `exports.User`
require('fs').readdirSync(__dirname + '/').forEach(function(file) {
if (file.match(/\.js$/) !== null && file !== 'index.js') {
var name = file.replace('.js', '');
exports[name] = require('./' + file);
}
});
Then you can require this directory from any where else.
Another option is to use the package require-dir which let's you do the following. It supports recursion as well.
var requireDir = require('require-dir');
var dir = requireDir('./path/to/dir');
I have a folder /fields full of files with a single class each, ex:
fields/Text.js -> Test class
fields/Checkbox.js -> Checkbox class
Drop this in fields/index.js to export each class:
var collectExports, fs, path,
__hasProp = {}.hasOwnProperty;
fs = require('fs');
path = require('path');
collectExports = function(file) {
var func, include, _results;
if (path.extname(file) === '.js' && file !== 'index.js') {
include = require('./' + file);
_results = [];
for (func in include) {
if (!__hasProp.call(include, func)) continue;
_results.push(exports[func] = include[func]);
}
return _results;
}
};
fs.readdirSync('./fields/').forEach(collectExports);
This makes the modules act more like they would in Python:
var text = new Fields.Text()
var checkbox = new Fields.Checkbox()
One more option is require-dir-all combining features from most popular packages.
Most popular require-dir does not have options to filter the files/dirs and does not have map function (see below), but uses small trick to find module's current path.
Second by popularity require-all has regexp filtering and preprocessing, but lacks relative path, so you need to use __dirname (this has pros and contras) like:
var libs = require('require-all')(__dirname + '/lib');
Mentioned here require-index is quite minimalistic.
With map you may do some preprocessing, like create objects and pass config values (assuming modules below exports constructors):
// Store config for each module in config object properties
// with property names corresponding to module names
var config = {
module1: { value: 'config1' },
module2: { value: 'config2' }
};
// Require all files in modules subdirectory
var modules = require('require-dir-all')(
'modules', // Directory to require
{ // Options
// function to be post-processed over exported object for each require'd module
map: function(reqModule) {
// create new object with corresponding config passed to constructor
reqModule.exports = new reqModule.exports( config[reqModule.name] );
}
}
);
// Now `modules` object holds not exported constructors,
// but objects constructed using values provided in `config`.
I know this question is 5+ years old, and the given answers are good, but I wanted something a bit more powerful for express, so i created the express-map2 package for npm. I was going to name it simply express-map, however the people at yahoo already have a package with that name, so i had to rename my package.
1. basic usage:
app.js (or whatever you call it)
var app = require('express'); // 1. include express
app.set('controllers',__dirname+'/controllers/');// 2. set path to your controllers.
require('express-map2')(app); // 3. patch map() into express
app.map({
'GET /':'test',
'GET /foo':'middleware.foo,test',
'GET /bar':'middleware.bar,test'// seperate your handlers with a comma.
});
controller usage:
//single function
module.exports = function(req,res){
};
//export an object with multiple functions.
module.exports = {
foo: function(req,res){
},
bar: function(req,res){
}
};
2. advanced usage, with prefixes:
app.map('/api/v1/books',{
'GET /': 'books.list', // GET /api/v1/books
'GET /:id': 'books.loadOne', // GET /api/v1/books/5
'DELETE /:id': 'books.delete', // DELETE /api/v1/books/5
'PUT /:id': 'books.update', // PUT /api/v1/books/5
'POST /': 'books.create' // POST /api/v1/books
});
As you can see, this saves a ton of time and makes the routing of your application dead simple to write, maintain, and understand. it supports all of the http verbs that express supports, as well as the special .all() method.
npm package: https://www.npmjs.com/package/express-map2
github repo: https://github.com/r3wt/express-map
Expanding on this glob solution. Do this if you want to import all modules from a directory into index.js and then import that index.js in another part of the application. Note that template literals aren't supported by the highlighting engine used by stackoverflow so the code might look strange here.
const glob = require("glob");
let allOfThem = {};
glob.sync(`${__dirname}/*.js`).forEach((file) => {
/* see note about this in example below */
allOfThem = { ...allOfThem, ...require(file) };
});
module.exports = allOfThem;
Full Example
Directory structure
globExample/example.js
globExample/foobars/index.js
globExample/foobars/unexpected.js
globExample/foobars/barit.js
globExample/foobars/fooit.js
globExample/example.js
const { foo, bar, keepit } = require('./foobars/index');
const longStyle = require('./foobars/index');
console.log(foo()); // foo ran
console.log(bar()); // bar ran
console.log(keepit()); // keepit ran unexpected
console.log(longStyle.foo()); // foo ran
console.log(longStyle.bar()); // bar ran
console.log(longStyle.keepit()); // keepit ran unexpected
globExample/foobars/index.js
const glob = require("glob");
/*
Note the following style also works with multiple exports per file (barit.js example)
but will overwrite if you have 2 exports with the same
name (unexpected.js and barit.js have a keepit function) in the files being imported. As a result, this method is best used when
your exporting one module per file and use the filename to easily identify what is in it.
Also Note: This ignores itself (index.js) by default to prevent infinite loop.
*/
let allOfThem = {};
glob.sync(`${__dirname}/*.js`).forEach((file) => {
allOfThem = { ...allOfThem, ...require(file) };
});
module.exports = allOfThem;
globExample/foobars/unexpected.js
exports.keepit = () => 'keepit ran unexpected';
globExample/foobars/barit.js
exports.bar = () => 'bar run';
exports.keepit = () => 'keepit ran';
globExample/foobars/fooit.js
exports.foo = () => 'foo ran';
From inside project with glob installed, run node example.js
$ node example.js
foo ran
bar run
keepit ran unexpected
foo ran
bar run
keepit ran unexpected
One module that I have been using for this exact use case is require-all.
It recursively requires all files in a given directory and its sub directories as long they don't match the excludeDirs property.
It also allows specifying a file filter and how to derive the keys of the returned hash from the filenames.
Require all files from routes folder and apply as middleware. No external modules needed.
// require
const { readdirSync } = require("fs");
// apply as middleware
readdirSync("./routes").map((r) => app.use("/api", require("./routes/" + r)));
I'm using node modules copy-to module to create a single file to require all the files in our NodeJS-based system.
The code for our utility file looks like this:
/**
* Module dependencies.
*/
var copy = require('copy-to');
copy(require('./module1'))
.and(require('./module2'))
.and(require('./module3'))
.to(module.exports);
In all of the files, most functions are written as exports, like so:
exports.function1 = function () { // function contents };
exports.function2 = function () { // function contents };
exports.function3 = function () { // function contents };
So, then to use any function from a file, you just call:
var utility = require('./utility');
var response = utility.function2(); // or whatever the name of the function is
Can use : https://www.npmjs.com/package/require-file-directory
Require selected files with name only or all files.
No need of absoulute path.
Easy to understand and use.
Using this function you can require a whole dir.
const GetAllModules = ( dirname ) => {
if ( dirname ) {
let dirItems = require( "fs" ).readdirSync( dirname );
return dirItems.reduce( ( acc, value, index ) => {
if ( PATH.extname( value ) == ".js" && value.toLowerCase() != "index.js" ) {
let moduleName = value.replace( /.js/g, '' );
acc[ moduleName ] = require( `${dirname}/${moduleName}` );
}
return acc;
}, {} );
}
}
// calling this function.
let dirModules = GetAllModules(__dirname);
Create an index.js file in your folder with this code :
const fs = require('fs')
const files = fs.readdirSync('./routes')
for (const file of files) {
require('./'+file)
}
And after that you can simply load all the folder with require("./routes")
If you include all files of *.js in directory example ("app/lib/*.js"):
In directory app/lib
example.js:
module.exports = function (example) { }
example-2.js:
module.exports = function (example2) { }
In directory app create index.js
index.js:
module.exports = require('./app/lib');

Resources