I am using the environment variable and arguments parsing module called nconf for my node.js Express web server.
https://github.com/indexzero/nconf
I decided that the best way to make the nconf data global was to simply attach it to the process variable (as in process.env), is this a good idea or bad idea? Will it slow down execution in weighing down "process"?
Here is my code:
var nconf = require('nconf');
nconf.argv()
.env()
.file({ file: './config/config.json' });
nconf.defaults({
'http': {
'port': 3000
}
});
process.nconf = nconf;
//now I can retrieve config settings anywhere like so process.nconf.get('key');
frankly, I kind of like this solution. Now I can retrieve the config data anywhere, without having to require a module. But there may be downsides to this...and it could quite possibly be a very bad idea. IDK.
It won't slow down the execution, but feels "smelly". It's hard to discover, and it will be difficult to test, if you ever decide you need to.
A better solution would be to attach settings to a module and use require() to import it wherever needed.
The best solution would be to just pass your settings object to the classes or modules that need it. Either directly, or as part of some kind of "global context".
Eg.
var global = {
settings: {
port: 8080
}
}
//...
global.api = new Api(global);
//...
function Api(global) {
var port = global.settings.port;
}
UPDATE: more info on why the original pattern is bad:
1) Discoverability
You attach your settings to process.settings and go off to a different project. A year later, someone else takes over or you need to update things. Will you remember you attached your settings to process.nconf? Or was it process.settings?
Now imagine you have 10 different global things, attached under different names, on different places.
It's not as bad as attaching directly to the global context, but it's certainly better to clearly see where the stuff you're using is coming from (constructor or module).
2) Testing
You decide you need to test your module. So now you need to tweak your settings for each test instead of loading them from a file or argv. How do you do that?
In case of the global process.nconf or require("settings") patterns, you need to do something like this:
function canOpenAPIOnTheConfiguredPort(done) {
var nconfSaveApiPort = process.nconf.api.port;
process.nconf.api.port = '1234';
var api = new Api();
test.assertEqual(api.port, '1234');
process.nconf.api.port = nconfSaveApiPort;
done();
}
As your application grows, this quickly becomes annoying (eg. imagine having to mock 10 things). In comparison, here's how you do it using the dependency injection (constructor) pattern.
function canOpenAPIOnTheConfiguredPort(done) {
var api = new Api({
port: '1234'
});
test.assertEqual(api.port, '1234');
done();
}
Notice that nconf is a singleton.
I use to configure at the very beginning of the program and then when I need a setting in another file I do:
var nconf = require ('nconf');
nconf.get('x');
Related
In node, module imports (aka require()) are hard coded in every file (aka module) which requires that import. This is can be tens or in our case hundreds of duplicate imports. What we are "requiring" in a dynamic way are mainly services, e.g. "playerService" with find, update, get methods etc. but could also be domain objects or persistence libraries
The crux is we have 3 versions of this "playerService" js file. One which does everything locally (in memory) for development, one which does everything with a local database (test), and one which does everything with an external system via an API (live). The switch in this case in on environment (dev, test or live).
It is worth noting we use classes everywhere we can because we find functions which return functions of functions etc. to be unreadable/maintainable (we are java developers really struggling with js)
We are also exclusively using web sockets in our node app -there is no http code.
So our services look like this:
const Player = require("./player")
class PlayerService {
constructor(timeout) {
this.timeout= 3000 // todo, put in a config file
if (timeout != null) {this.timeout= timeout}
}
updatePlayer(player) {
// logic to lookup player in local array and change it for dev version.
// test version would lookup player in DB and update it.
}
}
module.exports = PlayerService
We are familiar with dependency injection with Grails and spring, but haven't found anything comprehensible (see below) for node. We are not javascript nor node gurus unfortunately, despite extensive reading.
Currently we are considering one of these options, but would like to hear any better suggestions:
option 1:
Hard code the "dev" requires, e.g. require("./dev/playerSerice")
have a jenkins build server rewrite the source code in every file to require("./test/playerSerice").
option 2:
Hard code the "dev" requires, e.g. require("./playerSerice")
have a jenkins build server swap the file /test/playerService/playerService" to ./playerService.
Obviously these make it hard for developers to run the test or pro versions on their local machines without hacking the source.
option 3:
1. put the required module paths in a single config file.
2. swap out just the config file. E.g.
let Config = require("./config")
let PlayerService = require(Config.playerService)
We have tried to make this dependent on env and have a global config which the development, test and prod configs over ride these, but have not found an elegant way to do this. One way might be to duplicate this code at the top of every module:
let env = process.env.NODE_ENV || 'development'
let config = require('./config.'+env);
let PlayerService = require("./" + Config.playerService)
Then in config.development.js:
var config = require("./config.global")
config.PlayerService = "devPlayerService"
module.exports = config
Option 4:
Perhaps something like this would work:
let env = process.env.NODE_ENV || 'development'
require("./" + env + "/playerService")
all the above solutions suffer from lack of singletons - services are stateless. We are guessing that node is going to construct a new copy of each service for each request (or web socket message in our case). Is there a way to minimise this?
Obviously some simple, readable, and officially maintained form of Dependency injection would be nice, with some way to switch between which set of classes were injected.
We have read the following posts:
https://blog.risingstack.com/dependency-injection-in-node-js/ resultant code is unreadable (for us at least). The example being so contrived doesn't help, team is just some sort of proxy wrapper around User, not a service or anything useful. What are options? Why options?
https://medium.com/#Jeffijoe/dependency-injection-in-node-js-2016-edition-f2a88efdd427
But found them incomprehensible. E.g. the examples have keywords which come from thin air - they dont seem to be javascript or node commands and are not explained in the documentation where they come from.
And looked at these projects:
https://github.com/jaredhanson/electrolyte
https://www.npmjs.com/package/node-dependency-injection
https://www.npmjs.com/package/di
but they seemed to be either abandoned (di), not maintained or we just cant figure them out (electrolyte).
Is there some standard or simple di solution that many people are using, ideally documented for mortals and with a non "express" dependent example?
UPDATE 1
It seems the pattern I am using to create my services creates a new instance very time it is used/called. Services should be singletons. The simple solution is to add this to the bottom of my services:
let playerService = new PlayerService();
module.exports = playerService;
Apparently, this only creates one instance of the object, no matter now many times require("./playerService") is called.
For keeping the configuration per env, the right way is probably (similar to what you suggested)- Keeping a config/env directory and putting there a file per env, ie development.js, test.js etc, and in each of them putting the right values. eg:
module.exports = {
playerService: 'dev/PlayerService'
}
and require it:
let env = process.env.NODE_ENV || 'development'
, envConfig = require("./config/" + env)
, playerService = require(envConfig.playerService)
You can also have the all in one file like this:
config.js:
module.exports = {
development:{
playerService: '....'
},
test:{
playerService: '....'
}
}
and require it:
let env = process.env.NODE_ENV || 'development'
, config = require("./config")
, playerService = require(config[env][playerService])
This is a common use-case.
Or, if you have all services in directories per env, ie one directory for dev, one for test etc, you don't need the config, you can require like that:
let env = process.env.NODE_ENV || 'development'
, playerService = require('./' + env + '/playerServcie')
Making the services singleton in node js should be simple, have a look at the following:
https://blog.risingstack.com/fundamental-node-js-design-patterns/
https://www.sitepoint.com/javascript-design-patterns-singleton/
and this
Hope this helps.
I'm working on a MEAN (Mongo Express.js Angular Node.js) CRUD application. I have it working but everything is in one .js file. The single source code file is quite large. I want to refactor the code so CRUD functionality is in different source code files. Reading through other posts, I've got a working model but am not sure it is the right way in Node using Mongo to get it done.
Here's the code so far:
<pre>
var express = require('express');
var bodyParser = require('body-parser');
var app = express();
var path = require('path');
var db;
var connect = 'mongodb://<<mddbconnect string>>;
const MongoClient = require('mongodb').MongoClient;
var ObjectID = require("mongodb").ObjectID;
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: true }));
app.use(express.static(__dirname + '/'));
// viewed at http://localhost:<<port referecnes in app.listen>>
app.get('/', (req, res) => {
res.sendFile(path.join(__dirname + '/index.html'));
});
MongoClient.connect(connect, (err, database) => {
if (err) return console.log(err)
db = database
app.listen(3000, () => {
console.log('listening on 3000' + Date() );
// Here's the require for the search function in another source code file.
var searchroute = require('./serverSearch')(app, db);
})
})
//Handlers
The rest of the CRUD application functions with app.post, app.get. These are other functions I want to move into different source code files, like serverSearch.js.
</pre>
The code I separated right now is the search functionality which is inside of the MongoClient.connection function. This function has to successfully execute to make sure the variable 'db' is valid before passing both variables 'app' and 'db' to the the search function built out in the source code file serverSearch.js.
I could now build out my other CRUD functions in separate files in put them in the same area as 'var searchroute = require('./serverSearch)(app,db);
Is this the best way to separate code in a MEAN application where the main app and db vars need to be instantiated then passed to functions in other source code files?
What you are basically describing is modular coding heading towards "services" perhaps even micro-services. There are a few factors to keep in mind for your system. (I have no doubt that there are many other approaches to this btw). Basically in most NodeJS systems I have worked on (not all) I try to apply the following architecture in development and then bring over as much as possible I to production.
Create a directory under the main one. I usually use some type of name that points to the term functions. In this directory I maintain function and /or class files divided into categories. Function wrappers for DB would be held in DB functions. This file would only contain functions for the DB. Security functions in another file. Helper functions in another. Time manipulation in another. I am sure you get the idea. These are all wrapped in module exports
Now in any file in my project where say I would need DB and helpers I will start it by:
let nhelpers = require("helpfuncs");
let ndb = require("dbfuncs");
Obviously names are different.
And btw I divide all the NPM packages in the same way under an environment directory.
Maintaining that kind of structure allows you to maintain sane order over the code, logical chaining in any decent IDE, and having relevant methods show up in your IDE without having to remember every function name and all the methods within.
It also allows you to write an orderly system of micro-services making sure each part dies exactly what you want and allows for sane debugging.
It took me awhile to settle on this method and refine it.
It paid off for me. Hope this helps.
Edit to clarify for the OP:
When it comes to the process.env variables I became a great fan of dotenv https://www.npmjs.com/package/dotenv
This little package has saved me an incredible amount of headaches. Of course you will have to decide if you include it in production or not. I have seen arguments for both, but i think in a well set up AWS, Google, Azure environment (or in Docker of course) I am of the opinion it can safely be used.
A couple of caveats.
Do not leave your dotenv file in the root. Move it somewhere else in your directory structure. It is simple and I actually put it in the same directory as all my environment files and helper files.
Remember it is simply a text file. So an IDE will not pick up your specific env variables in chaining. (Unless someone knows of a trick which I would love to hear about)
If you put env variables like access info to your DB system or other sensitive stuff, HASH THEM FIRST, put the hash in your env and have a function in your code which specifically just does the hash to the string. Do not under any conditions leave sensitive information in your environment file without hashing it first.
The final Gatcha. These are not PHP magic globals which cannot be overwritten. If you lose track and overwrite one of those process.env variables in your code it will take the new value until you restart your node app and it reads from the dotenv file again. (But then again that is the rule with all environment variables not only user defined ones)
Any typos above excuse me. Done from my cell in the train.
I have a module for connecting to my DB and perform actions.
Then in my main script (app.js) I instantiate it like
var DBConn = require('./common/DBConn');
And from that script it works fine. But then I've got some other scripts that handle routes, and I want to perform some DB stuff on those, but if I use DBConn it returns an error saying "DBConn is not defined".
Then I can just instantiate another DBConn in these other js files, but this would mean I am creating a connection for each file, right? But I want these other scripts to use the DBConn object from the app.js, so that I'm not constantly establishing a connection to the DB and then closing it... (unless this is a good idea, but to me it makes more sense to have just one "global" object dealing with the connection over all the app and that's it).
(BTW: I'm using Express)
You want to require() your module in each file. Node will cache the module.
Typically, the context of a DB connection is abstracted away behind stores or repositories and your other modules interact with those. In cases where people are directly requiring modules like mongoose, they'll require mongoose everywhere but only call the connection code within their main application entry point (app.js/server.js/whatever).
https://nodejs.org/api/modules.html#modules_caching
Modules are cached after the first time they are loaded. This
means (among other things) that every call to require('foo') will get
exactly the same object returned, if it would resolve to the same
file.
Multiple calls to require('foo') may not cause the module code to be
executed multiple times. This is an important feature. With it,
"partially done" objects can be returned, thus allowing transitive
dependencies to be loaded even when they would cause cycles.
If you want to have a module execute code multiple times, then export
a function, and call that function.
You could use a singleton to solve this issue for you. Please remember however that singletons come with their own set of problems (a good discussuon on singletons can be found here What is so bad about singletons?).
That said once you take into consideration the pro's and cons of singletons they can be a fantastic solution to exactly this problem.
Example:
// database.js
var singleton = function singleton() {
this.DbConnection = {};
function setup(database, username, password) {
this.DbConnection = MakeDbConnection()
};
};
singleton.instance = null;
singleton.getInstance = function(){
if(this.instance === null){
this.instance = new singleton();
}
return this.instance;
};
module.exports = singleton.getInstance();
This can then be called elsewhere in your application like so...
// main.js
var db = require("./database");
db.setup('db', 'root', 'pass');
db.DbConnection();
Once you have called db.setup() the first time, the db connection will be available just with the following:
// elsewhere.js
var db = require("./database");
db.DbConnection();
I'm new to Node but I figured I'd jump right in and start converting a PHP app into Node/Express. It's a bilingual app that uses gettext with PO/MO files. I found a Node module called node-gettext. I'd rather not convert the PO files into another format right now, so it seems this library is my only option.
So my concern is that right now, before every page render, I'm doing something like this:
exports.home_index = function(req, res)
{
var gettext = require('node-gettext'),
gt = new gettext();
var fs = require('fs');
gt.textdomain('de');
var fileContents = fs.readFileSync('./locale/de.mo');
gt.addTextdomain('de', fileContents);
res.render(
'home/index.ejs',
{ gt: gt }
);
};
I'll also be using the translations in classes, so with how it's set up now I'd have to load the entire translation file again every time I want to translate something in another place.
The translation file is about 50k and I really don't like having to do file operations like this on every page load. In Node/Express, what would be the most efficient way to handle this (aside from a database)? Usually a user won't even be changing their language after the first time (if they're changing it from English).
EDIT:
Ok, I have no idea if this is a good approach, but it at least lets me reuse the translation file in other parts of the app without reloading it everywhere I need to get translated text.
In app.js:
var express = require('express'),
app = express(),
...
gettext = require('node-gettext'),
gt = new gettext();
Then, also in app.js, I create the variable app.locals.gt to contain the gettext/translation object, and I include my middleware function:
app.locals.gt = gt;
app.use(locale());
In my middleware file I have this:
mod
module.exports = function locale()
{
return function(req, res, next)
{
// do stuff here to populate lang variable
var fs = require('fs');
req.app.locals.gt.textdomain(lang);
var fileContents = fs.readFileSync('./locales/' + lang + '.mo');
req.app.locals.gt.addTextdomain(lang, fileContents);
next();
};
};
It doesn't seem like a good idea to assign the loaded translation file to app, since depending on the current request that file will be one of two languages. If I assigned the loaded translation file to app instead of a request variable, can that mix up users' languages?
Anyway, I know there's got to be a better way of doing this.
The simplest option would be to do the following:
Add this in app.js:
var languageDomains = {};
Then modify your Middleware:
module.exports = function locale()
{
return function(req, res, next)
{
// do stuff here to populate lang variable
if ( !req.app.locals.languageDomains[lang] ) {
var fs = require('fs');
var fileContents = fs.readFileSync('./locales/' + lang + '.mo');
req.app.locals.languageDomains[lang] = true;
req.app.locals.gt.addTextdomain(lang, fileContents);
}
req.textdomain = req.app.locals.gt.textdomain(lang);
next();
};
};
By checking if the file has already been loaded you are preventing the action from happening multiple times, and the domain data will stay resident in the server's memory. The downside to the simplicity of this solution is that if you ever change the contents of your .mo files whilst the server is running, the changes wont be taken into account. However, this code could be extended to keep an eye on the mtime of the files, and reload accordingly, or make use of fs.watchFile — if required:
if ( !req.app.locals.languageDomains[lang] ) {
var fs = require('fs'), filename = './locales/' + lang + '.mo';
var fileContents = fs.readFileSync(filename);
fs.watchFile(filename, function (curr, prev) {
req.app.locals.gt.addTextdomain(lang, fs.readFileSync(filename));
});
req.app.locals.languageDomains[lang] = true;
req.app.locals.gt.addTextdomain(lang, fileContents);
}
Warning: It should also be noted that using sync versions of functions outside of server initialisation is not a good idea because it can freeze the thread. You'd be better off changing your sync loading to the async equivalent.
After the above changes, rather than passing gt to your template, you should be able to use req.textdomain instead. It seems that the gettext library supports a number of requests directly on each domain object, which means you hopefully don't need to refer to the global gt object on a per request basis (which will be changing it's default domain on each request):
Each domain supports:
getTranslation
getComment
setComment
setTranslation
deleteTranslation
compilePO
compileMO
Taken from here:
https://github.com/andris9/node-gettext/blob/e193c67fdee439ab9710441ffd9dd96d027317b9/lib/domain.js
update
A little bit of further clarity.
Once the server has loaded the file into memory the first time, it should remain there for all subsequent connections it receives (for any visitor/request) because it is stored globally and wont be garbage collected — unless you remove all references to the data, which would mean gettext would need to have some kind of unload/forget domain method.
Node is different to PHP in that its environment is shared and wraps its own HTTP server (if you are using something like Express), which means it is very easy to remember data globally as it has a constant environment that all the code is executed within. PHP is always executed after the HTTP server has received and dealt with the request (e.g. Apache). Each PHP response is then executed in its own separate run-time, which means you have to rely on databases, sessions and cache stores to share even simple information and most resources.
further optimisations
Obviously with the above you are constantly running translations on each page load. Which means the gettext library will still be using the translation data resident in memory, which will take up processing time. To get around this, it would be best to make sure your URLs have something that makes them unique for each different language i.e. my-page/en/ or my.page.fr or even jp.domain.co.uk/my-page and then enable some kind of full page caching using something like memcached or express-view-cache. However, once you start caching pages you need to make certain there aren't any regions that are user specific, if so, you need to start implement more complicated systems that are sensitive to these areas.
Remember: The golden rule of optimisation, don't do so before you need to... basically meaning I wouldn't worry about page caching until you know it's going to be an issue, but it is always worth bearing in mind what your options are, as it should shape your code design.
update 2
Just to illustrate a bit further on the behaviour of a server running in JavaScript, and how the global behaviour is not just a property of req.app, but in fact any object that is further up the scope chain.
So, as an example, instead of adding var languageDomains = {}; to your app.js, you could instantiate it further up the scope of wherever your middleware is placed. It's best to keep your global entities in one place however, so app.js is the better place, but this is just for illustration.
var languageDomains = {};
module.exports = function locale()
{
/// you can still access languageDomains here, and it will behave
/// globally for the entire server.
languageDomains[lang]
}
So basically, where-as with PHP, the entire code-base is re-executed on each request — so the languageDomains would be instantiated a-new each time — in Node the only part of the code to be re-executed is the code within locale() (because it is triggered as part of a new request). This function will still have a reference to the already existing and defined languageDomains via the scope chain. Because languageDomains is never reset (on a per request basis) it will behave globally.
Concurrent users
Node.js is single threaded. This means that in order for it to be concurrent i.e. handle multiple requests at the "same" time, you have to code your app in such a way that each little part can be executed very quickly and then slip into a waiting state, whilst another part of another request is dealt with.
This is the reason for the asynchronous and callback nature of Node, and the reason to avoid Sync calls whilst your app is running. Any one Sync request could halt or freeze execution of the thread and delay handling for all other requests. The reason why I state this is to give you a better idea of how multiple users might interact with your code (and global objects).
Basically once a request is being dealt with by your server, it is it's only focus, until that particular execution cycle ends i.e. your request handler stops calling other code that needs to run synchronously. Once that happens the next queued item is dealt with (a callback or something), this could be part of another request, or it could be the next part in the current request.
I'm writing a node module to consume a REST API for a service. For all intents and purposes we might as well say it's twitter (though it's not).
The API is not small. Over a dozen endpoints. Given that I want to offer convenience methods for each of the endpoints I need to split up the code over multiple files. One file would be far too large.
Right now I am testing the pattern I will outline below, but would appreciate any advice as to other means by which I might break up this code. My goal essentially is to extend the prototype of a single object, but do so using multiple files.
Here's the "model" I'm using so far, but don't think is really a good idea:
TwitterClient.js
function TwitterClient(){
this.foo = "bar";
}
require("fs").readdirSync("./endpoints").forEach(function(file) {
require("./endpoints/" + file)(TwitterClient);
});
var exports = module.exports = TwitterClient;
endpoints/endpointA.js etc
module.exports = function(TwitterClient){
TwitterClient.prototype.someMethod = function(){
//do things here
}
}
The basic idea obviously is that any file in the endpoints folder is automatically loaded and the TwitterClient is passed in to it, so that it's prototype can be accessed/extended.
I don't plan to stick with this pattern because for some reason it seems like a bad idea to me.
Any suggestions of better patterns are very much appreciated, cheers