I'm developing an express app that provides a REST api, it uses mongodb through mongoskin. I wanted a layer that splits routing from db acess. I have seen an example that creates a database bridge by creating a module file, an example models/profiles.js:
var mongo = require('mongoskin'),
db = mongo.db('localhost:27017/profiler'),
profs = db.collection('profiles');
exports.examplefunction = function (info, cb) {
//code that acess the profs collection and do the query
}
later this module is required in the routing files.
My question is: If I use this aproach for creating one module for each collection, will it be efficient? Do I have an issue of connecting and disconnecting multiple(unnecessary) times from mongo by doing that?
I was thiking that maybe exporting the db variable from one module to the others that handle each collection would solve the suposed issue, but I'm not sure.
Use a single connection and then create your modules passing in the shared db instance. You want to avoid setting up separate db pools for each module. One of doing this is to construct the module as a class.
exports.build = function(db) {
return new MyClass(db);
}
var MyClass = function(db) {
this.db = db;
}
MyClass.doQuery = function() {
}
Related
I am trying to implement a common module for MongoDB connection using mongoose. and want to use the connection in other application for database operation. but facing issue when trying to use common database module. operation is halted / hanging after creating db connection. here is my codebase.
When I am using module specific dababase connection, then it is working fine, but when I am using common database connection it is hanging
Common DB Module
'use strict'
const mongoose = require('mongoose');
const DBOptions = require('./DBOption');
require("dotenv").config();
mongoose.Promise = global.Promise;
let isConnected;
const connectToDatabase = (MONGODB_URL) => {
if (isConnected) {
console.log('using existing database connection');
return Promise.resolve();
}
console.log('using new database connection');
console.log('DBOptions >> '+JSON.stringify(DBOptions));
return mongoose.connect(MONGODB_URL, DBOptions)
.then(db => {
console.log('db.connections[0].readyState >> '+db.connections[0].readyState);
isConnected = db.connections[0].readyState;
});
};
module.exports = connectToDatabase;
API Controller
const dbConnection = require('../DB/connection') // Internal Class
const DBConnection = require('as-common-util').connectToDatabase; // Common Class
/**
*
*/
app.get('/usr/alluser', async (req, res) => {
try {
//await dbConnection(process.env.MONGODB_URL) // This is working
await DBConnection(process.env.MONGODB_URL) // Code is hanging for this
let allUsers = await UserService.getAllUser()
console.log("All Users >> " + allUsers)
if (allUsers) {
return res.status(200).send(
new APIResponse({
success: true,
obj: allUsers
})
)
}
} catch (error) {
console.log(error)
}
})
It is hanging at following position
using new database connection
DBOptions >>
{"useNewUrlParser":true,"useUnifiedTopology":true,"useCreateIndex":true,"useFindAndModify":false,"autoIndex":false,"poolSize":10,"serverSelectionTimeoutMS":5000,"socketTimeoutMS":45000,"family":4}
db.connections[0].readyState >> 1
I am confused why same code is not working for common module.
This kind of pattern is not how Mongoose is meant to be used. Under the hood, Mongoose passes the underlying connection to the models in your module without the user really knowing anything about what is going on. That's why you can do magic stuff like MyModel.find() without ever having to create a model object yourself, or pass a db connection object to it.
If your db connection is in another module though, Mongoose won't be able to make those connections between your models and the MongoDB client connection since the models are no longer being registered on the mongoose object that is actually connected, and as a result, any requests you make using your models will break, since they will always be trying to connect through the object in your module.
There are other reasons why this won't, and shouldn't, work though. You're not supposed to be able to split a client. Doing so would make it unclear where communication along a client is coming from, or going to. You could change your function to make it return an established client connection. But your Mongoose models still wouldn't work. You would just be left with raw mongodb. If you want to do that, you might as well just uninstall Mongoose and use the mongodb library. Ultimately, you don't really gain anything from initializing the connection in a shared module. Initializing a connection is just a couple lines of code.
I doubt it's the connection that you want to share, rather it's the models (I'm guessing). You can put those in a shared module, and export them as a kind of connector function that injects the a given Mongoose instance into the models. See: Defining Mongoose Models in Separate Module.
This is a little hard articulate so I hope my title isn't too terrible.
I have a frontend/backend React/Node.js(REST API) Web app that I want to add Redis support to for storing retrieving app global settings and per-user specific settings (like language preference, last login, etc... simple stuff) So I was considering adding a /settings branch to my backend REST API to push/pull this information from a redis instance.
This is where my Node.js inexperience comes through. I'm looking at using the ioredis client and it seems too easy. If I have a couple of helpers (more than one .js which will call upon redis) will constructing the client as a const in each be safe to do? Or is there another recommended approach to reusing a single instance of it be the way to go?
Here's a sample of what I'm thinking of doing. Imagine if I had 3 helper modules that require access to the redis client. Should I declare them as const in each? Or centralize them in a single helper module, and get the client from it? Is there a dis-advantage to doing either?
const config = require('config.json');
const redis_url = config.redis_url;
//redis setup
const Redis = require('ioredis');
const redis = new Redis(redis_url);
module.exports = {
test
}
async function test(id) {
redis.get(id, function (err, result) {
if (err) {
console.error(err);
throw(err);
} else {
return result;
}
});
Thank you.
If no redis conflicts...
If the different "helper" modules you are referring to have no conflicts when interacting with redis, such as overwriting / using the same redis keys, then I can't see any reason not to use the same redis instance (as outlined by garlicman) and export this to the different modules in which it is used.
Otherwise use separate redis databases...
If you do require separate redis database connections, redis ships with 16 databases so you can specify which to connect to when creating a new instance - see below:
const redis = new Redis({ // SET UP CONFIG FOR CONNECTION TO REDIS
port: 6379, // Redis port
host: 127.0.0.1, // Redis host
family: 4, // 4 (IPv4) or 6 (IPv6)
db: 10, // Redis database to connect to
});
Normally what I would do (in Java say) is implement any explicit class with singleton access the hold the connection and any connection error/reconnect handling.
All modules in Node.js are already singletons I believe, but what I will probably go with will be a client class to hold it and my own access related methods. Something like:
const config = require('config.json');
const Redis = require("ioredis");
class Redis {
constructor(){
client = new Redis(config.redis_url);
}
get(key) {
return this.client.get(key);
}
set(key, value, ttl) {
let rp;
if (ttl === 0) {
rp = this.client.set(key, value);
}
else {
rp = this.client.set(key, value)
.then(function(res) {
this.client.expire(key, ttl);
});
}
return rp;
}
}
module.exports = new Redis;
I'll probably include a data_init() method to check and preload an initial key/value structure on first connect.
I am writing a NodeJS API with express that uses Sequelize as an ORM to connect to a postgres database.
I have several API files containing the endpoint functions for each model, and I group related functions in these files.
In these files, I load the db connection by requiring the models folder, which contains all the model definitions, and an index file that instantiates & exports the database connection with the models.
I instantiate this at the top of any file that needs access to the database. My problem is that when I enter an endpoint, I can access the database connection perfectly. But when I call any function from another file that also accesses the database, all of my models from the required file are undefined, and it throws an error.
/* api/fooApi.js */
const db = require('../models')
const logApi = require('../logApi')
async function createFoo(req, res, next) {
try {
// db.foo and db.log are defined here, and accessible
const foo = await db.foo.create(req.body)
const log = await logApi.logCreation('foo', foo)
}
}
module.exports = { createFoo }
/* api/logApi.js */
const db = require('../models')
async function logCreation(recordType, record) {
// db.foo and db.log are not defined here when the function is called from fooApi.js
const log = await db.log.create({
event: 'create',
type: recordType,
details: `${recordType} record created by user ${record.createdBy}`
})
return log
}
module.exports = { logCreation }
When I enter the endpoint function createFoo(), I have full access to everything that I expect to be in db, including db.foo.create(). But in logCreation(), I cannot access these same functions.
Many different models will access the logCreation function, so it needs to be defined in one place. The require statement at the top of the file is exactly the same as that in fooApi, but when I debug the function, db is an empty object without any of the properties it should have.
If I pass db as an argument to logCreation, then the function works, but I'd like to avoid this if I can, as it would involve a major restructuring.
Previously to having thing set up this way, I had the following in each api file:
let db = null
function init (dbConn) {
db = dbConn
}
At setup, I would call init() in every API file using the same instance as the argument. However, this was a super clunky way of doing things that I wanted to move away from.
So my question is: What is the correct way to set up Sequelize so that I can access the database across multiple files?
I am using implicit transactions using namespace and cls-hooked as directed in the docs.
I solved my problem - this had nothing to do with how I was importing the file, and was entirely due to a circular dependency because a helper function on one of the models was accessing a helper function in an API file.
I am working on a node.js app where I need to connect to more than one databases. One of the database is central database which contains information common to all. And then there are country level databases where data is stored according to the countries.
I am using sequelize ORM in the app.
Database is postgresql.
Framework is express.
The problem is I want to decide on runtime based on the request which database to use and models should automatically connect to the appropriate database. I have seen this question but didn't found it helpful.
I have also checked in another forums but didn't find anything.
You need to create objects corresponding to each of your database, and at each this object you need to instantiate the Sequelize. Further, for each sequelize instance you need to import the models (assumming that all those databases have exactly the same tables and model representations).
import Sequelize from 'sequelize';
let connectionsArray = [
'postgres://user:pass#example.com:5432/country1',
'postgres://user:pass#example.com:5432/country2',
'postgres://user:pass#example.com:5432/country3',
];
let country1DB, country2DB, country3DB;
country1DB = country2DB = country3DB = {};
country1DB.Sequelize = country2DB.Sequelize = country3DB.Sequelize = Sequelize;
country1DB.sequelize = new Sequelize(connectionsArray[0]);
country2DB.sequelize = new Sequelize(connectionsArray[1]);
country3DB.sequelize = new Sequelize(connectionsArray[2]);
// here you need to access the models path, maybe with fs module
// iterate over every model and import it into every country sequelize instance
// let's assume that models' paths are in simple array
models.forEach(modelFile => {
let model1DB = country1DB.sequelize.import(modelFile);
let model2DB = country2DB.sequelize.import(modelFile);
let model3DB = country3DB.sequelize.import(modelFile);
country1DB[model1DB.name] = model1DB;
country2DB[model2DB.name] = model2DB;
country3DB[model3DB.name] = model3DB;
});
// now every country?DB object has it's own sequelize instance and all model definitions inside
export {
country1DB,
country2DB,
country3DB
};
This is just some example code, it would need refactor to be useful (introduce some loops etc.). It should just show you the idea of how to use multiple databases within single application. If you would like to use e.g. country1 database somewhere, you would simply do
import { country1DB } from './databases';
country1DB.User.findAll({...});
Above code would perform SELECT * FROM users in previously specified country1 database. Example express route could look like below:
import * as databases from './databases';
app.get('/:dbIndex/users', (req, res) => {
databases['country' + req.params.dbIndex + 'DB'].User.find().then(user => {
res.json(user.toJSON());
});
});
Or, even better, you could write some middleware function that would be run before every request and which would be responsible for choosing proper database for further operations.
Knowing that Express.js pretty much leaves it to developer on deciding app structure, and after reading quite a few suggestions on SO (see link1 and link2 for example) as well as checking the example in official repo, I am still not sure if what I am doing is the best way forward.
Say I am using Redis extensively in my app, and that I have multiple "models" that require redis client to run query, would it be better to init redis client in the main app.js, like this:
var db = redis.createClient();
var models = require('./models')(db);
var routes = require('./controllers')(models);
or would it be better to just init redis in each model, then let each controller require models of interests?
The latter approach is what I am using, which looks less DRY. But is passing models instance around the best way? Note that I am loading multiple models/controllers here - I am not sure how to modify my setup to pass the redis client correctly to each models.
//currently in models/index.js
exports.home = require('./home.js');
exports.users = require('./user.js');
TL;DR, my questions are:
where best to init redis client in a MVC pattern app?
how to pass this redis client instance to multiple models with require('./models')(db)
Update:
I tried a different approach for index.js, use module.exports to return an object of models/controllers instead:
module.exports = function(models){
var routes = {};
routes.online = require('./home.js')(models);
routes.users = require('./user.js')(models);
return routes;
};
Seems like a better idea now?
Perhaps it's useful if I share how I recently implemented a project using Patio, a SQL ORM. A bit more background: the MVC-framework I was using was Locomotive, but that's absolutely not a requirement (Locomotive doesn't have an ORM and it leaves implementing how you handle models and databases to the developer, similar to Express).
Locomotive has a construct called 'initializers', which are just JS files which are loaded during app startup; what they do is up to the developer. In my project, one initializer configured the database.
The initializer established the actual database connection, also took care of loading all JS files in the model directory. In pseudocode:
registry = require('model_registry'); // see below
db = createDatabaseConnection();
files = fs.readDirSync(MODEL_DIRECTORY);
for each file in files:
if filename doesn't end with '.js':
continue
mod = require(path.join(MODEL_DIRECTORY, filename));
var model = mod(db);
registry.registerModel(model);
Models look like this:
// models/mymodel.js
module.exports = function(db ) {
var model = function(...) { /* model class */ };
model.modelName = 'MyModel'; // used by registry, see below
return model;
};
The model registry is a very simple module to hold all models:
module.exports = {
registerModel : function(model) {
if (! model.hasOwnProperty('modelName'))
throw Error('[model registry] models require a modelName property');
this[model.modelName] = model;
}
};
Because the model registry stores the model classes in this (which is module.exports), they can then be imported from other files where you need to access the model:
// mycontroller.js
var MyModel = require('model_registry').MyModel;
var instance = new MyModel(...);
Disclaimer: this worked for me, YMMV. Also, the code samples above don't take into account any asynchronous requirements or error handling, so the actual implementation in my case was a bit more elaborate.