My CMS is written in Node.js + Express + Mongoose. It's a multisite CMS and each site has its own database. So I needed to make Mongoose to switch the connection at every HTTP request. I looked around for some solution, or someone who had my same condition, but without success. So this is my solution:
HTTP Request:
router.get('/site/pages', function (req, res) {
pagesBiz.list(req.session, function(err, list) {
//do stuff
});
});
BIZ component (pagesBiz):
module.exports = {
list: function(session, callback) {
Page(session.database).find(callback);
}
}
Model (Page)
var cached = {};
var getModel = function (database) {
if(! cached[database]) {
var conn = mongoose.createConnection('mongodb://localhost/' + database);
cached[database] = conn.model('Page', pageSchema);
}
return cached[database];
}
module.exports = function(database) {
return getModel(database);
}
So how does it works? When the user logs in a new user session is created and bound to the user via the session cookie (I use MongoStore + Express Session). The session contains the name of the database that is used to dynamically instantiate the Mongoose models.
It works perfectly, users of different sites read from their own databases without the risk of "collisions", on the other side I have to pass the session (or the database name) around the functions, but I guess this is how Node.js works in a multithreaded context.
The only problem is when I use the populate method(), I get this error:
f:\www\rudsjs\node_modules\mongoose\lib\connection.js:625
throw new MongooseError.MissingSchemaError(name);
^ MissingSchemaError: Schema hasn't been registered for model "User". Use mongoose.model(name, schema)
at NativeConnection.Connection.model (f:\www\rudsjs\node_modules\mongoose\lib\connection.js:625:11)
at populate (f:\www\rudsjs\node_modules\mongoose\lib\model.js:2136:24)
at Function.Model.populate (f:\www\rudsjs\node_modules\mongoose\lib\model.js:2101:5)
at Object.cb (f:\www\rudsjs\node_modules\mongoose\lib\query.js:1159:16)
at Object._onImmediate (f:\www\rudsjs\node_modules\mongoose\node_modules\mquery\lib\utils.js:137:16)
at processImmediate [as _immediateCallback] (timers.js:336:15)
Process finished with exit code 8
I tried to preload the model before the populate() call using this:
User(session.database);
but the problem seems to be related to the way Mongoose caches the models (that is not mine), I looked at connection.js
model = this.base.models[name];
if (!model) {
throw new MongooseError.MissingSchemaError(name);
}
so I'd need a way to insert my model inside this.base.models. Do you have any idea? Do you especially have a better solution to implement a multi-site/multi-database environment?
I found a solution. I was caching the models and not the connections, so I was spawning a new Mongoose connection for each model, that's why Mongoose wasn't able to load other models using populate().
My solution is: use a global Cache object that stores the connections that will be used to build all the models.
Cache global object
module.exports = {
Cache: {
database: {}
}
}
Model (Page)
module.exports = function(session) {
if(! Cache.database[session.database]) {
debug("Create connection to " + session.database);
var conn = mongoose.createConnection('mongodb://localhost/' + session.database);
Cache.database[session.database] = conn;
}
debug("[rud] " + session.database + " model created");
return Cache.database[session.database].model('Page', pageSchema);
}
As you can see all models now will share the same connection stored in Cache.database[session.database]
Related
I'm developing a project in loopback3 where I need to create accounts for multiple companies, where each compnay has its own database, I'm fully aware that the loopback3 docs has a section where they explain how to create datasources programmatically and how to create models from that datasource, and I've used that to create the following code which receives in the request a parameter which i called dbname and this one changes the linking to the wanted datasource..
userclinic.js
Userclinic.observe('before save', async (ctx, next) => {
const dbname = ctx.instance.dbname; // database selection
const dbfound = await Userclinic.app.models.Clinics.findOne({where:{dbname}}) // checking if that database really exist in out registred clients databases
if( dbfound ){ // if database found
await connectToDatasource(dbname, Userclinic) // link the model to that database
} else { // otherwise
next(new Error('cancelled...')) // cancel the save
}
})
utils.js (from where i export my connectToDatasource method)
const connectToDatasource = (dbname, Model) => {
console.log("welcome");
var DataSource = require('loopback-datasource-juggler').DataSource;
var dataSource = new DataSource({
connector: require('loopback-connector-mongodb'),
host: 'localhost',
port: 27017,
database: dbname
});
Model.attachTo(dataSource);
}
module.exports = {
connectToDatasource
}
So my problem is that the datasource is actually really changing but the save happens in the previous datasource that was selected (which means it saves the instance to the old database) and doesn't save to the new one till I send the request again. so chaging the datasource is taking two requests to happen and it's also saving the instance in both databases.
I guess that when the request happen loopback checks the datasource related to that model first before allowing any action on that model, I really need to get this done by tonight and I wish someone can help out.
PS: if anyone has a solution to this or knows how to associate multiple clients (users) to multiple databases (programmatically of course) in any way using loopback 3 I'm all ears (eyes).
Thanks in advance.
I am trying to implement a common module for MongoDB connection using mongoose. and want to use the connection in other application for database operation. but facing issue when trying to use common database module. operation is halted / hanging after creating db connection. here is my codebase.
When I am using module specific dababase connection, then it is working fine, but when I am using common database connection it is hanging
Common DB Module
'use strict'
const mongoose = require('mongoose');
const DBOptions = require('./DBOption');
require("dotenv").config();
mongoose.Promise = global.Promise;
let isConnected;
const connectToDatabase = (MONGODB_URL) => {
if (isConnected) {
console.log('using existing database connection');
return Promise.resolve();
}
console.log('using new database connection');
console.log('DBOptions >> '+JSON.stringify(DBOptions));
return mongoose.connect(MONGODB_URL, DBOptions)
.then(db => {
console.log('db.connections[0].readyState >> '+db.connections[0].readyState);
isConnected = db.connections[0].readyState;
});
};
module.exports = connectToDatabase;
API Controller
const dbConnection = require('../DB/connection') // Internal Class
const DBConnection = require('as-common-util').connectToDatabase; // Common Class
/**
*
*/
app.get('/usr/alluser', async (req, res) => {
try {
//await dbConnection(process.env.MONGODB_URL) // This is working
await DBConnection(process.env.MONGODB_URL) // Code is hanging for this
let allUsers = await UserService.getAllUser()
console.log("All Users >> " + allUsers)
if (allUsers) {
return res.status(200).send(
new APIResponse({
success: true,
obj: allUsers
})
)
}
} catch (error) {
console.log(error)
}
})
It is hanging at following position
using new database connection
DBOptions >>
{"useNewUrlParser":true,"useUnifiedTopology":true,"useCreateIndex":true,"useFindAndModify":false,"autoIndex":false,"poolSize":10,"serverSelectionTimeoutMS":5000,"socketTimeoutMS":45000,"family":4}
db.connections[0].readyState >> 1
I am confused why same code is not working for common module.
This kind of pattern is not how Mongoose is meant to be used. Under the hood, Mongoose passes the underlying connection to the models in your module without the user really knowing anything about what is going on. That's why you can do magic stuff like MyModel.find() without ever having to create a model object yourself, or pass a db connection object to it.
If your db connection is in another module though, Mongoose won't be able to make those connections between your models and the MongoDB client connection since the models are no longer being registered on the mongoose object that is actually connected, and as a result, any requests you make using your models will break, since they will always be trying to connect through the object in your module.
There are other reasons why this won't, and shouldn't, work though. You're not supposed to be able to split a client. Doing so would make it unclear where communication along a client is coming from, or going to. You could change your function to make it return an established client connection. But your Mongoose models still wouldn't work. You would just be left with raw mongodb. If you want to do that, you might as well just uninstall Mongoose and use the mongodb library. Ultimately, you don't really gain anything from initializing the connection in a shared module. Initializing a connection is just a couple lines of code.
I doubt it's the connection that you want to share, rather it's the models (I'm guessing). You can put those in a shared module, and export them as a kind of connector function that injects the a given Mongoose instance into the models. See: Defining Mongoose Models in Separate Module.
I have Bookshelf.js models (a general model with parse() and format() functions and a model for each table that extends the general model) and routes that use these models. The database is PostgreSQL. But I have users in a SQLite database and each user has its own PostgreSQL connection options. I can retrieve the current user and expose its database connection options in the request, so every route knows how to connect to PostgreSQL. But how can I connect and use my Bookshelf.js models in a route? This is an example of what I am trying to do in a routes file:
var express = require('express'),
knex = require('knex'),
// This should be the file that exposes all models.
// Every model exposed, extends the general model.
// But it already requires a connection when init Bookshelf.js...
dataBookshelf = require('../models'),
router = express.Router();
router.get('/items/:id',function(req,res,next) {
// In req.db I have PostgreSQL connection options for the current user.
var connection = req.db;
// Make a connection to PostgreSQL.
var db = dataBookshelf(knex(connection));
// Use Items model.
var Items = db.Items;
Items.read(req.params.id).then(function(item) {
res.json(item);
}).catch(Items.NotFoundError,function() {
var err = new Error('Item not found.');
res.status(400).json(err);
});
});
module.exports = router;
If there is another way to save the current user connection options globally and to use them in the module that exposes Bookshelf.js instance, instead of passing them in the request, it should be better and safer.
I have a problem testing my mongoose models
I have a structure like
app
models
Address
User
Organization
test
Both models User and Organization need to know the model Address. My models are structured like:
module.exports = function (mongoose, config) {
var organizationSchema = new mongoose.Schema({
name : {
type : String
},
addresses : {
type : [mongoose.model('Address')]
}
});
var Organization = mongoose.model('Organization', organizationSchema);
return Organization;
};
In my normal app i require Address before requiring User and Organization and everything is fine. I now wrote tests for User and Organization. In order to have the Address model registered i call require('../models/Address.js') This works fine if i run one test. But if i run all tests in a batch i get an error because i tried to register Address twice.
OverwriteModelError: Cannot overwrite Address model once compiled.
How do i solve this problem?
The problem is that you cant set mongoose model twice. The easiest way to solve your problem is to take advantage of node.js require function.
Node.js caches all calls to require to prevent your model from initializing twice. But you wrapping your models with functions. Unwrapping them will solve your problem:
var mongoose = require('mongoose');
var config = require('./config');
var organizationSchema = new mongoose.Schema({
name : {
type : String
},
addresses : {
type : [mongoose.model('Address')]
}
});
module.exports = mongoose.model('Organization', organizationSchema);
Alternative solution is to make sure that each model initialized only once. For example, you can initialize all you modules before running your tests:
Address = require('../models/Address.js');
User = require('../models/User.js');
Organization = require('../models/Organization.js');
// run your tests using Address, User and Organization
Or you can add try catch statement to your models to handle this special case:
module.exports = function (mongoose, config) {
var organizationSchema = new mongoose.Schema({
name : {
type : String
},
addresses : {
type : [mongoose.model('Address')]
}
});
try {
mongoose.model('Organization', organizationSchema);
} catch (error) {}
return mongoose.model('Organization');
};
Update: In our project we have /models/index.js file to handle everything. First, it calls mongoose.connect to establish connection. Then it requires every model in models directory and creates a dictionary of it. So, when we need some model (e.g. user) we requires it by calling require('/models').user.
Best solution (IMO):
try {
mongoose.model('config')
} catch (_) {
mongoose.model('config', schema)
}
This question already has an answer, but for a unique way to accomplish this check out https://github.com/fbeshears/register_models. This sample project uses a register_models.js that includes all models from an array of file names. It works really well and you end up with all your models pretty much wherever you need them. Keep in mind node.js's cache will cache objects in your project while it's running.
I use try/catch to tackle this problem and it works ok. But i think this not the best way of doing this.
try{
var blog = mongoose.model('blog', Article);
} catch (error) {}
I fixed this by patching r.js and making sure that all my modules used localRequire calls.
Checkout the thread here
https://github.com/jrburke/requirejs/issues/726
I'm developing an express app that provides a REST api, it uses mongodb through mongoskin. I wanted a layer that splits routing from db acess. I have seen an example that creates a database bridge by creating a module file, an example models/profiles.js:
var mongo = require('mongoskin'),
db = mongo.db('localhost:27017/profiler'),
profs = db.collection('profiles');
exports.examplefunction = function (info, cb) {
//code that acess the profs collection and do the query
}
later this module is required in the routing files.
My question is: If I use this aproach for creating one module for each collection, will it be efficient? Do I have an issue of connecting and disconnecting multiple(unnecessary) times from mongo by doing that?
I was thiking that maybe exporting the db variable from one module to the others that handle each collection would solve the suposed issue, but I'm not sure.
Use a single connection and then create your modules passing in the shared db instance. You want to avoid setting up separate db pools for each module. One of doing this is to construct the module as a class.
exports.build = function(db) {
return new MyClass(db);
}
var MyClass = function(db) {
this.db = db;
}
MyClass.doQuery = function() {
}