Anyone got a migrate module that they use to migrate mongodb data with the mongoose plugin?
I am currently using the 'migrate' module and it works great except for the fact that I need to create/destroy my connection in each up/down.
I.E.
// Setup mongoose
var mongoose = require('mongoose')
, Role = require('../models/role')
, User = require('../models/user');
exports.up = function(next) {
// get a brand new connection for this patch.
mongoose.connect('mongodb://localhost/sagedb');
var adminUser = {
username: 'admin',
password: 'admin'
};
User.createUser(adminUser, function(err, user) {
if (err) {
mongoose.disconnect(); // Make sure to close connection
return next(err);
}
mongoose.disconnect(next); // Make sure to close connection
});
};
exports.down = function(next) {
mongoose.connect('mongodb://localhost/sagedb'); // new connection for down
User.getUserByUsername('admin', function(err, user) {
if (err) {
mongoose.disconnect(function() { // make sure to close connection
return next(err);
});
}
if (!user) {
mongoose.disconnect(); // make sure to close connection
return next();
}
User.deleteUser(user, function(err, user) {
console.log('deleted user');
mongoose.disconnect(next); // make sure to close connection
});
});
};
Probably a much better way to do this. Wondering if the only option is to create my own module that starts the connection once and closes it when all patches are complete.
I have seen mongoose-migrate which tracks migration in database collection. Not really specific to mongoose IMHO, I would rather still use the .migrate file but only have to open the connection once.
The reason of the issue is that you have connection "connected" each time, on every ,migration
That is why you have to disconnect.
The same situation if you replace connect with mongoose.createConnection. you will need to close it.
How to solve?
move
var mongoose = require('mongoose')
, Role = require('../models/role')
, User = require('../models/user');
into module like db
var mongoose = require('mongoose')
, Role = require('../models/role')
, User = require('../models/user');
module.exports = mongoose
and just require it
var mongoose = require('./db')
So you will have:
Single connection
All models loaded in one place
Clean code in
migrations
You can also try my migrate-mongoose migration framework which provides the mongoose connection right out of the box.
in your up or down function you can just access your models like this
this('user').findOne({ name: 'Sergey' });
It also persists your migrations to the database instead of the file system.
You also have east migration framework that is very powerful and it also has mongoDB adaptors:
https://github.com/okv/east
Then you will crate migration with command:
east create my_migration_name
And then your migration scripts will look like this:
exports.migrate = function(client, done) {
var db = client.db;
db.........
done();
};
exports.rollback = function(client, done) {
var db = client.db;
db.........
done();
};
Related
I'm doing a Node.js project that contains sub projects. One sub project will have one Mongodb database and Mongoose will be use for wrapping and querying db. But the problem is
Mongoose doesn't allow to use multiple databases in single mongoose instance as the models are build on one connection.
To use multiple mongoose instances, Node.js doesn't allow multiple module instances as it has caching system in require(). I know disable module caching in Node.js but I think it is not the good solution as it is only need for mongoose.
I've tried to use createConnection() and openSet() in mongoose, but it was not the solution.
I've tried to deep copy the mongoose instance (http://blog.imaginea.com/deep-copy-in-javascript/) to pass new mongoose instances to the sub project, but it throwing RangeError: Maximum call stack size exceeded.
I want to know is there anyways to use multiple database with mongoose or any workaround for this problem? Because I think mongoose is quite easy and fast. Or any other modules as recommendations?
According to the fine manual, createConnection() can be used to connect to multiple databases.
However, you need to create separate models for each connection/database:
var conn = mongoose.createConnection('mongodb://localhost/testA');
var conn2 = mongoose.createConnection('mongodb://localhost/testB');
// stored in 'testA' database
var ModelA = conn.model('Model', new mongoose.Schema({
title : { type : String, default : 'model in testA database' }
}));
// stored in 'testB' database
var ModelB = conn2.model('Model', new mongoose.Schema({
title : { type : String, default : 'model in testB database' }
}));
I'm pretty sure that you can share the schema between them, but you have to check to make sure.
Pretty late but this might help someone. The current answers assumes you are using the same file for your connections and models.
In real life, there is a high chance that you are splitting your models into different files. You can use something like this in your main file:
mongoose.connect('mongodb://localhost/default');
const db = mongoose.connection;
db.on('error', console.error.bind(console, 'connection error:'));
db.once('open', () => {
console.log('connected');
});
which is just how it is described in the docs. And then in your model files, do something like the following:
import mongoose, { Schema } from 'mongoose';
const userInfoSchema = new Schema({
createdAt: {
type: Date,
required: true,
default: new Date(),
},
// ...other fields
});
const myDB = mongoose.connection.useDb('myDB');
const UserInfo = myDB.model('userInfo', userInfoSchema);
export default UserInfo;
Where myDB is your database name.
One thing you can do is, you might have subfolders for each projects. So, install mongoose in that subfolders and require() mongoose from own folders in each sub applications. Not from the project root or from global. So one sub project, one mongoose installation and one mongoose instance.
-app_root/
--foo_app/
---db_access.js
---foo_db_connect.js
---node_modules/
----mongoose/
--bar_app/
---db_access.js
---bar_db_connect.js
---node_modules/
----mongoose/
In foo_db_connect.js
var mongoose = require('mongoose');
mongoose.connect('mongodb://localhost/foo_db');
module.exports = exports = mongoose;
In bar_db_connect.js
var mongoose = require('mongoose');
mongoose.connect('mongodb://localhost/bar_db');
module.exports = exports = mongoose;
In db_access.js files
var mongoose = require("./foo_db_connect.js"); // bar_db_connect.js for bar app
Now, you can access multiple databases with mongoose.
As an alternative approach, Mongoose does export a constructor for a new instance on the default instance. So something like this is possible.
var Mongoose = require('mongoose').Mongoose;
var instance1 = new Mongoose();
instance1.connect('foo');
var instance2 = new Mongoose();
instance2.connect('bar');
This is very useful when working with separate data sources, and also when you want to have a separate database context for each user or request. You will need to be careful, as it is possible to create a LOT of connections when doing this. Make sure to call disconnect() when instances are not needed, and also to limit the pool size created by each instance.
Mongoose and multiple database in single node.js project
use useDb to solve this issue
example
//product databse
const myDB = mongoose.connection.useDb('product');
module.exports = myDB.model("Snack", snackSchema);
//user databse
const myDB = mongoose.connection.useDb('user');
module.exports = myDB.model("User", userSchema);
A bit optimized(for me atleast) solution. write this to a file db.js and require this to wherever required and call it with a function call and you are good to go.
const MongoClient = require('mongodb').MongoClient;
async function getConnections(url,db){
return new Promise((resolve,reject)=>{
MongoClient.connect(url, { useUnifiedTopology: true },function(err, client) {
if(err) { console.error(err)
resolve(false);
}
else{
resolve(client.db(db));
}
})
});
}
module.exports = async function(){
let dbs = [];
dbs['db1'] = await getConnections('mongodb://localhost:27017/','db1');
dbs['db2'] = await getConnections('mongodb://localhost:27017/','db2');
return dbs;
};
I have been using this method and it works great for me until now.
const mongoose = require('mongoose');
function makeNewConnection(uri) {
const db = mongoose.createConnection(uri, {
useNewUrlParser: true,
useUnifiedTopology: true
});
db.on('error', function (error) {
console.log(`MongoDB :: connection ${this.name} ${JSON.stringify(error)}`);
db.close().catch(() => console.log(`MongoDB :: failed to close connection ${this.name}`));
});
db.on('connected', function () {
mongoose.set('debug', function (col, method, query, doc) {
console.log(`MongoDB :: ${this.conn.name} ${col}.${method}(${JSON.stringify(query)},${JSON.stringify(doc)})`);
});
console.log(`MongoDB :: connected ${this.name}`);
});
db.on('disconnected', function () {
console.log(`MongoDB :: disconnected ${this.name}`);
});
return db;
}
// Use
const db1 = makeNewConnection(MONGO_URI_DB1);
const db2 = makeNewConnection(MONGO_URI_DB2);
module.exports = {
db1,
db2
}
I am creating a multi-tenant Saas App. I was advised by many to keep my separate clients on separate databases, for better security and easier management.
How do we connect multiple databases to the Node app?
I know how to make my app run with a single database connection to mongodb, but not sure about multiple connections.
The mongoose docs mentions the following solutions for multiple connections:
export schema pattern (https://mongoosejs.com/docs/connections.html#multiple_connections)
connection pools (which has only up to 5 connections, which may not be ideal as I may have hundreds of clients in the future)
Another way which I tried (and it works!), is connecting to mongodb during a node API call and executing my logic, as shown below. The code below is a test route for registering a user with name and email. dbutils() is a function that I call to connect to mongodb, using mongoose.connect(). I am not sure if this is a good practice to connect during the API call.
router.post('/:db/register', async (req,res, next) => {
const startTime = new Date();
try {
if(!req.body.name) {
throw new Error("Name required");
}
if(!req.body.email) {
throw new Error("Email required");
}
await dbutils(req.params.db);// connect to db
const session = await mongoose.startSession();
session.startTransaction();
const newUser = new User({
name: req.body.name,
email: req.body.email,
})
await newUser.save({session});
await session.commitTransaction();
session.endSession();
const endTime = new Date();
const diff = endTime.getTime() - startTime.getTime();
return res.json({
newUser: {
email: req.body.email,
name: req.body.name
},
db: req.params.db,
timeElapsed: diff,
});
} catch(ex) {
return next(ex);
}
})
My dbutils() code
const mongoose = require('mongoose');
const mongoURI = "mongodb://PC:27017,PC:27018,PC:27019";
module.exports = async function(db) {
try {
await mongoose.connect(
`${mongoURI}/${db}`,
{
useNewUrlParser: true,
useCreateIndex: true,
useFindAndModify: false,
useUnifiedTopology: true,
}
)
} catch(ex) {
throw ex
}
}
I would be very happy for any recommendation or solution to this problem. Thank you very much in advance for your answer.
It is never a good idea to connect to your DB in an API call, you will be wasting a lot of resources, and deplaying the API responses as well.
The best way for you would be connect to multiple databases when Application starts, along with connection pooling configuration.
You can specify which schema belongs to which connection, and maintain separate DB collections.
You can use below code to work with multiple connections, and pooling:
const connection1 = mongoose.createConnection('mongodb://username:password#host1:port1[?options]',{
poolSize: 10
});
const connection2 = mongoose.createConnection('mongodb://username:password#host2:port2[?options]',{
poolSize: 10
});
Models/Schema on connection 1 can be created as below:
//User schema on connection 1
const userSchema = new Schema({ ... });
const UserModel = connection1.model('User', userSchema);
module.exports = UserModel;
Models/Schema on connection 2 can be created as below:
//Product schema on connection 2
const productSchema = new Schema({ ... });
const ProductModel = connection2.model('Product', productSchema);
module.exports = ProductModel;
For better performance, you can also have shared DB clusters for each DB, and use the cluster to connect to your database.
const conn = mongoose.createConnection('mongodb://[username:password#]host1[:port1][,host2[:port2],...[,hostN[:portN]]][/[database][?options]]', options);
For detailed information, Please read Mongoose Multiple Connections, and Connection Pooling
I'm doing a Node.js project that contains sub projects. One sub project will have one Mongodb database and Mongoose will be use for wrapping and querying db. But the problem is
Mongoose doesn't allow to use multiple databases in single mongoose instance as the models are build on one connection.
To use multiple mongoose instances, Node.js doesn't allow multiple module instances as it has caching system in require(). I know disable module caching in Node.js but I think it is not the good solution as it is only need for mongoose.
I've tried to use createConnection() and openSet() in mongoose, but it was not the solution.
I've tried to deep copy the mongoose instance (http://blog.imaginea.com/deep-copy-in-javascript/) to pass new mongoose instances to the sub project, but it throwing RangeError: Maximum call stack size exceeded.
I want to know is there anyways to use multiple database with mongoose or any workaround for this problem? Because I think mongoose is quite easy and fast. Or any other modules as recommendations?
According to the fine manual, createConnection() can be used to connect to multiple databases.
However, you need to create separate models for each connection/database:
var conn = mongoose.createConnection('mongodb://localhost/testA');
var conn2 = mongoose.createConnection('mongodb://localhost/testB');
// stored in 'testA' database
var ModelA = conn.model('Model', new mongoose.Schema({
title : { type : String, default : 'model in testA database' }
}));
// stored in 'testB' database
var ModelB = conn2.model('Model', new mongoose.Schema({
title : { type : String, default : 'model in testB database' }
}));
I'm pretty sure that you can share the schema between them, but you have to check to make sure.
Pretty late but this might help someone. The current answers assumes you are using the same file for your connections and models.
In real life, there is a high chance that you are splitting your models into different files. You can use something like this in your main file:
mongoose.connect('mongodb://localhost/default');
const db = mongoose.connection;
db.on('error', console.error.bind(console, 'connection error:'));
db.once('open', () => {
console.log('connected');
});
which is just how it is described in the docs. And then in your model files, do something like the following:
import mongoose, { Schema } from 'mongoose';
const userInfoSchema = new Schema({
createdAt: {
type: Date,
required: true,
default: new Date(),
},
// ...other fields
});
const myDB = mongoose.connection.useDb('myDB');
const UserInfo = myDB.model('userInfo', userInfoSchema);
export default UserInfo;
Where myDB is your database name.
One thing you can do is, you might have subfolders for each projects. So, install mongoose in that subfolders and require() mongoose from own folders in each sub applications. Not from the project root or from global. So one sub project, one mongoose installation and one mongoose instance.
-app_root/
--foo_app/
---db_access.js
---foo_db_connect.js
---node_modules/
----mongoose/
--bar_app/
---db_access.js
---bar_db_connect.js
---node_modules/
----mongoose/
In foo_db_connect.js
var mongoose = require('mongoose');
mongoose.connect('mongodb://localhost/foo_db');
module.exports = exports = mongoose;
In bar_db_connect.js
var mongoose = require('mongoose');
mongoose.connect('mongodb://localhost/bar_db');
module.exports = exports = mongoose;
In db_access.js files
var mongoose = require("./foo_db_connect.js"); // bar_db_connect.js for bar app
Now, you can access multiple databases with mongoose.
As an alternative approach, Mongoose does export a constructor for a new instance on the default instance. So something like this is possible.
var Mongoose = require('mongoose').Mongoose;
var instance1 = new Mongoose();
instance1.connect('foo');
var instance2 = new Mongoose();
instance2.connect('bar');
This is very useful when working with separate data sources, and also when you want to have a separate database context for each user or request. You will need to be careful, as it is possible to create a LOT of connections when doing this. Make sure to call disconnect() when instances are not needed, and also to limit the pool size created by each instance.
Mongoose and multiple database in single node.js project
use useDb to solve this issue
example
//product databse
const myDB = mongoose.connection.useDb('product');
module.exports = myDB.model("Snack", snackSchema);
//user databse
const myDB = mongoose.connection.useDb('user');
module.exports = myDB.model("User", userSchema);
A bit optimized(for me atleast) solution. write this to a file db.js and require this to wherever required and call it with a function call and you are good to go.
const MongoClient = require('mongodb').MongoClient;
async function getConnections(url,db){
return new Promise((resolve,reject)=>{
MongoClient.connect(url, { useUnifiedTopology: true },function(err, client) {
if(err) { console.error(err)
resolve(false);
}
else{
resolve(client.db(db));
}
})
});
}
module.exports = async function(){
let dbs = [];
dbs['db1'] = await getConnections('mongodb://localhost:27017/','db1');
dbs['db2'] = await getConnections('mongodb://localhost:27017/','db2');
return dbs;
};
I have been using this method and it works great for me until now.
const mongoose = require('mongoose');
function makeNewConnection(uri) {
const db = mongoose.createConnection(uri, {
useNewUrlParser: true,
useUnifiedTopology: true
});
db.on('error', function (error) {
console.log(`MongoDB :: connection ${this.name} ${JSON.stringify(error)}`);
db.close().catch(() => console.log(`MongoDB :: failed to close connection ${this.name}`));
});
db.on('connected', function () {
mongoose.set('debug', function (col, method, query, doc) {
console.log(`MongoDB :: ${this.conn.name} ${col}.${method}(${JSON.stringify(query)},${JSON.stringify(doc)})`);
});
console.log(`MongoDB :: connected ${this.name}`);
});
db.on('disconnected', function () {
console.log(`MongoDB :: disconnected ${this.name}`);
});
return db;
}
// Use
const db1 = makeNewConnection(MONGO_URI_DB1);
const db2 = makeNewConnection(MONGO_URI_DB2);
module.exports = {
db1,
db2
}
In the example I am working with is this code:
//lets require/import the mongodb native drivers.
var mongodb = require('mongodb');
//We need to work with "MongoClient" interface in order to connect to a mongodb server.
var MongoClient = mongodb.MongoClient;
// Connection URL. This is where your mongodb server is running.
var url = 'mongodb://localhost:27017/my_database_name';
// Use connect method to connect to the Server
MongoClient.connect(url, function (err, db) {
if (err) {
console.log('Unable to connect to the mongoDB server. Error:', err);
} else {
//HURRAY!! We are connected. :)
console.log('Connection established to', url);
// Get the documents collection
var collection = db.collection('users');
//Create some users
var user1 = {name: 'modulus admin', age: 42, roles: ['admin', 'moderator', 'user']};
var user2 = {name: 'modulus user', age: 22, roles: ['user']};
var user3 = {name: 'modulus super admin', age: 92, roles: ['super-admin', 'admin', 'moderator', 'user']};
// Insert some users
collection.insert([user1, user2, user3], function (err, result) {
if (err) {
console.log(err);
} else {
console.log('Inserted %d documents into the "users" collection. The documents inserted with "_id" are:', result.length, result);
}
//Close connection
db.close();
});
}
});
As you may see, he is doing an operation in the connect function. I would like to keep it modular and separate the connection from DB operations.
My suggestion would be to make a singleton on db variable and reuse that one. At least that's what I would do in Java to which I am used to.
However, I am not sure as in the example he hasn't suggested anything like that.
I would recommend against maintaining one connection if you want any kind of scalability.
There are a number of options for connection pooling, etc, but most folks who spend any time at all with Node and MongoDB end up moving to Mongoose at some point.
In addition to adding a nice schema layer, it offers connection abstraction so that you can default to a shared connection by calling mongoose.connect(), or you can create multiple connections or participate in connection pooling by calling mongoose.createConnection(). In both cases, you call it without a callback, and the mongoose machinery will defer subsequent calls to the module until after the connection is established, so that your code doesn't have to care.
Something like your use case might look like so:
// in your app.js or server.js file
var mongoose = require('mongoose');
mongoose.connect(config.db.url); // assuming you have some module that handles config variables
Then in ./models/user.js
const mongoose = require('mongoose'),
Schema = mongoose.Schema;
const UserSchema = new Schema({
name: String,
age: Number,
roles: [String]
});
mongoose.model('User',UserSchema);
finally, in lets say a seed function to create your initial batch of users:
const mongoose = require('mongoose'),
User = mongoose.model('User');
// create some users
var user1 = new User({name: 'modulus admin', age: 42, roles: ['admin', 'moderator', 'user']});
var user2 = new User({name: 'modulus user', age: 22, roles: ['user']});
user1.save(console.log);
user2.save(console.log);
I believe maintaining a single connection is the best as mentioned in another thread:
The primary comitter in node-mongodb-native says
You open do MongoClient.connect once when your app boots up and reuse the db object. It's not a singleton connection pool each .connect creates a new connection pool. So open it once an[d] reuse across all requests.
Say on server start initiate the mongo connection.
Server.js:
...
var db = require('./db');//require db.js
db.openMongoConnection(function(error)
{
if(error)
{
console.log(error);
console.log("cannot make the connection with database");
}
else
{
server.listen(7400);//say ur server listening on 7000 port
}
}
db.js
var db1;
var MongoClient = require('mongodb').MongoClient;
exports.openMongoConnection = function(callback)
{
MongoClient.connect(<YourUrl1>,function(err,dbInstance)
{
if(err)
{
callback(err);
}
else
{
db1 = dbInstance;
callback(null);
}
});
};
exports.getCollection = function(collectionName, callback){
dbInstance.collection(collectionName, function(err, collectionInstance){
if(err)
{
callback(err);
}
else
{
callback(null, collectionInstance)
}
});
}
Then you can call the getCollection to use at anytime by requiring dbInsance
I have an Node.js Express web app running.
I have two database connections, the system_db is created with "Mongoose.connect", the user_db is created with "Mongoose.createConnection". This seems to make them separate and have different connection pools. Although this could be the crux of the problem.
I have code dealing with with a Mongoose model
in the model file I have:
var authSchema = mongoose.Schema({
teamName: String,
league: String,
players: []
});
var Team = module.exports = mongoose.model('teams',authSchema);
in my main file I have two connections:
one is a system database
connection_uri = "mongodb://localhost:27017/";
system_db = mongoose.connect(connection_uri.concat("sparks"), {
native_parser : true
}, function(err){
if (err) throw err;
});
the other is a user database
user_db = mongoose.createConnection(connection_uri.concat(user_id));
then I have this piece of code in my main file which finds a Team object based off id:
app.param('team_id', function(req, res, next, team_id) {
Team.findOne(team_id, function(err, team) {
if (err) {
return next(err);
}
if (!team) {
throw new Error("no team matched");
}
req.team = team;
next();
});
});
the problem is that the app.param function above is not finding any matches for teams, even though they exist in a collection in the user_db database. this means, that I am thinking my model is pointing to the wrong database somehow? Somehow it must be pointing to this system_db instead of the user_db. Could this be right? How would I fix that?
The below method opens a connection to the system_db and is binded to the mongoose object,.i.e the current mongoose object is modified. The return type is not a Connection.
system_db = mongoose.connect(connection_uri.concat("sparks")...
Now when you again do:
user_db = mongoose.createConnection(connection_uri.concat(user_id));
This creates a new Connection to the user database and returns a connection, but does not modify the mongoose instance. The mongoose instance is still binded to the system_db database connection.
Since Team model is obtained from the same mongoose instance,
var Team = mongoose.model('teams',authSchema);
whatever operation is done on this model effectively occurs in the connection that the mongoose instance holds, and that is of the system_db database.
So you need to obtain the model from the user_db connection :
var user_db = mongoose.createConnection(..);
// retrieve the Team model
var Team= user_db.model('teams'); // not from mongoose instance.
use createConnection wherever you want to obtain a connection.