I have an Node.js Express web app running.
I have two database connections, the system_db is created with "Mongoose.connect", the user_db is created with "Mongoose.createConnection". This seems to make them separate and have different connection pools. Although this could be the crux of the problem.
I have code dealing with with a Mongoose model
in the model file I have:
var authSchema = mongoose.Schema({
teamName: String,
league: String,
players: []
});
var Team = module.exports = mongoose.model('teams',authSchema);
in my main file I have two connections:
one is a system database
connection_uri = "mongodb://localhost:27017/";
system_db = mongoose.connect(connection_uri.concat("sparks"), {
native_parser : true
}, function(err){
if (err) throw err;
});
the other is a user database
user_db = mongoose.createConnection(connection_uri.concat(user_id));
then I have this piece of code in my main file which finds a Team object based off id:
app.param('team_id', function(req, res, next, team_id) {
Team.findOne(team_id, function(err, team) {
if (err) {
return next(err);
}
if (!team) {
throw new Error("no team matched");
}
req.team = team;
next();
});
});
the problem is that the app.param function above is not finding any matches for teams, even though they exist in a collection in the user_db database. this means, that I am thinking my model is pointing to the wrong database somehow? Somehow it must be pointing to this system_db instead of the user_db. Could this be right? How would I fix that?
The below method opens a connection to the system_db and is binded to the mongoose object,.i.e the current mongoose object is modified. The return type is not a Connection.
system_db = mongoose.connect(connection_uri.concat("sparks")...
Now when you again do:
user_db = mongoose.createConnection(connection_uri.concat(user_id));
This creates a new Connection to the user database and returns a connection, but does not modify the mongoose instance. The mongoose instance is still binded to the system_db database connection.
Since Team model is obtained from the same mongoose instance,
var Team = mongoose.model('teams',authSchema);
whatever operation is done on this model effectively occurs in the connection that the mongoose instance holds, and that is of the system_db database.
So you need to obtain the model from the user_db connection :
var user_db = mongoose.createConnection(..);
// retrieve the Team model
var Team= user_db.model('teams'); // not from mongoose instance.
use createConnection wherever you want to obtain a connection.
Related
I'm doing a Node.js project that contains sub projects. One sub project will have one Mongodb database and Mongoose will be use for wrapping and querying db. But the problem is
Mongoose doesn't allow to use multiple databases in single mongoose instance as the models are build on one connection.
To use multiple mongoose instances, Node.js doesn't allow multiple module instances as it has caching system in require(). I know disable module caching in Node.js but I think it is not the good solution as it is only need for mongoose.
I've tried to use createConnection() and openSet() in mongoose, but it was not the solution.
I've tried to deep copy the mongoose instance (http://blog.imaginea.com/deep-copy-in-javascript/) to pass new mongoose instances to the sub project, but it throwing RangeError: Maximum call stack size exceeded.
I want to know is there anyways to use multiple database with mongoose or any workaround for this problem? Because I think mongoose is quite easy and fast. Or any other modules as recommendations?
According to the fine manual, createConnection() can be used to connect to multiple databases.
However, you need to create separate models for each connection/database:
var conn = mongoose.createConnection('mongodb://localhost/testA');
var conn2 = mongoose.createConnection('mongodb://localhost/testB');
// stored in 'testA' database
var ModelA = conn.model('Model', new mongoose.Schema({
title : { type : String, default : 'model in testA database' }
}));
// stored in 'testB' database
var ModelB = conn2.model('Model', new mongoose.Schema({
title : { type : String, default : 'model in testB database' }
}));
I'm pretty sure that you can share the schema between them, but you have to check to make sure.
Pretty late but this might help someone. The current answers assumes you are using the same file for your connections and models.
In real life, there is a high chance that you are splitting your models into different files. You can use something like this in your main file:
mongoose.connect('mongodb://localhost/default');
const db = mongoose.connection;
db.on('error', console.error.bind(console, 'connection error:'));
db.once('open', () => {
console.log('connected');
});
which is just how it is described in the docs. And then in your model files, do something like the following:
import mongoose, { Schema } from 'mongoose';
const userInfoSchema = new Schema({
createdAt: {
type: Date,
required: true,
default: new Date(),
},
// ...other fields
});
const myDB = mongoose.connection.useDb('myDB');
const UserInfo = myDB.model('userInfo', userInfoSchema);
export default UserInfo;
Where myDB is your database name.
One thing you can do is, you might have subfolders for each projects. So, install mongoose in that subfolders and require() mongoose from own folders in each sub applications. Not from the project root or from global. So one sub project, one mongoose installation and one mongoose instance.
-app_root/
--foo_app/
---db_access.js
---foo_db_connect.js
---node_modules/
----mongoose/
--bar_app/
---db_access.js
---bar_db_connect.js
---node_modules/
----mongoose/
In foo_db_connect.js
var mongoose = require('mongoose');
mongoose.connect('mongodb://localhost/foo_db');
module.exports = exports = mongoose;
In bar_db_connect.js
var mongoose = require('mongoose');
mongoose.connect('mongodb://localhost/bar_db');
module.exports = exports = mongoose;
In db_access.js files
var mongoose = require("./foo_db_connect.js"); // bar_db_connect.js for bar app
Now, you can access multiple databases with mongoose.
As an alternative approach, Mongoose does export a constructor for a new instance on the default instance. So something like this is possible.
var Mongoose = require('mongoose').Mongoose;
var instance1 = new Mongoose();
instance1.connect('foo');
var instance2 = new Mongoose();
instance2.connect('bar');
This is very useful when working with separate data sources, and also when you want to have a separate database context for each user or request. You will need to be careful, as it is possible to create a LOT of connections when doing this. Make sure to call disconnect() when instances are not needed, and also to limit the pool size created by each instance.
Mongoose and multiple database in single node.js project
use useDb to solve this issue
example
//product databse
const myDB = mongoose.connection.useDb('product');
module.exports = myDB.model("Snack", snackSchema);
//user databse
const myDB = mongoose.connection.useDb('user');
module.exports = myDB.model("User", userSchema);
A bit optimized(for me atleast) solution. write this to a file db.js and require this to wherever required and call it with a function call and you are good to go.
const MongoClient = require('mongodb').MongoClient;
async function getConnections(url,db){
return new Promise((resolve,reject)=>{
MongoClient.connect(url, { useUnifiedTopology: true },function(err, client) {
if(err) { console.error(err)
resolve(false);
}
else{
resolve(client.db(db));
}
})
});
}
module.exports = async function(){
let dbs = [];
dbs['db1'] = await getConnections('mongodb://localhost:27017/','db1');
dbs['db2'] = await getConnections('mongodb://localhost:27017/','db2');
return dbs;
};
I have been using this method and it works great for me until now.
const mongoose = require('mongoose');
function makeNewConnection(uri) {
const db = mongoose.createConnection(uri, {
useNewUrlParser: true,
useUnifiedTopology: true
});
db.on('error', function (error) {
console.log(`MongoDB :: connection ${this.name} ${JSON.stringify(error)}`);
db.close().catch(() => console.log(`MongoDB :: failed to close connection ${this.name}`));
});
db.on('connected', function () {
mongoose.set('debug', function (col, method, query, doc) {
console.log(`MongoDB :: ${this.conn.name} ${col}.${method}(${JSON.stringify(query)},${JSON.stringify(doc)})`);
});
console.log(`MongoDB :: connected ${this.name}`);
});
db.on('disconnected', function () {
console.log(`MongoDB :: disconnected ${this.name}`);
});
return db;
}
// Use
const db1 = makeNewConnection(MONGO_URI_DB1);
const db2 = makeNewConnection(MONGO_URI_DB2);
module.exports = {
db1,
db2
}
I was having an issue in my app that was simply from an oversight on the database name that I provided to my mongodb connection. While looking around I ran across the examples here:
https://wesleytsai.io/2015/08/02/mongodb-connection-pooling-in-nodejs/
So my question is why don't I get any kind of error thrown if I am providing a database that does not exist to the mongodb connection? I just need help interpreting/understanding the syntax below. On this line:
MongoClient.connect(mongoUrl, function(err, database) {
if( err ) throw err;
.
.
.
shouldn't an error be thrown since mongo can't connect if the database doesn't exist?
What I'm trying to do is setup some kind of error handling that can flag the fact the the database doesn't exist and likewise with the collection if that too does not exist ('expenses' should be 'test').
var mongodb = require('mongodb');
var MongoClient = mongodb.MongoClient;
var mongoUrl = 'mongodb://127.0.0.1:27017/expenses';
var db;
exports.connect = function(callback) {
MongoClient.connect(mongoUrl, function(err, database) {
if( err ) throw err;
db = database;
callback();
})
}
In mongodb, if you are connected to the datbase and then you provide a schema to a collection that you defined, then irrespective of the fact that whether that collection is there or not, mongodb creates a collection for you.
For example if you have the following code,
module.exports = mongoose.model('User', schema);
a collection named users is created in the database. So even if you dont have a collection, mongodb will create one for you.
If you want to check whether a database exists then follow like below:
var mongoose = require('mongoose')
, Admin = mongoose.mongo.Admin;
/// create a connection to ypur database
var connection = mongoose.createConnection(
'mongodb://user:password#localhost:port/your_database');
connection.on('open', function() {
// connection successful
new Admin(connection.db).listDatabases(function(err, result) {
console.log('listDatabases successful');
// the database lists are stored in result.databases
var allDatabases = result.databases;
// allDatabases contains the record of all the databases
// crosscheck this list with the database you want
});
});
I'm doing a Node.js project that contains sub projects. One sub project will have one Mongodb database and Mongoose will be use for wrapping and querying db. But the problem is
Mongoose doesn't allow to use multiple databases in single mongoose instance as the models are build on one connection.
To use multiple mongoose instances, Node.js doesn't allow multiple module instances as it has caching system in require(). I know disable module caching in Node.js but I think it is not the good solution as it is only need for mongoose.
I've tried to use createConnection() and openSet() in mongoose, but it was not the solution.
I've tried to deep copy the mongoose instance (http://blog.imaginea.com/deep-copy-in-javascript/) to pass new mongoose instances to the sub project, but it throwing RangeError: Maximum call stack size exceeded.
I want to know is there anyways to use multiple database with mongoose or any workaround for this problem? Because I think mongoose is quite easy and fast. Or any other modules as recommendations?
According to the fine manual, createConnection() can be used to connect to multiple databases.
However, you need to create separate models for each connection/database:
var conn = mongoose.createConnection('mongodb://localhost/testA');
var conn2 = mongoose.createConnection('mongodb://localhost/testB');
// stored in 'testA' database
var ModelA = conn.model('Model', new mongoose.Schema({
title : { type : String, default : 'model in testA database' }
}));
// stored in 'testB' database
var ModelB = conn2.model('Model', new mongoose.Schema({
title : { type : String, default : 'model in testB database' }
}));
I'm pretty sure that you can share the schema between them, but you have to check to make sure.
Pretty late but this might help someone. The current answers assumes you are using the same file for your connections and models.
In real life, there is a high chance that you are splitting your models into different files. You can use something like this in your main file:
mongoose.connect('mongodb://localhost/default');
const db = mongoose.connection;
db.on('error', console.error.bind(console, 'connection error:'));
db.once('open', () => {
console.log('connected');
});
which is just how it is described in the docs. And then in your model files, do something like the following:
import mongoose, { Schema } from 'mongoose';
const userInfoSchema = new Schema({
createdAt: {
type: Date,
required: true,
default: new Date(),
},
// ...other fields
});
const myDB = mongoose.connection.useDb('myDB');
const UserInfo = myDB.model('userInfo', userInfoSchema);
export default UserInfo;
Where myDB is your database name.
One thing you can do is, you might have subfolders for each projects. So, install mongoose in that subfolders and require() mongoose from own folders in each sub applications. Not from the project root or from global. So one sub project, one mongoose installation and one mongoose instance.
-app_root/
--foo_app/
---db_access.js
---foo_db_connect.js
---node_modules/
----mongoose/
--bar_app/
---db_access.js
---bar_db_connect.js
---node_modules/
----mongoose/
In foo_db_connect.js
var mongoose = require('mongoose');
mongoose.connect('mongodb://localhost/foo_db');
module.exports = exports = mongoose;
In bar_db_connect.js
var mongoose = require('mongoose');
mongoose.connect('mongodb://localhost/bar_db');
module.exports = exports = mongoose;
In db_access.js files
var mongoose = require("./foo_db_connect.js"); // bar_db_connect.js for bar app
Now, you can access multiple databases with mongoose.
As an alternative approach, Mongoose does export a constructor for a new instance on the default instance. So something like this is possible.
var Mongoose = require('mongoose').Mongoose;
var instance1 = new Mongoose();
instance1.connect('foo');
var instance2 = new Mongoose();
instance2.connect('bar');
This is very useful when working with separate data sources, and also when you want to have a separate database context for each user or request. You will need to be careful, as it is possible to create a LOT of connections when doing this. Make sure to call disconnect() when instances are not needed, and also to limit the pool size created by each instance.
Mongoose and multiple database in single node.js project
use useDb to solve this issue
example
//product databse
const myDB = mongoose.connection.useDb('product');
module.exports = myDB.model("Snack", snackSchema);
//user databse
const myDB = mongoose.connection.useDb('user');
module.exports = myDB.model("User", userSchema);
A bit optimized(for me atleast) solution. write this to a file db.js and require this to wherever required and call it with a function call and you are good to go.
const MongoClient = require('mongodb').MongoClient;
async function getConnections(url,db){
return new Promise((resolve,reject)=>{
MongoClient.connect(url, { useUnifiedTopology: true },function(err, client) {
if(err) { console.error(err)
resolve(false);
}
else{
resolve(client.db(db));
}
})
});
}
module.exports = async function(){
let dbs = [];
dbs['db1'] = await getConnections('mongodb://localhost:27017/','db1');
dbs['db2'] = await getConnections('mongodb://localhost:27017/','db2');
return dbs;
};
I have been using this method and it works great for me until now.
const mongoose = require('mongoose');
function makeNewConnection(uri) {
const db = mongoose.createConnection(uri, {
useNewUrlParser: true,
useUnifiedTopology: true
});
db.on('error', function (error) {
console.log(`MongoDB :: connection ${this.name} ${JSON.stringify(error)}`);
db.close().catch(() => console.log(`MongoDB :: failed to close connection ${this.name}`));
});
db.on('connected', function () {
mongoose.set('debug', function (col, method, query, doc) {
console.log(`MongoDB :: ${this.conn.name} ${col}.${method}(${JSON.stringify(query)},${JSON.stringify(doc)})`);
});
console.log(`MongoDB :: connected ${this.name}`);
});
db.on('disconnected', function () {
console.log(`MongoDB :: disconnected ${this.name}`);
});
return db;
}
// Use
const db1 = makeNewConnection(MONGO_URI_DB1);
const db2 = makeNewConnection(MONGO_URI_DB2);
module.exports = {
db1,
db2
}
I'm writing a Node module which whould perform queries to MongoDB.
My module should take as parameter (when I init it with new MyModule(db)) the MongoDB connection and use it inside it.
I was using standard MongoDB without any NPM module and I was passing in my db variable the connection to MongoDB. Now I'm switching to Mongoose and I can't find a way to pass the Mongoose connection to my module.
I don't want to initialize the Mongoose connection inside my module because I want to share it with my tests and with other modules.
How can I do? I've tried passing mongoose to my module but it doesn't work "is not a function".
Edit:
After reading the reply of #Neil Lunn I'm posting this sample of my module:
(function () {
"use strict";
/**
* various requires
*/
function TopicManager(dbURI) {
if (!(this instanceof TopicManager)) { return new TopicManager(dbURI); }
mongoose.connect(dbURI);
}
TopicManager.prototype.save = function (topics, done) {
var Topic = new mongoose.schema(
{
title: { type: String },
slug: { type: String, index: { unique: true } }
},
{collection : "topics"}
);
/**
* Use monguurl on "slug"
*/
mongoose.update(
{title: topic.title},
{upsert: true},
function (err, numberAffected, raw) {
if (err) { return done(err); }
return done(null, raw);
}
);
};
module.exports = TopicManager;
})();
It doesn't work because when I use it I get undefined is not a function when it runs new mongoose.
Generally speaking you don't do this. The mindset is a little different with mongoose than working with the native driver in it's raw form, and there are plenty of things helping under the hood to make things work a bit more seamlessly without diving into the gory details.
So the basic approach is when you define your "schema" and "models", these are bound to the default connection instance. Unless you have a specific reason for binding to another connection, this is what you should follow:
So you would have a Schema and model definition:
var mySchema = new Schema({
"name": String
});
module.exports = mongoose.model( "Model", mySchema, "collection" )
Where the "collection" part is optional otherwise the "model" name in the first argument is put to standard rules, usually lowercase and pluralized.
Then in your other code listing, you pull this in with require:
var Model = require('./models/mymodel.js');
And use your model objects as mongoose permits:
Model.find({ "field": "name"}, function(err,models) {
});
So it allows a bit more abstract handling than is done with the basic driver as the "models" themselves know how to bind to connections, or are otherwise explicitly bound to the connection you want as an optional parameter.
You can create a default Mongoose connection:
var mongoose = require('mongoose');
mongoose.connect('mongodb://localhost:27017/test');
var db = mongoose.connection;
If you want to open multiple connections you can use createConnection:
var dbconnection = mongoose.createConnection ('uri,uri');
This connection object can then be used for creating/retrieving models that are scoped only to this connection.
The Mongoose connection object has multiple events on which you attach handlers. You should check out the documentation for the full list of handlers you can use.
To get you started, a basic approach would be:
// log connection errors
db.on('error', console.error.bind(console, 'connection error:'));
// wait for opening the connection
db.once('open', function () {
// do something once connection to the MongoDB is open
});
If you open your connection using createConnection, you need to use it when creating models:
// dbconnection is created using createConnection
var MyModel = dbconnection.model('MyModel', MyModelSchema);
Anyone got a migrate module that they use to migrate mongodb data with the mongoose plugin?
I am currently using the 'migrate' module and it works great except for the fact that I need to create/destroy my connection in each up/down.
I.E.
// Setup mongoose
var mongoose = require('mongoose')
, Role = require('../models/role')
, User = require('../models/user');
exports.up = function(next) {
// get a brand new connection for this patch.
mongoose.connect('mongodb://localhost/sagedb');
var adminUser = {
username: 'admin',
password: 'admin'
};
User.createUser(adminUser, function(err, user) {
if (err) {
mongoose.disconnect(); // Make sure to close connection
return next(err);
}
mongoose.disconnect(next); // Make sure to close connection
});
};
exports.down = function(next) {
mongoose.connect('mongodb://localhost/sagedb'); // new connection for down
User.getUserByUsername('admin', function(err, user) {
if (err) {
mongoose.disconnect(function() { // make sure to close connection
return next(err);
});
}
if (!user) {
mongoose.disconnect(); // make sure to close connection
return next();
}
User.deleteUser(user, function(err, user) {
console.log('deleted user');
mongoose.disconnect(next); // make sure to close connection
});
});
};
Probably a much better way to do this. Wondering if the only option is to create my own module that starts the connection once and closes it when all patches are complete.
I have seen mongoose-migrate which tracks migration in database collection. Not really specific to mongoose IMHO, I would rather still use the .migrate file but only have to open the connection once.
The reason of the issue is that you have connection "connected" each time, on every ,migration
That is why you have to disconnect.
The same situation if you replace connect with mongoose.createConnection. you will need to close it.
How to solve?
move
var mongoose = require('mongoose')
, Role = require('../models/role')
, User = require('../models/user');
into module like db
var mongoose = require('mongoose')
, Role = require('../models/role')
, User = require('../models/user');
module.exports = mongoose
and just require it
var mongoose = require('./db')
So you will have:
Single connection
All models loaded in one place
Clean code in
migrations
You can also try my migrate-mongoose migration framework which provides the mongoose connection right out of the box.
in your up or down function you can just access your models like this
this('user').findOne({ name: 'Sergey' });
It also persists your migrations to the database instead of the file system.
You also have east migration framework that is very powerful and it also has mongoDB adaptors:
https://github.com/okv/east
Then you will crate migration with command:
east create my_migration_name
And then your migration scripts will look like this:
exports.migrate = function(client, done) {
var db = client.db;
db.........
done();
};
exports.rollback = function(client, done) {
var db = client.db;
db.........
done();
};