Mongoose: handling multiple databases when working with one model - node.js

What I want is to have arbitrary databases (50 for example) with the same collections (same schemas, exact models, different data) and 1 nodejs (expressjs + mongoose) web app.
Example simplified case:
I have:
a single web application (expressjs + mongoose) with User model.
50 domains 50 databases with users collection.
What behaviour I want to achieve:
GET /api/users/ http request is coming to one of domains (test-domain-39.myapp.com)
app gets the requested domain name (test-domain-39) and somehow mongoose understands that it wants to query database-39 when I just do User.find({isActive: true}) in users.controller
So I just want an abstraction. I pass db name to mongoose and continue to work with the User model (as we all usually do when having single DB connection) and mongoose, if needed, creates connection (if it's the first request to the specific db), keeps it alive for next requests in connection pool and etc.
What's the most simple and efficient way to accomplish that?
Thank's in advance!

IMHO, while this is possible with MongoDB, I wouldn't advise maintaining a separate database for each domain, especially if you are expecting to have a huge number of them. Have you considered a multi-tenant model instead?
The sample code below adds user 'Alex' into two different databases, "domainOne" and "domainTwo". Hope this helps
var mongoose = require('mongoose');
var personSchema = { name: String, domain : String };
var baseUri = 'mongodb://localhost/';
domains.forEach((domain) => {
var conn = mongoose.createConnection(baseUri + domain, (error) => {
if(error){
console.log('Ups! Database connection failed!');
return;
}
//Use the connection object to create your models,
//instead the mongoose object
//so that our data is saved into the database
//associated with this connection
var Person = conn.model('Person', personSchema);
//Lets add user 'Alex' into the database
(new Person({name : 'Alex', domain : domain })).save((error) => {
if(error){
console.log('Ups! Could not save person');
} else {
conn.close();
}
});
});
});

This is how I implemented my project:
// config/db.ts
import {createConnection} from 'mongoose'
const MONGO_URI = process.env.MONGO_URI
if (!MONGO_URI)
throw new Error(
'Please define the MONGO_URI environment variable inside .env'
)
const connections: any = {}
async function db(dbName: string) {
if (connections[dbName]) {
return connections[dbName]
} else {
connections[dbName] = createConnection(`${MONGO_URI}/${dbName}`)
return connections[dbName]
}
}
export default db
// models/Test.ts
import { Schema } from 'mongoose'
export interface ITest {
_id: Schema.Types.ObjectId
name: string
createdAt?: Date
}
const testSchema = new Schema<ITest>(
{
name: { type: String, required: true },
},
{ timestamps: true }
)
export default testSchema
// pages/api/test.ts
import nc from 'next-connect'
import db from '../../config/db'
import testSchema from '../../models/Test'
const handler = nc()
handler.get(
async (req: NextApiRequestExtended, res: NextApiResponseExtended) => {
try {
const conn = await db(req.headers['x-db-key'])
const model = await conn.model('Test', testSchema)
const data = await model.find({})
res.send(data)
} catch (error: any) {
res.status(500).json({ error: error.message })
}
}
)
export default handler

Related

Mongoose is not connecting to local database

This is my app.js file. Please help me out to connect it with my local database. Sometimes it gets connected to the database and logs to the console but it doesn't add any collection to the local database.
const mongoose = require('mongoose')
main().catch(err=>console.log(err))
async function main() {
await mongoose.connect("mongodb://localhost:27017/fruitsDB", {
useNewUrlParser: true,
useUnifiedTopology: true
});
//Creating new schema
const fruitSchema = new mongoose.Schema({
name: String,
rating: Number,
review: String
});
const Fruit = mongoose.model("Fruit", fruitSchema);
const fruit = new Fruit ({
name: "Apple",
rating: 7,
review: "Pretty solid"
});
await fruit.save()
}
Insist localhost use 127.0.0.1:27017 This will work for sure.
OR
This happened probably because the MongoDB service isn't started. Follow the below steps to start it:
Go to Control Panel and click on Administrative Tools.
Double-click on Services. A new window opens up.
Search MongoDB.exe. Right-click on it and select Start
The server will start. Now execute npm start again and the code might work this time.
You can use mongo connection like this in typescript for ES6.
Schema like below
import mongoose from "mongoose"
export const RequestLogsSchema = new mongoose.Schema(
{
request_id: String,
...
},
{
collection: "request_logs"
}
)
example connection like below
import mongoose from 'mongoose'
import { RequestLogsSchema } from './mongo-schemas/RequestLogsSchema'
export class MongoClient {
mongodb: any
constructor(private host: string) { }
async MongoConnect() {
return new Promise(async (resolve, _reject): Promise<void> => {
console.log('🟡 MongoDB Connecting !')
this.mongodb = await mongoose.connect(this.host).then(() => {
console.log('🟢 MongoDB Connected !')
resolve(true)
}).catch((err) => console.log(err))
})
}
}
export const schemas = {
RequestLogsModal: mongoose.model("RequestLogs", RequestLogsSchema),
...
}
new MongoClient('mongodb://username:password#localhost:27017/db_name?authSource=db_name').MongoConnect()
To save your data like
import { schemas } from '../connections/mongo'
const saver = (data) => {
const request_logs = new schemas.RequestLogsModal({
request_id: data.request_id,
...
})
await request_logs.save()
}

Why am I not able query ( User.findOne() ) my mongoDB after setting up a Schema?

I can't seem to query my MongoDB after setting up the MongoDB schema. I don't understand where I am going wrong with my MongoDB schema, any help is appreciated!
I WAS earlier successful in querying my mongodb before creating a schema, using queries like this:
const uri = process.env.MONGODB_URI;
const client = new MongoClient(uri);
const result = await client.db("inride_adverts").collection("adverts").findOne({OrgEmail: OrgEmailToSignIn});
However, according to a YouTube tutorial am following (10:40 mins), after setting up a mongodb schema, I am NOT successful in using the following query to interact with my mongodb:
User.findOne( {OrgEmail: signUpEmail} ).exec();
Find below my simple User Schema:
./models/User.js
import { mongoose} from 'mongoose';
const UserSchema = new mongoose.Schema({
OrgName: {
type: String,
required: true
},
OrgEmail: {
type: String,
required: true
},
OrgPwd: {
type: String,
required: true
}
}, { collection: 'adverts' });
export const User = mongoose.model('User', UserSchema);
Also, find below my server.js file
./server.js
import express from 'express';
import { User } from './models/User.js';
import mongodb from 'mongodb';
import { mongoose} from 'mongoose';
mongoose.connect(db, { useNewUrlParser: true })
mongoose.connect(db, { useNewUrlParser: true })
.then( () => console.log("MongoDB Connected..." ))
.catch(error => console.log(err))
app.route('/advertiserRegister')
.post( async (req, res) => {
let formData = req.body;
let signUpName = formData.signUpName;
let signUpEmail = formData.signUpEmail;
let signUpPwd = formData.signUpPwd;
console.log("signUpName: " +signUpName);
console.log("signUpEmail: " +signUpEmail);
console.log("signUpPwd: " +signUpPwd);
if(signUpPwd !== signUpPwdConfirm){
console.log("Passwords arent EQUAL!");
return 0;
} else {
try{
console.log("signUpEmail>>> : " + signUpEmail );
// Validation passed!
const testing = await User.findOne( {OrgEmail: signUpEmail} ).exec();
console.log("testing >>> : " ,testing );
res.redirect('/advertiserLogin')
} catch {
//console.error('error', code);
console.log("Error logging in ");
res.redirect('/advertiserRegister')
};
}
});
The server.js file yields:
MongoDB Connected...
signUpName: Merc Enterprise LTD
signUpEmail: hundredcent.a#gmail.com
signUpPwd: 555
signUpEmail>>> : hundredcent.a#gmail.com
testing >>> : **null**
Error logging in
Turns out that the reason I was NOT able to query my collection was due to the fact that I was unknowingly querying the incorrect collection being: adverts under the incorrect database being: inride_adverts.
I came to understand and realise that In my ./models/User.js, the mongoose.model('User', UserSchema); code creates a new database in Atlas MongoDB called test and creates a totally new collection called User.
Having understood this, I am able to populate and query this collection successfully!

What is the best way for a node app to maintain connection with hundred or even thousands of databases using mongoose?

I am creating a multi-tenant Saas App. I was advised by many to keep my separate clients on separate databases, for better security and easier management.
How do we connect multiple databases to the Node app?
I know how to make my app run with a single database connection to mongodb, but not sure about multiple connections.
The mongoose docs mentions the following solutions for multiple connections:
export schema pattern (https://mongoosejs.com/docs/connections.html#multiple_connections)
connection pools (which has only up to 5 connections, which may not be ideal as I may have hundreds of clients in the future)
Another way which I tried (and it works!), is connecting to mongodb during a node API call and executing my logic, as shown below. The code below is a test route for registering a user with name and email. dbutils() is a function that I call to connect to mongodb, using mongoose.connect(). I am not sure if this is a good practice to connect during the API call.
router.post('/:db/register', async (req,res, next) => {
const startTime = new Date();
try {
if(!req.body.name) {
throw new Error("Name required");
}
if(!req.body.email) {
throw new Error("Email required");
}
await dbutils(req.params.db);// connect to db
const session = await mongoose.startSession();
session.startTransaction();
const newUser = new User({
name: req.body.name,
email: req.body.email,
})
await newUser.save({session});
await session.commitTransaction();
session.endSession();
const endTime = new Date();
const diff = endTime.getTime() - startTime.getTime();
return res.json({
newUser: {
email: req.body.email,
name: req.body.name
},
db: req.params.db,
timeElapsed: diff,
});
} catch(ex) {
return next(ex);
}
})
My dbutils() code
const mongoose = require('mongoose');
const mongoURI = "mongodb://PC:27017,PC:27018,PC:27019";
module.exports = async function(db) {
try {
await mongoose.connect(
`${mongoURI}/${db}`,
{
useNewUrlParser: true,
useCreateIndex: true,
useFindAndModify: false,
useUnifiedTopology: true,
}
)
} catch(ex) {
throw ex
}
}
I would be very happy for any recommendation or solution to this problem. Thank you very much in advance for your answer.
It is never a good idea to connect to your DB in an API call, you will be wasting a lot of resources, and deplaying the API responses as well.
The best way for you would be connect to multiple databases when Application starts, along with connection pooling configuration.
You can specify which schema belongs to which connection, and maintain separate DB collections.
You can use below code to work with multiple connections, and pooling:
const connection1 = mongoose.createConnection('mongodb://username:password#host1:port1[?options]',{
poolSize: 10
});
const connection2 = mongoose.createConnection('mongodb://username:password#host2:port2[?options]',{
poolSize: 10
});
Models/Schema on connection 1 can be created as below:
//User schema on connection 1
const userSchema = new Schema({ ... });
const UserModel = connection1.model('User', userSchema);
module.exports = UserModel;
Models/Schema on connection 2 can be created as below:
//Product schema on connection 2
const productSchema = new Schema({ ... });
const ProductModel = connection2.model('Product', productSchema);
module.exports = ProductModel;
For better performance, you can also have shared DB clusters for each DB, and use the cluster to connect to your database.
const conn = mongoose.createConnection('mongodb://[username:password#]host1[:port1][,host2[:port2],...[,hostN[:portN]]][/[database][?options]]', options);
For detailed information, Please read Mongoose Multiple Connections, and Connection Pooling

Mongoose - multiple database connections

I want to understand how to switch between databases within mongoose global promise connection.
My current connection is established this way app.ts
import * as mongoose from 'mongoose';
...
try {
await mongoose.createConnection(`mongodb://localhost:27017/db1`, {
useNewUrlParser: true,
})
console.log("Connected")
} catch (error) {
console.log(error)
}
And then I am accessing it in different files some.model.ts
import { Schema, Document, model } from 'mongoose';
const SomeSchema: Schema = new Schema({
name: { type: String, required: true },
owner: { type: string, required: true }
});
export default model('Some', SomeSchema);
According to documentation.
So far we've seen how to connect to MongoDB using Mongoose's default connection. At times we may need multiple connections open to Mongo, each with different read/write settings, or maybe just to different databases for example. In these cases we can utilize mongoose.createConnection() which accepts all the arguments already discussed and returns a fresh connection for you.
const conn = mongoose.createConnection('mongodb://[username:password#]host1[:port1][,host2[:port2],...[,hostN[:portN]]][/[database][?options]]', options);
I can create multiple database connections like this
try {
const db1 = await mongoose.createConnection(`mongodb://localhost:27017/db1`, {
useNewUrlParser: true,
})
const db2 = await mongoose.createConnection(`mongodb://localhost:27017/db2`, {
useNewUrlParser: true,
})
console.log("Connected")
} catch (error) {
console.log(error)
}
I can see both connection in console.log(mongoose.connections)
But how can I specify what database should be used for the Model in some.model.ts?
import { Schema, Document, model } from 'mongoose';
const SomeSchema: Schema = new Schema({
name: { type: String, required: true },
owner: { type: string, required: true }
});
export default SPECIFY_DATABASE.model('Some', SomeSchema);
I have found other questions like this but there are connections created "localy", I need to use mongoose connection across many different files.
Thank you for answers, if you need more explanation please let me now.
You need to actually return the connection, and then register a given model to each of the connections. To clarify, you need:
something to create a (named, specific) connection
schemas
you create models by registering schemas to the given connections,
you also need something to orchestrate it.
Example, lets have a "db.js" file (I call mine "repo.js" usually) with a single export, a function that returns the initialized database Promise.
You'd use it by importing the function and awaiting for the db.
I have a bit of a longer example, so error handling etc is ommited for brevity.
import { createConnections } from './create-connections';
import { UsersSchema } from './users-schema';
import { PostsSchema } from './posts-schema';
let db: any;
export function getDatabase(): Promise<any> {
if (this.db) return Promise.resolve(db);
return createDatabases();
}
async function createDatabases() {
const { db1, db2 } = await createConnections('mongodb://localhost/db1', 'mongodb://localhost/db2');
const UserModel = db1.model('users', UserSchema);
const PostModel = db2.model('posts', PostSchema);
db = {
UserModel,
PostModel,
// also if you need this
connections: {
db1,
db2,
}
}
return db;
}
Now, I've used './create-connections' here, which is almost what you have:
// create-connection.js
const { createConnection } = require('mongoose');
// You create connections by calling this function and giving it the URL to the server
export function createConnections(url1, url2) {
const db1 = await createConnection(url1);
const db2 = await createConnection(url2);
return {
db1,
db2
}
}
Now, let's say you have two models: users and posts, let's have their schemas.
// users schema
import { Schema, Document } from 'mongoose';
export const UserSchema: Schema = new Schema({
name: { type: String, required: true },
});
// posts schema
import { Schema, Document } from 'mongoose';
export const PostSchema: Schema = new Schema({
text: { type: String, required: true },
owner: { type: SchemaID, required: true }
});
So now you need to bind it all in that fdirst file.
But how to use it? As I've said, since it's async, you always import it and use it as a simple async getDB:
// some controller, route handler, service etc.
import { getDatabase } from './get-database';
router.get('/users', async (req, res) => {
const User = await getDatabase().UserModel;
const users = await User.find();
return res.json(users);
});
router.post('/posts', async (req, res) {
const { text } = req.body;
const owner = req.user.id;
const Post = await getDatabase().PostModel;
const post = await Post.create({ text, owner });
return res.json(post);
});

Dynamically create collection with Mongoose

I want to give users the ability to create collections in my Node app. I have really only seen example of hard coding in collections with mongoose. Anyone know if its possible to create collections dynamically with mongoose? If so an example would be very helpful.
Basically I want to be able to store data for different 'events' in different collections.
I.E.
Events:
event1,
event2,
...
eventN
Users can create there own custom event and store data in that collection. In the end each event might have hundreds/thousands of rows. I would like to give users the ability to perform CRUD operations on their events. Rather than store in one big collection I would like to store each events data in a different collection.
I don't really have an example of what I have tried as I have only created 'hard coded' collections with mongoose. I am not even sure I can create a new collection in mongoose that is dynamic based on a user request.
var mongoose = require('mongoose');
mongoose.connect('localhost', 'events');
var schema = mongoose.Schema({ name: 'string' });
var Event1 = mongoose.model('Event1', schema);
var event1= new Event1({ name: 'something' });
event1.save(function (err) {
if (err) // ...
console.log('meow');
});
Above works great if I hard code 'Event1' as a collection. Not sure I create a dynamic collection.
var mongoose = require('mongoose');
mongoose.connect('localhost', 'events');
...
var userDefinedEvent = //get this from a client side request
...
var schema = mongoose.Schema({ name: 'string' });
var userDefinedEvent = mongoose.model(userDefinedEvent, schema);
Can you do that?
I believe that this is a terrible idea to implement, but a question deserves an answer. You need to define a schema with a dynamic name that allows information of 'Any' type in it. A function to do this may be a little similar to this function:
var establishedModels = {};
function createModelForName(name) {
if (!(name in establishedModels)) {
var Any = new Schema({ any: Schema.Types.Mixed });
establishedModels[name] = mongoose.model(name, Any);
}
return establishedModels[name];
}
Now you can create models that allow information without any kind of restriction, including the name. I'm going to assume an object defined like this, {name: 'hello', content: {x: 1}}, which is provided by the 'user'. To save this, I can run the following code:
var stuff = {name: 'hello', content: {x: 1}}; // Define info.
var Model = createModelForName(name); // Create the model.
var model = Model(stuff.content); // Create a model instance.
model.save(function (err) { // Save
if (err) {
console.log(err);
}
});
Queries are very similar, fetch the model and then do a query:
var stuff = {name: 'hello', query: {x: {'$gt': 0}}}; // Define info.
var Model = createModelForName(name); // Create the model.
model.find(stuff.query, function (err, entries) {
// Do something with the matched entries.
});
You will have to implement code to protect your queries. You don't want the user to blow up your db.
From mongo docs here: data modeling
In certain situations, you might choose to store information in
several collections rather than in a single collection.
Consider a sample collection logs that stores log documents for
various environment and applications. The logs collection contains
documents of the following form:
{ log: "dev", ts: ..., info: ... } { log: "debug", ts: ..., info: ...}
If the total number of documents is low you may group documents into
collection by type. For logs, consider maintaining distinct log
collections, such as logs.dev and logs.debug. The logs.dev collection
would contain only the documents related to the dev environment.
Generally, having large number of collections has no significant
performance penalty and results in very good performance. Distinct
collections are very important for high-throughput batch processing.
Say I have 20 different events. Each event has 1 million entries... As such if this is all in one collection I will have to filter the collection by event for every CRUD op.
I would suggest you keep all events in the same collection, especially if event names depend on client code and are thus subject to change. Instead, index the name and user reference.
mongoose.Schema({
name: { type: String, index: true },
user: { type: mongoose.Schema.Types.ObjectId, ref: 'User', index: true }
});
Furthermore I think you came at the problem a bit backwards (but I might be mistaken). Are you finding events within the context of a user, or finding users within the context of an event name? I have a feeling it's the former, and you should be partitioning on user reference, not the event name in the first place.
If you do not need to find all events for a user and just need to deal with user and event name together you could go with a compound index:
schema.index({ user: 1, name: 1 });
If you are dealing with millions of documents, make sure to turn off auto index:
schema.set('autoIndex', false);
This post has interesting stuff about naming collections and using a specific schema as well:
How to access a preexisting collection with Mongoose?
You could try the following:
var createDB = function(name) {
var connection = mongoose.createConnection(
'mongodb://localhost:27017/' + name);
connection.on('open', function() {
connection.db.collectionNames(function(error) {
if (error) {
return console.log("error", error)
}
});
});
connection.on('error', function(error) {
return console.log("error", error)
});
}
It is important that you get the collections names with connection.db.collectionNames, otherwise the Database won't be created.
This method works best for me , This example creates dynamic collection for each users , each collection will hold only corresponding users information (login details), first declare the function dynamicModel in separate file : example model.js
/* model.js */
'use strict';
var mongoose = require('mongoose'),
Schema = mongoose.Schema;
function dynamicModel(suffix) {
var addressSchema = new Schema(
{
"name" : {type: String, default: '',trim: true},
  "login_time" : {type: Date},
"location" : {type: String, default: '',trim: true},
}
);
return mongoose.model('user_' + suffix, addressSchema);
}
module.exports = dynamicModel;
In controller File example user.js,first function to create dynamic collection and second function to save data to a particular collection
/* user.js */
var mongoose = require('mongoose'),
function CreateModel(user_name){//function to create collection , user_name argument contains collection name
var Model = require(path.resolve('./model.js'))(user_name);
}
function save_user_info(user_name,data){//function to save user info , data argument contains user info
var UserModel = mongoose.model(user_name) ;
var usermodel = UserModel(data);
usermodel.save(function (err) {
if (err) {
console.log(err);
} else {
console.log("\nSaved");
}
});
}
yes we can do that .I have tried it and its working.
REFERENCE CODE:
app.post("/",function(req,res){
var Cat=req.body.catg;
const link= req.body.link;
const rating=req.body.rating;
Cat=mongoose.model(Cat,schema);
const item=new Cat({
name:link,
age:rating
});
item.save();
res.render("\index");
});
I tried Magesh varan Reference Code ,
and this code works for me
router.post("/auto-create-collection", (req, res) => {
var reqData = req.body; // {"username":"123","password":"321","collectionName":"user_data"}
let userName = reqData.username;
let passWord = reqData.password;
let collectionName = reqData.collectionName;
// create schema
var mySchema = new mongoose.Schema({
userName: String,
passWord: String,
});
// create model
var myModel = mongoose.model(collectionName, mySchema);
const storeData = new myModel({
userName: userName,
passWord: passWord,
});
storeData.save();
res.json(storeData);
});
Create a dynamic.model.ts access from some where to achieve this feature.
import mongoose, { Schema } from "mongoose";
export default function dynamicModelName(collectionName: any) {
var dynamicSchema = new Schema({ any: Schema.Types.Mixed }, { strict: false });
return mongoose.model(collectionName, dynamicSchema);
}
Create dynamic model
import dynamicModelName from "../models/dynamic.model"
var stuff = { name: 'hello', content: { x: 1 } };
var Model = await dynamicModelName('test2')
let response = await new Model(stuff).save();
return res.send(response);
Get the value from the dynamic model
var Model = dynamicModelName('test2');
let response = await Model.find();
return res.send(response);

Resources