Database Connection using common module is not working [ mongoose and mongodb ] - node.js

I am trying to implement a common module for MongoDB connection using mongoose. and want to use the connection in other application for database operation. but facing issue when trying to use common database module. operation is halted / hanging after creating db connection. here is my codebase.
When I am using module specific dababase connection, then it is working fine, but when I am using common database connection it is hanging
Common DB Module
'use strict'
const mongoose = require('mongoose');
const DBOptions = require('./DBOption');
require("dotenv").config();
mongoose.Promise = global.Promise;
let isConnected;
const connectToDatabase = (MONGODB_URL) => {
if (isConnected) {
console.log('using existing database connection');
return Promise.resolve();
}
console.log('using new database connection');
console.log('DBOptions >> '+JSON.stringify(DBOptions));
return mongoose.connect(MONGODB_URL, DBOptions)
.then(db => {
console.log('db.connections[0].readyState >> '+db.connections[0].readyState);
isConnected = db.connections[0].readyState;
});
};
module.exports = connectToDatabase;
API Controller
const dbConnection = require('../DB/connection') // Internal Class
const DBConnection = require('as-common-util').connectToDatabase; // Common Class
/**
*
*/
app.get('/usr/alluser', async (req, res) => {
try {
//await dbConnection(process.env.MONGODB_URL) // This is working
await DBConnection(process.env.MONGODB_URL) // Code is hanging for this
let allUsers = await UserService.getAllUser()
console.log("All Users >> " + allUsers)
if (allUsers) {
return res.status(200).send(
new APIResponse({
success: true,
obj: allUsers
})
)
}
} catch (error) {
console.log(error)
}
})
It is hanging at following position
using new database connection
DBOptions >>
{"useNewUrlParser":true,"useUnifiedTopology":true,"useCreateIndex":true,"useFindAndModify":false,"autoIndex":false,"poolSize":10,"serverSelectionTimeoutMS":5000,"socketTimeoutMS":45000,"family":4}
db.connections[0].readyState >> 1
I am confused why same code is not working for common module.

This kind of pattern is not how Mongoose is meant to be used. Under the hood, Mongoose passes the underlying connection to the models in your module without the user really knowing anything about what is going on. That's why you can do magic stuff like MyModel.find() without ever having to create a model object yourself, or pass a db connection object to it.
If your db connection is in another module though, Mongoose won't be able to make those connections between your models and the MongoDB client connection since the models are no longer being registered on the mongoose object that is actually connected, and as a result, any requests you make using your models will break, since they will always be trying to connect through the object in your module.
There are other reasons why this won't, and shouldn't, work though. You're not supposed to be able to split a client. Doing so would make it unclear where communication along a client is coming from, or going to. You could change your function to make it return an established client connection. But your Mongoose models still wouldn't work. You would just be left with raw mongodb. If you want to do that, you might as well just uninstall Mongoose and use the mongodb library. Ultimately, you don't really gain anything from initializing the connection in a shared module. Initializing a connection is just a couple lines of code.
I doubt it's the connection that you want to share, rather it's the models (I'm guessing). You can put those in a shared module, and export them as a kind of connector function that injects the a given Mongoose instance into the models. See: Defining Mongoose Models in Separate Module.

Related

next.js and mongodb coherence?

I googled a lot but still have no clear solution to my issue.
Connecting to MongoDB, usually you establish a connection and after the job is done you close it.
Since next.js (and probably node.js) is single threaded. Sometimes it happens that there are two requests processed async while one request established the connection to the database, the otherone is closing the exact same connection. So the first request runs into an Topology closed exception. I have the feeling that the mongodb driver client is shared.
Is there something I did not understood correct in this?
try {
await client.connect()
const database = client.db("test")
const collection = database.collection("test")
const newDataset = await collection.insertOne({})
return newDataset.insertedId.toString()
} finally {
await client.close();
}
As in the comments stated, ive seen a lot of examples & questions here on stackoverflow where in each received request (example below) a database connection is established. This has no benefits and is "bad" because it just takes time and makes no sense. E.g:
app.get("/", (req, res) => {
MongoClient.connect("...", (err, client) => {
// do what ever you want here
client.close();
});
});
If you application needs a database connection, establish the connection "in the startup phase" and keep the connection open. There is no reason to open and close the database connection for each request.
const mongodb = require("monogdb");
const express = require("express");
const app = express();
// some custom init stuff
// e.g. require your route handler etc.
mongodb.MongoClient("...", (err, client) => {
// do what ever you want with the db connection now
// e.g. monkey patch it, so you can use it in other files
// (There are better ways to handle that)
mongodb.client = client;
// or the better way
// pass it as function parameter
require("./routes")(app, client);
app.listen(8080, () => {
console.log("http server listening");
});
});
As you can see in the code above, we first create a database connection and then do other stuff. This has some advantages:
If your credentials are invalid, your application is not externeal reachable because the http server is not started
You have a single connection for all requests
Database queries are potential faster because you dont have to wait to establish first a db connection
NOTE: the code above was "inline coded" here and is not tested.
But i think its illustrated the concept behind my statement.

Create Bookshelf.js models without connection and make connection in route files

I have Bookshelf.js models (a general model with parse() and format() functions and a model for each table that extends the general model) and routes that use these models. The database is PostgreSQL. But I have users in a SQLite database and each user has its own PostgreSQL connection options. I can retrieve the current user and expose its database connection options in the request, so every route knows how to connect to PostgreSQL. But how can I connect and use my Bookshelf.js models in a route? This is an example of what I am trying to do in a routes file:
var express = require('express'),
knex = require('knex'),
// This should be the file that exposes all models.
// Every model exposed, extends the general model.
// But it already requires a connection when init Bookshelf.js...
dataBookshelf = require('../models'),
router = express.Router();
router.get('/items/:id',function(req,res,next) {
// In req.db I have PostgreSQL connection options for the current user.
var connection = req.db;
// Make a connection to PostgreSQL.
var db = dataBookshelf(knex(connection));
// Use Items model.
var Items = db.Items;
Items.read(req.params.id).then(function(item) {
res.json(item);
}).catch(Items.NotFoundError,function() {
var err = new Error('Item not found.');
res.status(400).json(err);
});
});
module.exports = router;
If there is another way to save the current user connection options globally and to use them in the module that exposes Bookshelf.js instance, instead of passing them in the request, it should be better and safer.

Is it necessary to close mongodb connection in nodejs?

I'm new to nodejs and mongodb. in mongodb native driver website they close connection after each request but it seems like to be very slow and problematic in high traffic websites. I'm just curious to know is it necessary to do that or I can declare a global variable and reference that to DB like this:
var mongodbClient = require('mongodb').MongoClient;
var db;
function connect() {
mongodbClient.connect('connection string', function (err, mdb) {
db = mdb;
});
}
connect();
function insert(query, collection, fn) {
db.collection(collection)
.insert(query, function (er, da) {
fn(er, da);
});
}
function find(query, collection, fn) {
db.collection(collection)
.find(query).toArray(function (er, da) {
fn(er, da);
});
}
I don't want to use mongoose and prefer to learn and understand what's going on under the hood.
The examples available in documentation are not actually good for real life use cases. If you are using a server framework you can normally connect to mongo and share reference to the connection throughout application. I use hapi and connect to server via a plugin which allows me to store the handle to open connection. This allows you to clean up on shutdown of server. Their are many modules for managing mongo such as mongoose, waterline or wadofgum-mongodb which I have recently written.

Mongoose dynamic models, cannot use populate()

My CMS is written in Node.js + Express + Mongoose. It's a multisite CMS and each site has its own database. So I needed to make Mongoose to switch the connection at every HTTP request. I looked around for some solution, or someone who had my same condition, but without success. So this is my solution:
HTTP Request:
router.get('/site/pages', function (req, res) {
pagesBiz.list(req.session, function(err, list) {
//do stuff
});
});
BIZ component (pagesBiz):
module.exports = {
list: function(session, callback) {
Page(session.database).find(callback);
}
}
Model (Page)
var cached = {};
var getModel = function (database) {
if(! cached[database]) {
var conn = mongoose.createConnection('mongodb://localhost/' + database);
cached[database] = conn.model('Page', pageSchema);
}
return cached[database];
}
module.exports = function(database) {
return getModel(database);
}
So how does it works? When the user logs in a new user session is created and bound to the user via the session cookie (I use MongoStore + Express Session). The session contains the name of the database that is used to dynamically instantiate the Mongoose models.
It works perfectly, users of different sites read from their own databases without the risk of "collisions", on the other side I have to pass the session (or the database name) around the functions, but I guess this is how Node.js works in a multithreaded context.
The only problem is when I use the populate method(), I get this error:
f:\www\rudsjs\node_modules\mongoose\lib\connection.js:625
throw new MongooseError.MissingSchemaError(name);
^ MissingSchemaError: Schema hasn't been registered for model "User". Use mongoose.model(name, schema)
at NativeConnection.Connection.model (f:\www\rudsjs\node_modules\mongoose\lib\connection.js:625:11)
at populate (f:\www\rudsjs\node_modules\mongoose\lib\model.js:2136:24)
at Function.Model.populate (f:\www\rudsjs\node_modules\mongoose\lib\model.js:2101:5)
at Object.cb (f:\www\rudsjs\node_modules\mongoose\lib\query.js:1159:16)
at Object._onImmediate (f:\www\rudsjs\node_modules\mongoose\node_modules\mquery\lib\utils.js:137:16)
at processImmediate [as _immediateCallback] (timers.js:336:15)
Process finished with exit code 8
I tried to preload the model before the populate() call using this:
User(session.database);
but the problem seems to be related to the way Mongoose caches the models (that is not mine), I looked at connection.js
model = this.base.models[name];
if (!model) {
throw new MongooseError.MissingSchemaError(name);
}
so I'd need a way to insert my model inside this.base.models. Do you have any idea? Do you especially have a better solution to implement a multi-site/multi-database environment?
I found a solution. I was caching the models and not the connections, so I was spawning a new Mongoose connection for each model, that's why Mongoose wasn't able to load other models using populate().
My solution is: use a global Cache object that stores the connections that will be used to build all the models.
Cache global object
module.exports = {
Cache: {
database: {}
}
}
Model (Page)
module.exports = function(session) {
if(! Cache.database[session.database]) {
debug("Create connection to " + session.database);
var conn = mongoose.createConnection('mongodb://localhost/' + session.database);
Cache.database[session.database] = conn;
}
debug("[rud] " + session.database + " model created");
return Cache.database[session.database].model('Page', pageSchema);
}
As you can see all models now will share the same connection stored in Cache.database[session.database]

efficiency of mongodb/mongoskin access by multiple modules approach?

I'm developing an express app that provides a REST api, it uses mongodb through mongoskin. I wanted a layer that splits routing from db acess. I have seen an example that creates a database bridge by creating a module file, an example models/profiles.js:
var mongo = require('mongoskin'),
db = mongo.db('localhost:27017/profiler'),
profs = db.collection('profiles');
exports.examplefunction = function (info, cb) {
//code that acess the profs collection and do the query
}
later this module is required in the routing files.
My question is: If I use this aproach for creating one module for each collection, will it be efficient? Do I have an issue of connecting and disconnecting multiple(unnecessary) times from mongo by doing that?
I was thiking that maybe exporting the db variable from one module to the others that handle each collection would solve the suposed issue, but I'm not sure.
Use a single connection and then create your modules passing in the shared db instance. You want to avoid setting up separate db pools for each module. One of doing this is to construct the module as a class.
exports.build = function(db) {
return new MyClass(db);
}
var MyClass = function(db) {
this.db = db;
}
MyClass.doQuery = function() {
}

Resources