I am using mongodb-native-driver in express.js app. I have around 6 collections in the database, so I have created 6 js files with each having a collection as a javascript object (e.g function collection(){}) and the prototypes functions handling all the manipulation on those collections. I thought this would be a good architecture.
But the problem I am having is how to connect to the database? Should I create a connection in each of this files and use them? I think that would be an overkill as the connect in mongodb-native-driver creates a pool of connections and having several of them would not be justified.
So how do I create a single connection pool and use it in all the collections.js files? I want to have the connection like its implemented in mongoose. Let me know if any of my thought process in architecture of the app is wrong.
Using Mongoose would solve these problems, but I have read in several places thats it slower than native-driver and also I would prefer a schema-less models.
Edit: I created a module out of models. Each collection was in a file and it took the database as an argument. Now in the index.js file I called the database connection and kept a variable db after I got the database from the connection. (I used the auto-reconnect feature to make sure that the connection wasn't lost). In the same index.js file I exported each of the collections like this
exports.model1 = require('./model1').(db)
exprorts.model2 = require('./model2').(db)
This ensured that the database part was handled in just one module and the app would just call function that each model.js file exported like save(), fincdbyid() etc (whatever you do in the function is upto you to implement).
how to connect to the database?
In order to connect using the MongoDB native driver you need to do something like the following:
var util = require('util');
var mongodb = require('mongodb');
var client = mongodb.MongoClient;
var auth = {
user: 'username',
pass: 'password',
host: 'hostname',
port: 1337,
name: 'databaseName'
};
var uri = util.format('mongodb://%s:%s#%s:%d/%s',
auth.user, auth.pass, auth.host, auth.port, auth.name);
/** Connect to the Mongo database at the URI using the client */
client.connect(uri, { auto_reconnect: true }, function (err, database) {
if (err) throw err;
else if (!database) console.log('Unknown error connecting to database');
else {
console.log('Connected to MongoDB database server at:');
console.log('\n\t%s\n', uri);
// Create or access collections, etc here using the database object
}
});
A basic connection is setup like this. This is all I can give you going on just the basic description of what you want. Post up some code you've got so far to get more specific help.
Should I create a connection in each of this files and use them?
No.
So how do I create a single connection pool and use it in all the collections.js files?
You can create a single file with code like the above, lets call it dbmanager.js connecting to the database. Export functions like createUser, deleteUser, etc. which operate on your database, then export functionality like so:
module.exports = {
createUser: function () { ; },
deleteUser: function () { ; }
};
which you could then require from another file like so:
var dbman = require('./dbmanager');
dbman.createUser(userData); // using connection established in `dbmanager.js`
EDIT: Because we're dealing with JavaScript and a single thread, the native driver indeed automatically handles connection pooling for you. You can look for this in the StackOverflow links below for more confirmation of this. The OP does state this in the question as well. This means that client.connect should be called only once by an instance of your server. After the database object is successfully retrieved from a call to client.connect, that database object should be reused throughout the entire instance of your app. This is easily accomplished by using the module pattern that Node.JS provides.
My suggestion is to create a module or set of modules which serves as a single point of contact for interacting with the database. In my apps I usually have a single module which depends on the native driver, calling require('mongodb'). All other modules in my app will not directly access the database, but instead all manipulations must be coordinated by this database module.
This encapsulates all of the code dealing with the native driver into a single module or set of modules. The OP seems to think there is a problem with the simple code example I've posted, describing a problem with a "single large closure" in my example. This is all pretty basic stuff, so I'm adding clarification as to the basic architecture at work here, but I still do not feel the need to change any code.
The OP also seems to think that multiple connections could possibly be made here. This is not possible with this setup. If you created a module like I suggest above then the first time require('./dbmanager') is called it will execute the code in the file dbmanager.js and return the module.exports object. The exports object is cached and is also returned on each subsequent call to require('./dbmanager'), however, the code in dbmanager.js will only be executed the first require.
If you don't want to create a module like this then the other option would be to export only the database passed to the callback for client.connect and use it directly in different places throughout your app. I recommend against this however, regardless of the OPs concerns.
Similar, possibly duplicate Stackoverflow questions, among others:
How to manage mongodb connections in nodejs webapp
Node.JS and MongoDB, reusing the DB object
Node.JS - What is the right way to deal with MongoDB connections
As accepted answer says - you should create only one connection for all incoming requests and reuse it, but answer is missing solution, that will create and cache connection. I wrote express middleware to achieve this - express-mongo-db. At first sight this task is trivial, and most people use this kind of code:
var db;
function createConnection(req, res, next) {
if (db) { req.db = db; next(); }
client.connect(uri, { auto_reconnect: true }, function (err, database) {
req.db = db = databse;
next();
});
}
app.use(createConnection);
But this code lead you to connection-leak, when multiple request arrives at the same time, and db is undefined. express-mongo-db solving this by holding incoming clients and calling connect only once, when module is required (not when first request arrives).
Hope you find it useful.
I just thought I would add in my own method of MongoDB connection for others interested or having problems with different methods
This method assumes you don't need authentication(I use this on localhost)
Authentication is still easy to implement
var MongoClient = require('mongodb').MongoClient;
var Server = require('mongodb').Server;
var client = new MongoClient(new Server('localhost',27017,{
socketOptions: {connectTimeoutMS: 500},
poolSize:5,
auto_reconnect:true
}, {
numberOfRetries:3,
retryMilliseconds: 500
}));
client.open(function(err, client) {
if(err) {
console.log("Connection Failed Via Client Object.");
} else {
var db = client.db("theDbName");
if(db) {
console.log("Connected Via Client Object . . .");
db.logout(function(err,result) {
if(!err) {
console.log("Logged out successfully");
}
client.close();
console.log("Connection closed");
});
}
}
});
Credit goes to Brad Davley which goes over this method in his book (page 231-232)
Related
I am working on an application that needs to connect to different MongoDB databases based on the customer that is accessing my application. Each MongoDB database is located on separate servers.
For example, if my application has 3 different customers:
john (mongodb://john:john#server1/john)
tom (mongodb://tom:tom#server2/tom)
harry (mongodb://harry:harry#server3/harry)
when john uses my application, Express.js should connect to mongodb://john:john#server1/john or when harry uses my application, it should connect to mongodb://harry:harry#server3/harry.
I am wondering what an optimized way would be to achieve this. Normally, when we have just one MongoDB database instance, we put the mongoose.connect method inside the app.js and that gets called when the Express.js server starts, but in this case I don't want to connect all three databases at once since that will not be the optimized way.
According to the Mongoose docs, it does support connections to multiple hosts.
const conn1 = mongoose.createConnection('host1', options)
const conn2 = mongoose.createConnection('host2', options)
Let me give you an example (this is Express code) :
// CONNECTION SETTER MIDDLEWARE
app.use(async function (req, res, next) {
// Let's just assume you identified the user earlier with your favourite
// auth method and put their identity in req.user
const userCon = await mongoose.createConnection(`mongodb://${req.user.username}:${req.user.password}#${req.user.host}/${req.user.database}`) // Here you have it!
// Saving the connection in the request so the next
// middlewares/handlers can access it
req.user.connection = userCon
// Thank you, next (middleware/handler)
next()
});
While mongoose.connect() sets the default connection, mongoose.createConnection() only returns a connection.
This is what I would try, and this is Mongoose's recommended way to handle multiple connections. You should note that you can't share Models across different connections, but you can share Schemas.
If you want to just initialize the connection and connect later, the Mongoose docs also provide an example.
// initialize now, connect later
db = mongoose.createConnection();
db.openUri('localhost', 'database', port, [opts]);
I was trying to understand how to use the passport.js in Mongodb native driver,Express now the problem is that all the reference or tutorials are showing the LOCAL-STRATEGY by using mongoose which creates a schema or model........so now iam STUCK
Take a look at the mongodb documentation for their Nodejs driver.
mongoDB Node Driver
Sorry for being here for a little bit late, but maybe my answer would be helpful to others who seeking answer for this kind of question.
I assume that you were struggling with these problems:
How to reuse database connection among your project
You can define your MongoClient object once and reuse this across multiple modules in your project as follow:
dbUtil.js hold definition of MongoClient object:
const MongoClient = require('mongodb').MongoClient;
const SERVER_URI = // link to your server, for example: 'http://localhost:27017';
const DB_NAME = // database name, for example: 'test';
/* #WHY?:
(option 1?) : to let the server assign objectId instead of Node driver,
(option 2 & 3?) : to get rid of deprecation warnings
*/
const clientObj = new MongoClient(`${SERVER_URI}/${DB_NAME}`, {
forceServerObjectId: true,
useNewUrlParser: true,
useUnifiedTopology: true
});
module.exports = {
client: clientObj,
dbName: DB_NAME
}
In another module where you need to use the defined connection:
const { client, dbName } = require('dbUtil');
// Because client.connect() return a Promise, you should wrap everything
// inside an immediately-invoked expression like this
(async () => {
await client.connect(); // at first you need to open the connection client
const dbO = await client.db(dbName); // get the connection to database
/* perform database operations, for example:
dbO.collection(users).insertOne({ name:'mongoadmin' });
*/
client.close(); // remember to close the connection when you're done
})();
So instead of the Mongoose way of using User.find().exec(), in Mongo native driver you have to activate connection to Client first and then use client.dbO.collection('users') (which return a Promise).
What the heck is Passport and why it's needed for your project
Passport is authentication middleware for Express that support authentication from Facebook, Google, JWT,... and many other authentication strategies. It can be helpful when you need to you want to support authentication from multiple authentication portal. However, it's not a must-have.
Sometimes applying another layer of abstraction from third-party libraries not only bring no sense to you & your project, but also over-complicate your existed code base. You'd chose not to use Mongoose and adapted MongoDb native driver instead, stated that you didn't need schema & model stuffs. For the same logic, I don't see any necessity of adapting Passport. This link can be helpful to you in some way: another Stackoverflow post
To apply authentication using JSON web token to your Express routes, you need to do these following steps:
Generate token for user signed in
Verify token
Define protected routes and write middlewares for those
All these tasks can be done without any third-party modules/libraries!
I believe your question stems from using mongodb schema validation instead of mongoose schema. You can use another means of authentication like JWT which does not directly need models for its authentication.
I have a web app under development. I found CW Buechler's tutorial on how to make a simple one very useful but have a couple of niggling worries that I'd like to know whether they're real issues, or things I can ignore.
The way I connect my routes to my database is straight from the tutorials.
in app.js, this code instantiates the database, and attaches a reference to it to every req object that flows thru the middleware.
// wire up the database
var mongo = require('mongodb');
var db = require('monk')('localhost:27017/StarChamber');
----------8<-------
// Make our db accessible to our router
app.use(function(req,res,next){
req.db = db;
next();
});
And in the middleware it get used like this:
app.get('/', function (req, res) {
var db = req.db;
var collection = db.get('myCollection');
// do stuff to produce results
res.json (results);
});
So, to my niggling worries:
Passing the db to the routes by attaching it to the req is pretty convenient, but does it impact performance? Would it be better to have a reference to it in my router file that I could just use? What's the code to do this?
Is it good practice to drop the collection after its been used? The Tutorial doesn't do so, but a collection.drop() call before exiting the route handler looks beneficial, otherwise I think I'll just rack up lots of open connections with the db.
Thanks as ever!
No, it won't impact performance. It's a convenient method to pass a reference to db around, but with Monk it doesn't seem to be especially necessary. See below for an alternative setup.
You are confusing collections with connections. The former are the MongoDB-equivalent of "tables" in SQL, so dropping them doesn't seem to make sense since that would basically throw away all the data in your database table. As for connections: through various layers of indirection, Monk seems to be using the official MongoDB Node driver, which handles connections itself (by means of a connection pool). So there's no need to handle it yourself.
For an alternative way of passing the Monk database handle around: you can place it in a separate module:
// database.js
module.exports = require('monk')('localhost:27017/StarChamber');
And in each module where you require the handle, you can import it:
var db = require('./database');
I'm going through a few error scenarios, trying to understand how to handle those.
In the case where there is no database connection, a Mongoose Model.find(...) call seems to hang. Below the example code. I would have assumed that the callback is invoked with an err object, but it is not.
How can I prevent the model call to hang? Do I manually have to check the readyState each time I access a model?
// app.js
// Let's use a non-existing host so connecting fails:
// (callback is invoked with err object)
mongoose.connect('mongodb://localhostXXX/blog', function(err){ ... });
BlogPost = mongoose.model('BlogPost', BlogPostSchema);
// api.js
exports.list_posts = function(req, res) {
// Ready state is '0' = disconnected (since we used a wrong hostname)
console.log('DB ready state: ' + BlogPost.db.readyState);
// This will not invoke the callback:
BlogPost.find(function(err, threads) {
// Never called...
});
}
It is not an answer, but hope it will help you to find solution. Had very similar issue with
mongoose.createConnection
while using passport module, found out that it works fine with
mongoose.connect
Since you're already using an error handler in the connect call, a sensible thing would be to quit your app when the DB is not up, or activate some middleware that responds with a pretty 500 Internal Server Error.
Mongoose uses node-mongodb-native under the hood for connections to mongodb, you might find other useful connection options in there. :)
EDIT: try setting socketOptions.socketTimeoutMS. Apparently there is no default timeout set. See http://mongodb.github.com/node-mongodb-native/api-generated/server.html.
I don't have node on my work machine to try out the exact syntax for you, but you will probably have to use mongoose.Connection, which has an open() method that accepts options to pass through to node-mongodb-native. I don't think mongoose.connect() accepts these options, but I could be wrong.
In order to solve this problem you need to do 3 tasks:
Configure the bufferMaxEntries:0 in the options.db section (for more details see here)
So when disable bufferMaxEntries this causes mongoose to stop buffer commands and retrying to send them when the server is down.
Configure the autoReconnect:false in the options.db section
disable autoReconnet in the db level. (see more information here)
if you are working with mongodb replicaset then you need to disable bufferCommands in the schema level (for each schema you create)
var schema = new Schema({..}, { bufferCommands: false });
Note: Please read the edited portion of this post before answering, it might save you time and answers one of my questions.
The problem I'm having is pretty simple but I'm pretty new to this overall and I'm having issues figuring out how to implement a mongodb database connection properly in a node/express app.
I'm using express 3.x and am basing my app layout around this project supplied by the author of Express:
https://github.com/expressjs/express/tree/d8caf209e38a214cb90b11ed59fd15b717b3f9bc/examples/blog (now removed from repo)
I have no interest in making a blog however the way the app is structured appears to be quite nice. The routes are separated and everything is organized nicely.
My problem is I might have 5-6 different route js files and each route js file might have anywhere between 1 and 15 routes; of those routes 1 or 15 might want to access the db.
So my problem is it seems like a really terrible idea to do a db.open(...) every single time I want to query the db. I should mention at this point I'm using the native mongo-db driver (npm install mongodb).
I would also need to include a file like this:
http://pastebin.com/VzFsPyax
...in all of those route files and all of my model files. Then I'm also dealing with dozens upon dozens of open connections.
Is there a way I can structure my app in such a way where I only make 1 connection and it stays open for the duration of the session (having a new one made every request would be bad too)?
If so, how can I do this? If you know the answer please post a code sample using tj's blog app (the one linked earlier in this post) structure as a base guide. Basically have a way where the routes and models can use the db freely while being in separate files than the db open code.
Thanks.
EDIT
I made some progress on solving one of my issues. If you look at tj's blog example he initializes his routes in the app.js like so:
require('./routes/site')(app);
require('./routes/post')(app);
And in the routes js file it starts like this:
module.exports = function(app){
I stumbled on a project earlier today where I saw someone pass 2 variables in the modules.exports call -> function(app, db). Then figured wow could it be that easy, do I need to just adjust my routes to be (app, db) too? Yeah, it seems so.
So now part 1 of the problem is solved. I don't have to require a mongo.js file with the connection boilerplate in every route file. At the same time it's flexible enough where I can decide to pick and choose which route files pass a db reference. This is standard and has no downside right?
Part 2 of the problem (the important one unfortunately) still exists though.
How can I bypass having to do a db.open(...) around every query I make and ideally only make a connection once per session?
Other solution is to pass database to the router via request, like this:
app.js
var db = openDatabase();
var app = express();
app.all('*', function(request, response, next)
{
request.database = db;
next();
});
app.get('/api/user/:id', Users.getByID);
users.js
var Users =
{
getByID: function(request, response)
{
request.database.collection('users').findOne(...)
response.send(user);
}
};
module.exports = Users;
I made a very simple module hub for this case that replaces the use of a global space.
In app.js you can create db connection once:
var hub = require('hub');
hub.db = new Db('foobar', new Server('10.0.2.15', 27017, {}), {native_parser: false});
And use it from any other files:
var hub = require('hub');
// hub.db - here link to db connection
This method uses a feature of 'require'. Module is only loaded for the first time and all the other calls gets a reference to an already loaded instance.
UPDATE
That's what I mean:
In main file like app.js we create Db connection, open it and store into hub:
app.js:
var hub = require('hub');
hub.mongodb = require('mongodb');
hub.mongodbClient = new hub.mongodb.Db('foobar', new hub.mongodb.Server('10.0.2.15', 27017, {}), {native_parser: false});
hub.mongodbClient.open(function(error) {
console.log('opened');
});
Now in any other file (message for example) we have access to opened connection and can simple use it:
message.js:
var hub = require('hub');
var collection = new hub.mongodb.Collection(hub.mongodbClient, 'message');
module.exports.count = function(cb) {
collection.count({}, function(err, count) {
cb(err, count);
});
};
Really silly. In the documentation it seems like db.open requires to be wrapped around whatever is using it, but in reality you can use it without a callback.
So the answer is to just do a db.open() in your database connection module, app.js file or where ever you decide to setup your db server/connection.
As long as you pass a reference to the db in the files using it, you'll have access to an "opened" db connection ready to be queried.