I wrote what is possibly the smallest wrapper around node-mongodb-native wrapper. But, I feel that it needs improving.
It's so small it fits here comfortably:
function MongoWrapper() {
this.db = null;
};
var mongoWrapper;
module.exports = exports = mongoWrapper = new MongoWrapper;
// This means that you can do `new include('mongoWrapper').MongoWrapper()`
mongoWrapper.MongoWrapper = MongoWrapper;
// ObjectId is the most handy method of all. This will work with
// native BSON or Pure BSON
mongoWrapper.ObjectId = function() {
if (!mongo.BSONNative || !mongo.BSONNative.ObjectID) {
return function(id) {
return mongo.BSONPure.ObjectID.createFromHexString(id);
};
}
return function(id) {
return new mongo.BSONNative.ObjectID(id);
};
}();
MongoWrapper.prototype.connect = function(url, options, cb ){
var that = this;
var MongoClient = mongo.MongoClient;
MongoClient.connect( url, function( err, db ){
if( err ) {
console.log( err );
} else {
that.db = db;
}
cb( err, db );
});
}
Now... The "problem" with this is that I need to wrap my whole server in a callback:
mw.connect('mongodb://localhost/hotplate', {}, function( err, db ){
app.configure(function(){
app.set('port', process.env.PORT || 3000);
app.set('views', __dirname + '/views');
...
app.use(express.session({
// secret: settings.cookie_secret,
secret: 'woodchucks are nasty animals',
store: new MongoStore({
// db: settings.db
// db: hotplate.get('db').client
db: db
})
}));
Other drivers (like Mongoose, or even mongojs) manage not to force to use the callback. I looked at their code, and... well, I didn't quite get it. Mongojs in particular seems to use a library for promises, but I am having trouble understanding it.
Note that express.session for example wants, as a parameter, a fully working connection (which is what I do here). Without using the connection, you cannot actually be sure that the connection will be set.
So: what's the easiest way to get rid of the need for a callback?
The basic idea, I suppose, would be to "clone" the mongodb API calls, wrapping them with code to handle the possibility that the "db" variable is not set. But... how would that work?
Any help would be greatly appreciated!
Merc.
Eventually, you'll hit the situation where you absolutely need to wait for the connection to complete before continuing as it is async. And without a callback, it won't work (as the MongoClient requires a callback).
You could use an Event to wrap it -- but that's just a different type of callback really (conceptually). That's what Mongoose does -- it raises an event when the connection is ready, open.
Using Node.js, there isn't a solution that doesn't involve either an event or callback, somewhere (that's an intentional design choice of Node and the MongoDB driver). It's an async connection in the driver. You just need to delay some of the express setup until after the connection is opened. It only needs to happen at app startup.
Realize this question is a little old, but I use this tiny little wrapper to do the "lifting" and for a small amount of sugar so my db code is a little less verbose. Things like findById without having to wrap the ObjectId, and findArray without having to toArray() a query. Check it out:
https://github.com/dmcaulay/mongo-wrapper
Related
In the docs for Application there's a note on using context to expose widely used properties in ctx:
For example, to add a reference to your database from ctx:
app.context.db = db();
app.use(async ctx => {
console.log(ctx.db);
});
However, it also notes that relying more on ctx "could be considered an anti-pattern". Fair enough.
Then we have koa-knex-realworld-example which suggests something like this:
module.exports = function (app) {
// ...
const db = require('knex')(config.db)
app.db = db
// ...
return async function (ctx, next) {
if (ctx.app.migration && promise) {
await promise
}
return next()
}
}
And passes the app instance to the middleware:
app.use(db(app))
That feels strange to me. Having to pass app to its own middleware seems a little backwards, but maybe it's a known Koa pattern?
Finally, we have koa-knex which exposes knex as this.knex in middleware further down the stack:
app.use(_.get('/:userid', function *(userid) {
this.body = {
profile: yield this.knex('users').where('id', userid);
};
});
That one's a bit dated (generators, which are fine but not the default API for Koa anymore) and chucks Knex in global:
global.__knex || (global.__knex = Knex.initialize({
// ...
}));
this.knex = global.__knex;
yield next;
What's the alternative?
In summary, I think I'm pretty comfortable putting a reference to Knex in the context, and I don't think I'll be tempted to pollute it with too much else. However, if I wanted an alternative, is there a middleware approach to making Knex available in Koa2 that is preferable to the above options?
Currently trying to implement a different approach to connecting to my database using promises and pooling. This is what I have as of the moment:
// databaseConnection.js
var configDB = require('./database.js');
var mysql = require('promise-mysql');
var pool = mysql.createPool(configDB.connectionData);
function getSqlConnection() {
return pool.getConnection(configDB.connectionData).disposer(function(connection) {
connection.release();
});
}
module.exports = getSqlConnection;
Then I use the query like this:
#sqlQuery.js
var Promise = require("bluebird");
var getSqlConnection = require('./databaseConnection')
Promise.using(getSqlConnection(), function(connection) {
return connection.query("SELECT * FROM EXAMPLE_TABLE").then(function(row) {
return process(rows);
}
}
I'm using this library which is just node-mysql wrapped with BlueBird promises. With that, I wanted to take advantage of BlueBird's disposing and using capability so I would only be connected to the DB when I needed to be.
Currently though I'm getting an error from Connection.js of mysql stating: cb is not a function. Based on this question I have somewhat of an idea of what I'm doing wrong but I'm not sure how I would go about using that with BlueBird's dispose/using paradigm. Thanks in advance for anyone that can help!
Huge lack of oversight on my part. The following line:
return pool.getConnection(configDB.connectionData).disposer...
should be:
return pool.getConnection().disposer...
Sorry about that. Still getting an error for connection.release not being a function which is strange but at least I can move forward with debugging that.
[This question is quite vague, I apologize for it. I'm trying to address my various troubles by answering the question myself]
I am building a Node.js app which has to perform various tasks at given intervals. Here is the global scaffold (involves bluebird promises and mongoose for DB interactions) :
var Promise = require("bluebird");
var mongoose = require('mongoose');
mongoose.Promise = require('bluebird');
// Personal modules
var bootApp = require(...);
var doStuffA = require(...);
var doStuffB = require(...);
var doStuffC = require(...);
// running locally, but meant to be deployed at some point
mongoose.connect('mongodb://localhost:27017/myDatabase');
var db = mongoose.connection;
db.on('error', () => {
console.log("Error : lost connection !"));
process.exit(1);
});
db.once('open', () => {
bootApp() // always start by booting
.then( () => { // then start the infinite loop of events
setInterval(doStuffA, 1000*60*60); // 1x/1h
setInterval(doStuffB, 1000*60*10); // 1x/10min
setInterval(doStuffC, 1000*60*3); // 1x/3min
}).catch((e) => { // errors are handled by doStuffX(), so we should never catch anything here
console.log(e.message);
process.exit(1);
});
});
Each module doStuffX is a function returning a Promise, handling its own errors, and should finish at some point.
Expected behaviour for the entire app :
The app should be able to run forever
The app should try to doStuffX() at the given interval, regardless of whether it succeeded or failed last time.
[Optional :] The app should close smoothly without retrying any doStuff upon receiving a "shut down" signal.
My question : how to build a clean scaffold for such an app ? Can I get rid of setInterval and use promises instead ? One of my main concerns is to make sure the previous instance of doStuffX() is finished before starting the next one, even if it involves "killing" it in some way.
I am open to any link about scaffolding apps, but PLEASE DO NOT GIVE ME AN ANSWER/LINK INVOLVING EXPRESS : I don't need Express, since my app doesn't receive any request. (everything I found so far starts with Express :/)
If you don't want to start the next doStuffX() until the previous one is done, then you can replace your setInterval() with repeated setTimeout() calls.
function runA() {
setTimeout(function() {
doStuffA().then(runA).catch(function(err) {
// decide what to do differently if doStuffA has an error
});
}, 1000*60*60);
}
runA();
You could also add a timeout to this so that if doStuffA() doesn't respond within a certain amount of time, then you take some other action. This would involve using another timer and a timeout flag.
[I answer my own question, trying to put here everything I changed afterwards, in case someone falls into this page someday...]
For the Mongoose part of the scaffold, here is what I got so far for a reliable long-term DB connection :
The Mongoose documentation gives a fancy way to ensure the driver will never give up on trying to reconnect with reconnectTries
I don't really understand socketOptions and keepalive which seem related to replicas, so I leave them out of my code for now
Since Mongoose should autoreconnect whenever something goes wrong, I'll keep the db.once('open') as the access to the app code itself, even though I don't really understand yet the difference with db.on('connected')
I recommend reading this.
var Promise = require("bluebird");
var mongoose = require('mongoose');
mongoose.Promise = require('bluebird');
// Personal modules
var bootApp = require(...);
var doStuffA = require(...);
var doStuffB = require(...);
var doStuffC = require(...);
// running locally, but meant to be deployed at some point
var uri = 'mongodb://localhost:27017/myDatabase';
// the added option makes sure the app will always try to reconnect...
mongoose.connect(uri, { server: { reconnectTries: Number.MAX_VALUE } });
var db = mongoose.connection;
db.on('error', () => {
console.log("Error with Mongoose connection."));
});
db.once('open', () => {
bootApp() // always start by booting
.then( () => { // then start the infinite loop of events
//////////////////////////////////
/// Here goes the actual stuff ///
//////////////////////////////////
}).catch((e) => { // errors are handled by doStuffX(), so we should never catch anything here
console.log(e.message);
});
});
Now, for the actual repetitive stuff, my objective is to make sure everything runs smoothly, and that no process gets stuck. About the changes I made :
The methods used are not native ES6 but are specific to bluebird. You can read about .timeout() and .delay() which I find very useful for chaining timeouts and intervals in a clean code.
In my mind, .then(runA, runA) should always launch ONE UNIQUE instance of runA but I'm concerned about whether I could actually end up launching two parallel instances...
// Instead of using setInterval in a bluebird promised environment...
setInterval(doStuffA, 1000*60*60); // 1x/1h
// I would have liked a full promise chain, but as jfriend00 stated,
// It will end up crashing because the initial promise is never resolved...
function runA() {
return doStuffA()
.timeout(1000*60*30) // kill the running instance if it takes longer than 30min
.delay(1000*60*60) // wait 60min
.then(runA, runA); // whatever the outcome, restart the process
}
runA();
// Therefore, a solution like jfriend00's seems like the way to go :
function runA() {
setTimeout(function() {
doStuffA()
.timeout(1000*60*30)
.then(runA, runA)
}, 1000*60*60);
}
runA();
I'm running Express on Node.js and am wondering how I can effectively pass a single database connection context object between distinct Node modules (think of them sort of like application models).
I'd like to do this to be able to start a database transaction in one model and preserve it across calls to other affected models, for the duration of a single HTTP request.
I've seen people attempt to solve this using per-request database connections exposed as middleware before my route is run (taking from a connection pool, then running another piece of middleware after the routes to return the connection to the pool). That unfortunately means explicitly passing around a context object to all the affected functions, which is inelegant and clunky.
I've also seen people talking about the continuation-local-storage and AsyncWrap modules, but I'm unclear how they can solve my particular problem. I tried working with continuation-local-storage briefly but because I primarily use promises and generators in my code, it wasn't able to pass state back from the run method (it simply returns the context object passed into its callback).
Here's an example of what I'm trying to do:
// player-routes.js
router.post('/items/upgrade', wrap(function *(req, res) {
const result = yield playerItem.upgrade(req.body.itemId);
res.json();
});
// player-item.js
const playerItem = {
upgrade: Promise.coroutine(function *(user, itemId) {
return db.withTransaction(function *(conn) {
yield db.queryAsync('UPDATE player_items SET level = level + 1 WHERE id = ?', [itemId]);
yield player.update(user);
return true;
});
})
};
module.exports = playerItem;
// player.js
const player = {
update(user) {
return db.queryAsync('UPDATE players SET last_updated_at = NOW() WHERE id = ?', [user.id]);
})
};
module.exports = player;
// db.js
db.withTransaction = function(genFn) {
return Promise.using(getTransactionConnection(), conn => {
return conn.beginTransactionAsync().then(() => {
const cr = Promise.coroutine(genFn);
return Promise
.try(() => cr(conn))
.then(res => {
return conn.commitAsync().thenReturn(res);
}, err => {
return conn.rollbackAsync()
.then(() => logger.info('Transaction successfully rolled back'))
.catch(e => logger.error(e))
.throw(err);
});
});
});
};
A couple of notes here:
The wrap function is just a little piece of wrapper middleware that allows me to use generators/yield in my routes.
The db module is also just a small wrapper around the popular mysql module, that has been promisified.
What I'd like to do, probably in db.queryAsync, is check if there's a conn object set on the current context (which I'd set around the return Promise... call in db.withTransaction). If so, use that connection to do all subsequent database calls until the context goes out of scope.
Unfortunately, wrapping the return Promise... call in the CLS namespace code didn't allow me to actually return the promise -- it just returned the context object, which is incorrect in my case. It looks like most usages of CLS rely on not actually returning anything from inside the run callback. I also looked at cls-bluebird, but that didn't seem to do what I need it to do, either.
Any ideas? I feel like I'm close, but it's just not all hooking up exactly how I need it to.
I'm using the node-mongodb-native to connect to a local MongoDB instance. I'm having a little trouble wrapping my head around how to handle the connections. I've attempted to abstract the MongoDB stuff into a custom Database module:
Database.js
var mongo = require('mongodb');
var Database = function() { return this; };
Database.prototype.doStuff = function doStuff(callback) {
mongo.connect('mongodb://127.0.0.1:27017/testdb', function(err, conn) {
conn.collection('test', function(err, coll) {
coll.find({}, function(err, cursor) {
cursor.toArray(function(err, items) {
conn.close();
return callback(err, items);
});
});
});
});
};
// Testing
new Database().doStuff(function(err, items) {
console.log(err, items);
});
Is a new connection required for each method? That seems like it would get expensive awfully quick. I imagined that perhaps the connection would be established in the constructor and subsequent calls would leverage the existing connection.
This next question may be more of a design question, but considering how connection setup and tear-down may be expensive operations, I'm considering adding a Database object that is global to my application that can be leveraged to make calls to the database. Does this seem reasonable?
Please note that the code above was roughly taken from here. Thanks for your help.
You don't need a new connection for each method - you can open it once and use it for subsequent calls. The same applies to the individual collection variables - you can cache the result of a single call to collection() and this will let you only need those callbacks once, leaving them out everywhere else.