There are some database operations in my codebase that need to occur in multiple API endpoints that require atomicity across multiple collections.
Is it possible to contain these operations in a function and pass the session to the function? Javascript is not passed by reference but I'm thinking if you pass the session object and then return it you should be able to preserve continuity in your main scope. Thinking of implementing code something like this:
let session = await mongoose.startSession();
session = await functionWithDbOperations(session);
//continue implementation
The function would be defined elsewhere:
export const functionWithDbOperations = async (session) => {
await session.withTransaction(async () => {
// various db operations
// more db operations
});
return session;
}
Will this result in bugs? I'm not too well versed in Mongodb's implementation of atomicity across multiple DB accesses.
Related
When working with a big application that has several tables and several DB operations it's very difficult to keep track of what transactions are occurring. To workaround this we started by passing around a trx object.
This has proven to be very messy.
For example:
async getOrderById(id: string, trx?: Knex.Transaction) { ... }
Depending on the function calling getOrderById it will either pass a trx object or not. The above function will use trx if it is not null.
This seems simple at first, but it leads to mistakes where if you're in the middle of a transaction in one function and call another function that does NOT use a transaction, knex will hang with famous Knex: Timeout acquiring a connection. The pool is probably full.
async getAllPurchasesForUser(userId: string) {
..
const trx = await knex.transaction();
try {
..
getPurchaseForUserId(userId); // Forgot to make this consume trx, hence Knex timesout acquiring connection.
..
}
Based on that, I'm assuming this is not a best practice, but I would love if someone from Knex developer team could comment.
To improve this we're considering to instead use knex.transactionProvider() that is accessed throughout the app wherever we perform DB operations.
The example on the website seems incomplete:
// Does not start a transaction yet
const trxProvider = knex.transactionProvider();
const books = [
{title: 'Canterbury Tales'},
{title: 'Moby Dick'},
{title: 'Hamlet'}
];
// Starts a transaction
const trx = await trxProvider();
const ids = await trx('catalogues')
.insert({name: 'Old Books'}, 'id')
books.forEach((book) => book.catalogue_id = ids[0]);
await trx('books').insert(books);
// Reuses same transaction
const sameTrx = await trxProvider();
const ids2 = await sameTrx('catalogues')
.insert({name: 'New Books'}, 'id')
books.forEach((book) => book.catalogue_id = ids2[0]);
await sameTrx('books').insert(books);
In practice here's how I'm thinking about using this:
SingletonDBClass.ts:
const trxProvider = knex.transactionProvider();
export default trxProvider;
Orders.ts
import trx from '../SingletonDBClass';
..
async getOrderById(id: string) {
const trxInst = await trx;
try {
const order = await trxInst<Order>('orders').where({id});
trxInst.commit();
return order;
} catch (e) {
trxInst.rollback();
throw new Error(`Failed to fetch order, error: ${e}`);
}
}
..
Am I understanding this correctly?
Another example function where a transaction is actually needed:
async cancelOrder(id: string) {
const trxInst = await trx;
try {
trxInst('orders').update({ status: 'CANCELED' }).where({ id });
trxInst('active_orders').delete().where({ orderId: id });
trxInst.commit();
} catch (e) {
trxInst.rollback();
throw new Error(`Failed to cancel order, error: ${e}`);
}
}
Can someone confirm if I'm understanding this correctly? And more importantly if this is a good way to do this. Or is there a best practice I'm missing?
Appreciate your help knex team!
No. You cannot have global singleton class returning the transaction for your all of your internal functions. Otherwise you are trying always to use the same transaction for all the concurrent users trying to do different things in the application.
Also when you once commit / rollback the transaction returned by provider, it will not work anymore for other queries. Transaction provider can give you only single transaction.
Transaction provider is useful in a case, where you have for example middleware, which provides transaction for request handlers, but it should not be started, since it might not be needed so you don't want yet allocate a connection for it from pool.
Good way to do your stuff is to pass transcation or some request context or user session around, so that each concurrent user can have their own separate transactions.
for example:
async cancelOrder(trxInst, id: string) {
try {
trxInst('orders').update({ status: 'CANCELED' }).where({ id });
trxInst('active_orders').delete().where({ orderId: id });
trxInst.commit();
} catch (e) {
trxInst.rollback();
throw new Error(`Failed to cancel order, error: ${e}`);
}
}
Depending on the function calling getOrderById it will either pass a trx object or not. The above function will use trx if it is not null.
This seems simple at first, but it leads to mistakes where if you're in the middle of a transaction in one function and call another function that does NOT use a transaction, knex will hang with famous Knex: Timeout acquiring a connection. The pool is probably full.
We usually do it in a way that if trx is null, query throws an error, so that you need to explicitly pass either knex / trx to be able to execute the method and in some methods trx is actually required to be passed.
Anyhow if you really want to force everything to go through single transaction in a session by default you could create API modules in a way that for each user session you create an API instance which is initialized with transaction:
const dbForSession = new DbService(trxProvider);
const users = await dbForSession.allUsers();
and .allUsers() does something like return this.trx('users');
I'm trying to use findAndUpdate from within the object class but it's not working.
Getting the following error
findByIdAndUpdate is not a function
The code
Inside
gameScheme.methods.makeNextRound = function() {
First thing, assign self (Before any lines)
var self = this;
Returning a Promise
return self.findByIdAndUpdate(
self._id,
{ $push: { "rounds": { storyTitle: story.name } } }
).exec();
Cannot know how to get self properly from Mongoose since this method is defined before
// we need to create a model using it
let Game = mongoose.model('Game', gameScheme);
// make this available to our users in our Node applications
module.exports = Game;
On a side note, just using update doesn't work. The rounds do not get concatenated.
Also I think it's very unwise to use game.rouds[0] = "some round" or pushing directly and saving due to nodes async nature. (document miss match)
According to the Mongoose documentation for Schemas you define .methods and .statics on the schema. However, MongoDB methods are called directly on the model.
gameScheme.methods.makeNextRound = function () {
return this.model("Game").findByIdAndUpdate(this._id,
};
Important: the actual interaction with the data happens with the Model that you obtain through mongoose.model or db.model. That's the object that you can instantiate or that you can call .find(), .findOne(), etc upon. Don't confuse schemas and actual models!
I'm running Express on Node.js and am wondering how I can effectively pass a single database connection context object between distinct Node modules (think of them sort of like application models).
I'd like to do this to be able to start a database transaction in one model and preserve it across calls to other affected models, for the duration of a single HTTP request.
I've seen people attempt to solve this using per-request database connections exposed as middleware before my route is run (taking from a connection pool, then running another piece of middleware after the routes to return the connection to the pool). That unfortunately means explicitly passing around a context object to all the affected functions, which is inelegant and clunky.
I've also seen people talking about the continuation-local-storage and AsyncWrap modules, but I'm unclear how they can solve my particular problem. I tried working with continuation-local-storage briefly but because I primarily use promises and generators in my code, it wasn't able to pass state back from the run method (it simply returns the context object passed into its callback).
Here's an example of what I'm trying to do:
// player-routes.js
router.post('/items/upgrade', wrap(function *(req, res) {
const result = yield playerItem.upgrade(req.body.itemId);
res.json();
});
// player-item.js
const playerItem = {
upgrade: Promise.coroutine(function *(user, itemId) {
return db.withTransaction(function *(conn) {
yield db.queryAsync('UPDATE player_items SET level = level + 1 WHERE id = ?', [itemId]);
yield player.update(user);
return true;
});
})
};
module.exports = playerItem;
// player.js
const player = {
update(user) {
return db.queryAsync('UPDATE players SET last_updated_at = NOW() WHERE id = ?', [user.id]);
})
};
module.exports = player;
// db.js
db.withTransaction = function(genFn) {
return Promise.using(getTransactionConnection(), conn => {
return conn.beginTransactionAsync().then(() => {
const cr = Promise.coroutine(genFn);
return Promise
.try(() => cr(conn))
.then(res => {
return conn.commitAsync().thenReturn(res);
}, err => {
return conn.rollbackAsync()
.then(() => logger.info('Transaction successfully rolled back'))
.catch(e => logger.error(e))
.throw(err);
});
});
});
};
A couple of notes here:
The wrap function is just a little piece of wrapper middleware that allows me to use generators/yield in my routes.
The db module is also just a small wrapper around the popular mysql module, that has been promisified.
What I'd like to do, probably in db.queryAsync, is check if there's a conn object set on the current context (which I'd set around the return Promise... call in db.withTransaction). If so, use that connection to do all subsequent database calls until the context goes out of scope.
Unfortunately, wrapping the return Promise... call in the CLS namespace code didn't allow me to actually return the promise -- it just returned the context object, which is incorrect in my case. It looks like most usages of CLS rely on not actually returning anything from inside the run callback. I also looked at cls-bluebird, but that didn't seem to do what I need it to do, either.
Any ideas? I feel like I'm close, but it's just not all hooking up exactly how I need it to.
I have the following queries, which starts with the GetById method firing up, once that fires up and extracts data from another document, it saves into the race document.
I want to be able to cache the data after I save it for ten minutes. I have taken a look at cacheman library and not sure if it is the right tool for the job. what would be the best way to approach this ?
getById: function(opts,callback) {
var id = opts.action;
var raceData = { };
var self = this;
this.getService().findById(id,function(err,resp) {
if(err)
callback(null);
else {
raceData = resp;
self.getService().getPositions(id, function(err,positions) {
self.savePositions(positions,raceData,callback);
});
}
});
},
savePositions: function(positions,raceData,callback) {
var race = [];
_.each(positions,function(item) {
_.each(item.position,function(el) {
race.push(el);
});
});
raceData.positions = race;
this.getService().modelClass.update({'_id' : raceData._id },{ 'positions' : raceData.positions },callback(raceData));
}
I have recently coded and published a module called Monc. You could find the source code over here. You could find several useful methods to store, delete and retrieve data stored into the memory.
You may use it to cache Mongoose queries using simple nesting as
test.find({}).lean().cache().exec(function(err, docs) {
//docs are fetched into the cache.
});
Otherwise you may need to take a look at the core of Mongoose and override the prototype in order to provide a way to use cacheman as you original suggested.
Create a node module and force it to extend Mongoose as:
monc.hellocache(mongoose, {});
Inside your module you should extend the Mongoose.Query.prototype
exports.hellocache = module.exports.hellocache = function(mongoose, options, Aggregate) {
//require cacheman
var CachemanMemory = require('cacheman-memory');
var cache = new CachemanMemory();
var m = mongoose;
m.execAlter = function(caller, args) {
//do your stuff here
}
m.Query.prototype.exec = function(arg1, arg2) {
return m.execAlter.call(this, 'exec', arguments);
};
})
Take a look at Monc's source code as it may be a good reference on how you may extend and chain Mongoose methods
I will explain with npm redis package which stores key/value pairs in the cache server. keys are queries and redis stores only strings.
we have to make sure that keys are unique and consistent. So key value should store query and also name of the model that you are applying the query.
when you query, inside the mongoose library, there is
function Query(conditions, options, model, collection) {} //constructor function
responsible for query. inside this constructor,
Query.prototype.exec = function exec(op, callback) {}
this function is responsible executing the queries. so we have to manipulate this function and have it execute those tasks:
first check if we have any cached data related to the query
if yes respond to request right away and return
if no we need to respond to request and update our cache and then respond
const redis = require("client");
const redisUrl = "redis://127.0.0.1:6379";
const client = redis.createClient(redisUrl);
const util = require("util");
//client.get does not return promise
client.get = util.promisify(client.get);
const exec = mongoose.Query.prototype.exec;
//mongoose code is written using classical prototype inheritance for setting up objects and classes inside the library.
mongoose.Query.prototype.exec = async function() {
//crate a unique and consistent key
const key = JSON.stringify(
Object.assign({}, this.getQuery(), {
collection: this.mongooseCollection.name
})
);
//see if we have value for key in redis
const cachedValue = await redis.get(key);
//if we do return that as a mongoose model.
//the exec function expects us to return mongoose documents
if (cachedValue) {
const doc = JSON.parse(cacheValue);
return Array.isArray(doc)
? doc.map(d => new this.model(d))
: new this.model(doc);
}
const result = await exec.apply(this, arguments); //now exec function's original task.
client.set(key, JSON.stringify(result),"EX",6000);//it is saved to cache server make sure capital letters EX and time as seconds
};
if we store values as array of objects we need to make sure that each object is individullay converted to mongoose document.
this.model is a method inside the Query constructor and converts object to a mongoose document.
note that if you are storing nested values instead of client.get and client.set, use client.hset and client.hget
Now we monkey patched
Query.prototype.exec
so you do not need to export this function. wherever you have a query operation inside your code, mongoose will execute above code
How do I ensure that SQLite doesn't interleave queries from multiple concurrent node.js/Express HTTP requests into a single transaction?
I would want DB queries from different requests to be executed in separate transactions, isolated from each other, allowing each to be committed or rolled back independently of each other. Given node.js's single-thread (and single-DB-connection?) characteristics this seems particularly problematic.
I've been trawling through dozens of web pages and docs and haven't found any clear (to me) explanation of how (or whether) this scenario is handled. For instance the description of db.serialize() only states that one query will be executed at a time - it says nothing about the separation of queries belonging to different transactions.
Any pointers appreciated!
I faced the exact same issue. Here is how I solved it:
Step 1: Dnsure all database access is done via a set of library functions, where the database is passed in as the first input parameter.
Step 2: When you create the database object (using new sqlite3.Database('path')), wrap the object with some extra meta-information:
Here is how I wrapped mine:
var wrap = { db_obj : new sqlite3.Database('path'), locked : "UNLOCKED", work_queue : [] }
Step 3: Create a function that will be used to run db functions. If the database is locked, this function will defer their execution until later. I called mine "runAsync":
function runAsync( db, fn, callBack) {
if (db.locked === "LOCKED" ) {
db.work_queue.push( function() { runAsync( db, fn, callBack ); });
return;
}
fn(db,callBack);
}
We can see that this function checks the state of the wrapped db object. If it is locked, we defer execution by storing the function in the "work queue" to be executed later (by doing fn()).
Step 4: Create a function that will be used to run db functions exclusively. If the database is locked, this function will defer execution until later, else it will lock the database, run its function and then unlock the database. I called mine: runAsyncExclusive():
function runAsyncExclusive( db, fn, callBack ) {
if(db.locked === "LOCKED") {
db.work_queue.push( function() { runAsyncExclusive( db, fn, callBack) });
return;
}
var exclusiveDb = {
db_obj : db.db_obj,
locked : "UNLOCKED",
work_queue : []
};
db.locked = "LOCKED";
fn(exclusiveDb, function(err,res) {
db.locked = "UNLOCKED";
var workItems = db.work_queue;
_.each(workItems, function(fn) { fn(); });
db.work_queue = [];
callBack(err,res);
})
}
The exclusiveDb object that is passed into the function will allow exclusive access to the db. This object can itself be locked, allowing for arbitrarily deep nested locking.
Step 5: Adjust your library functions such that they call asyncRun() and asyncRunExclusive() where appropriate:
function get(db,sql,params,callBack) {
db.get(sql,params,callBack);
}
Becomes...
function get(db,sql,params,callBack) {
runAsync( db, function(db,cb) { db.get(sql,params,cb); }, callBack );
}
Voila!
(Apologies if this wasn't clear enough)