Mongo DB 4.0 Transactions With Mongoose & NodeJs, Express - node.js

I am developing an application where I am using MongoDB as database with Nodejs + Express in application layer, I have two collections, namely
users
transactions
Here i have to update wallet of thousands of users with some amount and if successful create a new document with related info for each transaction, This is My code :
userModel.update({_id : ObjectId(userId)}, {$inc : {wallet : 500}}, function (err, creditInfo) {
if(err){
console.log(err);
}
if(creditInfo.nModified > 0) {
newTransModel = new transModel({
usersId: ObjectId(userId),
amount: winAmt,
type: 'credit',
});
newTransModel.save(function (err, doc) {
if(err){
Cb(err);
}
});
}
});
but this solution is not atomic there is always a possibility of user wallet updated with amount but related transaction not created in transactions collection resulting in financial loss.
I have heard that recently MongoDB has added Transactions support in its 4.0 version, I have read the MongoDB docs but couldn't get it to successfully implement it with mongoose in Node.js, can anyone tell me how this above code be reimplemented using the latest Transactions feature of MongoDB which have these functions
Session.startTransaction()
Session.abortTransaction()
Session.commitTransaction()
MongoDB Docs : Click Here

with mongoose in Node.js, can anyone tell me how this above code be reimplemented using the latest Transactions feature
To use MongoDB multi-documents transactions support in mongoose you need version greater than v5.2. For example:
npm install mongoose#5.2
Mongoose transactional methods returns a promise rather than a session which would require to use await. See:
Transactions in Mongoose
Blog: A Node.JS Perspective on MongoDB 4.0: Transactions
For example, altering the example on the resource above and your example, you can try:
const User = mongoose.model('Users', new mongoose.Schema({
userId: String, wallet: Number
}));
const Transaction = mongoose.model('Transactions', new mongoose.Schema({
userId: ObjectId, amount: Number, type: String
}));
await updateWallet(userId, 500);
async function updateWallet(userId, amount) {
const session = await User.startSession();
session.startTransaction();
try {
const opts = { session };
const A = await User.findOneAndUpdate(
{ _id: userId }, { $inc: { wallet: amount } }, opts);
const B = await Transaction(
{ usersId: userId, amount: amount, type: "credit" })
.save(opts);
await session.commitTransaction();
session.endSession();
return true;
} catch (error) {
// If an error occurred, abort the whole transaction and
// undo any changes that might have happened
await session.abortTransaction();
session.endSession();
throw error;
}
}
is not atomic there is always a possibility of user wallet updated with amount but related transaction not created in transactions collection resulting in financial loss
You should also consider changing your MongoDB data models. Especially if the two collections are naturally linked. See also Model data for Atomic Operations for more information.
An example model that you could try is Event Sourcing model. Create a transaction entry first as an event, then recalculate the user's wallet balance using aggregation.
For example:
{tranId: 1001, fromUser:800, toUser:99, amount:300, time: Date(..)}
{tranId: 1002, fromUser:77, toUser:99, amount:100, time: Date(..)}
Then introduce a process to calculate the amount for each users per period as a cache depending on requirements (i.e. per 6 hours). You can display the current user's wallet balance by adding:
The last cached amount for the user
Any transactions for the user occur since the last cached amount. i.e. 0-6 hours ago.

Related

MongoDB TTL deleting documents later then specified delay time

I am trying to delete a document in my mongoDB collection using TTL feature of it, and it works as expected, but not fully. It deletes the document later than the specified time.
I specified delete time 10seconds, but sometimes it takes 20seconds to delete it, sometimes 50seconds. Maybe I am making some mistake. I have used UTC date format, and tried with my local area date format too, but still the same. How do I resolve this problem?
One more thing that I want to ask, lets say somehow it works, and I have thousands of documents in my collection, will it decrease the performance of my database for requests and response handling??
because I am not deleting the whole collection, but individual documents, so keeping track of them might decrease performance, am I right?
Here is what I tried.
this is my schema
const mongoose = require('mongoose');
const tokens = mongoose.Schema({
token: String,
createdAt:{
type: Date,
expires: 10,
}
});
module.exports = {
authTokens: mongoose.model('Authtoken', tokens)
}
THIS HOW I AM CREATING DOCUMENT IN COLLECTION
app.post('/createToken', async (req, res) => {
try{
//create document in authTokens Collection
await authTokens.create({
token: req.body.token,
createdAt: new Date().toUTCString() //getting utc value of date
});
}
catch (err) {
res.send("Could not send token!");
console.error(err);
return;
}
res.send("Document Created Successfully");
return;
});
Can anyone help please
Thanks

Is the database gets locked when making a transaction with mongodb?

I'm having a quite complex data structure, and logic, where I'm using many findOneAndUpdates for atomicity, and the whole process can only be successful if there are no errors. If there are errors, all changes need to be rolled back.
My application is not specifically for this, but it may demonstrate the problem. Let's say it's an ecommerce system, and two people are looking at the same items. The same sneakers, and t-shirt for example, but only one is available of both (let's say there's no basket, so we only know about these orders when they come in).
The rule is the order can only be successful if after subtracting the ordered amount of the available inventories, their amounts remain 0, or greater. So the order is either completely fulfilled, or not fulfilled at all.
This is my first time playing with transactions, and cannot even understand them (and the more I read about them, the more unanswered questions I have, so become more confused).
I was playing with the thought of what if the following happens in order:
first transaction starts
second transaction starts
first transaction updates items
second transaction updates items
first transaction commits
second transaction commits
This is an example code of how it would look like:
type OrderItem = {
itemId: string;
amount: string;
};
type Order = OrderItem[];
const makeOrder = async (order: Order) => {
const session = await startSession();
session.startTransaction();
let orderSuccessful = true;
await Promise.all(
order.map(async ({ itemId, amount }) => {
const updatedItem = await ItemModel.findByIdAndUpdate(
itemId,
{ $inc: { amount: -amount } },
{ new: true, session },
);
if (updatedItem.amount < 0) orderSuccessful = false;
}),
);
if (!orderSuccessful) {
await session.abortTransaction();
await session.endSession();
throw 'Items missing from order.';
}
await session.commitTransaction();
await session.endSession();
};
And two orders come in this form: [ { itemId: 'sneakers', amount: 1 }, { itemId: 'tShirt', amount: 1 }]. And exactly that much inventory we have in the database.
So basically there would be two sessions in parallel, and the changes would only be reflected if the transactions are commited.
But my question was at both transactions (because they aren't commited yet, and the sessions "don't know about each other", at both transactions the state at the time of findOneAndUpdate is that there's still one available of each items, and I basically "lose" the benefit of it, because even though there's no gap between read and update, it's only true for that session.
I was doing some playaround, and realised that it's not the case.
console.time('until timeout to update outside session');
new Promise((resolve) => {
setTimeout(async () => {
console.timeEnd('until timeout to update outside session'); // 2nd
console.time('updating the same order outside session');
const updatedOutsideOrder = await OrderModel.findByIdAndUpdate(
order._id,
{
$inc: { value: -1 },
},
{ new: true },
).exec();
console.timeEnd('updating the same order outside session'); // 5th
console.log('updatedOutsideOrder', updatedOutsideOrder); // 6th
resolve(true);
}, 1000);
});
const session = await startSession();
session.startTransaction();
const updatedInsideOrder = await OrderModel.findByIdAndUpdate(
order._id,
{
$inc: { value: -1 },
},
{ new: true, session },
).exec();
console.log('updatedInsideOrder', updatedInsideOrder); // 1st
await new Promise((resolve) => {
setTimeout(() => {
resolve(true);
console.log('timeout to make sure the update outside finishes before commit'); // 3rd
}, 5000);
});
await session.commitTransaction();
await session.endSession();
console.log(
'orderAfter transaction',
await OrderModel.findById(order._id).exec(),
); // 4th
I was surprised when I noticed that mongoose actually waits to do anything if a transaction is in progress. I guess the database is "locked".
This has raised a lot of questions.
what if multiple instances of the api are deployed, and mongoose won't know about the sessions in different instances?
if to the previous question the answer is it's not mongoose, but the database is sending the signal to mongoose that it's currently locked, how is it going to be solved when the database will need to be available through the whole world?
the most important question is how this will look like when there will be thousands of orders per second? If the whole transaction takes more than a millisecond, the delay between every request will grow as time goes.
I'm going round and round with this problem for months, and cannot find the solution, so any help would be appreciated.

Nodejs mongodb find and update multiple documents in transaction

I have a mongo 4.2 replica set. I have N processes running concurrently and trying to read a collection. This collection is like a queue. I'd like to read 100 elements and update them in a transaction so other processes won't try to read those.
My code goes:
const collection = client.db("test").collection(TEST_COLLECTION);
const session = client.startSession();
try {
let data = null;
await session.withTransaction(async () => {
console.log("starting transaction")
data = await collection.find({ runId: null }, { _id: 1, limit: 100 }).toArray();
const idList = data.map(item => item._id.toHexString());
await collection.updateMany(
{ runId: { $in: idList } },
{ $set: { runId: runId } },
{ session });
console.log("Successful transaction")
});
data.map(item => {
// process element one by one and update them (no need for transaction here)
})
} catch (e) {
console.error("The transaction was aborted due to an unexpected error: " + e);
} finally {
await session.endSession();
console.log("Closing transaction")
}
this is the code I've got right now. The thing is that find() won't accept options so I can't pass the session. This means it won't be part of the transaction.
the mongodb documentations states that: When using the drivers, each operation in the transaction must be associated with the session (i.e. pass in the session to each operation).
So I'm assuming that this is actually not transactional only the update part which not solves my problem. Is there any way to include both in my transaction? Any ideas on this? Other/better options?
Thanks
EDIT:
So I was staring at my question for 15 minutes when it hit me. If I update first using the transaction. Then querying with the runId even outside of the transaction I can achieve my goal. Am I right? Is it so easy?
EDIT2:
Edit1 was stupid now I can't limit to 100 items. Back to the start.
EDIT3:
I'am using native mongodb nodejs driver.
To use a find in a transaction, pass the session using the session method:
doc = await Customer.findOne({ name: 'Test' }).session(session);
See Transactions in Mongoose

What is the best way to slowly (500/second) update many documents in a MongoDB database with a new field and value?

I have a MongoDB database that I want to update about 100,000 documents with a "score" for each on a daily basis. The challenge with the way I have implemented it is that it tries to update them really really fast (about 2,000 updates per second) and my MongoDB limits are set to only 500 updates per second (M5 tier) so MongoDB is sporadically throwing an error back to me (I confirmed with MongoDB support that this why I'm getting the error sometimes).
Is there a way to perhaps batch the updates or a better way to do what I'm doing?
Here's the code I am using. If I just turn it off when I get an error and start it back up it will eventually update all the documents, but that's an unsustainable solution:
await client
.db("test")
.collection("collection_name")
.find({ score: { $exists: false } })
.forEach(async data => {
await client
.db("test")
.collection("collection_name")
.updateOne(
{ _id: data._id },
{
$set: {
score: GetScore(data)
}
}
);
});
client.close();
One problem might be that the callback to forEach is likely not awaited from the mongo library, therefore multiple of your queries will be issued concurrently - query two will be issued before query one is finished etc.
You could use a combination of next and hasNext on the cursor combined with awaiting a a promise that resolves later (might not be needed) instead of doing forEach, like so:
var cursor = await client
.db("test")
.collection("collection_name")
.find({ score: { $exists: false } });
while(await cursor.hasNext()) {
var data = await cursor.next();
await client
.db("test")
.collection("collection_name")
.updateOne(
{ _id: data._id },
{
$set: {
score: GetScore(data)
}
}
);
}
Docs: http://mongodb.github.io/node-mongodb-native/3.5/api/Cursor.html#next
http://mongodb.github.io/node-mongodb-native/3.5/api/Cursor.html#hasNext
Again, the "sleep" might actually not be necessary when you get your queries to run sequentially.

MongoDB: handling auto-incrementing model id's instead of Mongo's native ObjectID

Due to a management decision, we are using userId for the users collection, postId for the posts collection, and topicId for the topics collection, instead of '_id' for each collection as the unique identifier.
This causes a few problems getting started - one of the problems I have encountered is with upserts -
Using Mongoose, we have a schema that restricts userId to be a unique value - but when doing an update on a user model, with upsert set to true, MongoDB appears to only look at the ObjectIds of a collection to see if the same one exists - it doesn't check to see if a model already exists with the same userId - therefore Mongo does an insert instead of an update.
let me illustrate this with some data:
let's say the user's collection has one document:
{
_id:'561b0fad638e99481ab6d84a'
userId:3,
name:'foo'
}
we then run:
User.update({userId:3},{"$set":{name:'bar'},{upsert:true},function(err,resp){
if(err){
// "errMessage": "insertDocument :: caused by :: 11000 E11000 duplicate key error index: app42153482.users.$userId_1 dup key: { : 3 }",
}
});
one would think that MongoDB would find the existing document with userId:3 and udpate it, so there must be something I am doing wrong since it's giving me the duplicate key error?
Typically the default value ObjectId is more ideal for the _id. Here, in this situation you can either override the default _id or you can have your own field for id(like userId in your case).
Use a separate counters collection to track the last number sequence used. The _id field contains the sequence name and the seq field contains the last value of the sequence.
Insert into the counters collection, the initial value for the userid:
db.counters.insert( {
_id: "userid",
seq: 0 } )
Create a getNextSequence function that accepts a name of the sequence. The function uses the findAndModify() method to atomically increment the seq value and return this new value:
function getNextSequence(name) {
var ret = db.counters.findAndModify(
{
query: { _id: name },
update: { $inc: { seq: 1 } },
new: true
}
);
return ret.seq;
}
Use this getNextSequence() function during insert().
db.users.insert(
{
_id: getNextSequence("userid"),
name: "Sarah C."
}
)
db.users.insert(
{
_id: getNextSequence("userid"),
name: "Bob D."
}
)
This way you can maintain as many sequences as you want in the same counter collection. For the upsert issue, check out the Optimistic Loop block in this link Create an auto-increment sequence field.
The second approach is to use a mongoose middleware like mongodb-autoincrement.
Hope it helps.
I don't know which versions of MongoDB and Mongoose you are using, but I couldn't reproduce your problem with MongoDB 3.0 and Mongoose 4.1.10.
I made a sample for you which will create and save a new user, update (using upsert) it, and create another one through an upsert. Try running this code:
"use strict";
var mongoose=require("mongoose");
var Schema = require('mongoose').Schema;
var ObjectId = mongoose.Schema.Types.ObjectId;
// Connect to test
mongoose.connect("mongodb://localhost:27017/test");
// Lets create your schema
var userSchema = new Schema({
_id: ObjectId,
userId: {type: Number, unique: true },
name: String
});
var User = mongoose.model("User", userSchema, "Users");
User.remove() // Let's prune our collection to start clean
.then( function() {
// Create our sample record
var myUser = new User({
_id:'561b0fad638e99481ab6d84a',
userId:3,
name:'foo'
});
return myUser.save();
})
.then( function() {
// Now its time to update (upsert userId 3)
return User.update({userId:3},{"$set":{name:'bar'}},{upsert:true});
})
.then( function() {
// Now its time to insert (upsert userId 4)
return User.update({userId:4},{"$set":{name:'bee'}},{upsert:true});
})
.then( function() {
// Lets show what we have inserted
return User.find().then(function(data) {console.log(data)});
})
.catch( function(err) {
// Show errors if anything goes wrong
console.error("ERROR", err);
})
.then( function() {
mongoose.disconnect();
});
Following the documentation (of MongoDB 3.0) upsert:true will only not insert a non-existing document if your query conditions match on the _id field.
See: https://docs.mongodb.org/manual/reference/method/db.collection.update/#mongodb30-upsert-id
Why are you not using the user_name for a user as unique id?
Because auto-incrementing fields as ids are a bad practice to use in a mongodb environment, especially if you want to use sharding
=> all your inserts will occur on the latest shard
=> the mongodb cluster will have to rebalance often / redistribute the data around.
(Currently this will not occur on your system as you still use the generated _id field)
You can off course also create a unique index on the user_id field:
https://docs.mongodb.org/manual/core/index-unique/#index-type-unique

Resources