Is the database gets locked when making a transaction with mongodb? - node.js

I'm having a quite complex data structure, and logic, where I'm using many findOneAndUpdates for atomicity, and the whole process can only be successful if there are no errors. If there are errors, all changes need to be rolled back.
My application is not specifically for this, but it may demonstrate the problem. Let's say it's an ecommerce system, and two people are looking at the same items. The same sneakers, and t-shirt for example, but only one is available of both (let's say there's no basket, so we only know about these orders when they come in).
The rule is the order can only be successful if after subtracting the ordered amount of the available inventories, their amounts remain 0, or greater. So the order is either completely fulfilled, or not fulfilled at all.
This is my first time playing with transactions, and cannot even understand them (and the more I read about them, the more unanswered questions I have, so become more confused).
I was playing with the thought of what if the following happens in order:
first transaction starts
second transaction starts
first transaction updates items
second transaction updates items
first transaction commits
second transaction commits
This is an example code of how it would look like:
type OrderItem = {
itemId: string;
amount: string;
};
type Order = OrderItem[];
const makeOrder = async (order: Order) => {
const session = await startSession();
session.startTransaction();
let orderSuccessful = true;
await Promise.all(
order.map(async ({ itemId, amount }) => {
const updatedItem = await ItemModel.findByIdAndUpdate(
itemId,
{ $inc: { amount: -amount } },
{ new: true, session },
);
if (updatedItem.amount < 0) orderSuccessful = false;
}),
);
if (!orderSuccessful) {
await session.abortTransaction();
await session.endSession();
throw 'Items missing from order.';
}
await session.commitTransaction();
await session.endSession();
};
And two orders come in this form: [ { itemId: 'sneakers', amount: 1 }, { itemId: 'tShirt', amount: 1 }]. And exactly that much inventory we have in the database.
So basically there would be two sessions in parallel, and the changes would only be reflected if the transactions are commited.
But my question was at both transactions (because they aren't commited yet, and the sessions "don't know about each other", at both transactions the state at the time of findOneAndUpdate is that there's still one available of each items, and I basically "lose" the benefit of it, because even though there's no gap between read and update, it's only true for that session.
I was doing some playaround, and realised that it's not the case.
console.time('until timeout to update outside session');
new Promise((resolve) => {
setTimeout(async () => {
console.timeEnd('until timeout to update outside session'); // 2nd
console.time('updating the same order outside session');
const updatedOutsideOrder = await OrderModel.findByIdAndUpdate(
order._id,
{
$inc: { value: -1 },
},
{ new: true },
).exec();
console.timeEnd('updating the same order outside session'); // 5th
console.log('updatedOutsideOrder', updatedOutsideOrder); // 6th
resolve(true);
}, 1000);
});
const session = await startSession();
session.startTransaction();
const updatedInsideOrder = await OrderModel.findByIdAndUpdate(
order._id,
{
$inc: { value: -1 },
},
{ new: true, session },
).exec();
console.log('updatedInsideOrder', updatedInsideOrder); // 1st
await new Promise((resolve) => {
setTimeout(() => {
resolve(true);
console.log('timeout to make sure the update outside finishes before commit'); // 3rd
}, 5000);
});
await session.commitTransaction();
await session.endSession();
console.log(
'orderAfter transaction',
await OrderModel.findById(order._id).exec(),
); // 4th
I was surprised when I noticed that mongoose actually waits to do anything if a transaction is in progress. I guess the database is "locked".
This has raised a lot of questions.
what if multiple instances of the api are deployed, and mongoose won't know about the sessions in different instances?
if to the previous question the answer is it's not mongoose, but the database is sending the signal to mongoose that it's currently locked, how is it going to be solved when the database will need to be available through the whole world?
the most important question is how this will look like when there will be thousands of orders per second? If the whole transaction takes more than a millisecond, the delay between every request will grow as time goes.
I'm going round and round with this problem for months, and cannot find the solution, so any help would be appreciated.

Related

Do I really have to pass the session as a param when doing Mongo Node transaction

When doing a transaction like
await ( await this.client )
.withSession(async (session) => {
try {
session.startTransaction();
await collection1.updateMany({ id }, { $set: { done: true } }, { session });
await collection2.updateMany({ someId, test: { $exists: true } }, { $set: { test: [] } }, { session });
session.commitTransaction();
} catch (err) {
session.abortTransaction();
throw new Error(`Failed`);
}
});
Why do I have to pass the { session } as a param for the 2 updates?
The documentation doesn't seem to explain why that is, shouldn't everything between a start, stop session use that session, including await.collection1?
Thank you
This is totally normal in all the DBMS that i worked with.
As far as i know, the reason behind that is that you dont always want to add every DB-transaction to the started Transaction, because it might lead to reduced throughput and blocking.
So you only want to add Transactions that change the state of a Document and which are essential. E.g. most of the time you dont want to add simple read operations to the transactions and block the document with that

Race condition between two request in mongodb

This is eCommerce site and I'm using mongodb as a database, users can place order and each order can have multiple products. Product is a seperate table that contains quantityLeft of each product. There's a situation that when two concurrent requests comes and tries to buy the same product the ordered items in orders table exceeds the available quantity in product table.
Product Table
{
_id: '56e33c56ddec541556a61763',
name: 'Chocolate',
quantityLeft: 1
}
In product's table only 1 chocolate left if one request comes at a time it works fine. Request comes check the order.quantity and handle if there's enough product available.
But when 2 requests comes exactly the same time issue occurs both the request query the database to get the product and check the quantityLeft and found that only 1 chocolate is available and passes the check that enough quantity is still present in inventory and places the order. But in actual 2 orders are placed and quantity we have is only 1.
Order Table
{
_id: '60e33c56ddec541556a61595',
items: [{
_id: '56e33c56ddec541556a61763',
quantity: 1
}]
}
I tried to put both the queries to get the Product detail and place order in same transaction but it doesn't work. Something like this
const session = await mongoose.startSession({ defaultTransactionOptions: { readConcern: { level: 'local' }, writeConcern: { w: 1 } } })
await session.withTransaction(async () => {
const promiseArray = order.items.map((item) => Product.find({ _id: item._id }, { session })
const products = Promise.all(promiseArray)
const productById = {}
products.forEach((product) => {
productById[product._id] = product
})
order.items.forEach((item) => {
if (productById[item].quantityLeft < order.item) {
throw new Error('Not enough quantity')
}
})
await Order.create(order, {session})
}, { readConcern: { level: 'local' }, writeConcern: { w: 1 } });
I'm using nodejs (14.16), mongodb as database npm package is mongoose (5.9).

Nodejs mongodb find and update multiple documents in transaction

I have a mongo 4.2 replica set. I have N processes running concurrently and trying to read a collection. This collection is like a queue. I'd like to read 100 elements and update them in a transaction so other processes won't try to read those.
My code goes:
const collection = client.db("test").collection(TEST_COLLECTION);
const session = client.startSession();
try {
let data = null;
await session.withTransaction(async () => {
console.log("starting transaction")
data = await collection.find({ runId: null }, { _id: 1, limit: 100 }).toArray();
const idList = data.map(item => item._id.toHexString());
await collection.updateMany(
{ runId: { $in: idList } },
{ $set: { runId: runId } },
{ session });
console.log("Successful transaction")
});
data.map(item => {
// process element one by one and update them (no need for transaction here)
})
} catch (e) {
console.error("The transaction was aborted due to an unexpected error: " + e);
} finally {
await session.endSession();
console.log("Closing transaction")
}
this is the code I've got right now. The thing is that find() won't accept options so I can't pass the session. This means it won't be part of the transaction.
the mongodb documentations states that: When using the drivers, each operation in the transaction must be associated with the session (i.e. pass in the session to each operation).
So I'm assuming that this is actually not transactional only the update part which not solves my problem. Is there any way to include both in my transaction? Any ideas on this? Other/better options?
Thanks
EDIT:
So I was staring at my question for 15 minutes when it hit me. If I update first using the transaction. Then querying with the runId even outside of the transaction I can achieve my goal. Am I right? Is it so easy?
EDIT2:
Edit1 was stupid now I can't limit to 100 items. Back to the start.
EDIT3:
I'am using native mongodb nodejs driver.
To use a find in a transaction, pass the session using the session method:
doc = await Customer.findOne({ name: 'Test' }).session(session);
See Transactions in Mongoose

What is the best way to slowly (500/second) update many documents in a MongoDB database with a new field and value?

I have a MongoDB database that I want to update about 100,000 documents with a "score" for each on a daily basis. The challenge with the way I have implemented it is that it tries to update them really really fast (about 2,000 updates per second) and my MongoDB limits are set to only 500 updates per second (M5 tier) so MongoDB is sporadically throwing an error back to me (I confirmed with MongoDB support that this why I'm getting the error sometimes).
Is there a way to perhaps batch the updates or a better way to do what I'm doing?
Here's the code I am using. If I just turn it off when I get an error and start it back up it will eventually update all the documents, but that's an unsustainable solution:
await client
.db("test")
.collection("collection_name")
.find({ score: { $exists: false } })
.forEach(async data => {
await client
.db("test")
.collection("collection_name")
.updateOne(
{ _id: data._id },
{
$set: {
score: GetScore(data)
}
}
);
});
client.close();
One problem might be that the callback to forEach is likely not awaited from the mongo library, therefore multiple of your queries will be issued concurrently - query two will be issued before query one is finished etc.
You could use a combination of next and hasNext on the cursor combined with awaiting a a promise that resolves later (might not be needed) instead of doing forEach, like so:
var cursor = await client
.db("test")
.collection("collection_name")
.find({ score: { $exists: false } });
while(await cursor.hasNext()) {
var data = await cursor.next();
await client
.db("test")
.collection("collection_name")
.updateOne(
{ _id: data._id },
{
$set: {
score: GetScore(data)
}
}
);
}
Docs: http://mongodb.github.io/node-mongodb-native/3.5/api/Cursor.html#next
http://mongodb.github.io/node-mongodb-native/3.5/api/Cursor.html#hasNext
Again, the "sleep" might actually not be necessary when you get your queries to run sequentially.

Mongo DB 4.0 Transactions With Mongoose & NodeJs, Express

I am developing an application where I am using MongoDB as database with Nodejs + Express in application layer, I have two collections, namely
users
transactions
Here i have to update wallet of thousands of users with some amount and if successful create a new document with related info for each transaction, This is My code :
userModel.update({_id : ObjectId(userId)}, {$inc : {wallet : 500}}, function (err, creditInfo) {
if(err){
console.log(err);
}
if(creditInfo.nModified > 0) {
newTransModel = new transModel({
usersId: ObjectId(userId),
amount: winAmt,
type: 'credit',
});
newTransModel.save(function (err, doc) {
if(err){
Cb(err);
}
});
}
});
but this solution is not atomic there is always a possibility of user wallet updated with amount but related transaction not created in transactions collection resulting in financial loss.
I have heard that recently MongoDB has added Transactions support in its 4.0 version, I have read the MongoDB docs but couldn't get it to successfully implement it with mongoose in Node.js, can anyone tell me how this above code be reimplemented using the latest Transactions feature of MongoDB which have these functions
Session.startTransaction()
Session.abortTransaction()
Session.commitTransaction()
MongoDB Docs : Click Here
with mongoose in Node.js, can anyone tell me how this above code be reimplemented using the latest Transactions feature
To use MongoDB multi-documents transactions support in mongoose you need version greater than v5.2. For example:
npm install mongoose#5.2
Mongoose transactional methods returns a promise rather than a session which would require to use await. See:
Transactions in Mongoose
Blog: A Node.JS Perspective on MongoDB 4.0: Transactions
For example, altering the example on the resource above and your example, you can try:
const User = mongoose.model('Users', new mongoose.Schema({
userId: String, wallet: Number
}));
const Transaction = mongoose.model('Transactions', new mongoose.Schema({
userId: ObjectId, amount: Number, type: String
}));
await updateWallet(userId, 500);
async function updateWallet(userId, amount) {
const session = await User.startSession();
session.startTransaction();
try {
const opts = { session };
const A = await User.findOneAndUpdate(
{ _id: userId }, { $inc: { wallet: amount } }, opts);
const B = await Transaction(
{ usersId: userId, amount: amount, type: "credit" })
.save(opts);
await session.commitTransaction();
session.endSession();
return true;
} catch (error) {
// If an error occurred, abort the whole transaction and
// undo any changes that might have happened
await session.abortTransaction();
session.endSession();
throw error;
}
}
is not atomic there is always a possibility of user wallet updated with amount but related transaction not created in transactions collection resulting in financial loss
You should also consider changing your MongoDB data models. Especially if the two collections are naturally linked. See also Model data for Atomic Operations for more information.
An example model that you could try is Event Sourcing model. Create a transaction entry first as an event, then recalculate the user's wallet balance using aggregation.
For example:
{tranId: 1001, fromUser:800, toUser:99, amount:300, time: Date(..)}
{tranId: 1002, fromUser:77, toUser:99, amount:100, time: Date(..)}
Then introduce a process to calculate the amount for each users per period as a cache depending on requirements (i.e. per 6 hours). You can display the current user's wallet balance by adding:
The last cached amount for the user
Any transactions for the user occur since the last cached amount. i.e. 0-6 hours ago.

Resources