I have a mongo 4.2 replica set. I have N processes running concurrently and trying to read a collection. This collection is like a queue. I'd like to read 100 elements and update them in a transaction so other processes won't try to read those.
My code goes:
const collection = client.db("test").collection(TEST_COLLECTION);
const session = client.startSession();
try {
let data = null;
await session.withTransaction(async () => {
console.log("starting transaction")
data = await collection.find({ runId: null }, { _id: 1, limit: 100 }).toArray();
const idList = data.map(item => item._id.toHexString());
await collection.updateMany(
{ runId: { $in: idList } },
{ $set: { runId: runId } },
{ session });
console.log("Successful transaction")
});
data.map(item => {
// process element one by one and update them (no need for transaction here)
})
} catch (e) {
console.error("The transaction was aborted due to an unexpected error: " + e);
} finally {
await session.endSession();
console.log("Closing transaction")
}
this is the code I've got right now. The thing is that find() won't accept options so I can't pass the session. This means it won't be part of the transaction.
the mongodb documentations states that: When using the drivers, each operation in the transaction must be associated with the session (i.e. pass in the session to each operation).
So I'm assuming that this is actually not transactional only the update part which not solves my problem. Is there any way to include both in my transaction? Any ideas on this? Other/better options?
Thanks
EDIT:
So I was staring at my question for 15 minutes when it hit me. If I update first using the transaction. Then querying with the runId even outside of the transaction I can achieve my goal. Am I right? Is it so easy?
EDIT2:
Edit1 was stupid now I can't limit to 100 items. Back to the start.
EDIT3:
I'am using native mongodb nodejs driver.
To use a find in a transaction, pass the session using the session method:
doc = await Customer.findOne({ name: 'Test' }).session(session);
See Transactions in Mongoose
Related
I'm having a quite complex data structure, and logic, where I'm using many findOneAndUpdates for atomicity, and the whole process can only be successful if there are no errors. If there are errors, all changes need to be rolled back.
My application is not specifically for this, but it may demonstrate the problem. Let's say it's an ecommerce system, and two people are looking at the same items. The same sneakers, and t-shirt for example, but only one is available of both (let's say there's no basket, so we only know about these orders when they come in).
The rule is the order can only be successful if after subtracting the ordered amount of the available inventories, their amounts remain 0, or greater. So the order is either completely fulfilled, or not fulfilled at all.
This is my first time playing with transactions, and cannot even understand them (and the more I read about them, the more unanswered questions I have, so become more confused).
I was playing with the thought of what if the following happens in order:
first transaction starts
second transaction starts
first transaction updates items
second transaction updates items
first transaction commits
second transaction commits
This is an example code of how it would look like:
type OrderItem = {
itemId: string;
amount: string;
};
type Order = OrderItem[];
const makeOrder = async (order: Order) => {
const session = await startSession();
session.startTransaction();
let orderSuccessful = true;
await Promise.all(
order.map(async ({ itemId, amount }) => {
const updatedItem = await ItemModel.findByIdAndUpdate(
itemId,
{ $inc: { amount: -amount } },
{ new: true, session },
);
if (updatedItem.amount < 0) orderSuccessful = false;
}),
);
if (!orderSuccessful) {
await session.abortTransaction();
await session.endSession();
throw 'Items missing from order.';
}
await session.commitTransaction();
await session.endSession();
};
And two orders come in this form: [ { itemId: 'sneakers', amount: 1 }, { itemId: 'tShirt', amount: 1 }]. And exactly that much inventory we have in the database.
So basically there would be two sessions in parallel, and the changes would only be reflected if the transactions are commited.
But my question was at both transactions (because they aren't commited yet, and the sessions "don't know about each other", at both transactions the state at the time of findOneAndUpdate is that there's still one available of each items, and I basically "lose" the benefit of it, because even though there's no gap between read and update, it's only true for that session.
I was doing some playaround, and realised that it's not the case.
console.time('until timeout to update outside session');
new Promise((resolve) => {
setTimeout(async () => {
console.timeEnd('until timeout to update outside session'); // 2nd
console.time('updating the same order outside session');
const updatedOutsideOrder = await OrderModel.findByIdAndUpdate(
order._id,
{
$inc: { value: -1 },
},
{ new: true },
).exec();
console.timeEnd('updating the same order outside session'); // 5th
console.log('updatedOutsideOrder', updatedOutsideOrder); // 6th
resolve(true);
}, 1000);
});
const session = await startSession();
session.startTransaction();
const updatedInsideOrder = await OrderModel.findByIdAndUpdate(
order._id,
{
$inc: { value: -1 },
},
{ new: true, session },
).exec();
console.log('updatedInsideOrder', updatedInsideOrder); // 1st
await new Promise((resolve) => {
setTimeout(() => {
resolve(true);
console.log('timeout to make sure the update outside finishes before commit'); // 3rd
}, 5000);
});
await session.commitTransaction();
await session.endSession();
console.log(
'orderAfter transaction',
await OrderModel.findById(order._id).exec(),
); // 4th
I was surprised when I noticed that mongoose actually waits to do anything if a transaction is in progress. I guess the database is "locked".
This has raised a lot of questions.
what if multiple instances of the api are deployed, and mongoose won't know about the sessions in different instances?
if to the previous question the answer is it's not mongoose, but the database is sending the signal to mongoose that it's currently locked, how is it going to be solved when the database will need to be available through the whole world?
the most important question is how this will look like when there will be thousands of orders per second? If the whole transaction takes more than a millisecond, the delay between every request will grow as time goes.
I'm going round and round with this problem for months, and cannot find the solution, so any help would be appreciated.
I want to make a transaction of MongoDB by Mongoose.
Mongoose version is ^5.8.9
Here is code to explain more details.
Below is my question.
how to match session on each collection. because I looking forward to use a session for each collection. however i fail to find a solution.
If one collection of User or Score, Transaction should be rollback and change work never to apply collection.
Is it possible ?
// here is just sample. it is not able to run by below the code
const session = await User.startSession();
await session.startTransaction();
try {
const user = await User.create(object);
const score = await Score.updateOne({userId : user._id }, {score : { $inc :5 }} );
// if one of User or Score will be failed, transaction should be rollback.
await session.commitTransaction();
} catch(e) {
console.log(e)
await session.abortTransaction();
} finally {
await session.endSession();
}
I have a MongoDB database that I want to update about 100,000 documents with a "score" for each on a daily basis. The challenge with the way I have implemented it is that it tries to update them really really fast (about 2,000 updates per second) and my MongoDB limits are set to only 500 updates per second (M5 tier) so MongoDB is sporadically throwing an error back to me (I confirmed with MongoDB support that this why I'm getting the error sometimes).
Is there a way to perhaps batch the updates or a better way to do what I'm doing?
Here's the code I am using. If I just turn it off when I get an error and start it back up it will eventually update all the documents, but that's an unsustainable solution:
await client
.db("test")
.collection("collection_name")
.find({ score: { $exists: false } })
.forEach(async data => {
await client
.db("test")
.collection("collection_name")
.updateOne(
{ _id: data._id },
{
$set: {
score: GetScore(data)
}
}
);
});
client.close();
One problem might be that the callback to forEach is likely not awaited from the mongo library, therefore multiple of your queries will be issued concurrently - query two will be issued before query one is finished etc.
You could use a combination of next and hasNext on the cursor combined with awaiting a a promise that resolves later (might not be needed) instead of doing forEach, like so:
var cursor = await client
.db("test")
.collection("collection_name")
.find({ score: { $exists: false } });
while(await cursor.hasNext()) {
var data = await cursor.next();
await client
.db("test")
.collection("collection_name")
.updateOne(
{ _id: data._id },
{
$set: {
score: GetScore(data)
}
}
);
}
Docs: http://mongodb.github.io/node-mongodb-native/3.5/api/Cursor.html#next
http://mongodb.github.io/node-mongodb-native/3.5/api/Cursor.html#hasNext
Again, the "sleep" might actually not be necessary when you get your queries to run sequentially.
I am developing an application where I am using MongoDB as database with Nodejs + Express in application layer, I have two collections, namely
users
transactions
Here i have to update wallet of thousands of users with some amount and if successful create a new document with related info for each transaction, This is My code :
userModel.update({_id : ObjectId(userId)}, {$inc : {wallet : 500}}, function (err, creditInfo) {
if(err){
console.log(err);
}
if(creditInfo.nModified > 0) {
newTransModel = new transModel({
usersId: ObjectId(userId),
amount: winAmt,
type: 'credit',
});
newTransModel.save(function (err, doc) {
if(err){
Cb(err);
}
});
}
});
but this solution is not atomic there is always a possibility of user wallet updated with amount but related transaction not created in transactions collection resulting in financial loss.
I have heard that recently MongoDB has added Transactions support in its 4.0 version, I have read the MongoDB docs but couldn't get it to successfully implement it with mongoose in Node.js, can anyone tell me how this above code be reimplemented using the latest Transactions feature of MongoDB which have these functions
Session.startTransaction()
Session.abortTransaction()
Session.commitTransaction()
MongoDB Docs : Click Here
with mongoose in Node.js, can anyone tell me how this above code be reimplemented using the latest Transactions feature
To use MongoDB multi-documents transactions support in mongoose you need version greater than v5.2. For example:
npm install mongoose#5.2
Mongoose transactional methods returns a promise rather than a session which would require to use await. See:
Transactions in Mongoose
Blog: A Node.JS Perspective on MongoDB 4.0: Transactions
For example, altering the example on the resource above and your example, you can try:
const User = mongoose.model('Users', new mongoose.Schema({
userId: String, wallet: Number
}));
const Transaction = mongoose.model('Transactions', new mongoose.Schema({
userId: ObjectId, amount: Number, type: String
}));
await updateWallet(userId, 500);
async function updateWallet(userId, amount) {
const session = await User.startSession();
session.startTransaction();
try {
const opts = { session };
const A = await User.findOneAndUpdate(
{ _id: userId }, { $inc: { wallet: amount } }, opts);
const B = await Transaction(
{ usersId: userId, amount: amount, type: "credit" })
.save(opts);
await session.commitTransaction();
session.endSession();
return true;
} catch (error) {
// If an error occurred, abort the whole transaction and
// undo any changes that might have happened
await session.abortTransaction();
session.endSession();
throw error;
}
}
is not atomic there is always a possibility of user wallet updated with amount but related transaction not created in transactions collection resulting in financial loss
You should also consider changing your MongoDB data models. Especially if the two collections are naturally linked. See also Model data for Atomic Operations for more information.
An example model that you could try is Event Sourcing model. Create a transaction entry first as an event, then recalculate the user's wallet balance using aggregation.
For example:
{tranId: 1001, fromUser:800, toUser:99, amount:300, time: Date(..)}
{tranId: 1002, fromUser:77, toUser:99, amount:100, time: Date(..)}
Then introduce a process to calculate the amount for each users per period as a cache depending on requirements (i.e. per 6 hours). You can display the current user's wallet balance by adding:
The last cached amount for the user
Any transactions for the user occur since the last cached amount. i.e. 0-6 hours ago.
I am looking for a different ways to update 2 collections at the same time, in the following case i updating number of cases for a doctor
My Mongo DB document structure:
Doctors Collection:
{
"_id" : ObjectId("5a18a637346826574416a588"),
"doc_fname" : "Bob",
"doc_lname" : "Smith",
"numOfCases" : NumberInt(6),
"__v" : NumberInt(0)
}
CustomerCases Collection:
{
"_id" : ObjectId("5a18c02221344b58585105ea"),
"doctor_id" : ObjectId("5a18a637346826574416a588"),
"cust_fname" : "test",
"cust_lname" : "test",
"case_type" : "Crowns",
"__v" : NumberInt(0)
}
NodeJS Mongoose Code:
var NewCustomerCase = req.body;
CustomerCases.create(NewCustomerCase, function(err,CustomerCase){
if(err){
throw err;
}
else{
var query = {_id:NewCustomerCase.doctor_id};
// if the field doesnt exsit $set will set q new field
var update = {
'$inc': {'numOfCases': 1}
}
// when true return the updated document
var options = {new:true};
Doctors.findOneAndUpdate(query,update,options,function(err,updatedDocter){
if(err){
throw err;
}
});
res.json(CustomerCase);
}
});
This approach works fine, is there a better way to approach this problem, as far as i know its not possible to execute 2 collection at the same time
If the documents you are updating don't depend on each other you can update them both at the same time. For example by making two promises and then running them at the same time with Promise.all():
const customerPromise = CustomerCases.create(NewCustomerCase);
const doctorPromise = Doctors.findOneAndUpdate(query, update, options);
Promise.all([customerPromise, doctorPromise])
.then((result) => {
return res.status(200).json(result);
}).catch((err) => {
return res.status(400).json(err);
});
The promise returned by Promise.all() will be resolved when both the customer and doctor promises are resolved. The documentation is here.
The problem occurs if there is an error. If the customer fails to update the doctor will still increment numOfCases by 1. If the doctor document fails to update the customer case will still be made. So in fact, you do have some dependencies between the documents, and Promise.all() is maybe not such as good solution after all. This is often where transactions come in to play, but transactions in MongoDB is outside the scope of this answer, so I'll just post this link instead.
You already run the DB operations sequentially with callbacks. Here is a more modern way of doing it based on async/await:
try {
const newCustomerCaseFromDb = await CustomerCases.create(NewCustomerCase);
const doctorUpdated = await Doctors.findOneAndUpdate(query, update, options);
return res.status(200).json({ newCustomerCaseFromDb, doctorUpdated });
} catch(err) {
return res.status(400).json(err);
}
A function that uses await needs the async keyword.
The customer case is created first. If it fails it jumps into the catch block. If it succeeds it continues with finding and updating the doctor. Using await makes it look like synchronous code.
These are my thoughts:-
Use promises instead of callback.
Use of promise.all to concatenate your requests and execute them simultaneously.