This is eCommerce site and I'm using mongodb as a database, users can place order and each order can have multiple products. Product is a seperate table that contains quantityLeft of each product. There's a situation that when two concurrent requests comes and tries to buy the same product the ordered items in orders table exceeds the available quantity in product table.
Product Table
{
_id: '56e33c56ddec541556a61763',
name: 'Chocolate',
quantityLeft: 1
}
In product's table only 1 chocolate left if one request comes at a time it works fine. Request comes check the order.quantity and handle if there's enough product available.
But when 2 requests comes exactly the same time issue occurs both the request query the database to get the product and check the quantityLeft and found that only 1 chocolate is available and passes the check that enough quantity is still present in inventory and places the order. But in actual 2 orders are placed and quantity we have is only 1.
Order Table
{
_id: '60e33c56ddec541556a61595',
items: [{
_id: '56e33c56ddec541556a61763',
quantity: 1
}]
}
I tried to put both the queries to get the Product detail and place order in same transaction but it doesn't work. Something like this
const session = await mongoose.startSession({ defaultTransactionOptions: { readConcern: { level: 'local' }, writeConcern: { w: 1 } } })
await session.withTransaction(async () => {
const promiseArray = order.items.map((item) => Product.find({ _id: item._id }, { session })
const products = Promise.all(promiseArray)
const productById = {}
products.forEach((product) => {
productById[product._id] = product
})
order.items.forEach((item) => {
if (productById[item].quantityLeft < order.item) {
throw new Error('Not enough quantity')
}
})
await Order.create(order, {session})
}, { readConcern: { level: 'local' }, writeConcern: { w: 1 } });
I'm using nodejs (14.16), mongodb as database npm package is mongoose (5.9).
Related
I am working on an HRM project using MERN stack I have a medical expense section in which employees can request expense coverage (limit = 6000), how can I sum all requested amounts by employee and check if each employee is exceeding the limit before requesting here is my code:-
router.post("/", async (req, res) => {
// check if employee has pending request
let medicalExpenseRequest = await MedicalExpenseRequest.findOne({
"employee._id": req.body.employeeId,
"medicalExpense._id": req.body.medicalExpenseId,
status: "Pending",
});
if (medicalExpenseRequest)
return res
.status(400)
.send("Oops there is pending medical request for this employee!");
// this blocks the employee to request more than the limit at once // medicalexpense.allowed == 6000
if (req.body.amount > medicalExpense.allowedAmount)
return res
.status(400)
.send(
`Sorry can't request more than ${medicalExpense.allowedAmount} ETB`
);
// keep track of each employee request
//I want to check if the sum of all amounts requested by employee is greater than the limit and block if it is true
let taken;
const coveredExpense = await MedicalExpenseRequest.aggregate([
// { $match: { _id: { $eq: req.body.employeeId } } },
{
$group: {
_id: "$employee._id",
totalTaken: { $sum: "$amount" },
},
},
]);
// process the request successfully
});
NB: I tried MongoDB aggregates but I could not able to achieve the target
i figured out by my self what i did is to filter through coveredExpense and check if taken amount is greater than the allowed amount
coveredExpense.filter((expense) => {
taken = expense.totalTaken;
emp_id = expense._id;
});
if (
taken >= medicalExpense.allowedAmount &&
medicalRequesEmployee &&
medicalRequesEmployee.employee._id.equals(emp_id)
) {
return res.status(400).send("You have exceeded the limit");
}
Background: Im developing an app that shows analytics for inventory management.
It gets an office EXCEL file uploaded, and as the file uploads the app convert it to an array of JSONs. Then, it comapers each json object with the objects in the DB, change its quantity according to the XLS file, and add a timestamp to the stamps array which contain the changes in qunatity.
For example:
{"_id":"5c3f531baf4fe3182cf4f1f2",
"sku":123456,
"product_name":"Example",
"product_cost":10,
"product_price":60,
"product_quantity":100,
"Warehouse":4,
"stamps":[]
}
after the XLS upload, lets say we sold 10 units, it should look like that:
{"_id":"5c3f531baf4fe3182cf4f1f2",
"sku":123456,
"product_name":"Example",
"product_cost":10,
"product_price":60,
"product_quantity":90,
"Warehouse":4,
"stamps":[{"1548147562": -10}]
}
Right now i cant find the right commands for mongoDB to do it, Im developing in Node.js and Angular, Would love to read some ideas.
for (let i = 0; i < products.length; i++) {
ProductsDatabase.findOneAndUpdate(
{"_id": products[i]['id']},
//CHANGE QUANTITY AND ADD A STAMP
...
}
You would need two operations here. The first will be to get an array of documents from the db that match the ones in the JSON array. From the list you compare the 'product_quantity' keys and if there is a change, create a new array of objects with the product id and change in quantity.
The second operation will be an update which uses this new array with the change in quantity for each matching product.
Armed with this new array of updated product properties, it would be ideal to use a bulk update for this as looping through the list and sending
each update request to the server can be computationally costly.
Consider using the bulkWrite method which is on the model. This accepts an array of write operations and executes each of them of which a typical update operation
for your use case would have the following structure
{ updateOne :
{
"filter" : <document>,
"update" : <document>,
"upsert" : <boolean>,
"collation": <document>,
"arrayFilters": [ <filterdocument1>, ... ]
}
}
So your operations would follow this pattern:
(async () => {
let bulkOperations = []
const ids = products.map(({ id }) => id)
const matchedProducts = await ProductDatabase.find({
'_id': { '$in': ids }
}).lean().exec()
for(let product in products) {
const [matchedProduct, ...rest] = matchedProducts.filter(p => p._id === product.id)
const { _id, product_quantity } = matchedProduct
const changeInQuantity = product.product_quantity - product_quantity
if (changeInQuantity !== 0) {
const stamps = { [(new Date()).getTime()] : changeInQuantity }
bulkOperations.push({
'updateOne': {
'filter': { _id },
'update': {
'$inc': { 'product_quantity': changeInQuantity },
'$push': { stamps }
}
}
})
}
}
const bulkResult = await ProductDatabase.bulkWrite(bulkOperations)
console.log(bulkResult)
})()
You can use mongoose's findOneAndUpdate to update the existing value of a document.
"use strict";
const ids = products.map(x => x._id);
let operations = products.map(xlProductData => {
return ProductsDatabase.find({
_id: {
$in: ids
}
}).then(products => {
return products.map(productData => {
return ProductsDatabase.findOneAndUpdate({
_id: xlProductData.id // or product._id
}, {
sku: xlProductData.sku,
product_name: xlProductData.product_name,
product_cost: xlProductData.product_cost,
product_price: xlProductData.product_price,
Warehouse: xlProductData.Warehouse,
product_quantity: productData.product_quantity - xlProductData.product_quantity,
$push: {
stamps: {
[new Date().getTime()]: -1 * xlProductData.product_quantity
}
},
updated_at: new Date()
}, {
upsert: false,
returnNewDocument: true
});
});
});
});
Promise.all(operations).then(() => {
console.log('All good');
}).catch(err => {
console.log('err ', err);
});
I created a many-to-many association by sequelize in my koa app. But I had no idea on how to create additional attributes in the junction table. Thanks.
I referred to the official doc of sequelize but didn't find a solution. In brief:
"an order can have many items"
"an item can exist in many orders"
Then I created OrderItems as junction table.
But I have trouble in inserting value into the junction table
// definitions
const Item = sequelize.define('item', itemSchema);
const Order = sequelize.define('order', orderSchema);
// junction table
const OrderItems = sequelize.define('order_item', {
item_quantity: { type: Sequelize.INTEGER } // number of a certain item in a certain order.
});
// association
Item.belongsToMany(Order, { through: OrderItems, foreignKey: 'item_id' });
Order.belongsToMany(Item, { through: OrderItems, foreignKey: 'order_id' });
// insert value
const itemVals = [{ name: 'foo', price: 6 }, { name: 'bar', price: 7 }];
const orderVals = [
{
date: '2019-01-06',
items: [{ name: 'foo', item_quantity: 12 }]
},
{
date: '2019-01-07',
items: [{ name: 'foo', item_quantity: 14 }]
}
]
items = Item.bulkCreate(itemVals)
orders = Order.bulkCreate(orderVals)
//Questions here: create entries in junction table
for (let order of orders) {
const itemsInOrder = Item.findAll({
where: {
name: {
[Op.in]: order.items.map(item => item.name)
}
}
})
order.addItems(itemsInOrder, {
through: {
item_quantity: 'How to solve here?'
}
})
}
// my current manual solution:
// need to configure column names in junction table manually.
// Just like what we do in native SQL.
const junctionValList =[]
for (let orderVal of orderVals) {
orderVal.id = (await Order.findOne(/* get order id */)).dataValues.id
for (let itemVal of orderVal.items) {
itemVal.id = (await Item.findOne(/* get item id similarly */)).dataValues.id
const entyInJunctionTable = {
item_id: itemVal.id,
order_id: orderVal.id,
item_quantity: itemVal.item_quantity
}
junctionValList.push(entyInJunctionTable)
}
}
OrderItems.bulkCreate(junctionValList).then(/* */).catch(/* */)
In case that this script it's for seeding purpose you can do something like this:
/*
Create an array in which all promises will be stored.
We use it like this because async/await are not allowed inside of 'for', 'map' etc.
*/
const promises = orderVals.map((orderVal) => {
// 1. Create the order
return Order.create({ date: orderVal.date, /* + other properties */ }).then((order) => {
// 2. For each item mentioned in 'orderVal.items'...
return orderVal.items.map((orderedItem) => {
// ...get the DB instance
return Item.findOne({ where: { name: orderedItem.name } }).then((item) => {
// 3. Associate it with current order
return order.addItem(item.id, { through: { item_quantity: orderedItem.item_quantity } });
});
});
});
});
await Promise.all(promises);
But it's not an efficient way to do it in general. First of all, there are a lot of nested functions. But the biggest problem is that you associate items with the orders, based on their name and it's possible that in the future you will have multiple items with the same name.
You should try to use an item id, this way you will be sure about the outcome and also the script it will be much shorter.
I have a nodejs API server deployed in a Kubernetes cluster.
Users can send in bids on auction items.
To prevent a bid from overriding another there needs to be some synchronization.
I am seeing the following for an incoming bid:
start a transaction that reads the current bid and compares it to the incoming bid and updates the record
create an aggregation that does the same as above
I don't know which way to go. I also understand that you need to lock the document with either IX or X.
For a RDBMS you would create a transaction that locks the record and releases it after update but I don't know how it works for MongoDB.
Product.findById(productId)
.then(productmatch => {
if (productmatch.currentPrice > price) throw Error('no go')
const bid = new Bid({
price,
date: Date.now(),
user: user._id
})
return Product.findByIdAndUpdate(productId, {
$push: {
bids: bid
},
currentPrice: price,
currentUser: user,
currentBid: bid
}, {
new: true
})
.then(product => {
if (!product) throw Error(`no go`)
return bid._id.toString()
})
})
After a little more research I came up with this solution, however I do not know if its 100% reliable but I believe this method will lock the document and not let any other threads read the document between the query and update operations.
var query = {
_id: productId,
closed: false,
currentPrice: {
$lt: price
}
},
update = {
$push: {
bids: bid
},
currentPrice: price,
currentUser: user,
currentBid: bid
},
options = {
new: true
};
return Product.findOneAndUpdate(query, update, options)
.then(product => {
if (!product) throw Error(`no product found with id ${productId}`)
return bid._id.toString()
})
I have been struggling with the questions for a months now still no solution.
Basically I have 2 mongodb database structures.
One is called Users and another is called Items.
One user can have multiple Items.
User structure is simple =
Users = [{
_id: 1,
name: "Sam",
email: "sam#gmail.com",
group: "Rangers"
},
{
_id: 2,
name: "Michael",
email: "michael#gmail.com"
group: "Muse"
},
{
_id: 3,
name: "John",
email: "john#gmail.com"
group: "Merchant"
},
.....
]
The Items structures are as follows and each item is assigned to a user.
Items = [
{
_id: 1,
user_id: 1,
item_name: "Flying Sword",
timestamp: ...
},
{
_id: 3,
user_id: 1,
item_name: "Invisible Cloak",
timestamp: ...
},
{
_id: 4,
user_id: 2,
item_name: "Iron Shield"
},
{
_id: 5,
user_id: 7,
item_name: "Splashing Gun",
timestamp: ...
},
...
]
I want to run a mongoose query that queries the user as primary object.
And upon the returning the results of the user object I want to query the all the Items objects with the filtered users and append them as subdocuments to each user objects previously queried.
For example I want to query
Users.find({group: "Muse"}, function(err, users){
I DON"T KNOW WHAT TO WRITE INSIDE
})
Basically the results should be:
[
{
_id: 4,
name: "Jack",
email: "jack#gmail.com",
group: "Muse",
items: [
{
_id: 8
name: "Magic Wand",
user_id: 4,
timestamp: ...
}
{
_id: 12
name: "Blue Potion",
user_id: 4,
timestamp: ...
},
{
_id: 18
name: "Teleportation Scroll",
user_id: 4,
timestamp: ...
}
]
}
.....
More USERS of similar structure
]
Each user will return a maximum of three items which are sorted by timestamp.
Thanks in advance, I tried so many times and failed.
This is a multiple step question. So lets list out the steps:
Get a list of user documents that match a particular group.
Get a list of item documents that are assigned to each matched user from step 1.
Assign the appropriate item documents to a new property on the corresponding user document.
This can be tackled a few ways. A first pass might be to retrieve all the user documents and then iterating over them in memory retrieving the list of item documents for each user and appending that list to the user document. If your lists are smallish this shouldn't be too much of an issue but as scale comes into play and this becomes a larger list it could become a memory hog.
NOTE: all of the following code is untested so it might have typos or the like.
Users.find({group: "Muse"}, function(err, users){
var userIDs;
if (err) {
// do error handling
return;
}
userIDs = users.map(function (user) { return user._id; });
Items.find({user_id: {$in: userIDs}}, function (err, items) {
if (err) {
// do error handling
return;
}
users.forEach(function (user) {
user.items = items.filter(function (item) {
return item.user_id === user._id;
});
});
// do something with modified users object
});
});
While this will solve the problem there are plenty of improvements that can be made to make it a bit more performant as well as "clean".
For instance, lets use promises since this involves async operations anyway. Assuming Mongoose is configured to use the native Promise object or a then/catch compliant library
Users.find({group: "Muse"}).exec().then(function(users) {
var userIDs = users.map(function(user) {
return user._id;
});
// returns a promise
return Promise.all([
// include users for the next `then`
// avoids having to store it outside the scope of the handlers
users,
Items.find({
user_id: {
$in: userIDs
}
}).exec()
]);
}).then(function(results) {
var users = results[0];
var items = results[1];
users.forEach(function(user) {
user.items = items.filter(function(item) {
return item.user_id === user._id;
});
});
return users;
}).catch(function (err) {
// do something with errors from either find
});
This makes it subjectively a bit more readable but doesn't really help since we are doing a lot of manipulation in memory. Again, this might not be a concern if the document collections are smallish. However if is, there is a tradeoff that can be made with breaking up the request for items into one-per-user. Thus only working on chunks of the item list at a time.
We will also use Bluebird's map to limit the number of concurrent requests for items.
Users.find({group: "Muse"}).exec().then(function(users) {
return bluebird.map(users, function(user) {
return Items.find({user_id: user._id}).exec().then(function (items) {
user.items = items;
return user;
});
}, {concurrency: 5});
}).then(function(users) {
// do something with users
}).catch(function(err) {
// do something with errors from either find
});
This limits the amount of in memory manipulation for items but still leaves us iterating over users in memory. That can be tackled as well by using mongoose streams but I will leave that up to you to explore on your own (there are also other questions already on SO on how to use streams).
This makes it subjectively a bit more readable but doesn't really help since we are doing a lot of manipulation in memory. Again, this might not be a concern if the document collections are smallish. However if is, there is a tradeoff that can be made with breaking up the request for items into one-per-user. Thus only working on chunks of the item list at a time.