how can i get sum of all amounts requested - mongoDB - node.js

I am working on an HRM project using MERN stack I have a medical expense section in which employees can request expense coverage (limit = 6000), how can I sum all requested amounts by employee and check if each employee is exceeding the limit before requesting here is my code:-
router.post("/", async (req, res) => {
// check if employee has pending request
let medicalExpenseRequest = await MedicalExpenseRequest.findOne({
"employee._id": req.body.employeeId,
"medicalExpense._id": req.body.medicalExpenseId,
status: "Pending",
});
if (medicalExpenseRequest)
return res
.status(400)
.send("Oops there is pending medical request for this employee!");
// this blocks the employee to request more than the limit at once // medicalexpense.allowed == 6000
if (req.body.amount > medicalExpense.allowedAmount)
return res
.status(400)
.send(
`Sorry can't request more than ${medicalExpense.allowedAmount} ETB`
);
// keep track of each employee request
//I want to check if the sum of all amounts requested by employee is greater than the limit and block if it is true
let taken;
const coveredExpense = await MedicalExpenseRequest.aggregate([
// { $match: { _id: { $eq: req.body.employeeId } } },
{
$group: {
_id: "$employee._id",
totalTaken: { $sum: "$amount" },
},
},
]);
// process the request successfully
});
NB: I tried MongoDB aggregates but I could not able to achieve the target

i figured out by my self what i did is to filter through coveredExpense and check if taken amount is greater than the allowed amount
coveredExpense.filter((expense) => {
taken = expense.totalTaken;
emp_id = expense._id;
});
if (
taken >= medicalExpense.allowedAmount &&
medicalRequesEmployee &&
medicalRequesEmployee.employee._id.equals(emp_id)
) {
return res.status(400).send("You have exceeded the limit");
}

Related

MongoDB updateOne with push operator in nested object takes a long time to update

My model is as follows:
username: String,
password: String,
email: String,
portfolios: [,
{
portfolioName: String,
stocks: [
ticker: String,
amount: Number,
price: Number
]
}
]
The following post request finds the respective user and portfolio to push a stock to, then adds it.
The request does work but not consistently. Sometimes it works instantly but when doesn't I have to spam the request 5-10 ten times until it starts to work again. It is important to note that when it doesn't work nothing inside the post request even works, like any console.log(). But after spamming the request 5-10 times then everything works as usual
app.post('/api/stocks', async (req, res) => {
try {
const username = req.body.username
const portfolioName = req.body.portfolioName
const ticker = req.body.ticker
const amount = req.body.amount
const price = req.body.price
const stock = {
ticker: ticker,
amount: amount,
price: price
}
const insertStock = await UserModel.updateOne(
{
"username": username
},
{
$push: {
'portfolios.$[updatePortfolio].stocks': stock,
}
},
{
"arrayFilters": [
{"updatePortfolio.portfolioName": portfolioName}
]
}
)
if (insertStock) {
return res.json({status: 200})
} else {
return res.json({status: 500})
}
} catch (err) {
console.log(err)
}
})
Is there a performance issue with what I'm trying to accomplish because the data is nested? Or is it an issue with MongoDB? I am using a free database

Handling concurrent request that finds and update the same resource in Node Js & Mongo DB?

I have a function in node that runs after a clicking the checkout button. It checks the availability of the items in cart and if the item is available it will deduct it from the inventory.
I'm currently testing with two users clicking the checkout button at the same time. Both users have the exact same content in their cart (10 apples each) which gives a total of 20 apples, but there are only 10 apples in inventory.
If there is no item in cart it should return an error to the user but both orders are going through.
NOTE: This works if there is a 1 second delay between the clicks.
What can i do to prevent this?
// Check if items in inventory
const availability = await checkInventory(store, cart, seller);
if (!availability.success) {
return res.status(400).json({
success: false,
type: 'unavailable',
errors: availability.errors,
});
}
// Deduct Inventory
const inventory = await deductInventory(store, seller, cart);
if (!inventory) {
return next(new ErrorResponse('Server Error', 500));
}
checkInventory
exports.checkInventory = asyncHandler(async (store, cart, seller) => {
let isAvailable = true;
const unavailableProducts = [];
const inventory = await Inventory.find({
$and: [
{
store: store,
user: seller,
},
],
});
const products = inventory[0].products;
cart.forEach((item) => {
const product = products.find(
(product) => product._id.toString() === item.productId
);
if (!item.hasvariation) {
if (product.stock < item.qty) {
isAvailable = false;
unavailableProducts.push(
`${item.title} is not available, only ${product.stock} left available`
);
}
}
if (item.hasvariation) {
const variation = product.variations.find(
(variation) => variation._id.toString() === item.variationId
);
const option = variation.options.find(
(option) => option._id.toString() === item.optionId
);
if (option.stock < item.qty) {
isAvailable = false;
unavailableProducts.push(
`${item.title} is not available, only ${product.stock} left available`
);
}
}
});
return {
success: isAvailable,
errors: unavailableProducts,
};
});
deductInventory
exports.deductInventory = asyncHandler(async (store, seller, cart) => {
const inventory = await Inventory.findOne({
$and: [
{
store: store,
user: seller,
},
],
});
const products = inventory.products;
cart.forEach((item) => {
const product = products.find(
(product) => product._id.toString() === item.productId
);
if (!item.hasvariation) {
product.stock = product.stock - item.qty;
}
if (item.hasvariation) {
const variation = product.variations.find(
(variation) => variation._id.toString() === item.variationId
);
const option = variation.options.find(
(option) => option._id.toString() === item.optionId
);
option.stock = option.stock - item.qty;
}
});
const saveInventory = await Inventory.findOneAndUpdate(
{
$and: [
{
store: store,
user: seller,
},
],
},
{
$set: { products: products },
},
{ new: true, runValidator: true }
);
if (!saveInventory) {
return {
success: false,
errors: ['Server Error'],
};
}
return {
success: true,
};
});
The problem is that the 2 checkout calls run at (almost) the same time and your routine is not thread-safe. Both calls read a copy of the inventory data in memory. So both calls get a products.stock=10 and based on that local info you check and set the products counter by calculating the new amount in your function (stock-qty) and use an update query to set it as a fixed value (so both calls update the products.stock to 0). Resulting in your concurrency issues.
What you should do is let mongodb handle the concurrency for you.
There are several ways to handle concurrency but you could for example use the $inc to decrease the stock amount directly in mongo. That way the stock amount in the db can never be wrong.
result = await update({stock: {$ge: 10}}, {$inc: {stock : -10}})
As I added a filter to the query the order amount can not be lower than 0 plus you can now check the result of the update call to see if the update modified any documents. If it did not (result.nModified==0) you know the inventory was too low and you can report that back to the user.
https://docs.mongodb.com/manual/reference/operator/update/inc/
https://docs.mongodb.com/manual/reference/method/db.collection.update/#std-label-writeresults-update

Race condition between two request in mongodb

This is eCommerce site and I'm using mongodb as a database, users can place order and each order can have multiple products. Product is a seperate table that contains quantityLeft of each product. There's a situation that when two concurrent requests comes and tries to buy the same product the ordered items in orders table exceeds the available quantity in product table.
Product Table
{
_id: '56e33c56ddec541556a61763',
name: 'Chocolate',
quantityLeft: 1
}
In product's table only 1 chocolate left if one request comes at a time it works fine. Request comes check the order.quantity and handle if there's enough product available.
But when 2 requests comes exactly the same time issue occurs both the request query the database to get the product and check the quantityLeft and found that only 1 chocolate is available and passes the check that enough quantity is still present in inventory and places the order. But in actual 2 orders are placed and quantity we have is only 1.
Order Table
{
_id: '60e33c56ddec541556a61595',
items: [{
_id: '56e33c56ddec541556a61763',
quantity: 1
}]
}
I tried to put both the queries to get the Product detail and place order in same transaction but it doesn't work. Something like this
const session = await mongoose.startSession({ defaultTransactionOptions: { readConcern: { level: 'local' }, writeConcern: { w: 1 } } })
await session.withTransaction(async () => {
const promiseArray = order.items.map((item) => Product.find({ _id: item._id }, { session })
const products = Promise.all(promiseArray)
const productById = {}
products.forEach((product) => {
productById[product._id] = product
})
order.items.forEach((item) => {
if (productById[item].quantityLeft < order.item) {
throw new Error('Not enough quantity')
}
})
await Order.create(order, {session})
}, { readConcern: { level: 'local' }, writeConcern: { w: 1 } });
I'm using nodejs (14.16), mongodb as database npm package is mongoose (5.9).

MongoDB - find one and add a new property

Background: Im developing an app that shows analytics for inventory management.
It gets an office EXCEL file uploaded, and as the file uploads the app convert it to an array of JSONs. Then, it comapers each json object with the objects in the DB, change its quantity according to the XLS file, and add a timestamp to the stamps array which contain the changes in qunatity.
For example:
{"_id":"5c3f531baf4fe3182cf4f1f2",
"sku":123456,
"product_name":"Example",
"product_cost":10,
"product_price":60,
"product_quantity":100,
"Warehouse":4,
"stamps":[]
}
after the XLS upload, lets say we sold 10 units, it should look like that:
{"_id":"5c3f531baf4fe3182cf4f1f2",
"sku":123456,
"product_name":"Example",
"product_cost":10,
"product_price":60,
"product_quantity":90,
"Warehouse":4,
"stamps":[{"1548147562": -10}]
}
Right now i cant find the right commands for mongoDB to do it, Im developing in Node.js and Angular, Would love to read some ideas.
for (let i = 0; i < products.length; i++) {
ProductsDatabase.findOneAndUpdate(
{"_id": products[i]['id']},
//CHANGE QUANTITY AND ADD A STAMP
...
}
You would need two operations here. The first will be to get an array of documents from the db that match the ones in the JSON array. From the list you compare the 'product_quantity' keys and if there is a change, create a new array of objects with the product id and change in quantity.
The second operation will be an update which uses this new array with the change in quantity for each matching product.
Armed with this new array of updated product properties, it would be ideal to use a bulk update for this as looping through the list and sending
each update request to the server can be computationally costly.
Consider using the bulkWrite method which is on the model. This accepts an array of write operations and executes each of them of which a typical update operation
for your use case would have the following structure
{ updateOne :
{
"filter" : <document>,
"update" : <document>,
"upsert" : <boolean>,
"collation": <document>,
"arrayFilters": [ <filterdocument1>, ... ]
}
}
So your operations would follow this pattern:
(async () => {
let bulkOperations = []
const ids = products.map(({ id }) => id)
const matchedProducts = await ProductDatabase.find({
'_id': { '$in': ids }
}).lean().exec()
for(let product in products) {
const [matchedProduct, ...rest] = matchedProducts.filter(p => p._id === product.id)
const { _id, product_quantity } = matchedProduct
const changeInQuantity = product.product_quantity - product_quantity
if (changeInQuantity !== 0) {
const stamps = { [(new Date()).getTime()] : changeInQuantity }
bulkOperations.push({
'updateOne': {
'filter': { _id },
'update': {
'$inc': { 'product_quantity': changeInQuantity },
'$push': { stamps }
}
}
})
}
}
const bulkResult = await ProductDatabase.bulkWrite(bulkOperations)
console.log(bulkResult)
})()
You can use mongoose's findOneAndUpdate to update the existing value of a document.
"use strict";
const ids = products.map(x => x._id);
let operations = products.map(xlProductData => {
return ProductsDatabase.find({
_id: {
$in: ids
}
}).then(products => {
return products.map(productData => {
return ProductsDatabase.findOneAndUpdate({
_id: xlProductData.id // or product._id
}, {
sku: xlProductData.sku,
product_name: xlProductData.product_name,
product_cost: xlProductData.product_cost,
product_price: xlProductData.product_price,
Warehouse: xlProductData.Warehouse,
product_quantity: productData.product_quantity - xlProductData.product_quantity,
$push: {
stamps: {
[new Date().getTime()]: -1 * xlProductData.product_quantity
}
},
updated_at: new Date()
}, {
upsert: false,
returnNewDocument: true
});
});
});
});
Promise.all(operations).then(() => {
console.log('All good');
}).catch(err => {
console.log('err ', err);
});

Synchronized update of documents using Mongoose

I have a nodejs API server deployed in a Kubernetes cluster.
Users can send in bids on auction items.
To prevent a bid from overriding another there needs to be some synchronization.
I am seeing the following for an incoming bid:
start a transaction that reads the current bid and compares it to the incoming bid and updates the record
create an aggregation that does the same as above
I don't know which way to go. I also understand that you need to lock the document with either IX or X.
For a RDBMS you would create a transaction that locks the record and releases it after update but I don't know how it works for MongoDB.
Product.findById(productId)
.then(productmatch => {
if (productmatch.currentPrice > price) throw Error('no go')
const bid = new Bid({
price,
date: Date.now(),
user: user._id
})
return Product.findByIdAndUpdate(productId, {
$push: {
bids: bid
},
currentPrice: price,
currentUser: user,
currentBid: bid
}, {
new: true
})
.then(product => {
if (!product) throw Error(`no go`)
return bid._id.toString()
})
})
After a little more research I came up with this solution, however I do not know if its 100% reliable but I believe this method will lock the document and not let any other threads read the document between the query and update operations.
var query = {
_id: productId,
closed: false,
currentPrice: {
$lt: price
}
},
update = {
$push: {
bids: bid
},
currentPrice: price,
currentUser: user,
currentBid: bid
},
options = {
new: true
};
return Product.findOneAndUpdate(query, update, options)
.then(product => {
if (!product) throw Error(`no product found with id ${productId}`)
return bid._id.toString()
})

Resources