So, I'm building an Ecommerce Api with Koajs and Koa router, and i'm now handling orders payment request, the thing is, before the actual paying, the status of my order model change to 'Waiting for payment', and it is at this point when my API changes the stock of my products.
What I thought is that when this happen, for my response to be faster, the product update could be done asynchronously, as maybe the request may need to process a lot of products updates, and the client can recieve a response and then just focus on paying while the products are updating. But, what if two people make the a request with the same product and while updating it enters in a conflict as the sum of two quantities exceeds the actual product stock? (the shoppingCart checks not to surpass the limits of product stock but once this is done it cant be changed)
For this may be better to just make the client wait and do the process with await? or there is a way that can handle this type of problems better?
My code looks somewhat like this:
router.post('api.orders.payment', '/:id/pay', async (ctx) => {
const order = await ctx.orm.order.findByPk(ctx.params.id);
order.status = 'Waiting for payment';
await order.save({ fields: ['status', 'total'] });
// Update all the products
updateProducts(ctx, order);
ctx.body = ctx.jsonSerializer('order', {
attributes: ['status', 'total'],
topLevelLinks: {
self: `${ctx.origin}${ctx.router.url('api.orders.show', { id: order.id })}`,
},
}).serialize(order);
And the updateProduct function looks like this:
async function updateProducts(ctx, order) {
const shoppingCart = await ctx.orm.orderProduct.findAll({ where: { orderId: order.id } });
let product = null;
shoppingCart.forEach(async (element) => {
product = await ctx.orm.product.findByPk(element.productId);
product.stock -= element.quantity;
await product.save({ fields: ['stock'] });
});
}
I'm using Sequelize, Koa, Koa-router, jsonserializer, and my product model has attributes "id, stock", my order model has attributes "id", "total", and OrderProduct that represents my cart model has attributes "orderId", "productId" and "quantity".
Related
I have users and companies and want to store a company for each user and all of the users of each company in Firebase.
user={
"id":"tjkdEnc3skdm2Jjknd"
"name":"Adam",
"street":"Sideway 4",
"company":"dHend4sdkn25"
}
companies={
"id":"dHend4sdkn25",
"name":"Comp Ltd.",
"members":[
{
"id":"tjkdEnc3skdm2Jjknd"
"name":"Adam"
},{
"id":"dfjnUkJKB3sdn8n2kj"
"name":"Berta"
}
]
}
All explanations say that duplicate data is the best way to deal with and so I want to write some cloud functions to keep thigs in sync when editing on one of the sides.
Basically I started with
exports.userChangedCompany = functions.firestore
.document('users/{userId}')
.onUpdate((change, context) => {
const data = change.after.data();
const previousData = change.before.data();
if (data.company == previousData.company) {
return null;
}
else{
admin.firestore().doc('companies/'+data.company).set({ ... });
}
});
to update the companies when a user changed the company. Unfortunately I haven't found any hint how to set the new company-data properly.
Can someone please help me?
It sounds like you just need to remove user from members array of old company and add in that array of new company. You just need IDs of both companies.
async function updateCompanies(userId, username, oldCompanyId, newCompanyId) {
const companiesRef = await admin.firestore().collection("companies")
const userObj = {id: userId, name: username}
// Removing from old company and adding in new company
await Promise.all([
companiesRef.doc(oldCompanyId).update({members: admin.firestore.FieldValue.arrayRemove(userObj)}),
companiesRef.doc(newCompanyId).update({members: admin.firestore.FieldValue.arrayUnion(userObj)})
])
return true
}
You can just call this function in your cloud function. Just make sure you pass correct params. The reason why you need to pass the username as well is you have array of objects (members) and hence you need the complete object to add/remove using arrayUnion/arrayRemove.
I am trying to construct a query to return data from multiple tables and build them into a single array of objects to return to the client. I have two tables, incidents and sources. Each source has an incident_id that corresponds to an incident in the first table.
Since there can be more than one source I want to query for the incidents, then on each incident add a key of source that has a value of the array of associated sources. The desired final structure is this:
{
"incident_id": 1,
"id": "wa-olympia-1",
"city": "Olympia",
"state": "Washington",
"lat": 47.0417,
"long": -122.896,
"title": "Police respond to broken windows with excessive force",
"desc": "Footage shows a few individuals break off from a protest to smash City Hall windows. Protesters shout at vandals to stop.\n\nPolice then arrive. They arrest multiple individuals near the City Hall windows, including one individual who appeared to approach the vandals in an effort to defuse the situation.\n\nPolice fire tear gas and riot rounds at protesters during the arrests. Protesters become agitated.\n\nAfter police walk arrestee away, protesters continue to shout at police. Police respond with a second bout of tear gas and riot rounds.\n\nA racial slur can be heard shouted, although it is unsure who is shouting.",
"date": "2020-05-31T05:00:00.000Z",
"src": ['http://google.com']
}
Here is the route as it stands:
router.get('/showallincidents', (req, res) => {
Incidents.getAllIncidents()
.then((response) => {
const incidents = response.map((incident) => {
const sources = Incidents.createSourcesArray(incident.incident_id);
return {
...incident,
src: sources,
};
});
res.json(incidents);
})
.catch((err) => {
res.status(500).json({ message: 'Request Error' });
});
});
Here are the models I currently have:
async function getAllIncidents() {
return await db('incidents');
}
async function createSourcesArray(incident_id) {
const sources = await db('sources')
.select('*')
.where('sources.incident_id', incident_id);
return sources;
}
When this endpoint is hit I get a "too many connections" error. Please advise.
I found a solution. What I decided to do was to query the two tables independently. Then I looped through the first result array, and within that loop looped through the second array, checking for the foreign key they share, then when I found a match, added those results to an array on the original object, then returned a new array with the objects of the first with the associated data from the second. Models are unchanged, here is the updated route.
router.get('/showallincidents', async(req, res) => {
try {
const incidents = await Incidents.getAllIncidents();
const sources = await Incidents.getAllSources();
const responseArray = []
// Reconstructs the incident object with it's sources to send to front end
incidents.forEach((incident) => {
incident['src'] = []
sources.forEach(source => {
if (source.incident_id === incident.incident_id) {
incident.src.push(source)
}
})
responseArray.push(incident)
})
res.json(responseArray)
} catch (e) {
res.status(500).json({
message: 'Request Error'
});
}
});
Are the two tables in the same database? If that is the case, it is much more efficent to do the primary/foreign key match by doing an SQL join. What you have implemented is "nested loop join" which might not be the optimal way to match depending on the value distribution of the primary key. You can search SQL join algorithms to see examples and pro/cons.
If the tables are in different databases then indeed client side join is likely your only option. Though again if you know something about the underlying distribution it might be better to do a hash join
I have a table called HOUSE. And it has a column named STATUS.
I also have a table called TASK and it also has a column named STATUS.
Each house has many tasks. And if there's one task that has a status of inProgress, the house status shall be inProgress. And if all of the tasks are done, then house is done.
I want this status column of the house be dependent on the status of its all tasks.
When I call /getHouses, here's what I do to add a property called status to each house object, because currently I have no STATUS column in the HOUSE table.
exports.getMyHouses = (req, res) => {
const page = myUtil.parser.tryParseInt(req.query.page, 0)
const limit = myUtil.parser.tryParseInt(req.query.limit, 10)
db.House.findAndCountAll({
where: { userId: req.user.id },
include: [
{
model: db.Task,
as: "task",
include: [
{
model: db.Photo,
as: "photos"
}
]
},
{
model: db.Address,
as: "address"
}
],
offset: limit * page,
limit: limit,
order: [["id", "ASC"]],
})
.then(data => {
let newData = JSON.parse(JSON.stringify(data))
const houses = newData.rows
for (let house of houses) {
house.status = "done"
const tasks = house.task
for (let task of tasks) {
if (task.status == "inProgress") {
house.status = "inProgress"
break
}
}
}
res.json(myUtil.response.paging(newData, page, limit))
})
.catch(err => {
console.log("Error get houses: " + err.message)
res.status(500).send({
message: "An error has occured while retrieving data."
})
})
}
EDIT: I just realized that perhaps I can update the house's status column each time there's an update in the task's status. I've never thought about this before.
But I would still love it if anyone could confirm that this is a good strategy or if there's a better one.
The option you have is viable as long as filtering by the house's status isn't something you require. This would essentially be called a virtual field (since it isn't something directly from the database). If you do need to filter by this field, you'd then need to query for all the tasks InProgress and get the unique house IDs.
You could update the house's status column on task update too but you could run into some race conditions if, for example, multiple requests were being made to update tasks to the same house. Make sure to run a transaction here if you were too. Querying/filtering for houses with InProgress tasks would be much faster since you can query it directly. However, updates would be slower since you'd need to run a task update, a count query on tasks, and an update query on the house.
Both have it's pro's and con's, it mainly depends on your application design's requirement.
I have an API that in order to insert a new item it needs to be validated. The validation basically is a type validator(string, number, Date, e.t.c) and queries the database that checks if the "user" has an "item" in the same date, which if it does the validation is unsuccessful.
Pseudocode goes like this:
const Item = require("./models/item");
function post(newDoc){
let errors = await checkForDocErrors(newDoc)
if (errors) {
throw errors;
}
let itemCreated = await Item.create(newDoc);
return itemCreated;
}
My problem is if I do two concurrent requests like this:
const request = require("superagent");
// Inserts a new Item
request.post('http://127.0.0.1:5000/api/item')
.send({
"id_user": "6c67ea36-5bfd-48ec-af62-cede984dff9d",
"start_date": "2019-04-02",
"name": "Water Bottle"
})
/*
Inserts a new Item, which shouldn't do. Resulting in two items having the
same date.
*/
request.post('http://127.0.0.1:5000/api/item')
.send({
"id_user": "6c67ea36-5bfd-48ec-af62-cede984dff9d",
"start_date": "2019-04-02",
"name": "Toothpick"
})
Both will be successful, which it shouldn't be since an "user" cannot have two "items" in the same date.
If I execute the second one after the first is finished, everything works as expected.
request.post('http://127.0.0.1:5000/api/item') // Inserts a new Item
.send({
"id_user": "6c67ea36-5bfd-48ec-af62-cede984dff9d",
"start_date": "2019-04-02",
"name": "Water Bottle"
})
.then((res) => {
// It is not successful since there is already an item with that date
// as expected
request.post('http://127.0.0.1:5000/api/item')
.send({
"id_user": "6c67ea36-5bfd-48ec-af62-cede984dff9d",
"start_date": "2019-04-02",
"name": "Toothpick"
})
})
To avoid this I send one request with an array of documents, but I want to prevent this issue or at least make less likely to happen.
SOLUTION
I created a redis server. Used the package redis-lock and wrapped around the POST route.
var client = require("redis").createClient()
var lock = require("redis-lock")(client);
var itemController = require('./controllers/item');
router.post('/', function(req, res){
let userId = "";
if (typeof req.body === 'object' && typeof req.body.id_user === 'string') {
userId = req.body.id_user;
}
lock('POST ' + req.path + userId, async function(done){
try {
let result = await itemController.post(req.body)
res.json(result);
} catch (e) {
res.status(500).send("Server Error");
}
done()
})
}
Thank you.
Explain
That is a race condition.
two or more threads can access shared data and they try to change it at the same time
What is a race condition?
Solution:
There are many ways to prevent conflict data in this case, a lock is 1 option.
You can lock on application level or database level... but I prefer you read this thread before chose any of them.
Optimistic vs. Pessimistic locking
Quick solution: pessimistic-lock https://www.npmjs.com/package/redis-lock
You should create a composite index or a composite primary key that includes the id_user and the start_date fields. This will ensure that no documents for the same user with the same date can be created, and the database will throw an error if you'll try to do it.
Composite index with mongoose
You could also use transactions. To do it, you should execute the find and the create methods inside a transaction, to ensure that no concurrent queries on the same document will be executed.
Mongoose transactions tutorial
More infos
I would go with an unique composite index, that in your specific case should be something like
mySchema.index({user_id: 1, start_date: 1}, {unique: true});
I am attempting to create an instance and multiple related instances with a many to many relation using a junction table.
While creating the multiple related instances, I need to add a value to a property on the junction table as well. I don't know if it is my lack of knowledge of sequelize or promises that is causing my problem.
The code I am using is below. This code does add the items to the database, but I need to redirect after the operation has completed, which is not working.
Basically, I need to create a Recipe. Once that is created, I need to create Ingredients and relate them to that Recipe. The ingredients are stored in an array coming from a form on an HTML page. While relating the Ingredients, I need to add the ingredient_quantity to the RecipeIngredients table, which is the through part of the relationship (the junction table).
global.db.Recipe.belongsToMany(
global.db.Ingredient,
{
as: 'Ingredients',
through: global.db.RecipeIngredients,
foreignKey: 'recipe_id'
});
global.db.Ingredient.belongsToMany(
global.db.Recipe,
{
as: 'Recipes',
through: global.db.RecipeIngredients,
foreignKey: 'ingredient_id'
});
router.post('/new', ensureLoggedIn, bodyParser.json(), function (req, res) {
var recipeName = req.body.recipe_name;
var steps = req.body.steps;
var ingredients = req.body.ingredients;
var ingredientQty = {};
var currentIngredient;
var ingredientsToAdd = [];
db.Recipe.create({
recipe_name: recipeName,
directions: steps,
FamilyId: req.user.FamilyId,
CreatedBy: req.user._id
})
.then(function (recipe) {
for (var i = 0; i < ingredients.length; i++) {
currentIngredient = ingredients[i];
ingredientQty[currentIngredient.ingredient_name] =
currentIngredient.quantity;
db.Ingredient.findOrCreate({
where: {
ingredient_name: currentIngredient.ingredient_name,
FamilyId: req.user.FamilyId
}
})
.spread(function (ingredient, created) {
if (created) {
console.log("Added Ingredient to DB: " +
currentIngredient.ingredient_name);
}
ingredient.Recipes = {
ingredient_quantity:
ingredientQty[ingredient.ingredient_name]
};
ingredient.CreatedBy = req.user._id;
recipe.addIngredient(ingredient)
.then(function () {
console.log("Added Ingredient " + ingredient.ingredient_name
+ " to Recipe " + recipe.recipe_name);
});
})
}
})
.finally(function(recipe){
res.redirect('/recipes');
});
});
Any help would be greatly appreciated. I know that I am running into issues because of trying to use promises inside of a loop, I just don't know how else I can accomplish this.
Using sequelize, you can create objects along with its associated objects in one step, provided all objects that you're creating are new. This is also called nested creation. See this link and scroll down to section titled "Creating with associations"
Coming to your issue, you've a many-to-many relationship between Recipe and Ingredient, with RecipeIngredients being the join table.
Suppose you've a new Recipe object which you want to create, like:
var myRecipe = {
recipe_name: 'MyRecipe',
directions: 'Easy peasy',
FamilyId: 'someId',
CreatedBy: 'someUserId'
}
And an array of Ingredient objects, like:
var myRecipeIngredients = [
{ ingredient_name: 'ABC', FamilyId: 'someId'},
{ ingredient_name: 'DEF', FamilyId: 'someId'},
{ ingredient_name: 'GHI', FamilyId: 'someId'}]
// associate the 2 to create in 1 step
myRecipe.Ingredients = myRecipeIngredients;
Now, you can create myRecipe and its associated myRecipeIngredients in one step as shown below:
Recipe.create(myRecipe, {include: {model: Ingredient}})
.then(function(createdObjects){
res.json(createdObjects);
})
.catch(function(err){
next(err);
});
And that is all !!
Sequelize will create 1 row in Recipe, 3 rows in Ingredient and 3 rows in RecipeIngredients to associate them.
I was able to fix the problem that I was having. The answers here helped me come up with my solution.
I am posting the solution below in case anyone else runs into a similar issue. I created a variable to store the Promise from Recipe.create(), I used Promise.map to findOrCreate all of the ingredients from the form data. Because findOrCreate returns an array containing Promise and a boolean for if the item was created, I then had to get the actual ingredients out of the results of the Promise.map function. So I used the JavaScript array.map() function to get the first item from the arrays. And finally, use Promise.map again to add each Ingredient to the Recipe
var ingredients = req.body.ingredients,
recipeName = req.body.recipeName,
ingredientsQty = {}; // Used to map the ingredient and quantity for the
// relationship, because of the Junction Table
var recipe = models.Recipe.create({recipe_name: recipeName});
// Use Promise.map to findOrCreate all ingredients from the form data
Promise.map(ingredients, function(ing){
ingredientsQty[ing.ingredient_name] = ing.ingredient_quantity;
return models.Ingredient.findOrCreate({ where: { ingredient_name: ing.ingredient_name}});
})
// Use JavaScript's Array.prototype.map function to return the ingredient
// instance from the array returned by findOrCreate
.then(function(results){
return results.map(function(result){
return result[0];
});
})
// Return the promises for the new Recipe and Ingredients
.then(function(ingredientsInDB){
return Promise.all([recipe, ingredientsInDB]);
})
// Now I can use Promise.map again to create the relationship between the /
// Recipe and each Ingredient
.spread(function(addedRecipe, ingredientsToAdd){
recipe = addedRecipe;
return Promise.map(ingredientsToAdd, function(ingredientToAdd){
ingredientToAdd.RecipeIngredients = {
ingredient_quantity: ingredientsQty[ingredientToAdd.ingredient_name]
};
return recipe.addIngredient(ingredientToAdd);
});
})
// And here, everything is finished
.then(function(recipeWithIngredients){
res.end
});