Tables:
users
user_category
categories
user_id (PK)
user_id (PK)
category_id (PK)
name
category_id (PK)
name
I'm having problems with concurrency on node, I'll describe an example to reproduce the error I have:
When receiving 5 requests in parallel to perform a PUT on the same existing user and their categories:
PUT users/userId123
body
{
"name": "other name"
"categories" : [
"categoryId123",
"categoryId456",
"categoryId789",
]
}
Upon receiving this payload in a simulation, first a DELETE query is performed on user_category and then an INSERT on user_category to insert the new categories.
It turns out that as the node is a single thread, the main thread competes, and when performing 5 identical requests in parallel, it is happening to perform two DELETES of two different requests and then the 2 INSERTs, which will generate a primary key error in the table user_category, I already tried to use a transaction for these two queries, but the error still occurs, blocking the main thread until doing these two operations is not an option defied the loss in performance.
Attempt-1
Below is an excerpt from UserController code that does this
if (Array.isArray(user.categories)) {
await userCategoriesRepository.deleteHardByUserId(user.userId)
const userCategories = []
for (const categoryId of user.categories) {
userCategories.push({userId: user.userId, categoryId})
}
await userCategoriesRepository.save(userCategories)
}
UserRepository:
save (userCategories: ICreateUserCategoryDTO[]): Promise<void> {
try {
return this.repository.insert(userCategories)
} catch (error) {
throw new Error(error)
}
}
deleteHardByUserId (userId: string): Promise<void> {
try {
return this.repository.delete({ userId })
} catch (error) {
throw new Error(error)
}
}
Attempt-2 (Transaction)
Below is an excerpt from UserController code that does this
if (Array.isArray(user.categories)) {
const userCategories = []
for (const categoryId of user.categories) {
userCategories.push({userId: user.userId, categoryId})
}
await userCategoriesRepository.updateWithTransaction(userId, userCategories)
}
UserRepository:
async updateWithTransaction (userId: string, userCategories: ICreateUserCategoryDTO[]) {
try {
return getManager().transaction(async transaction => {
await transaction.delete(UserCategory, { userId })
await transaction.insert(UserCategory, userCategories)
})
} catch (error) {
throw new Error(error)
}
}
QUERY LOGGING
case: 1 request (SUCESS)
query: START TRANSACTION // REQUEST 1
query: DELETE FROM "user_category" WHERE "user_id" = $1 -- PARAMETERS: ["userId123"] // REQUEST 1
query: INSERT INTO "user_category"("user_id", "category_id") VALUES ($1, $2) -- PARAMETERS: ["userId123", "categoryId123"] // REQUEST 1
query: COMMIT // REQUEST 1
case: 2 or more requests parallel (ERROR PK)
query: START TRANSACTION // REQUEST 1
query: DELETE FROM "user_category" WHERE "user_id" = $1 -- PARAMETERS: ["userId123"] // REQUEST 1
query: START TRANSACTION // REQUEST 2
query: DELETE FROM "user_category" WHERE "user_id" = $1 -- PARAMETERS: ["userId123"] // REQUEST 2
query: INSERT INTO "user_category"("user_id", "category_id") VALUES ($1, $2) -- PARAMETERS: ["userId123", "categoryId123"] // REQUEST 1
query: COMMIT // REQUEST 1
query: INSERT INTO "user_category"("user_id", "category_id") VALUES ($1, $2), ($3, $4) -- PARAMETERS: ["userId123", "categoryId123", "userId123", "categoryId456"] // REQUEST 2 --> PK ERROR
query: COMMIT // REQUEST 2
Details:
node 16.13.1
typeorm 4.3.2
pg 8.4.0
npm 8.3.0
Postgres database 13.2
Related
The basic question, is why is it changing the property job.companyHandle to job.companyhandle?
In my Job model, I have this function:
static async create({ title, salary, equity, companyHandle }) {
const duplicateCheck = await db.query(
`SELECT title
FROM jobs
WHERE title=$1 AND company_handle=$2`,
[title, companyHandle]);
if (duplicateCheck.rows[0])
throw new BadRequestError(`Duplicate job: ${title} at ${companyHandle}`);
const result = await db.query(
`INSERT INTO jobs
(title, salary, equity, company_handle)
VALUES ($1, $2, $3, $4)
RETURNING id, title, salary, equity, company_handle AS companyHandle`,
[
title,
salary,
equity,
companyHandle
],
);
const job = result.rows[0];
return job;
}
My jest testing code for that function looks like this:
describe("create", function () {
const newJob = {
title: "NJ1",
salary: 50000,
equity: "0.45",
companyHandle: "c3"
};
test("works", async function () {
let job = await Job.create(newJob);
expect(job).toEqual({ ...newJob, id: expect.any(Number) });
const result = await db.query(
`SELECT id, title, salary, equity, company_handle AS companyHandle
FROM jobs
WHERE title = 'NJ1'`);
expect(result.rows).toEqual([
{
id: expect.any(Number),
title: "NJ1",
salary: 50000,
equity: "0.45",
companyHandle: "c3"
},
]);
});
... another test ...
});
So I've tried removing the AS companyHandle from the return statement and adding
job.companyHandle = job.company_handle;
delete job.company_handle;
into the model right before the return. In some of the other tests, this returns the proper companyHandle property, but in the first test case, it's still coming back as companyhandle. I don't understand why it's returning a lowercase.
A quick view of the jest results:
create › works
expect(received).toEqual(expected) // deep equality
- Expected - 1
+ Received + 1
## -1,8 +1,8 ##
Array [
Object {
- "companyHandle": "c3",
+ "companyhandle": "c3",
"equity": "0.45",
"id": Any<Number>,
"salary": 50000,
"title": "NJ1",
},
Also, I know a bunch of this code probably isn't necessary, but I've seen too many posts without enough context, so I decided to err on the side of over sharing.
As was mentioned in the comments, I made the mistake of not double quoting my identifiers.
I was too focused in on one case, and missed all of the other returns where I had not double quoted.
Thank you #Crontab and #Adrian Klaver
Below iam calling addUpdateDailyLeads with an array like
[{
"yyyymmdd": "20191124",
"admin_login":"rasheed.s",
"category":"PO",
"amount":10,
"office_id":10000,
"new_leads_attempted":10
},
{
"yyyymmdd": "20191124",
"admin_login":"rasheed.s",
"category":"PO",
"amount":10,
"office_id":10000,
"new_leads_attempted":10
},
{
"yyyymmdd": "20191125",
"admin_login":"prajeed.av",
"category":"FHL",
"amount":10,
"office_id":10000,
"new_leads_attempted":10
}
]
So,key 0 should insert,
key 1 should update because duplicate key constratint,
key 2 will insert,
but im getting duplicate key constraint error on key 1,because array map not waiting for the query to be executed .
const addUpdateDailyLeads = async (req, res) => {
let admin_login,category,office_id,new_leads_attempted,yyyymmdd,where,values;
let data = req.body;
req.body.map(async function(item,i){
admin_login = item.admin_login,
category = item.category,
office_id = item.office_id,
new_leads_attempted = item.new_leads_attempted,
yyyymmdd = item.yyyymmdd;
where = {yyyymmdd:yyyymmdd, admin_login:admin_login, category:category};
values = {yyyymmdd:yyyymmdd, admin_login:admin_login, category:category,office_id:office_id,new_leads_attempted:new_leads_attempted,update_date:moment().format('YYYYMMDDHHmmss')};
console.log("calling ",i);
let chck = await addUpdateDailyLeadsCollection({where:where,values:values})
console.log("")
console.log("called")
})
res.json({ code: '200', message: `Advisor Daily Leads Updated ${admin_login}` });
}
const addUpdateDailyLeadsCollection = async data => {
let transaction;
let where = data.where
let values = data.values
var Sequelize = require("sequelize");
console.log("startef 1");
await AdvisorLeads.findOne({ where: where }, { useMaster: true }).then( async(data)=>{
console.log("waited");
if(data){
await data.update({new_leads_attempted: Sequelize.literal('new_leads_attempted + '+values.new_leads_attempted)}).then(data=>{
console.log("updated")
return Promise.resolve(1);
})
}else{
AdvisorLeads.create(values).then(data=>{
console.log("inserted")
return Promise.resolve(1);
})
}
})
};
final output on console
calling 0
startef 1
waiting 1
calling 1
startef 1
waiting 1
calling 2
startef 1
waiting 1
waited
waited
waited
called
called
called
inserted
inserted
My expected output like
calling 0
startef 1
waiting 1
waited
inserted
called
calling 1
startef 1
waiting 1
waited
updated
called
calling 2
startef 1
waiting 1
waited
inserted
called
Finally whait i need is to wait for each item ,execute all queries and then process next item
I think you can solve by using await on the update and create statements....
But also take a look at the UPSERT method, which could simplify your code quite a bit. From
The Sequelize API Reference: "Insert or update a single row. An update will be executed if a row which matches the supplied values on either the primary key or a unique key is found."
Addendum: for synchronizing async/await, there are many ways to do this, as detailed in this post. Here's some code I set up following the ES7 method:
let params = [{id : 1, sal : 10}, {id : 44, sal: 30}, {id : 1, sal : 20}];
async function doUpsertArrayInSequence(myParams) {
let results = [];
for (let i = 0; i < myParams.length; i++) {
let x = await User.findByPk(myParams[i].id).then(async (u) => {
if (u != null) {
await u.update({ sal : u.sal + myParams[i].sal});
} else {
await User.create({id: myParams[i].id, sal: myParams[i].sal});
}
});
results.push(x);
}
return results;
}
await doUpsertArrayInSequence(params).then(function(result) {
User.findAll().then(proj => {
res.send(proj);
next();
});
})
From the log, I can see
a) a SELECT, followed by an UPDATE or INSERT for each row (in sequence).
b) the 2nd occurrence of id=1 reflects the update of the 1st occurrence.
c) the final findAll reflects all inserts and updates.
HTH
I have an API that in order to insert a new item it needs to be validated. The validation basically is a type validator(string, number, Date, e.t.c) and queries the database that checks if the "user" has an "item" in the same date, which if it does the validation is unsuccessful.
Pseudocode goes like this:
const Item = require("./models/item");
function post(newDoc){
let errors = await checkForDocErrors(newDoc)
if (errors) {
throw errors;
}
let itemCreated = await Item.create(newDoc);
return itemCreated;
}
My problem is if I do two concurrent requests like this:
const request = require("superagent");
// Inserts a new Item
request.post('http://127.0.0.1:5000/api/item')
.send({
"id_user": "6c67ea36-5bfd-48ec-af62-cede984dff9d",
"start_date": "2019-04-02",
"name": "Water Bottle"
})
/*
Inserts a new Item, which shouldn't do. Resulting in two items having the
same date.
*/
request.post('http://127.0.0.1:5000/api/item')
.send({
"id_user": "6c67ea36-5bfd-48ec-af62-cede984dff9d",
"start_date": "2019-04-02",
"name": "Toothpick"
})
Both will be successful, which it shouldn't be since an "user" cannot have two "items" in the same date.
If I execute the second one after the first is finished, everything works as expected.
request.post('http://127.0.0.1:5000/api/item') // Inserts a new Item
.send({
"id_user": "6c67ea36-5bfd-48ec-af62-cede984dff9d",
"start_date": "2019-04-02",
"name": "Water Bottle"
})
.then((res) => {
// It is not successful since there is already an item with that date
// as expected
request.post('http://127.0.0.1:5000/api/item')
.send({
"id_user": "6c67ea36-5bfd-48ec-af62-cede984dff9d",
"start_date": "2019-04-02",
"name": "Toothpick"
})
})
To avoid this I send one request with an array of documents, but I want to prevent this issue or at least make less likely to happen.
SOLUTION
I created a redis server. Used the package redis-lock and wrapped around the POST route.
var client = require("redis").createClient()
var lock = require("redis-lock")(client);
var itemController = require('./controllers/item');
router.post('/', function(req, res){
let userId = "";
if (typeof req.body === 'object' && typeof req.body.id_user === 'string') {
userId = req.body.id_user;
}
lock('POST ' + req.path + userId, async function(done){
try {
let result = await itemController.post(req.body)
res.json(result);
} catch (e) {
res.status(500).send("Server Error");
}
done()
})
}
Thank you.
Explain
That is a race condition.
two or more threads can access shared data and they try to change it at the same time
What is a race condition?
Solution:
There are many ways to prevent conflict data in this case, a lock is 1 option.
You can lock on application level or database level... but I prefer you read this thread before chose any of them.
Optimistic vs. Pessimistic locking
Quick solution: pessimistic-lock https://www.npmjs.com/package/redis-lock
You should create a composite index or a composite primary key that includes the id_user and the start_date fields. This will ensure that no documents for the same user with the same date can be created, and the database will throw an error if you'll try to do it.
Composite index with mongoose
You could also use transactions. To do it, you should execute the find and the create methods inside a transaction, to ensure that no concurrent queries on the same document will be executed.
Mongoose transactions tutorial
More infos
I would go with an unique composite index, that in your specific case should be something like
mySchema.index({user_id: 1, start_date: 1}, {unique: true});
I'm new to node(express) and pg-promise, and have not been able to figure out how to add the result of each nested query(loop) into the main json array result query.
I have two tables: Posts and comments.
CREATE TABLE post(
id serial,
content text not null,
linkExterno text,
usuario VARCHAR(50) NOT NULL REFERENCES usuarios(alias) ON UPDATE cascade ON DELETE cascade,
multimedia text,
ubicacation VARCHAR(100),
likes integer default 0,
time VARCHAR default now(),
reported boolean default false,
PRIMARY KEY (id) );
CREATE TABLE comment(
id serial,
idPost integer NOT NULL REFERENCES post(id) ON UPDATE cascade ON DELETE cascade,
acount VARCHAR(50) NOT NULL REFERENCES users(alias) ON UPDATE cascade ON DELETE cascade,
content text NOT NULL,
date date default now(),
PRIMARY KEY (id));
So I want to add the result of each comments to each post and return the posts.
I have this, but doesn't work:
con.task(t => {
return t.any('select *, avatar from post, users where user= $1 and user = alias ORDER BY time DESC LIMIT 10 OFFSET $2', [username, pos])
.then(posts => {
if(posts.length > 0){
for (var post of posts){
post.coments = t.any('select * from comment where idPost = $1 ', post.id);
}
}
});
}).then(posts => {
res.send(posts);
}).catch(error => {
console.log(error);
});
Any suggestions?
PD: I think my question is kind of similar to this one:
get JOIN table as array of results with PostgreSQL/NodeJS
ANSWERS:
Option 1 (best choice):
Making a single query through JSON to psql (JSON query)
See answer by #vitaly-t
OR
Getting the nested data asynchronously using ajax.
Option 2:
function buildTree(t) {
return t.map("select *, avatar from publicacion, usuarios where usuario = $1 and usuario = alias ORDER BY hora DESC LIMIT 10 OFFSET $2", [username, cantidad], posts => {
return t.any('select * from comentario where idPublicacion = $1', posts.id)
.then(coments => {
posts.coments = coments;
console.log(posts.coments);
return posts;
});
}).then(t.batch); // settles the array of generated promises
}
router.get('/publicaciones', function (req, res) {
cantidad = req.query.cantidad || 0; //num de publicaciones que hay
username = req.session.user.alias;
con.task(buildTree)
.then(data => {
res.send(data);
})
.catch(error => {
console.log(error);
});
});
Option 3(async):
try{
var posts = await con.any('select *, avatar from post, users where user = $1 and user = alias ORDER BY time DESC LIMIT 10 OFFSET $2', [username, q])
for (var post of posts){
post.coments = await con.any('select * from comment where idPublictcion = $1', post.id);
}
}catch(e){
console.log(e);
}
I'm the author of pg-promise ;)
con.task(t => {
const a = post => t.any('SELECT * FROM comment WHERE idPost = $1', post.id)
.then(comments => {
post.comments = comments;
return post;
});
return t.map('SELECT *, avatar FROM post, users WHERE user = $1 AND user = alias ORDER BY time DESC LIMIT 10 OFFSET $2', [username, pos], a)
.then(t.batch);
})
.then(posts => {
res.send(posts);
})
.catch(error => {
console.log(error);
});
Also see this question: get JOIN table as array of results with PostgreSQL/NodeJS.
UPDATE
In case you do not want to go all the way with the JSON query approach, then the following will scale much better than the original solution, as we concatenate all child queries, and then execute them as one query:
con.task(async t => {
const posts = await t.any('SELECT *, avatar FROM post, users WHERE user = $1 AND user = alias ORDER BY time DESC LIMIT 10 OFFSET $2', [username, pos]);
const a = post => ({query: 'SELECT * FROM comment WHERE idPost = ${id}', values: post});
const queries = pgp.helpers.concat(posts.map(a));
await t.multi(queries)
.then(comments => {
posts.forEach((p, index) => {
p.comments = comments[index];
});
});
return posts;
})
.then(posts => {
res.send(posts);
})
.catch(error => {
console.log(error);
});
See API:
helpers.concat
Database.multi
If you want structured (nested) data, without having to
A) re-write your sql using json function, or split it out into multiple task queries, or
B) refactor your code to use the API of a heavy ORM
you could check out sql-toolkit. It's a node library built for pg-promise which allows you to write regular native SQL and receive back properly structured (nested) pure business objects. It's strictly an enhancement toolkit on top of pg-promise, and does not seek to abstract out pg-promise (you still set up pg-promise and can use it directly).
For example:
class Article extends BaseDAO {
getBySlug(slug) {
const query = `
SELECT
${Article.getSQLSelectClause()},
${Person.getSQLSelectClause()},
${ArticleTag.getSQLSelectClause()},
${Tag.getSQLSelectClause()}
FROM article
JOIN person
ON article.author_id = person.id
LEFT JOIN article_tags
ON article.id = article_tags.article_id
LEFT JOIN tag
ON article_tags.tag_id = tag.id
WHERE article.slug = $(slug);
`;
return this.one(query, { slug });
// OUTPUT: Article {person: Person, tags: Tags[Tag, Tag, Tag]}
}
The select clause uses the business object "getSQLSelectClause" methods to save tedium in typing the columns, as well as ensure no collisions of names (nothing magical going on, and could just be written out instead).
The this.one is a call into sql-toolkits base DAO class. It is responsible for structuring the flat result records into a nice nested structure.
(Also notice that it is "one" which matches our mental model for the SQL. The DAO methods for one, oneOrNone, many, and any ensure their count against the number of generated top level business objects - not the number of rows the sql expression returns!)
Check out the repository for details on how to set it up on top of pg-promise. (Disclamer, I am the author of sql-toolkit.)
You can use await but it will work sync.
return t.any('select *, avatar from post, users where user= $1 and user = alias ORDER BY time DESC LIMIT 10 OFFSET $2', [username, pos])
.then(posts => {
if(posts.length > 0){
for (var post of posts){
post.coments = await t.any('select * from comment where idPost = ', post.id);
}
}
return posts;
});
Actually i recommend you to use orm tools like bookshelf, knex, typeorm
I'm a newbie at Node JS, and I'm using NodeJS (v. 8.7.0), sqlite3 and Express.
I have two tables in a SQLite database:
releases (id, title, image)
links (id, url)
Each "release" has one or more "links" associated with it.
I can get all the releases using:
dbh.all("SELECT * FROM releases ORDER BY id DESC", (err, rows) => { ... })
And I can get all the links for a given release using:
dbh.all("SELECT * FROM links WHERE id = ?", (err, rows) => { ... })
But I can't figure out how to add a "links" property to each "release", which contains their corresponding links, so that I can feed the resulting object to Mustache, and generate a HTML page.
I know that storing hierarchical data inside of a relational database is not the best idea, and I could easily do this using PHP, but I really want to learn how to use NodeJS.
This is what I've come up so far:
var sqlite3 = require("sqlite3")
function main() {
db = new sqlite3.Database("releases.sqlite3")
all = []
db.each(
"SELECT * FROM releases ORDER BY id DESC",
(err, release) => {
release.links = []
db.all("SELECT url FROM links WHERE id = ?", [release.id], (err, links) => {
links = links.map((e) => { return e.url })
release.links = links
// line above: tried
// links.forEach((e) => { release.links.push(e.url) })
// too, but that didn't work either.
})
all.push(release)
},
(complete) => { console.log(all) }
)
}
main()
Though, when I run it, it inevitably shows:
links: []
Every time. How can I fix this?
Thank you in advance.
Edit 1:
This SQL snippet generates the database, and populates it with some data.
CREATE TABLE `links` ( `id` TEXT, `url` TEXT );
CREATE TABLE `releases` ( `id` TEXT, `title` TEXT, `image` TEXT );
INSERT INTO links VALUES
('rel-001', 'https://example.com/mirror1'),
('rel-001', 'https://example.com/mirror2');
INSERT INTO releases VALUES
('rel-001', 'Release 001', 'https://example.com/image.jpg');
The goal is to have something like this:
{
releases:[
{
id:'rel-001',
title:'Release 001',
image:'https://example.com/image.jpg',
links:[
'https://example.com/mirror1',
'https://example.com/mirror2'
]
}
]
}
try to see if both queries are being executed by adding console.log in the callbacks, moreover you should push the links only within the second callback since before the callback is fired the value is not existing, thus you are trying to push an empty value, also you don't need to initialize release.links = [], all will be only filled after all queries are executed, so therefore we need to execute console.log(all); in the last child callback:
function main() {
all = []
var parentComplete = false;
db.each("SELECT * FROM releases ORDER BY id DESC", (err, release) => {
db.all("SELECT url FROM links WHERE id = ?", [release.id], (err, links) => {
release.links = links.map(e => e.url);
all.push(release);
if (parentComplete){
console.log(all);
}
})
},
(complete) => {
parentComplete = true;
}
)
}
main();
p.s. in order to get the result you want you will need to initialize all as an object all = {releases:[]}
function main() {
all = {releases:[]};
var parentComplete = false;
db.each("SELECT * FROM releases ORDER BY id DESC", (err, release) => {
db.all("SELECT url FROM links WHERE id = ?", [release.id], (err, links) => {
release.links = links.map(e => e.url);
all.releases.push(release);
if (parentComplete){
console.log(all);
}
})
},
(complete) => {
parentComplete = true;
}
)
}
main();