I am using Sequelize in my node js server. I am ending up with validation errors because my code tries to write the record twice instead of creating it once and then updating it since it's already in DB (Postgresql).
This is the flow I use when the request runs:
const latitude = req.body.latitude;
var metrics = await models.user_car_metrics.findOne({ where: { user_id: userId, car_id: carId } })
if (metrics) {
metrics.latitude = latitude;
.....
} else {
metrics = models.user_car_metrics.build({
user_id: userId,
car_id: carId,
latitude: latitude
....
});
}
var savedMetrics = await metrics();
return res.status(201).json(savedMetrics);
At times, if the client calls the endpoint very fast twice or more the endpoint above tries to save two new rows in user_car_metrics, with the same user_id and car_id, both FK on tables user and car.
I have a constraint:
ALTER TABLE user_car_metrics DROP CONSTRAINT IF EXISTS user_id_car_id_unique, ADD CONSTRAINT user_id_car_id_unique UNIQUE (car_id, user_id);
Point is, there can only be one entry for a given user_id and car_id pair.
Because of that, I started seeing validation issues and after looking into it and adding logs I realize the code above adds duplicates in the table (without the constraint). If the constraint is there, I get validation errors when the code above tries to insert the duplicate record.
Question is, how do I avoid this problem? How do I structure the code so that it won't try to create duplicate records. Is there a way to serialize this?
If you have a unique constraint then you can use upsert to either insert or update the record depending on whether you have a record with the same primary key value or column values that are in the unique constraint.
await models.user_car_metrics.upsert({
user_id: userId,
car_id: carId,
latitude: latitude
....
})
See upsert
PostgreSQL - Implemented with ON CONFLICT DO UPDATE. If update data contains PK field, then PK is selected as the default conflict key. Otherwise, first unique constraint/index will be selected, which can satisfy conflict key requirements.
Related
I'm creating the backend for a simple app which allows users to create, update, and delete products. Using Express as my framework, with Postgres as my DB and Prisma, which I'm new to, as my ORM. Users and products have a one-to-many relationship. Prisma's documentation states that when updating a record, you should use the update method - so to update the name of a product with a given ID, your code would look something like this:
export const updateProduct = async (req, res) => {
const [productID, newProductName, userID] = [req.params.id, req.body.name, res.locals.user.id];
const product = await prisma.product.update({
where: {
id: productID,
},
data: {
name: newProductName
}
});
res.status(200);
res.json(product);
};
However, there's a problem here - I'm not checking to see that the product with the provided ID belongs to the user that has sent the request to update it. I have the ID of the user who has sent the request in the variable userID, and each product in the DB has a field belongsToID which is set to the ID of the user that the product belongs to. I should theoretically therefore be able to modify my query to get the product with the specified ID and a matching belongsToID like so:
export const updateProduct = async (req, res) => {
const [productID, newProductName, userID] = [req.params.id, req.body.name, res.locals.user.id];
const product = await prisma.product.update({
where: {
id: productID,
belongsToID: userID
},
data: {
name: newProductName
}
});
res.status(200);
res.json(product);
};
That, however, does not work - I get the following error: Type '{ id: any; belongsToID: any; }' is not assignable to type 'ProductWhereUniqueInput'. Object literal may only specify known properties, and 'belongsToId' does not exist in type 'ProductWhereUniqueInput'.ts(2322).
It appears that when trying to do a 'findUnique', Prisma doesn't allow non-unique fields to be used in the query (even if the combination of both fields is unique, as is the case here). I do get that logically, my query doesn't make much sense - the ID alone is already enough to find a unique entry without the second field, so in that sense, the second field is totally redundant. Yet how else am I meant to check that the belongsToID is what it should be before updating the record? Is there somewhere else within the object passed to .update where I can provide a check to be performed on the retrieved record before performing the update?
I believe that creating an index would allow me to query for both fields at once - but why should I have to create an index when the ID (which is already indexed) alone is all I need to retrieve the record I need? What I really need is a way to perform a check on a retrieved record before performing the update when using Prisma.table_name.update(), not a way to query for something with a unique combination of fields.
I am trying to delete and update records in cosmosDB using my graphql/nodejs code and getting error - "Entity with the specified id does not exist in the system". Here is my code
deleteRecord: async (root, id) => {
const { resource: result } = await container.item(id.id, key).delete();
console.log(`Deleted item with id: ${id}`);
},
Somehow below code is not able to find record, even "container.item(id.id, key).read()" doesn't work.
await container.item(id.id, key)
But if I try to find record using query spec it works
await container.items.query('SELECT * from c where c.id = "'+id+'"' ).fetchNext()
FYI- I am able to fetch all records and create new item, so Connection to DB and reading/writing is not an issue.
What else can it be? Any pointer related to this will be helpful.
Thanks in advance.
It seems you pass the wrong key to item(id,key). According to the Note of this documentation:
In both the "update" and "delete" methods, the item has to be selected
from the database by calling container.item(). The two parameters
passed in are the id of the item and the item's partition key. In this
case, the parition key is the value of the "category" field.
So you need to pass the value of your partition key, not your partition key path.
For example, if you have document like below, and your partition key is '/category', you need to use this code await container.item("xxxxxx", "movie").
{
"id":"xxxxxx",
"category":"movie"
}
I am trying to query dynamodb using the following code:
const AWS = require('aws-sdk');
let dynamo = new AWS.DynamoDB.DocumentClient({
service: new AWS.DynamoDB(
{
apiVersion: "2012-08-10",
region: "us-east-1"
}),
convertEmptyValues: true
});
dynamo.query({
TableName: "Jobs",
KeyConditionExpression: 'sstatus = :st',
ExpressionAttributeValues: {
':st': 'processing'
}
}, (err, resp) => {
console.log(err, resp);
});
When I run this, I get an error saying:
ValidationException: Query condition missed key schema element: id
I do not understand this. I have defined id as the partition key for the jobs table and need to find all the jobs that are in processing status.
You're trying to run a query using a condition that does not include the primary key. This is how queries work in DynamoDB. You would need to do a scan for the info in your case, however, I don't think that is the best option.
I think you want to set up a global secondary index and use that to query for the processing status.
In another answer #smcstewart responded to this question. But he provides a link instead of commenting why this error occurs. I want to add a brief comment hoping it will save your time.
AWS docs on Querying a Table states that you can do WHERE condition queries (e.g. SQL query SELECT * FROM Music WHERE Artist='No One You Know') in the DynamoDB way, but with one important caveat:
You MUST specify an EQUALITY condition for the PARTITION key, and you can optionally provide another condition for the SORT key.
Meaning you can only use key attributes with Query. Doing it in any other way would mean that DynamoDB would run a full scan for you which is NOT efficient - less efficient than using Global secondary indexes.
So if you need to query on non-key attributes using Query is usually NOT an option - best option is using Global Secondary Indexes as suggested by #smcstewart.
I found this guide to be useful to create a Global secondary index manually.
If you need to add it using CloudFormation here is a relevant page.
I was getting this error for a different scenario. Here is my scenario.
(It's very unlikely that anyone else ends up with this case, but incase)
I had a query working on a Table (say table A). Table A had a partition key m_id and sort key u_id.
I had a query to fetch data using m_id. The query was working.
'''
var queryParams = {
ExpressionAttributeValues: {
':m_id': mId
},
KeyConditionExpression: 'm_id = :m_id',
TableName: "A"
};
let connections = await docClient.query(queryParams).promise();
'''
I created another Table say Table B. I made some errors in naming keys so I simply deleted and created a table with the same name again, Table B. Table B had partition key m_id, and sort key s_id.
I copied pasted the same query which I was using for Table A, I changed Table name only because partition key had the same name.
To my shock, I get this expectation.
"ValidationException: Query condition missed key schema element"
I rechecked all the names, I compared the query with the working query. Everything was fine.
I thought maybe because, I was deleting recreating Table B, it could be something with that. So I create a fresh Table with a new Name Table B2 with the same key names as Table B.
In my query that was throwing exceptions, I changed only the Table name from B to B2.
And the Exception was gone.
If you are getting this on a fresh table, where no query has worked earlier, creating a new Table with a new name is an option.
If you delete a Table only to change partition key names, it may be safer to use a new name for Table as well (Dynamo could be referring metadata by table names and not by internal identifiers, it is possible that old metadata stays even if you delete a table. Just a guess given I faced this case).
EDIT:2022-July-12
This error does not leave me. My own answer was helpful but one more case, there was a trailing space in name of Key in the table. And Dynamo does not even check for spaces in key names.
You have to create an global secondary index for the status field.
Then, you code could look like smth like this:
dynamo.query({
TableName: "Jobs",
IndexName: 'status',
KeyConditionExpression: '#s = :st',
ExpressionAttributeValues: {
':st': 'processing'
},
ExpressionAttributeNames: {
'#s': 'status',
},
}, (err, resp) => {
console.log(err, resp);
});
Note: scan operation is indeed very costly, especially if you table is huge in size
i solved the problem using AWS.DynamoDB.DocumentClient() with scan, for sample (nodejs):
var docClient = new AWS.DynamoDB.DocumentClient();
var params = {
TableName: "product",
FilterExpression: "#cg = :data",
ExpressionAttributeNames: {
"#cg": "categoria",
},
ExpressionAttributeValues: {
":data": category,
}
};
docClient.scan(params, onScan);
function onScan(err, data) {
if (err) {
// for the log in server
console.error("Unable to scan the table. Error JSON:", JSON.stringify(err, null, 2));
res.json(err);
} else {
console.log("Scan succeeded.");
res.json(data);
}
}
I have a Node application which runs in cluster mode with pm2.
I also have a function which checks if a specific row is in a db table. If the row is missing it creates the row otherwise a value is set and saved.
I only need one row for each combination of userId and groupId.
function someFunction()={
return Activation.findOne({ where: { userId: userId, groupId: groupId } })
.then(activationObject => {
if (!activationObject) {
return Activation.create({ userId: userId, groupId: groupId, activationTime: sequelize.fn('NOW') })
} else {
activationObject.activationTime = sequelize.fn('NOW');
return activationObject.save()
}
})
}
How can I avoid race conditions when running node in cluster mode?
Currently if first worker checks the row is available and the second checks at the same time both get no result and in the end we have two newly created rows instead of one.
I know that Sequelize provides a findOrCreate() method but I wanted an easy understandable example.
The easiest way would be to add a UNIQUE constraint for the combination of userId and groupId with an ON CONFLICT REPLACE clause, and always create a new row instead of updating. This will cause a newly inserted row with the new activationTime to replace the old row.
You can additionally check the number of rows inserted to tell whether the insert succeeded or not.
Example: UNIQUE (userId, groupId) ON CONFLICT REPLACE
How can I add a new row with the update operation
I am using following code
statuscollection.update({
id: record.id
}, {
id: record.id,
ip: value
}, {
upsert: true
}, function (err, result) {
console.log(err);
if (!err) {
return context.sendJson([], 404);
}
});
While calling this first one i will add the row
id: record.id
Then id:value then i have to add id:ggh
How can i add every new row by calling this function for each document I need to insert
By the structure of your code you are probably missing a few concepts.
You are using update in a case where you probably do not need to.
You seem to be providing an id field when the primary key for MongoDB would be _id. If that is what you mean.
If you are intending to add a new document on every call then you probably should be using insert. Your use of update with upsert has an intended usage of matching a document with the query criteria, if the document exists update the fields as specified, if not then insert a new document with the fields specified.
Unless that actually is your goal then insert is most certainly what you need. In that case you are likely to rely on the value of _id being populated automatically or by supplying your own unique value yourself. Unless you specifically want another field as an identifier that is not unique then you will likely want to be using the _id field as described before.