I am using the node aws-sdk, I have implemented a method to create or update an item in DynamoDb.
It works well based of off the Key (Id), and will either created or update the item.
My params are as follows:
let params = {
TableName: TableName,
Key: {
key: args.key
},
UpdateExpression: `set
userKey = :userKey,
content = :content`,
ExpressionAttributeValues: {
':userKey': args.userKey,
':content': args.content
},
ExpressionAttributeNames: {
}
};
I have since realised I need to conditionally check a secondary key on the update to ensure the userKey matches.
So I added:
ConditionExpression: 'userKey = :userKey',
Now the create doesn't work as it fails the condition, what is the correct way to do the create and conditional update in one statement?
My table definitionas are as follows:
AttributeDefinitions:
- AttributeName: key
AttributeType: S
- AttributeName: userKey
AttributeType: S
- AttributeName: timestamp
AttributeType: N
KeySchema:
- AttributeName: key
KeyType: HASH
You've got two options-
If you userKey is actually the sort key (https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.CoreComponents.html) of the table, then you can change your params as such:
Key: {
key: args.key
userKey: args.userKey
}
However if userKey is just another attribute, then you can extend the condition expression as such:
ConditionExpression: 'userKey = :userKey OR attribute_not_exists(userKey)'
Which will require either that userKey matches what you were expecting, or that it hasn't been set at all (which would be the case on an upsert).
Note- this would allow you to update an item with a key that did not also have a userKey. If you're concerned about that then you can extend the condition to:
ConditionExpression: 'userKey = :userKey OR(attribute_not_exists(key) AND attribute_not_exists(userKey))'
Related
I am trying to delete and then create a DynamoDB table using nodejs aws sdk (version 3.142.0) and I wanted to use the waiters (waitUntilTableNotExists / waitUntilTableExists), but I don't understand how they are supposed to be used and I cannot find a good example online.
Regards
Here is one way after a createTable command in aws-sdk-js-v3 to wait for the table to complete. A note is that if you do NOT use waitUntilTableExists and instead attempt to use DescribeTableCommand it will incorrectly report TableStatus == 'ACTIVE' even though you cannot Read/Write to the table, you must use waitUntilTableExists.
import {
CreateTableCommandInput,
CreateTableCommandOutput,
waitUntilTableExists
} from "#aws-sdk/client-dynamodb";
const client = new DynamoDBClient({ region: "us-east-1" });
const data = await client.send(
new CreateTableCommand({
TableName: tableName,
AttributeDefinitions: partitionAndSortKeyDefinitions,
KeySchema: columnSchema,
ProvisionedThroughput: {
ReadCapacityUnits: 4,
WriteCapacityUnits: 2,
},
})
);
// maxWaitTime - seconds
const results = await waitUntilTableExists({client: client, maxWaitTime: 120}, {TableName: tableName})
if (results.state == 'SUCCESS') {
return results.reason.Table
}
console.error(`${results.state} ${results.reason}`);
I have a NodeJS function that scan a table in DynamoDB (without primary sort key) and return the number of elements of the column sync that are null.
My table:
var params = {
AttributeDefinitions: [
{
AttributeName: "barname",
AttributeType: "S"
},
{
AttributeName: "timestamp",
AttributeType: "S"
}
],
KeySchema: [
{
AttributeName: "barname",
KeyType: "HASH"
},
{
AttributeName: "timestamp",
KeyType: "RANGE"
}
],
ProvisionedThroughput: {
ReadCapacityUnits: 1,
WriteCapacityUnits: 1
},
TableName: tableName
};
The function that count when sync==false
var dynamodb = new AWS.DynamoDB({apiVersion: '2012-08-10'});
async function getCountNoSync(type){
console.log(type)
var params = {
TableName: tableName,
FilterExpression: 'sync = :sync and billing = :billing',
ExpressionAttributeValues: {
':billing' : {S: type},
':sync' : {BOOL: false}
},
};
var count = 0;
await dynamodb.scan(params).promise()
.then(function(data){
count = data.Count;
})
.catch(function(err) {
count = 0;
console.log(err);
});
return count;
}
The function works fine If a have few elements in my table (eg. less than 150). If the number of elements are higher, the count variable is always 0. It loooks like the scan do not find all elements.
Any ideia?
Best regards
The reason that you do not find all the items where attribute sync == null is that the scan operation is only reading part of your table.
As the documentation states:
If the total number of scanned items exceeds the maximum dataset size limit of 1 MB, the scan stops and results are returned to the user as a LastEvaluatedKey value to continue the scan in a subsequent operation.
So if your table is several hundred of megabytes big, you need to call scan() multiple times and provide the LastEvaluatedKey to read the next "page" of your table. This process is also called "pagination".
But this will take a lot of time and the time this needs will just increase with your table size. The proper way of doing this would be to create an index of the sync field and then do a query() on that index.
You can read more about that in the AWS documentation:
Querying and Scanning a DynamoDB Table
Reference documentation for scan()
Paginating the Results
I try to update an item in dynamodb by adding a condition, without passing the key in the parameters.
And as soon as my condition is true update. Is it possible to do this?
Below an example of an item:
{
"id" : "bcc2f32e-305e-4469-88e2-463724b5c6a9",
"name" : "toto",
"email" : "toto#titi.com"
}
Where email is unique for items.
I tested this code and it works :
const name= "updateName";
const params = {
TableName: MY_TABLE,
Key: {
id
},
UpdateExpression: 'set #name = :name',
ExpressionAttributeNames: { '#name': 'name' },
ExpressionAttributeValues: { ':name': name },
ReturnValues: "ALL_NEW"
}
dynamoDb.update(params, (error, result) => {
if (error) {
res.status(400).json({ error: 'Could not update Item' });
}
res.json(result.Attributes);
})
But i want to do something like this (replace the Key by conditionExpression):
const params = {
TableName: MY_TABLE,
UpdateExpression: 'set #name = :name',
ConditionExpression: '#email = :email',
ExpressionAttributeNames: {
'#name': 'name',
'#email': 'email'
},
ExpressionAttributeValues: {
':name': name,
':email': email
},
ReturnValues: "ALL_NEW"
}
dynamoDb.update(params, (error, result) => {
if (error) {
res.status(400).json({ error: 'Could not update User' });
}
res.json(result.Attributes);
})
But this code doesn't work.
Any ideas?
You cannot update an item in DynamoDB without using the entire primary key (partition key, and sort key if present). This is because you must specify exactly one record for the update. See the documentation here.
If you want to find an item using a field that is not the primary key, then you can search using a scan (potentially slow and expensive) or by using a Global Secondary Index (GSI) on that field. Either of these methods requires that you do a separate request to find the item in question, and then use its primary key to perform the update.
It sounds like you want to do an update that waits for a condition. That's not how DynamoDb works; it cannot wait for anything (except consistency, I suppose, but that's somewhat different). What you can do is make a request with a condition, and if it fails the condition (returning immediately), make the request again later. If you do this you'll need to be careful to backoff appropriately, or you might end up making a lot of requests very quickly.
The key is a required parameter when doing updates; the condition expression can be used in addition to providing the key, but can't be used instead of the key.
Also, I am not sure you fully understand what the conditionExpression is for - its not like the 'where' clause in an SQL update statement (i.e. update mytable set name='test' where email='myemail.com'.
Instead, logically the conditionExpression in an update would be more like:
update mytable set name='test' where key='12345' but only if quantity >0 - for example,
i.e. you are telling dynamodb the exact key of the record you want updated, and once it finds it it uses the condition expression to determine if the update should proceed - i.e. find the record with id=12345, and change the name to 'test', only of the quantity is greater than 0.
It does not use the conditionExpression to find records to update.
I have a table that has:
userId(partition key) postId(sort key) category
I want to show all items filtered by category?
How should I do it?
My try
const params = {
TableName: "posts_reddit",
KeyConditionExpression: "userId = :userId",
FilterExpression: "category: category",
ExpressionAttributeValues: {
":userId": event.requestContext.identity.cognitoIdentityId,
":category": "engineering"
}
};
You missed a colon in your filter expression, it should be:
"FilterExpression: "category: :category"
The structure for the ExpressionAttributeValues is not right as well, you must provide which type you're passing as an argument. See ether example in the scan operation docs.
I'm writing a nodejs 5.7.1 application with aws-sdk for DynamoDB.
I have a table of events that I created with the following code:
var statsTableName='bingodrive_statistics';
var eventNameColumn = 'event_name';
var eventTimeColumn = 'event_time';
var eventDataColumn = 'event_data';
var params = {
TableName: statsTableName,
KeySchema: [ // The type of of schema. Must start with a HASH type, with an optional second RANGE.
{ // Required HASH type attribute
AttributeName: eventNameColumn,
KeyType: 'HASH',
},
{ // Optional RANGE key type for HASH + RANGE tables
AttributeName: eventTimeColumn,
KeyType: 'RANGE',
}
],
AttributeDefinitions: [ // The names and types of all primary and index key attributes only
{
AttributeName: eventNameColumn,
AttributeType: 'S', // (S | N | B) for string, number, binary
},
{
AttributeName: eventTimeColumn,
AttributeType: 'N'
}
],
ProvisionedThroughput: { // required provisioned throughput for the table
ReadCapacityUnits: 1,
WriteCapacityUnits: 1,
}
};
dynamodbClient.createTable(params, callback);
as you can see, I have a Hash + Range index. the range is on event_time.
now I want to scan or query for all the items between two specific dates.
so i'm sending the following params to the query function of dynamoDb:
{
"TableName": "bingodrive_statistics",
"KeyConditionExpression": "event_time BETWEEN :from_time and :to_time",
"ExpressionAttributeValues": {
":from_time": 1457275538691,
":to_time": 1457279138691
}
and i'm getting this error:
{
"message": "Query condition missed key schema element",
"code": "ValidationException",
"time": "2016-03-06T15:46:06.862Z",
"requestId": "5a672003-850c-47c7-b9df-7cd57e7bc7fc",
"statusCode": 400,
"retryable": false,
"retryDelay": 0
}
I'm new to dynamoDb. I don't know what's the best method, Scan or Query in my case. any information regarding the issue would be greatly appreciated.
You should use query. You can't use only range key if you want to query for values between two range keys, you need to use hash key as well since range key. It's because hash key (partition key) is used to select a physical partition where the data is stored, sorted by range key (sort key). From DynamoDB developer guide:
If the table has a composite primary key (partition key and sort key), DynamoDB calculates the hash value of the partition key in the same way as described in Data Distribution: Partition Key—but it stores all of the items with the same partition key value physically close together, ordered by sort key value.
Also, you should choose partition key that distributes well your data. If evenName has small total number of values, it might not be the best option (See Guidelines For Tables]
That said, if you already have eventName as your hash key and eventTime as your range Key, you should query (sorry for pseudo code, I use DynamoDBMapper normally):
hashKey = name_of_your_event
conditions = BETWEEN
attribute_values (eventTime1, eventTime2)
You don't need additional Local Secondary Index or Global Secondary Index for that. Note that GSI let's you query for columns that are not indexed with the table hash and range key, but to query data between the timestamps, you will still need a range key or will need to do a Scan otherwise.
Use this query
function getConversationByDate(req , cb) {
var payload = req.all; //05/09/2017
var params = {
TableName: "message",
IndexName: "thread_id-timestamp-index",
KeyConditionExpression: "#mid = :mid AND #time BETWEEN :sdate AND :edate",
ExpressionAttributeNames: {
"#mid": "thread_id",
"#time": "timestamp"
},
ExpressionAttributeValues: {
":mid": payload.thread_id,
":sdate": payload.startdate,
":edate": payload.enddate
}
};
req.dynamo.query(params, function (err, data) {
cb(err, data);
});
}