I have a dynamodb table with the following structure:
'Items': [
{'user_id': 'myuser',
'owes_to': {
'youruser': '10',
'user3': '20'
}
}]
Is it possible in dynamodb to create a new user key in owes_to or if it already exists add to the value of that key?
In example in two calls:
Add 'user4' with value 20, It doesnt exist so create the user4 key with value 20)
Next time update user4 value 20 to 40, This time key already existed so add the 20 to the existing value
It's possible, update expression will be similar to this:
update_request = {
Key={'user_id': ':id'},
UpdateExpression='SET owes_to.#user_name = if_not_exists(owes_to.#user_name, :default_val) + :val',
ExpressionAttributeNames={
'#user_name': <your_user_name>
}
ExpressionAttributeValues={
':id': 'myuser',
':val': <your_update_value>,
':default_val': 0
}
}
Note the if_not_exists(owes_to.#user_name, :default_val) construction. It will take value of the user if it exists or take value specified in :default_val if it doesn't.
More about adding and updating map values in official docs
Related
This issue has been raised before but so far I couldn't find a solution that worked in boto3. GSI is set on 'solutionId' and partition key being 'emp_id'. Basically, I just want to delete all records in the table without deleting the table itself. What am I missing here?
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb.html#DynamoDB.Client.batch_write_item
table_name = "solutions"
dynamodb_client = boto3.client('dynamodb')
dynamodb_resource = boto3.resource('dynamodb')
table = dynamodb_resource.Table(table_name)
data = table.scan()
delete_list = []
for item in data['Items']:
delete_list.append({
'DeleteRequest': {
'Key': {
'solutionId': {"S": f'{item["solutionId"]}'}
}
}
}
)
def list_spliter(list, size):
return (list[pos:pos + size] for pos in range(0, len(list), size))
for batch in list_spliter(delete_list, 25):
dynamodb_resource.batch_write_item(RequestItems={
f'{table_name}': batch
}
)
I think there are two small issues here:
you're using the high-level service resource interface so you don't need to explicitly tell DynamoDB what the attribute types are. They are inferred through automatic marshaling. So you can simply use "key" : "value" rather than "key": {"S": "value"} for the string keys
when deleting an item you need to provide the full primary key, to include both partition key and sort key
So, for example, if your partition and sort keys are named pk and sk:
'DeleteRequest': {
'Key': {
'pk': pk,
'sk': sk
}
}
I have this code, What I am trying to do is create a match and then add the match id to the property "playedMatches" in the player table.
My code creates the match then I add the ID of that created match to the array for "playedMatches" in the player table, everything works fine. However, when I want to push the second played match ID to the array, it deletes all the items in the array and replaces it for the new one created.
In MongoDB I would use the $push Update Operator. Is there something similar I can use for Dynamo DB?
I have also tried with createSet and I get errors every time.
What am I doing wrong?
This is the table structure:
id: xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx
img:
name: john doe
playedMatches: [xxxxx-xxxxx-xxxx-xxxx-xxxxx]
I want push new id's into playedMatches
const matchId = ['987654321'];
const duplicatedId = '987654321';
const putDataParams = {
TableName: table,
Key: {
id,
},
UpdateExpression: 'set playedMatches = list_append(playedMatches, :matchId)',
ConditionExpression: 'not contains (playedMatches, :duplicatedId)',
ExpressionAttributeValues: {
':matchId': matchId,
':duplicatedId': duplicatedId,
},
ReturnValues: 'UPDATED_NEW',
};
Thanks a lot!
I try to update an item in dynamodb by adding a condition, without passing the key in the parameters.
And as soon as my condition is true update. Is it possible to do this?
Below an example of an item:
{
"id" : "bcc2f32e-305e-4469-88e2-463724b5c6a9",
"name" : "toto",
"email" : "toto#titi.com"
}
Where email is unique for items.
I tested this code and it works :
const name= "updateName";
const params = {
TableName: MY_TABLE,
Key: {
id
},
UpdateExpression: 'set #name = :name',
ExpressionAttributeNames: { '#name': 'name' },
ExpressionAttributeValues: { ':name': name },
ReturnValues: "ALL_NEW"
}
dynamoDb.update(params, (error, result) => {
if (error) {
res.status(400).json({ error: 'Could not update Item' });
}
res.json(result.Attributes);
})
But i want to do something like this (replace the Key by conditionExpression):
const params = {
TableName: MY_TABLE,
UpdateExpression: 'set #name = :name',
ConditionExpression: '#email = :email',
ExpressionAttributeNames: {
'#name': 'name',
'#email': 'email'
},
ExpressionAttributeValues: {
':name': name,
':email': email
},
ReturnValues: "ALL_NEW"
}
dynamoDb.update(params, (error, result) => {
if (error) {
res.status(400).json({ error: 'Could not update User' });
}
res.json(result.Attributes);
})
But this code doesn't work.
Any ideas?
You cannot update an item in DynamoDB without using the entire primary key (partition key, and sort key if present). This is because you must specify exactly one record for the update. See the documentation here.
If you want to find an item using a field that is not the primary key, then you can search using a scan (potentially slow and expensive) or by using a Global Secondary Index (GSI) on that field. Either of these methods requires that you do a separate request to find the item in question, and then use its primary key to perform the update.
It sounds like you want to do an update that waits for a condition. That's not how DynamoDb works; it cannot wait for anything (except consistency, I suppose, but that's somewhat different). What you can do is make a request with a condition, and if it fails the condition (returning immediately), make the request again later. If you do this you'll need to be careful to backoff appropriately, or you might end up making a lot of requests very quickly.
The key is a required parameter when doing updates; the condition expression can be used in addition to providing the key, but can't be used instead of the key.
Also, I am not sure you fully understand what the conditionExpression is for - its not like the 'where' clause in an SQL update statement (i.e. update mytable set name='test' where email='myemail.com'.
Instead, logically the conditionExpression in an update would be more like:
update mytable set name='test' where key='12345' but only if quantity >0 - for example,
i.e. you are telling dynamodb the exact key of the record you want updated, and once it finds it it uses the condition expression to determine if the update should proceed - i.e. find the record with id=12345, and change the name to 'test', only of the quantity is greater than 0.
It does not use the conditionExpression to find records to update.
I am trying to create a dynamoDB table, with a secondary index with partition and sort key.
I can create the table without the secondary index, but haven't been able to find a way yet to add the secondary index
I've looked at both of these resources, but haven't found anything that actually shows me what code i need in my cdk python script to create the resource with a secondary index
https://docs.aws.amazon.com/cdk/api/latest/docs/#aws-cdk_aws-dynamodb.Table.html
https://docs.aws.amazon.com/cdk/api/latest/docs/aws-dynamodb-readme.html
This is the code that will create the table
table_name = 'event-table-name'
event_table = dynamoDB.Table(self, 'EventsTable',
table_name=table_name,
partition_key=Attribute(
name='composite',
type=AttributeType.STRING
),
sort_key=Attribute(
name='switch_ref',
type=AttributeType.STRING
),
removal_policy=core.RemovalPolicy.DESTROY,
billing_mode=BillingMode.PAY_PER_REQUEST,
stream=StreamViewType.NEW_IMAGE,
)
and this is the secondary index I need to attach to it
secondaryIndex = dynamoDB.GlobalSecondaryIndexProps(
index_name='mpan-status-index',
partition_key=Attribute(
name='field1',
type=AttributeType.STRING
),
sort_key=Attribute(
name='field2',
type=AttributeType.STRING
),
)
I've tried adding the block inside the table creation and tried calling the addSecondaryindex method on the table. But both fail either saying unexpected keyword or object has no attribute addGlobalSecondaryIndex
addGlobalSecondaryIndex should be called on the Table class.
The code below (in typescript) works perfectly for me:
const table = new ddb.Table(this, "EventsTable", {
tableName: "event-table-name",
partitionKey: { name: 'composite', type: ddb.AttributeType.STRING },
sortKey: { name: 'switch_ref', type: ddb.AttributeType.STRING },
removalPolicy: cdk.RemovalPolicy.DESTROY,
billingMode: BillingMode.PAY_PER_REQUEST,
stream: StreamViewType.NEW_IMAGE
});
table.addGlobalSecondaryIndex({
indexName: 'mpan-status-idex',
partitionKey: { name: 'field1', type: ddb.AttributeType.STRING },
sortKey: { name: 'field2', type: ddb.AttributeType.STRING }
});
For anyone looking for this and stumbling on it through google search:
create your table with the usual:
from aws_cdk import aws_dynamodb as dynamodb
from aws_cdk.aws_dynamodb import Attribute, AttributeType, ProjectionType
table = dynamodb.Table(self, 'tableID',
partition_key=Attribute(name='partition_key', type = AttributeType.STRING))
then add your global secondary indexes in much the same way:
table.add_global_secondary_index(
partition_key=Attribute(name='index_hash_key', type=AttributeType.NUMBER),
sort_key=Attribute(name='range_key', type=AttributeType.STRING),
index_name='some_index')
you can add projection attributes with they kwarg arguments:
projection_type = ProjectionType.INCLUDE,
non_key_attributes= ['list', 'of', 'attribute','names']
and projection_type defaults to All if you don't include it.
I know the docs are incomplete in lots of areas, but this is found here:
https://docs.aws.amazon.com/cdk/api/latest/python/aws_cdk.aws_dynamodb/Table.html?highlight=add_global#aws_cdk.aws_dynamodb.Table.add_global_secondary_index
Have you tried using the addGlobalSecondaryIndex method as in
event_table.addGlobalSecondaryIndex({indexName: "...", partitionKey: "...", ...})
Take a look at the documentation for the method.
aws_dynamodb.Table returns an ITable. To use the addGlobalSecondaryIndex, first cast to Table like so:
table = aws_dynamodb.Table(self, "Table",
partition_key=dynamodb.Attribute(name="id", type=dynamodb.AttributeType.STRING)
aws_dynamodb.Table(table).add_global_secondary_index(...)
This question already has answers here:
How to Ignore Duplicate Key Errors Safely Using insert_many
(3 answers)
Closed 5 years ago.
I'm using NodeJS with MongoDB and Express.
I need to insert records into a collection where email field is mandatory.
I'm using insertMany function to insert records. It works fine when unique emails are inserted, but when duplicate emails are entered, the operation breaks abruptly.
I tried using try catch to print the error message, but the execution fails as soon as a duplicate email is inserted. I want the execution to continue and store the duplicates. I want to get the final list of the records inserted/failed.
Error Message:
Unhandled rejection MongoError: E11000 duplicate key error collection: testingdb.gamers index: email_1 dup key:
Is there any way to handle the errors or is there any other approach apart from insertMany?
Update:
Email is a unique field in my collection.
If you want to continue inserting all the non-unique documents rather than stopping on the first error, considering setting the {ordered:false} options to insertMany(), e.g.
db.collection.insertMany(
[ , , ... ],
{
ordered: false
}
)
According to the docs, unordered operations will continue to process any remaining write operations in the queue but still show your errors in the BulkWriteError.
I can´t make comment, so goes as answer:
is you database collection using unique index for this field, or your schema has unique attribute for the field? please share more information about you code.
From MongoDb docs:
"Inserting a duplicate value for any key that is part of a unique index, such as _id, throws an exception. The following attempts to insert a document with a _id value that already exists:"
try {
db.products.insertMany( [
{ _id: 13, item: "envelopes", qty: 60 },
{ _id: 13, item: "stamps", qty: 110 },
{ _id: 14, item: "packing tape", qty: 38 }
] );
} catch (e) {
print (e);
}
Since _id: 13 already exists, the following exception is thrown:
BulkWriteError({
"writeErrors" : [
{
"index" : 0,
"code" : 11000,
"errmsg" : "E11000 duplicate key error collection: restaurant.test index: _id_ dup key: { : 13.0 }",
"op" : {
"_id" : 13,
"item" : "envelopes",
"qty" : 60
}
}
],
(some code omitted)
Hope it helps.
Since you know that the error is occurring due to duplicate key insertions, you can separate the initial array of objects into two parts. One with unique keys and the other with duplicates. This way you have a list of duplicates you can manipulate and a list of originals to insert.
let a = [
{'email': 'dude#gmail.com', 'dude': 4},
{'email': 'dude#yahoo.com', 'dude': 2},
{'email': 'dude#hotmail.com', 'dude': 2},
{'email': 'dude#gmail.com', 'dude': 1}
];
let i = a.reduce((i, j) => {
i.original.map(o => o.email).indexOf(j.email) == -1? i.original.push(j): i.duplicates.push(j);
return i;
}, {'original': [], 'duplicates': []});
console.log(i);
EDIT: I just realised that this wont work if the keys are already present in the DB. So you should probably not use this answer. But Ill just leave it here as a reference for someone else who may think along the same lines.
Nic Cottrell's answer is right.