The DynamoDB.DocumentClient automatically marshals & unmarshals values between JavaScript types DynamoDB's more descriptive AttributeMap type. However, when working with an Item that has a StringSet attribute, it does not seem to do the conversion automatically.
When adding a StringSet attribute to the table using DocumentClient, I use the createSet(...) method to convert the array to a Set. When retrieving the value back, what is the inverse of createSet(...)? Is the best practice to just access the Set's .values directly? And if so, is that documented somewhere?
Here's sample code adding an Item with a StringSet attribute, then retrieving that item:
const docClient = new DocumentClient();
const TableName = "StringSets-Example";
const PK = "Names";
const Values = ["Peter", "Paul", "Mary"];
const putParams = {
TableName,
Item: {
PK,
names: docClient.createSet(Values)
}
}
await docClient.put(putParams).promise();
// ... some time later, I can retrieve the value with ...
const getParams = {
TableName,
Key: { PK }
}
const result = await docClient.get(getParams).promise();
The result.Item there is a Set object, whereas I would expect it to be the same array I passed into createSet(...).
If interested in seeing this live, this repo has a fully-functioning example. Clone it, npm install, and run index.js and you'll see something like:
$ ./index.js
Running On: darwin 19.6.0
Node version: v12.20.0
AWS SDK version: 2.799.0
-------------------------
Creating table "StringSets-Example"
Waiting for "StringSets-Example" status to be "ACTIVE"
Table status is: CREATING
Table status is: ACTIVE
Put String Set "["Peter, "Paul, "Mary"]" into "StringSets-Example" with key "Names" and attribute "names"
Retrieved Item with key "Names" from "StringSets-Example"
The raw Item: {
PK: 'Names',
names: Set {
wrapperName: 'Set',
values: [ 'Mary', 'Paul', 'Peter' ],
type: 'String'
}
}
The raw Item.names.values: [ 'Mary', 'Paul', 'Peter' ]
-------------------------
Done. To clean up, run:
./src/deleteTable.js
The best solution I have here is to avoid the DocumentClient and the createSet(...) method. Here's a sample using AWS SDK V3:
const key = { PK: `SampleNames`, SK: `SampleNames` };
const names = new Set([`Peter`, `Paul`, `Mary`]);
const item = { ...key, names };
const marshalledItem = marshall(item);
console.log(`Raw item: ${inspect(item)}`)
console.log(`Marshalled item to PUT: ${inspect(marshalledItem, { depth: 4 })}`)
const client = new DynamoDBClient({});
await client.send(new PutItemCommand({
TableName: tableName,
Item: marshalledItem,
}));
const { Item } = await client.send(new GetItemCommand({
TableName: tableName,
Key: marshall(key),
}));
console.log(`Returned item: ${inspect(Item, { depth: 4 })}`);
console.log(`Unmarshalled returned item: ${inspect(unmarshall(Item))}`);
The console output from there is:
Raw item: {
PK: 'SampleNames',
SK: 'SampleNames',
names: Set { 'Peter', 'Paul', 'Mary' }
}
Marshalled item to PUT: {
PK: { S: 'SampleNames' },
SK: { S: 'SampleNames' },
names: { SS: [ 'Peter', 'Paul', 'Mary' ] }
}
Returned item: {
PK: { S: 'SampleNames' },
SK: { S: 'SampleNames' },
names: { SS: [ 'Mary', 'Paul', 'Peter' ] }
}
Unmarshalled returned item: {
PK: 'SampleNames',
SK: 'SampleNames',
names: Set { 'Mary', 'Paul', 'Peter' }
}
... which makes a lot more sense to me. I expect using the marshall/unmarshall methods from AWS SDK V2's Converter module would work similarly.
Related
I am using nodejs aws-sdk/clients/dynamodb library with dynamodb. I need to update an item in the table. Below is the sample code to update an item:
params = {
TableName:table,
Key:{
"year": year,
"title": title
},
UpdateExpression: "set info.rating = info.rating + :val",
ExpressionAttributeValues:{
":val": 1
},
ReturnValues:"UPDATED_NEW"
};
I will have to specify each attribute in info in UpdateExpression. My info object is very big and I am looking for an easier way to do that. Is there a build-in method to support update an object to dynamodb item? something like:
params = {
TableName:table,
Key:{
"year": year,
"title": title
},
Item: info
};
The answer given by E.J. Brennan is great for cases where it's ok to replace the entire item. DocumentClient eases the hassle of dealing with DynamoDB attribute types, but the example given uses the put method. According to the docs put passes through to putItem which
Creates a new item, or replaces an old item with a new item
That means that it's not going to help with partial updates to existing items where you don't already have the full record (and can get away with a full replacement). For partial updates you have to use updateItem, or it's DocumentClient counterpart, update.
The AWS labs has published a utility to help with constructing update expressions to use with updateItem. Since I generally prefer to use DocumentClient, I unmarshall values with the utility function provided by DynamoDB's Converter (yes, I know it's a bit a back and forth, but it makes testing easier).
const AWS = require('aws-sdk');
const db = new AWS.DynamoDB.DocumentClient();
const { UpdateExpression, ExpressionAttributes } = require('#aws/dynamodb-expressions');
const { unmarshall } = AWS.DynamoDB.Converter;
const updateExpressionProps = ({ category, classification }) => {
attributes = new ExpressionAttributes();
expression = new UpdateExpression();
expression.set('category', category);
expression.set('classification', classification);
return {
UpdateExpression: expression.serialize(attributes),
ExpressionAttributeNames: attributes.names,
ExpressionAttributeValues: unmarshall(attributes.values),
};
};
const updateRequest = async ({ id, subject, category, classification }) =>
await db
.update({
TableName: 'table-name',
Key: {
id,
subject,
},
...updateExpressionProps({ category, classification }),
})
.promise();
This bit of code only updates the category and classification attributes on the record identified with id and subject without the hassle of manually building a correct UpdateExpression string. This example could easily be generalized into something reusable throughout your project.
I wrote the following utility method to convert a given object to UpdateExpression (SET only), ExpressionAttributeNames, and ExpressionAttributeValues.
const convertToCompositePathObj = (obj: Record<any, any>) => {
const res: Record<string, string | number | []> = {};
const getPropertyPath = (obj: Record<any, any>, current = "") => {
for (let key in obj) {
const value = obj[key];
const newKey = current ? [current, key].join(".") : key;
if (value && typeof value === "object" && !Array.isArray(value)) {
getPropertyPath(value, newKey);
} else res[newKey] = value;
}
};
getPropertyPath(sampleObject);
return res;
};
const generateDynamoDbUpdateExpression = (obj: Record<any, any>) => {
const compositePathObj = convertToCompositePathObj(sampleObject);
let counter = 0;
let updateExpression = "SET ";
const expressionAttNamesMap: any = {};
const expressionAttValuesMap: any = {};
for (let k in compositePathObj) {
const newUpdateExpression = k
.split(".")
.map((item) => {
const attName = `#${item}`;
if (!expressionAttNamesMap[attName]) {
expressionAttNamesMap[attName] = item;
}
return attName;
})
.join(".")
.concat(`= :${counter} AND `);
expressionAttValuesMap[`:${counter}`] = compositePathObj[k];
counter += 1;
updateExpression += newUpdateExpression;
}
updateExpression = updateExpression.substring(0, updateExpression.length - 5);
return {
UpdateExpression: updateExpression,
ExpressionAttributeNames: expressionAttNamesMap,
ExpressionAttributeValues: expressionAttValuesMap
};
};
// example usage:
const sampleObject = {
name: {
first: "John",
last: "Doe"
},
address: {
line1: "123 test st.",
line2: "Apt 123",
city: "Los Angeless",
state: "CA",
zip: 92763
},
phone: 8675768475
};
console.log(generateDynamoDbUpdateExpression(sampleObject));
/**
*****OUTPUT*****
{
UpdateExpression: "SET #name.#first= :0 AND #name.#last= :1 AND #address.#line1= :2 AND #address.#line2= :3 AND #address.#city= :4 AND #address.#state= :5 AND #address.#zip= :6 AND #phone= :7",
ExpressionAttributeNames:{
#name: "name"
#first: "first"
#last: "last"
#address: "address"
#line1: "line1"
#line2: "line2"
#city: "city"
#state: "state"
#zip: "zip"
#phone: "phone"
},
ExpressionAttributeValues:{
:0: "John"
:1: "Doe"
:2: "123 test st."
:3: "Apt 123"
:4: "Los Angeless"
:5: "CA"
:6: 92763
:7: 8675768475
}
}
**/
PS: I wrote this in a hurry so please excuse the formatting and any types.
https://codesandbox.io/s/aws-dynamic-update-expression-set-33tye7?file=/src/index.ts
You could use the Document Client:
Version 2.2.0 of the AWS SDK for JavaScript introduces support for the
document client abstraction in the AWS.DynamoDB namespace. The
document client abstraction makes it easier to read and write data to
Amazon DynamoDB with the AWS SDK for JavaScript. Now you can use
native JavaScript objects without annotating them as AttributeValue
types.
For example:
var docClient = new AWS.DynamoDB.DocumentClient({region: 'us-west-2'});
var params = {
Item: {
hashkey: 'key',
boolAttr: true,
listAttr: [1, 'baz', true]
mapAttr: {
foo: 'bar'
}
},
TableName: 'table'
};
docClient.put(params, function(err, data){
if (err) console.log(err);
else console.log(data);
});
https://aws.amazon.com/blogs/developer/announcing-the-amazon-dynamodb-document-client-in-the-aws-sdk-for-javascript/
I'm trying to return only values where john is found from a DynamoDB database.
I'm able to return values where it contains name: john from a mapped list, however the problem am having is that it appears to also be returning other values as well.
Running select: 'count' returns 1 match which is correct but it doesn't return anything when used.
I'm assuming that count just returns a number and not a specific select where john is matched.
I'm writing this in NodeJS; am hoping someone can help me figure this out.
I know that the value I only want shown are json elements where name: john, anything else I want omitted from being shown.
Here's my result as of right now:
{
"Department": [
{
"employees": [
{
"name": "john"
},
{
"name": "sally"
}
]
}
],
"Count": 1
}
My code:
const AWS = require('aws-sdk'); // eslint-disable-line import/no-extraneous-dependencies
const dc = new AWS.DynamoDB.DocumentClient();
exports.handler = async (event, context, callback) => {
// Construct the params for filtering data through dynamodb
const params = {
FilterExpression: "contains(#department, :employees)",
ExpressionAttributeNames: {
"#department": "employees",
},
ExpressionAttributeValues: {
":employees": {
"name":"john"
}
},
ProjectionExpression: "#department",
TableName: 'mytable',
//Select: 'COUNT'
};
const resultDC = await dc.scan(params).promise();
var items = resultDC.Items;
var count = resultDC.Count;
var returned_list = {
'Department' : items,
'Count' : count,
};
// create a response
const response = {
statusCode: 200,
body: JSON.stringify(returned_list),
};
callback(null, response);
};
I suggest you to use a Local Secondary Index.
Take a look here.
I am attempting to delete data from a DynamoDB table.
If I delete data using a partition key, it works.
But when I delete multiple rows using any other fields, it fails.
var params = {
TableName: "test",
Key: {
dmac: dmac,
},
ConditionExpression: "dmac= :dmac"
};
docClient.delete( params, (error) => {
if (error) {
console.log( "Delete data fail" );
} else {
console.log( "Delete data Success" );
}
});
Items (or rows) in DynamoDB are uniquely identified by their primary key. A table can have a simple primary key (a partition key) or a composite primary key (a partition key plus a sort key).
To delete an item, you must provide the full primary key (whether it's a simple partition key or composite partition key plus sort key).
So, if you want to delete items that meet a specific condition, for example cars with maxspeed < 120, then issue a query or scan to identify those items, retrieve the primary keys, and then delete the items in a second operation.
To delete a single item, use DeleteItem. To delete multiple items, use BatchWriteItem. Despite the naming of BatchWriteItem, it can be used to put multiple items or to delete multiple items, and you can target one or more DynamoDB tables in the same API call.
Here is an AWS SDK v2 example of deleting multiple items:
const aws = require("aws-sdk");
const ddb = new aws.DynamoDB({ region: "us-east-1" });
(async () => {
const params = {
RequestItems: {
albums: []
}
};
params.RequestItems.albums.push({
DeleteRequest: {
Key: {
pk: { S: "The Who" },
sk: { S: "Tommy" }
}
}
});
params.RequestItems.albums.push({
DeleteRequest: {
Key: {
pk: { S: "The Beatles" },
sk: { S: "Abbey Road" }
}
}
});
await ddb.batchWriteItem(params).promise();
})();
Here is an AWS SDK v3 example of deleting multiple items:
const {
BatchWriteItemCommand,
DynamoDBClient
} = require("#aws-sdk/client-dynamodb");
(async () => {
const client = new DynamoDBClient({ region: "us-east-1" });
const params = {
RequestItems: {
albums: []
}
};
params.RequestItems.albums.push({
DeleteRequest: {
Key: {
pk: { S: "The Who" },
sk: { S: "Tommy" }
}
}
});
params.RequestItems.albums.push({
DeleteRequest: {
Key: {
pk: { S: "The Beatles" },
sk: { S: "Abbey Road" }
}
}
});
await client.send(new BatchWriteItemCommand(params));
})();
in DynamoDB you can only delete an item using its key (that is: the partition key and the sort key, if it is defined on the table). This is, for example, underlined by the fact that the Key attribute is a required attribute in the canonical specification of the delete operation. See: https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_DeleteItem.html#DDB-DeleteItem-request-Key
This means that if you want to delete an item using other attributes you must first lookup the item by the attributes you do have, extract the key from the returned item, and then delete the item using that key.
The standard solution for "looking up an item by attributes that are not the item's key" is to define a global secondary index (GSI) on the table with those attribute(s) defined as the GSI's key.
How to update quantity value based on title in the movies array and Item id (123)
I only manage to update value at the first layer like name (David), but don't know how to update the second layer with additional filter for the array (movies).
From:
Item:
{
id: 123,
name: 'David',
movies: [
{
id: 1,
title: 'The lord of the ring',
quantity: 1
},
{
id: 2,
title: 'Star Wars',
quantity: 1
}
]
}
To:
Item:
{
id: 123,
Name: 'David',
movies: [
{
id: 1,
title: 'The lord of the ring',
quantity: 2
},
{
id: 2,
title: 'Star Wars',
quantity: 1
}
]
}
By the way, I'm using aws DynamoDB document client in node.js, it will be nice if you can share me how you do it in your update parameter.
There is no way to update an object inside of a list without replacing it.
You probably want to restructure your table to emulate a relational data model. AWS has some documentation on this.
As an example, create your table like this:
aws dynamodb create-table \
--table-name movie-table \
--attribute-definitions AttributeName=rId,AttributeType=N AttributeName=rKey,AttributeType=S \
--key-schema AttributeName=rId,KeyType=HASH AttributeName=rKey,KeyType=RANGE
The table will have generically named hash and range keys. This script demonstrates how to structure the data and add to the "count":
const { DynamoDB } = require('aws-sdk');
const client = new DynamoDB.DocumentClient({ region: 'us-east-1' });
const addItem = (rId, rKey, attributes) => {
const item = { rId, rKey };
Object.assign(item, attributes);
return client.put({ TableName: 'movie-table', Item: item }).promise();
};
// NOTE: this is where the count attribute gets iterated
const addToCount = (rId, rKey) => client.update({
TableName: 'movie-table',
Key: { rId, rKey },
UpdateExpression: 'ADD #count :n',
ExpressionAttributeNames: { '#count': 'count' },
ExpressionAttributeValues: { ':n': 1 },
}).promise();
const run = async () => {
await addItem(123, 'USER|123', { name: 'David' });
await addItem(1, 'MOVIE|1', { title: 'The lord of the ring' });
await addItem(2, 'MOVIE|2', { title: 'Star Wars' });
await addItem(123, 'COUNT|1', { count: 1 });
await addItem(123, 'COUNT|2', { count: 1 });
await addToCount(123, 'COUNT|1');
};
run();
This is what the table looks like after the script runs:
I know this is a bit old but there is a way. Using the document client SDK, you can reference object properties and array elements in the UpdateExpression. However, you can't run any logic so you have to know/assume/expect that the element indexes are enough.
For example, you can do something like this:
let params = {
TableName: 'your-table-name',
Key: { id: 123 },
UpdateExpression: 'set movies[0].quantity = :x',
ExpressionAttributeValues: { ':x': 5 }
};
const client = AWS.DynamoDB.DocumentClient();
client.update(params);
NOTE: You cannot make the index an Expression Attribute Value. You would have to dynamically build that update expression based on the index you know has to be updated. It's not a perfect solution but it could get the job done.
For reference, I derived this from the base (non-DocumentClient) example from here: Adding Nested Map Attributes
Struggling to find an example of how to query a table to return rows with ids from a given list.
The query below throws on the inclusion of IN
var params = {
id: '7deb3df9-552b-47a4-aef3-ad601f141d50'
};
var p = {
TableName: 'players',
KeyConditionExpression: 'id IN (:id)',
ExpressionAttributeValues: buildQuery(params)
};
You can't use "IN" operator with KeyConditionExpression, please see details in this SO question
You may want to use batchGetItem instead of query, which is not so efficient though.
Here is how your params could look like:
var params = {
RequestItems: {
'players': {
Keys: [{
id: "7deb3df9-552b-47a4-aef3-ad601f141d50",
rangeKey: "<range key 1>" // <--- if your table has a range key, you must specify its value here
}, {
id: "<ANOTHER ID 2>",
rangeKey: "<range key 2>"
}, {
id: "<ANOTHER ID 3>",
rangeKey: "<range key 3>"
}]
}
}
};
dynamodbDoc.batchGet(params, function(err, data) {
});