Add a string to a list in a dynamodb table - node.js

So I have been working with dynamodb in a nodejs express app and I have a specific table that has a field which is just empty lists and I want to append a string to the list.
The table name is "dev_entrants" and here is an example of the table:
----------------------------------------
primary Key Sort Key
eventID | eventType | entrants
----------------------------------------
Qual-919-w5wm1xhnw | Qual | []
----------------------------------------
So I receive a post request and then route it through express and it comes to a function where after doing type checks and all that I try to add stuff to my table with:
import AWS from 'aws-sdk';
const docClient = new AWS.DynamoDB.DocumentClient({region: 'us-west-1'});
const db = {
docClient
};
...
const entrantsParams = {
'TableName' : 'dev_entrants',
'Key': {
'eventID' : 'Qual-919-w5wm1xhnw',
},
'UpdateExpression' : "SET #attrName = list_append(#attrName, :attrValue)",
'ExpressionAttributeNames' : {
'#attrName' : 'entrants'
},
'ExpressionAttributeValues' : {
':attrValue' : ['joe'],
}
};
const updateEntrantsPromise = db.docClient.update(entrantsParams).promise();
(For the purpose of this example I have replaced variables with the strings they represent)
I have spent 6 hours or so reading through different documentation, as well as on stack overflow trying to find the answer.
The current error i get is the provided key element does not match the schema. If I remove the brackets around the attrValue then I get wrong operand type. I know the key exists in the table as i copied and pasted it from there. Also I am succesfully adding things to the table from another function so my connection is working fine. Can anyone please help me out?

You need to include the eventType in the Key object because your table schema has a sort key. If your table has a sort/partition key then you need to include it along with the primary key. Try it with the following:
const entrantsParams = {
'TableName' : 'dev_entrants',
'Key': {
'eventID' : 'Qual-919-w5wm1xhnw',
'eventType' : 'Qual'
},
'UpdateExpression' : "SET #attrName = list_append(if_not_exists(#attrName, :empty_list), :attrValue)",
'ExpressionAttributeNames' : {
'#attrName' : 'entrants'
},
'ExpressionAttributeValues' : {
':attrValue' : ['joe'],
':empty_list': []
}
};

Related

dynamodb query: ValidationException: The number of conditions on the keys is invalid

I have the following schema where I am basically just trying to have a table with id as primary key, and both code and secondCode to be global secondary indexes to use to query the table.
resource "aws_dynamodb_table" "myDb" {
name = "myTable"
billing_mode = "PAY_PER_REQUEST"
hash_key = "id"
attribute {
name = "id"
type = "S"
}
attribute {
name = "code"
type = "S"
}
attribute {
name = "secondCode"
type = "S"
}
global_secondary_index {
name = "code-index"
hash_key = "code"
projection_type = "ALL"
}
global_secondary_index {
name = "second_code-index"
hash_key = "secondCode"
projection_type = "ALL"
}
}
When I try to look for one item by code
const toGet = Object.assign(new Item(), {
code: 'code_456',
});
item = await dataMapper.get<Item>(toGet);
locally I get
ValidationException: The number of conditions on the keys is invalid
and on the deployed instance of the DB I get
The provided key element does not match the schema
I can see from the logs that the key is not being populated
Serverless: [AWS dynamodb 400 0.082s 0 retries] getItem({ TableName: 'myTable', Key: {} })
Here is the class configuration for Item
#table(getEnv('MY_TABLE'))
export class Item {
#hashKey({ type: 'String' })
id: string;
#attribute({
indexKeyConfigurations: { 'code-index': 'HASH' },
type: 'String',
})
code: string;
#attribute({
indexKeyConfigurations: { 'second_code-index': 'HASH' },
type: 'String',
})
secondCode: string;
#attribute({ memberType: embed(NestedItem) })
nestedItems?: Array<NestedItem>;
}
class NestedItem {
#attribute()
name: string;
#attribute()
price: number;
}
I am using https://github.com/awslabs/dynamodb-data-mapper-js
I looked at the repo you linked for the package, I think you need to use the .query(...) method with the indexName parameter to tell DynamoDB you want to use that secondary index. Usuallly in DynamoDB, get operations use the default keys (in your case, you'd use get for queries on id, and query for queries on indices).
Checking the docs, it's not very clear - if you look at the GetItem reference, you'll see there's nowhere to supply an index name to actually use the index, whereas the Query operation allows you to supply one. As for why you need to query this way, you can read this: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.CoreComponents.html
The issue you are facing is due to calling a GetItem on an index, which is not possible. A GetItem must target a single item and an index can contain multiple items with the same key (unlike the base table), for this reason you can only use multi-item APIs on an index which are Query and Scan.

Design DynamoDB for Large data using array. Over 400k

I'm worked with node.js and for record event-time and event-data, I want to use AWS DynamoDB Table.
single data and one-key-data schema sample like under.
var want_to_update_data = [ // contain json more or same than 1
{ "event-time" : "2021-01-02 10:11:12", "event-data" : 19 },
...
];
var saved_data = { // DynamoDB Saved Data Sample with Key "ABCD"
"Key" : "ABCD",
"events" : [
...
{ "event-time" : "2020-12-28 09:30:17", "event-data" : 35 },
{ "event-time" : "2021-01-01 19:11:12", "event-data" : 16 },
{ "event-time" : "2021-01-02 10:11:12", "event-data" : 19 },
...
]
}
Upper data is only for sample. json contains more key-values.
For update single json, my server code likes under.
app.get('/insert', req, res) {
var update_key = req.body.update_key; // "ABCD"
var want_to_update_data = req.body.update_data; // [ {} ]
var updateExpression = 'SET events = list_append(events, :dl)'; // dl means data-list
var expressionAttributeValues = {':dl' : want_to_update_data }; // dl means data-list
AWS.config.update({
region: "region",
endpoint: "endpoint",
accessKeyId: "access_key",
secretAccessKey: "secret_key",
});
var params = {
TableName: "TableName",
Key: update_key,
UpdateExpression: updateExpression,
ExpressionAttributeValues: expressionAttributeValues,
ReturnValues: "UPDATED_NEW"
});
var dynamodbClient = new AWS.DynamoDB.DocumentClient();
dynamodbClient.update(params, function(err, data) {
if(err) { /* handle error */ }
else { /* handle after update complete */ }
});
});
Simply, It works, but AWS DynamoDB can't change over 4kb at once.
So, if array data is bigger than 4k, list_append not worked with ValidationException: Item size to update has exceeded the maximum allowed size error message.
I want to remain all events and want to sort event-time keys.
How can I change my DynamoDB schema?
Under code is just my guess.
Is it possible?
{
"Key" : "ABCD",
/*
"events" : [
...
{ "event-time" : "2020-12-28 09:30:17", "event-data" : 35 },
{ "event-time" : "2021-01-01 19:11:12", "event-data" : 16 },
{ "event-time" : "2021-01-02 10:11:12", "event-data" : 19 },
...
]
*/
"events" : { // is it possible events['some-date'] could be sort-key?
...
"2020-12-28 09:30:17" : 35,
"2021-01-01 19:11:12" : 16,
"2021-01-02 10:11:12" : 19,
...
}
}
/*
var updateExpression = 'SET events = list_append(events, :dl)'; // dl means data-list
var expressionAttributeValues = {':dl' : want_to_update_data }; // dl means data-list
*/
// under 2 line is just for sample. json key is not fixed
var event_time = '2021-01-02 10:11:12';
var single_json = want_to_update_data[event_time];
// if number-start key is not allow(event_time), prefix append could be like "D_2021~~"
var updateExpression = `SET events.${event_time} = :dl`;
var expressionAttributeValues = {':dl' : single_json }; // single json
Thank you for read my question.
I want to create new table for save my events json datas.
(not S3. using DynamoDB)
DynamoDB has an item size limit of 400kb, including the attribute names.
According to the docs
Number of Values in List, Map, or Set
There is no limit on the number of values in a List, a Map, or a Set, as long as the item containing the values fits within the 400 KB item size limit.
It sounds like your list of 4k events, plus whatever else you're saving in the item, exceeds this 400kb limit. You can check out this nifty calculator to get an idea of the size of your item.
Instead of storing events in a list item, you might want to store them in an item collection.
For example, instead of this:
You could store event data like this
If you need to fetch events by time, you might consider making the event_time your sort key.
You cannot store infinitely large amounts of data in a DynamoDB on single Key.
Just like what you wrote, periodically back up to S3.
In my case, I use redis-cache for check each array size and with node-schedule, back-up large datas to S3.

DynamoDB Scan with FilterExpression in nodejs

I'm trying to retrieve all items from a DynamoDB table that match a FilterExpression, and although all of the items are scanned and half do match, the expected items aren't returned.
I have the following in an AWS Lambda function running on Node.js 6.10:
var AWS = require("aws-sdk"),
documentClient = new AWS.DynamoDB.DocumentClient();
function fetchQuotes(category) {
let params = {
"TableName": "quotient-quotes",
"FilterExpression": "category = :cat",
"ExpressionAttributeValues": {":cat": {"S": category}}
};
console.log(`params=${JSON.stringify(params)}`);
documentClient.scan(params, function(err, data) {
if (err) {
console.error(JSON.stringify(err));
} else {
console.log(JSON.stringify(data));
}
});
}
There are 10 items in the table, one of which is:
{
"category": "ChuckNorris",
"quote": "Chuck Norris does not sleep. He waits.",
"uuid": "844a0af7-71e9-41b0-9ca7-d090bb71fdb8"
}
When testing with category "ChuckNorris", the log shows:
params={"TableName":"quotient-quotes","FilterExpression":"category = :cat","ExpressionAttributeValues":{":cat":{"S":"ChuckNorris"}}}
{"Items":[],"Count":0,"ScannedCount":10}
The scan call returns all 10 items when I only specify TableName:
params={"TableName":"quotient-quotes"}
{"Items":[<snip>,{"category":"ChuckNorris","uuid":"844a0af7-71e9-41b0-9ca7-d090bb71fdb8","CamelCase":"thevalue","quote":"Chuck Norris does not sleep. He waits."},<snip>],"Count":10,"ScannedCount":10}
You do not need to specify the type ("S") in your ExpressionAttributeValues because you are using the DynamoDB DocumentClient. Per the documentation:
The document client simplifies working with items in Amazon DynamoDB by abstracting away the notion of attribute values. This abstraction annotates native JavaScript types supplied as input parameters, as well as converts annotated response data to native JavaScript types.
It's only when you're using the raw DynamoDB object via new AWS.DynamoDB() that you need to specify the attribute types (i.e., the simple objects keyed on "S", "N", and so on).
With DocumentClient, you should be able to use params like this:
const params = {
TableName: 'quotient-quotes',
FilterExpression: '#cat = :cat',
ExpressionAttributeNames: {
'#cat': 'category',
},
ExpressionAttributeValues: {
':cat': category,
},
};
Note that I also moved the field name into an ExpressionAttributeNames value just for consistency and safety. It's a good practice because certain field names may break your requests if you do not.
I was looking for a solution that combined KeyConditionExpression with FilterExpression and eventually I worked this out.
Where aws is the uuid. Id is an assigned unique number preceded with the text 'form' so I can tell I have form data, optinSite is so I can find enquiries from a particular site. Other data is stored, this is all I need to get the packet.
Maybe this can be of help to you:
let optinSite = 'https://theDomainIWantedTFilterFor.com/';
let aws = 'eu-west-4:EXAMPLE-aaa1-4bd8-9ean-1768882l1f90';
let item = {
TableName: 'Table',
KeyConditionExpression: "aws = :Aw and begins_with(Id, :form)",
FilterExpression: "optinSite = :Os",
ExpressionAttributeValues: {
":Aw" : { S: aws },
":form" : { S: 'form' },
":Os" : { S: optinSite }
}
};

How to add a number to a DynamoDB number set using Node.js

I'm just trying to add a number to a number set in DynamoDB. This expression was working with an untyped list. But to save space since it will just be storing numbers I moved everything to plain number sets. Now no matter how much I tinker with it I can't get it to go through.
var phoneID = req.body.PhoneID;
var category = req.body.ratingCategory;
var ratingToAdd = [Number(req.body.rating)]
var dbparams = {
"TableName": "Venue_Ratings",
Key: {
"PhoneID" : phoneID
},
"UpdateExpression": "SET #categoryName = list_append(#categoryName, :rating)",
"ExpressionAttributeNames" : {
"#categoryName" : category
},
"ExpressionAttributeValues": {
":rating": ratingToAdd
},
"ReturnValues": "ALL_NEW"
};
This error is being thrown An operand in the update expression has an incorrect data type
I have also tried changing the update expression to an ADD expression instead like so ADD #categoryName :rating.
I've tried changing ratingToAdd to a plain number not in an array, a string in an array, and a plain string not in an array.
I'm calling the db using the docClient.update method.
I have verified that the sets in the db are in fact number sets and that they exist.
What am I missing here? Thanks for the help.
The below code should add the number to the Number Set (i.e. DynamoDB data type 'NS').
Use this function with ADD in UpdateExpression:-
docClient.createSet([Number(5)])
Code:-
var params = {
TableName : "Movies",
Key : {
"yearkey" : 2016,
"title" : "The Big New Movie 1"
},
UpdateExpression : "ADD #category :categorySet",
ExpressionAttributeNames: {
'#category' : 'category'
},
ExpressionAttributeValues: {':categorySet' : docClient.createSet( [Number(5)])},
ReturnValues: 'UPDATED_NEW'
};

how to use in operator in dynamo db

I have a user table with a field username. I need to write something equivalent to this in dynamo db: Select * from user where username in('a','b','c');
Adding more from code prosepective i have usernames in an array say var arr=['a','b','c'];
I so far tried this which is giving me zero result
this.dynamo.client.scanAsync({
TableName: this.dynamo.table('users'),
FilterExpression: 'username IN (:list)',
ExpressionAttributeValues: {
':list': arr.toString()
}
}).then((response) => {
console.log(response);
return {
userFriends: result.Item.friends
};
});
When I pass one element in array it give me result searching passed single element in user table but its not working with more than one element in array.
The individual users should be given as comma separated String variables. JavaScript array is equivalent to List in AWS DynamoDB data type. The DynamoDB can't compare the String data type in database with List attribute (i.e. Array in JavaScript).
var params = {
TableName : "Users",
FilterExpression : "username IN (:user1, :user2)",
ExpressionAttributeValues : {
":user1" : "john",
":user2" : "mike"
}
};
Construct the object from array for FilterExpression:-
Please refer the below code for forming the object dynamically based on Array value.
var titleValues = ["The Big New Movie 2012", "The Big New Movie"];
var titleObject = {};
var index = 0;
titleValues.forEach(function(value) {
index++;
var titleKey = ":titlevalue"+index;
titleObject[titleKey.toString()] = value;
});
var params = {
TableName : "Movies",
FilterExpression : "title IN ("+Object.keys(titleObject).toString()+ ")",
ExpressionAttributeValues : titleObject
};
Note:-
I don't think IN clause with 1000s of usernames is a good idea in terms of performance.

Resources