Design DynamoDB for Large data using array. Over 400k - node.js

Question

I'm worked with node.js and for record event-time and event-data, I want to use AWS DynamoDB Table.
single data and one-key-data schema sample like under.
var want_to_update_data = [ // contain json more or same than 1
{ "event-time" : "2021-01-02 10:11:12", "event-data" : 19 },
...
];
var saved_data = { // DynamoDB Saved Data Sample with Key "ABCD"
"Key" : "ABCD",
"events" : [
...
{ "event-time" : "2020-12-28 09:30:17", "event-data" : 35 },
{ "event-time" : "2021-01-01 19:11:12", "event-data" : 16 },
{ "event-time" : "2021-01-02 10:11:12", "event-data" : 19 },
...
]
}
Upper data is only for sample. json contains more key-values.
For update single json, my server code likes under.
app.get('/insert', req, res) {
var update_key = req.body.update_key; // "ABCD"
var want_to_update_data = req.body.update_data; // [ {} ]
var updateExpression = 'SET events = list_append(events, :dl)'; // dl means data-list
var expressionAttributeValues = {':dl' : want_to_update_data }; // dl means data-list
AWS.config.update({
region: "region",
endpoint: "endpoint",
accessKeyId: "access_key",
secretAccessKey: "secret_key",
});
var params = {
TableName: "TableName",
Key: update_key,
UpdateExpression: updateExpression,
ExpressionAttributeValues: expressionAttributeValues,
ReturnValues: "UPDATED_NEW"
});
var dynamodbClient = new AWS.DynamoDB.DocumentClient();
dynamodbClient.update(params, function(err, data) {
if(err) { /* handle error */ }
else { /* handle after update complete */ }
});
});
Simply, It works, but AWS DynamoDB can't change over 4kb at once.
So, if array data is bigger than 4k, list_append not worked with ValidationException: Item size to update has exceeded the maximum allowed size error message.
I want to remain all events and want to sort event-time keys.
How can I change my DynamoDB schema?
Under code is just my guess.
Is it possible?
{
"Key" : "ABCD",
/*
"events" : [
...
{ "event-time" : "2020-12-28 09:30:17", "event-data" : 35 },
{ "event-time" : "2021-01-01 19:11:12", "event-data" : 16 },
{ "event-time" : "2021-01-02 10:11:12", "event-data" : 19 },
...
]
*/
"events" : { // is it possible events['some-date'] could be sort-key?
...
"2020-12-28 09:30:17" : 35,
"2021-01-01 19:11:12" : 16,
"2021-01-02 10:11:12" : 19,
...
}
}
/*
var updateExpression = 'SET events = list_append(events, :dl)'; // dl means data-list
var expressionAttributeValues = {':dl' : want_to_update_data }; // dl means data-list
*/
// under 2 line is just for sample. json key is not fixed
var event_time = '2021-01-02 10:11:12';
var single_json = want_to_update_data[event_time];
// if number-start key is not allow(event_time), prefix append could be like "D_2021~~"
var updateExpression = `SET events.${event_time} = :dl`;
var expressionAttributeValues = {':dl' : single_json }; // single json
Thank you for read my question.
I want to create new table for save my events json datas.
(not S3. using DynamoDB)

Answer 1

DynamoDB has an item size limit of 400kb, including the attribute names.
According to the docs
Number of Values in List, Map, or Set
There is no limit on the number of values in a List, a Map, or a Set, as long as the item containing the values fits within the 400 KB item size limit.
It sounds like your list of 4k events, plus whatever else you're saving in the item, exceeds this 400kb limit. You can check out this nifty calculator to get an idea of the size of your item.
Instead of storing events in a list item, you might want to store them in an item collection.
For example, instead of this:
You could store event data like this
If you need to fetch events by time, you might consider making the event_time your sort key.

Answer 2

You cannot store infinitely large amounts of data in a DynamoDB on single Key.
Just like what you wrote, periodically back up to S3.
In my case, I use redis-cache for check each array size and with node-schedule, back-up large datas to S3.