I am trying to delete and then create a DynamoDB table using nodejs aws sdk (version 3.142.0) and I wanted to use the waiters (waitUntilTableNotExists / waitUntilTableExists), but I don't understand how they are supposed to be used and I cannot find a good example online.
Regards
Here is one way after a createTable command in aws-sdk-js-v3 to wait for the table to complete. A note is that if you do NOT use waitUntilTableExists and instead attempt to use DescribeTableCommand it will incorrectly report TableStatus == 'ACTIVE' even though you cannot Read/Write to the table, you must use waitUntilTableExists.
import {
CreateTableCommandInput,
CreateTableCommandOutput,
waitUntilTableExists
} from "#aws-sdk/client-dynamodb";
const client = new DynamoDBClient({ region: "us-east-1" });
const data = await client.send(
new CreateTableCommand({
TableName: tableName,
AttributeDefinitions: partitionAndSortKeyDefinitions,
KeySchema: columnSchema,
ProvisionedThroughput: {
ReadCapacityUnits: 4,
WriteCapacityUnits: 2,
},
})
);
// maxWaitTime - seconds
const results = await waitUntilTableExists({client: client, maxWaitTime: 120}, {TableName: tableName})
if (results.state == 'SUCCESS') {
return results.reason.Table
}
console.error(`${results.state} ${results.reason}`);
Related
I want to get data form dynamoDB, shorted by timestamp. Anyone can help? My code is given below.
const AWS = require("aws-sdk");
const dynamoDbClient = new AWS.DynamoDB.DocumentClient();
const USERS_TABLE = process.env.USERS_TABLE;
const getNews = async (req, res) => {
try {
//dynamodb params
const params = {
TableName: USERS_TABLE,
FilterExpression: "PK = :this",
ExpressionAttributeValues: { ":this": "newsTable" },
};
//get dynamodb data
const data = await dynamoDbClient.scan(params).promise();
res.status(200).send({ data: data });
} catch (e) {
return res.status(400).send({ message: e.message });
}
};
module.exports = { getNews };
Option 1: Keep Scan; Sort client-side
Works for small tables only. Single Scan call will scan only the first 1 MB of data in the table.
If you're doing scan operation as in your code example, it's impossible to get results sorted from DynamoDB. The only way to sort them is on client-side after you download all your data from database.
Replace:
res.status(200).send({ data: data });
With:
res.status(200).send({data: data.sort((a, b) => b.date - a.date)});
However, this is not recommended, since Scan operation without pagination will scan only 1st MB of data in your table. So you could get partial results. Possible solutions are:
Option 2: (recommended) Don't use Scan; Use Query; Sort by secondary key
This will work if you have your timestamp in the secondary key of the table
Don't use Scan; Use Query -- that way you can sort your data by SK (secondary key) by passing the ScanIndexForward: false to get the most recent results first.
Assuming you have such a table schema, where a timestamp is in the secondary key:
PK
SK
email
newsTable
2022-01-01
some-1#example.com
newsTable
2022-02-01
some-2#example.com
newsTable
2022-03-01
some-3#example.com
You can change your code from:
const params = {
TableName: USERS_TABLE,
FilterExpression: 'PK = :this',
ExpressionAttributeValues: {':this': 'newsTable'},
};
//get dynamodb data
const data = await dynamoDbClient.scan(params).promise();
To:
const params = {
TableName: USERS_TABLE,
KeyConditionExpression: 'PK = :this',
ExpressionAttributeValues: {':this': 'newsTable'},
ScanIndexForward: false,
};
//get dynamodb data
const data = await dynamoDbClient.query(params).promise();
And it will return results sorted from database already.
If you don't have a timestamp in your secondary key, and you cannot add it, you can add Local Secondary Index or Global Secondary Index.
Option 3: Keep Scan, but paginate; Sort client-side
Works if you cannot change DB schema and cannot switch your code to the Query operation.
Beware, it will be much more expensive, much slower. The larger the table, the slower it gets.
If you absolutely need to use Scan, you need to paginate through all the pages of the Scan operation, and then sort results in the JS code, like I described before. I've developed a handy library that makes scanning in parallel and supports pagination.
I have a NodeJS function that scan a table in DynamoDB (without primary sort key) and return the number of elements of the column sync that are null.
My table:
var params = {
AttributeDefinitions: [
{
AttributeName: "barname",
AttributeType: "S"
},
{
AttributeName: "timestamp",
AttributeType: "S"
}
],
KeySchema: [
{
AttributeName: "barname",
KeyType: "HASH"
},
{
AttributeName: "timestamp",
KeyType: "RANGE"
}
],
ProvisionedThroughput: {
ReadCapacityUnits: 1,
WriteCapacityUnits: 1
},
TableName: tableName
};
The function that count when sync==false
var dynamodb = new AWS.DynamoDB({apiVersion: '2012-08-10'});
async function getCountNoSync(type){
console.log(type)
var params = {
TableName: tableName,
FilterExpression: 'sync = :sync and billing = :billing',
ExpressionAttributeValues: {
':billing' : {S: type},
':sync' : {BOOL: false}
},
};
var count = 0;
await dynamodb.scan(params).promise()
.then(function(data){
count = data.Count;
})
.catch(function(err) {
count = 0;
console.log(err);
});
return count;
}
The function works fine If a have few elements in my table (eg. less than 150). If the number of elements are higher, the count variable is always 0. It loooks like the scan do not find all elements.
Any ideia?
Best regards
The reason that you do not find all the items where attribute sync == null is that the scan operation is only reading part of your table.
As the documentation states:
If the total number of scanned items exceeds the maximum dataset size limit of 1 MB, the scan stops and results are returned to the user as a LastEvaluatedKey value to continue the scan in a subsequent operation.
So if your table is several hundred of megabytes big, you need to call scan() multiple times and provide the LastEvaluatedKey to read the next "page" of your table. This process is also called "pagination".
But this will take a lot of time and the time this needs will just increase with your table size. The proper way of doing this would be to create an index of the sync field and then do a query() on that index.
You can read more about that in the AWS documentation:
Querying and Scanning a DynamoDB Table
Reference documentation for scan()
Paginating the Results
Running a Node.js serverless backend through AWS.
Main objective: to filter and list all LOCAL jobs (table items) that included the available services and zip codes provided to the filter.
Im passing in multiple zip codes, and multiple available services.
data.radius would be an array of zip codes = to something like this:[ '93901', '93902', '93905', '93906', '93907', '93912', '93933', '93942', '93944', '93950', '95377', '95378', '95385', '95387', '95391' ]
data.availableServices would also be an array = to something like this ['Snow removal', 'Ice Removal', 'Salting', 'Same Day Response']
I am trying to make an API call that returns only items that have a matching zipCode from the array of zip codes provided by data.radius, and the packageSelected has a match of the array data.availableServices provided.
API CALL
import * as dynamoDbLib from "./libs/dynamodb-lib";
import { success, failure } from "./libs/response-lib";
export async function main(event, context) {
const data = JSON.parse(event.body);
const params = {
TableName: "jobs",
FilterExpression: "zipCode = :radius, packageSelected = :availableServices",
ExpressionAttributeValues: {
":radius": data.radius,
":availableServices": data.availableServices
}
};
try {
const result = await dynamoDbLib.call("query", params);
// Return the matching list of items in response body
return success(result.Items);
} catch (e) {
return failure({ status: false });
}
Do I need to map the array of zip codes and available services first for this to work?
Should I be using comparison operators?
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/LegacyConditionalParameters.QueryFilter.html
Is a sort key value or partition key required to query and filter? (the table has a sort key and partition key but i would like to avoid using them in this call)
Im not 100% sure on how to go about this so if anyone could point me in the right direction that would be wonderful and greatly appreciated!!
I'm not sure what your dynamodb-lib refers to but here's an example of how you can scan for attribute1 in a given set of values and attribute2 in a different set of values. This uses the standard AWS JavaScript SDK, and specifically the high-level document client.
Note that you cannot use an equality (==) test here, you have to use an inclusion (IN) test. And you cannot use query, but must use scan.
const AWS = require('aws-sdk');
let dc = new AWS.DynamoDB.DocumentClient({'region': 'us-east-1'});
const data = {
radius: [ '93901', '93902', '93905', '93906', '93907', '93912', '93933', '93942', '93944', '93950', '95377', '95378', '95385', '95387', '95391' ],
availableServices: ['Snow removal', 'Ice Removal', 'Salting', 'Same Day Response'],
};
// These hold ExpressionAttributeValues
const zipcodes = {};
const services = {};
data.radius.forEach((zipcode, i) => {
zipcodes[`:zipcode${i}`] = zipcode;
})
data.availableServices.forEach((service, i) => {
services[`:services${i}`] = service;
})
// These hold FilterExpression attribute aliases
const zipcodex = Object.keys(zipcodes).toString();
const servicex = Object.keys(services).toString();
const params = {
TableName: "jobs",
FilterExpression: `zipCode IN (${zipcodex}) AND packageSelected IN (${servicex})`,
ExpressionAttributeValues : {...zipcodes, ...services},
};
dc.scan(params, (err, data) => {
if (err) {
console.log('Error', err);
} else {
for (const item of data.Items) {
console.log('item:', item);
}
}
});
I'm trying to retrieve all items from a DynamoDB table that match a FilterExpression, and although all of the items are scanned and half do match, the expected items aren't returned.
I have the following in an AWS Lambda function running on Node.js 6.10:
var AWS = require("aws-sdk"),
documentClient = new AWS.DynamoDB.DocumentClient();
function fetchQuotes(category) {
let params = {
"TableName": "quotient-quotes",
"FilterExpression": "category = :cat",
"ExpressionAttributeValues": {":cat": {"S": category}}
};
console.log(`params=${JSON.stringify(params)}`);
documentClient.scan(params, function(err, data) {
if (err) {
console.error(JSON.stringify(err));
} else {
console.log(JSON.stringify(data));
}
});
}
There are 10 items in the table, one of which is:
{
"category": "ChuckNorris",
"quote": "Chuck Norris does not sleep. He waits.",
"uuid": "844a0af7-71e9-41b0-9ca7-d090bb71fdb8"
}
When testing with category "ChuckNorris", the log shows:
params={"TableName":"quotient-quotes","FilterExpression":"category = :cat","ExpressionAttributeValues":{":cat":{"S":"ChuckNorris"}}}
{"Items":[],"Count":0,"ScannedCount":10}
The scan call returns all 10 items when I only specify TableName:
params={"TableName":"quotient-quotes"}
{"Items":[<snip>,{"category":"ChuckNorris","uuid":"844a0af7-71e9-41b0-9ca7-d090bb71fdb8","CamelCase":"thevalue","quote":"Chuck Norris does not sleep. He waits."},<snip>],"Count":10,"ScannedCount":10}
You do not need to specify the type ("S") in your ExpressionAttributeValues because you are using the DynamoDB DocumentClient. Per the documentation:
The document client simplifies working with items in Amazon DynamoDB by abstracting away the notion of attribute values. This abstraction annotates native JavaScript types supplied as input parameters, as well as converts annotated response data to native JavaScript types.
It's only when you're using the raw DynamoDB object via new AWS.DynamoDB() that you need to specify the attribute types (i.e., the simple objects keyed on "S", "N", and so on).
With DocumentClient, you should be able to use params like this:
const params = {
TableName: 'quotient-quotes',
FilterExpression: '#cat = :cat',
ExpressionAttributeNames: {
'#cat': 'category',
},
ExpressionAttributeValues: {
':cat': category,
},
};
Note that I also moved the field name into an ExpressionAttributeNames value just for consistency and safety. It's a good practice because certain field names may break your requests if you do not.
I was looking for a solution that combined KeyConditionExpression with FilterExpression and eventually I worked this out.
Where aws is the uuid. Id is an assigned unique number preceded with the text 'form' so I can tell I have form data, optinSite is so I can find enquiries from a particular site. Other data is stored, this is all I need to get the packet.
Maybe this can be of help to you:
let optinSite = 'https://theDomainIWantedTFilterFor.com/';
let aws = 'eu-west-4:EXAMPLE-aaa1-4bd8-9ean-1768882l1f90';
let item = {
TableName: 'Table',
KeyConditionExpression: "aws = :Aw and begins_with(Id, :form)",
FilterExpression: "optinSite = :Os",
ExpressionAttributeValues: {
":Aw" : { S: aws },
":form" : { S: 'form' },
":Os" : { S: optinSite }
}
};
I understand that I can create "Lists" only from primitive data types, so look at my (Node.js using AWS Document Client) code as pseudo code. My objective is to attach a JSON array to an item so that I can later retrieve/update/delete the device (and corresponding data) from the customer's record. I understand I may be able to use Maps to do this, but I'm a beginner and the documentation regarding how to do that using document client is unclear to me.
This is what I am trying to do:
var deviceData = {
'deviceID': deviceID,
'attributes': [
{'firmwareVersion': firmwareVersion},
{'productID': productID},
{'knickName': 'New Device'},
{'dateAdded': (new Date()).getTime()}
]
};
var newCustomerData = {
TableName: process.env.customerMasterFile,
Key: {
'email': email
},
ReturnValues: 'UPDATED_NEW',
UpdateExpression: 'ADD #device :device SET #customerEmailDomain = :customerEmailDomain, #friendlyName = :friendlyName, #created = :created, #updated = :updated',
ExpressionAttributeNames: {
'#device': 'deviceList',
'#customerEmailDomain': 'customerEmaiDomain',
'#friendlyName': 'friendlyName',
'#created': 'createAccountTime',
'#updated': 'updateAccountTime',
},
ExpressionAttributeValues: {
':device': docClient.createSet([deviceData]), // I know this is incorrect...
':customerEmailDomain': customerEmailDomain,
':friendlyName': friendlyName,
':created': (new Date()).getTime(),
':updated': (new Date()).getTime()
}
};
docClient.update(newCustomerData, function(err, data) {
if (err) console.log(err);
else console.log(data);
});
Normally, JSON data will be persisted as Map on DynamoDB. If you store JSON array on DynamoDB, it will be stored as "List of Map" data type on DynamoDB which will make it difficult to update, delete, retrieve without knowing the index of the List data type (i.e. device). It is not recommended to use "List of Map" if you need to accomplish update/delete without knowing the index of list (i.e. index of an array).
1) Changed to SET for all attributes including device
To store single JSON object as Map which will allow to update/delete without knowing the index of an array:-
var params = {
TableName: process.env.customerMasterFile,
Key: {
'email': email
},
ReturnValues: 'UPDATED_NEW',
UpdateExpression: 'SET #device = :device, #customerEmailDomain = :customerEmailDomain ,#friendlyName = :friendlyName, #created = :created, #updated = :updated',
ExpressionAttributeNames: {
'#device': 'deviceList',
'#customerEmailDomain': 'customerEmaiDomain',
'#friendlyName': 'friendlyName',
'#created': 'createAccountTime',
'#updated': 'updateAccountTime',
},
ExpressionAttributeValues: {
':device': deviceData,
':customerEmailDomain': customerEmailDomain,
':friendlyName': friendlyName,
':created': (new Date()).getTime(),
':updated': (new Date()).getTime()
}
};
Sample device as Map:-
Alternate Approach:-
Add device id as sort key of the table
The attributes email and device id forms the unique combination for an item on DynamoDB
You can accomplish the update/delete easily with this data model