I am very new to AWS and I have been reading the dynamoDb SDK documentation and the properties that you can specify when creating a Table are far more than the properties that you pass when creating a table using AWS CDK.
SDK example:
var AWS = require("aws-sdk");
AWS.config.update({
region: "us-west-2",
endpoint: "http://localhost:8000"
});
var dynamodb = new AWS.DynamoDB();
var params = {
TableName : "Movies",
KeySchema: [
{ AttributeName: "year", KeyType: "HASH"}, //Partition key
{ AttributeName: "title", KeyType: "RANGE" } //Sort key
],
AttributeDefinitions: [
{ AttributeName: "year", AttributeType: "N" },
{ AttributeName: "title", AttributeType: "S" }
],
ProvisionedThroughput: {
ReadCapacityUnits: 10,
WriteCapacityUnits: 10
}
};
dynamodb.createTable(params, function(err, data) {
if (err) {
console.error("Unable to create table. Error JSON:", JSON.stringify(err, null, 2));
} else {
console.log("Created table. Table description JSON:", JSON.stringify(data, null, 2));
}
});
CDK example:
import * as dynamodb from '#aws-cdk/aws-dynamodb';
const table = new dynamodb.Table(this, 'Hits', {
partitionKey: { name: 'path', type: dynamodb.AttributeType.STRING }
});
here are all the props you can set which are more high level table related settings:
https://docs.aws.amazon.com/cdk/api/latest/docs/#aws-cdk_aws-dynamodb.Table.html
so for example if I want to set the provision throughput in CDK how do I do it? or set AttributeDefinitions, or indexes?
Do I wait unit table creation is done and then modify the table properties via the SDK UpdateTable call?
https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/DynamoDB.html#updateTable-property
Billing Mode
DynamoDB supports two billing modes:
PROVISIONED - the default mode where the table and global secondary
indexes have configured read and write capacity.
PAY_PER_REQUEST - on-demand pricing and scaling. You only pay for what
you use and there is no read and write capacity for the table or its
global secondary indexes.
see the Billing Mode attribute:
cdk docs
Dynamodb is pretty much entirely implemented in CDK, but some of the properties are a bit more difficult to find if you aren't very familiar with.
Billing Mode is the property for Provisioned or on demand read/write capacity. It is a constant, so it would be used something like
billingMode: aws_dynamodb.BillingMode.PAY_PER_REQUEST
With CDK you often have to dive a little bit into the documentation to find what you want, but for the mainstream services - Lambda, S3, Dynamo - these are fully implemented in CDK.
And in any case, for other services that may not be, you can use any of the functions that start with Cfn as these are escape hatches that allow you to basically implement direct cloud formation template jsons from cdk
Related
I have an Azure CosmosDB SQP API account with one container "EmployeeContainer", with the partition key "personId". I have three different type of collections in this container. Their schema are as shown below:
Person Collection:
{
"Id": "1234569",
"personId" : "P1241234",
"FirstName": "The first name",
"LarstName": "The last name"
}
Person-Department Collection:
{
"Id": "923456757",
"personId" : "P1241234",
"departmentId": "dept01",
"unitId": "unit01",
"dateOfJoining": "joining date"
}
Department-Employees
{
"id": "678678",
"departmentId" : "dept01",
"departmentName": "IT",
"employees" : [
{ "personId": "P1241234" },
{ "personId": "P1241235" },
{ "personId": "P1241236" },
]
}
How will the data be stored in the logical partitions? PersonId is the partition key and all the collections have personId in it. So, the document in the person collection with the person id "P1241234" and the document in the Person-Department collection with person id "P1241234" will be stored in the same logical partition? How will be the data in the Department-Employees be stored?
This design is not optimal. You should combine Person and Person-Department into a single collection using personId as the partition key, then have a second container for departments that has departmentId as it's partition key with each person as another row and any other properties that you would need when querying that collection. Do not write code that queries both containers. Each should have all the data it needs to satisfy any request you make. Then use change feed to keep them in sync.
You can get more details on how to model this article here
Yes, that is true, documents with the same personId will be stored under the same logical partition (regardless of their type\schema). I'm not sure you can create documents without the partition key on a collection with a partition key, but if you can - all of them should be under the same logical partition (but I dont think you can create them).
I am following the AWS Node.js tutorial, but I cannot get the provided code to work.
Specifically, I am trying to create a “Movies” table on a local instance of DynamoDB and load it with some provided sample data. However, I am receiving ”Cannot do operations on a non-existent table” when I try to create the table, which seems a bit strange.
For my set up, I am running DynamoDB in one console window. The command I am using and output is as follows:
COMPUTERNAME:node-dyna-demo myUserName$ java -Djava.library.path=./dynamodb_local_latest/DynamoDBLocal_lib -jar ./dynamodb_local_latest/DynamoDBLocal.jar -sharedDb
Initializing DynamoDB Local with the following configuration:
Port: 8000
InMemory: false
DbPath: null
SharedDb: true
shouldDelayTransientStatuses: false
CorsParams: *
In a separate console, I am executing the following code:
AWS.config.update({
credentials: {
accessKeyId: ‘testAccessId’,
secretAccessKey: ‘testAccessKey’
},
region: 'us-west-2',
endpoint: 'http://localhost:8000'
})
const dynamodb = new AWS.DynamoDB()
const docClient = new AWS.DynamoDB.DocumentClient()
const dbParams = {
TableName : "Movies",
KeySchema: [ … ],
AttributeDefinitions: [ … ],
ProvisionedThroughput: { … }
}
dynamodb.createTable(dbParams, function(err, data) {
if (err) {
console.error(
'Unable to create table. Error JSON:',
JSON.stringify(err, null, 2)
)
} else {
console.log(
'Created table. Table description JSON:',
JSON.stringify(data, null, 2)
)
}
})
The error I get from execution is:
Unable to create table. Error JSON: {
"message": "Cannot do operations on a non-existent table",
"code": "ResourceNotFoundException",
"time": "2018-01-24T15:56:13.229Z",
"requestId": "c8744331-dd19-4232-bab1-87d03027e7fc",
"statusCode": 400,
"retryable": false,
"retryDelay": 9.419252980728942
}
Does anyone know a possible cause for this exception?
When Dynamodb local is started without -sharedDb flag, there is a possibility for occurrence of this kind of issue.
In your case local dynamodb is started as expected,
java -Djava.library.path=./dynamodb_local_latest/DynamoDBLocal_lib -jar ./dynamodb_local_latest/DynamoDBLocal.jar -sharedDb
Try the following solutions,
Solution 1: Remove the credential info in in AWS config
AWS.config.update({
region: 'us-west-2',
endpoint: 'http://localhost:8000'
})
Reason: If in case dynamodb local is running without -sharedDb flag, it will create new database with respect to that credential.
Solution 2: Creating table using the dynamodb shell (http://localhost:8000/shell/#) and verify whether its created or not using the following command
aws dynamodb list-tables --endpoint-url http://localhost:8000
The first answer gave me a hint for my solution. I am using Java in combination with maven. Hopefully my answer can help others. The AWS region might be conflicting. The Java code points to eu-west-1.
amazonDynamoDB = AmazonDynamoDBClientBuilder
.standard()
.withEndpointConfiguration(
new AwsClientBuilder
.EndpointConfiguration("http://localhost:" +
LocalDbCreationRule.port, "eu-west-1"))
.build();
But my AWS config in my home folder on my local machine on location
~/.aws/config
showed the following:
[default]
region = eu-central-1
I changed the region to eu-west-1, matching the Java code, in my local config and the issue was resolved.
In my serverless.yml file, I have specified a DynamoDB resource, something to this effect (see below). I'd like to know two things:
Why is it that I'm not seeing these tables get created when they don't exist, forcing me to manually enter AWS console and do so myself?
In my source code (nodejs), i'm not sure how I'd reference a table specified in the yml file instead of hardcoding it.
The two questions above roll up into a singular problem, which is that I'd like to be able to specify the tables in the yml and then when doing a "deploy", have a different table set created per environment.
i.e. If I deploy to "--stage Prod", then table would be "MyTable_Prod". If I deploy to "--stage Dev", then table would be "MyTable_Dev", etc...
Figuring this out would go a long way to making deployments much smoother :).
The serverless.yml section of interest is as follows:
resources:
Resources:
DynamoDbTable:
Type: AWS::DynamoDB::Table
Properties:
TableName: MyHappyFunTable
AttributeDefinitions:
- AttributeName: id
AttributeType: S
KeySchema:
- AttributeName: id
KeyType: HASH
ProvisionedThroughput:
ReadCapacityUnits: 5
WriteCapacityUnits: 5
DynamoDBIamPolicy:
Type: AWS::IAM::Policy
DependsOn: DynamoDbTable
Properties:
PolicyName: lambda-dynamodb
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- dynamodb:Query
- dynamodb:Scan
- dynamodb:GetItem
- dynamodb:PutItem
- dynamodb:UpdateItem
- dynamodb:DeleteItem
Resource: "arn:aws:dynamodb:${opt:region, self:provider.region}:*:table/${self:provider.environment.DYNAMODB_TABLE}"
Roles:
- Ref: IamRoleLambdaExecution
A sample of my horrid 'hardcoded' table names is as follows:
dbParms = {
TableName : "MyTable_Dev",
FilterExpression: "#tid = :tid and #owner = :owner",
ProjectionExpression: "#id, #name",
ExpressionAttributeNames: {
"#tid" : "tenantid",
"#id" : "id",
"#name" : "name",
"#owner" : "owner"
},
ExpressionAttributeValues: {
":tid": tenantId,
":owner": owner
}
};
Note the "MyTable_Dev" ... ideally i'd like that to be something like "MyTable_"
+ {$opt.stage} ... or something to that effect.
In my source code (nodejs), i'm not sure how I'd reference a table specified in the yml file instead of hardcoding it.
I would put your stage in an environment variable that your Lambda function has access to.
In your serverless.yml,
provider:
...
environment:
STAGE: {$opt:stage}
Then, in your code you can access it through process.env.STAGE.
const tableName = 'MyTable_' + process.env.STAGE
Is there any way to query a dynamodb table with multiple values for a single attribute?
TableName: "sdfdsgfdg"
IndexName: 'username-category-index',
KeyConditions: {
"username": {
"AttributeValueList": { "S": "aaaaaaa#gmail.com" }
,
"ComparisonOperator": "EQ"
},
"username": {
"AttributeValueList": { "S": "hhhhh#gmail.com" }
,
"ComparisonOperator": "EQ"
},
"category": {
"AttributeValueList": { "S": "Coupon" }
,
"ComparisonOperator": "EQ"
}
}
BachGetItem API can be used to get multiple items from DynamoDB table. However, it can't be used in your use case as you are getting the data from index.
The BatchGetItem operation returns the attributes of one or more items
from one or more tables. You identify requested items by primary key.
In API perspective, there is no other solution. You may need to look at data modelling perspective and design the table/index to satisfy your Query Access Pattern (QAP).
Also, please note that querying the index multiple times with partition key values (i.e. some small number) wouldn't impact the performance as long as it is handful of items.
Well, I am having an issue that I've been dealing with for the last 2 days and still seem to have no progress on.
Basically, I am trying to develop a skill for Amazon's echo dot, but my particular skill requires the use of persistent data. I took to the docs and found information on account linking and DynamoDB, account linking seemed to complex for a simple research project so I took to DynamoDB.
I used a lambda function, and it ran fine until I put the DynamoDB table line:
alexa.dynamoDBTableName = 'rememberThisDB';
That line completely stops my skill from working and returns the following message:
The remote endpoint could not be called, or the response it returned was invalid.
I honestly have no idea how to deal with it; I am completely new to the whole AWS concept so I don't even know how to get the actual error message that the Lambda function is returning.
I changed the role and gave it the following configuration:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:*:*:*"
},
{
"Effect": "Allow",
"Action": [
"dynamodb:DeleteItem",
"dynamodb:GetItem",
"dynamodb:PutItem",
"dynamodb:Scan",
"dynamodb:UpdateItem"
],
"Resource": "*Yes, I did put the correct ARN*"
}
]
}
But that didn't really change anything, it still just returned the same error.
The issue is, I'm not doing anything at all with DynamoDB, I am simply defining the dynamoDBTableName property of the alexa object, that's it.
Yes, the DynamoDB table exists.
I feel like my head is about to blow up, so any help would be greatly appreciated.
UPDATE: Found out how to see logs, here is the latest log: Error fetching user state: ValidationException: The provided key element does not match the schema, not sure why it would give that error since I never queried anything, the only thing I did was declare the table name.
Just to document the resolution of the question in the comments and so that this question doesn't remain "unanswered" on SO:
Assuming you're using the alexa-skills-kit-sdk-for-nodejs, your table should have a single userId string HASH key.
var newTableParams = {
AttributeDefinitions: [
{
AttributeName: 'userId',
AttributeType: 'S'
}
],
KeySchema: [
{
AttributeName: 'userId',
KeyType: 'HASH'
}
],
ProvisionedThroughput: {
ReadCapacityUnits: 5,
WriteCapacityUnits: 5
}
}
It turned out the user did not have the appropriate schema setup for their DynamoDB table.