I understand that I can create "Lists" only from primitive data types, so look at my (Node.js using AWS Document Client) code as pseudo code. My objective is to attach a JSON array to an item so that I can later retrieve/update/delete the device (and corresponding data) from the customer's record. I understand I may be able to use Maps to do this, but I'm a beginner and the documentation regarding how to do that using document client is unclear to me.
This is what I am trying to do:
var deviceData = {
'deviceID': deviceID,
'attributes': [
{'firmwareVersion': firmwareVersion},
{'productID': productID},
{'knickName': 'New Device'},
{'dateAdded': (new Date()).getTime()}
]
};
var newCustomerData = {
TableName: process.env.customerMasterFile,
Key: {
'email': email
},
ReturnValues: 'UPDATED_NEW',
UpdateExpression: 'ADD #device :device SET #customerEmailDomain = :customerEmailDomain, #friendlyName = :friendlyName, #created = :created, #updated = :updated',
ExpressionAttributeNames: {
'#device': 'deviceList',
'#customerEmailDomain': 'customerEmaiDomain',
'#friendlyName': 'friendlyName',
'#created': 'createAccountTime',
'#updated': 'updateAccountTime',
},
ExpressionAttributeValues: {
':device': docClient.createSet([deviceData]), // I know this is incorrect...
':customerEmailDomain': customerEmailDomain,
':friendlyName': friendlyName,
':created': (new Date()).getTime(),
':updated': (new Date()).getTime()
}
};
docClient.update(newCustomerData, function(err, data) {
if (err) console.log(err);
else console.log(data);
});
Normally, JSON data will be persisted as Map on DynamoDB. If you store JSON array on DynamoDB, it will be stored as "List of Map" data type on DynamoDB which will make it difficult to update, delete, retrieve without knowing the index of the List data type (i.e. device). It is not recommended to use "List of Map" if you need to accomplish update/delete without knowing the index of list (i.e. index of an array).
1) Changed to SET for all attributes including device
To store single JSON object as Map which will allow to update/delete without knowing the index of an array:-
var params = {
TableName: process.env.customerMasterFile,
Key: {
'email': email
},
ReturnValues: 'UPDATED_NEW',
UpdateExpression: 'SET #device = :device, #customerEmailDomain = :customerEmailDomain ,#friendlyName = :friendlyName, #created = :created, #updated = :updated',
ExpressionAttributeNames: {
'#device': 'deviceList',
'#customerEmailDomain': 'customerEmaiDomain',
'#friendlyName': 'friendlyName',
'#created': 'createAccountTime',
'#updated': 'updateAccountTime',
},
ExpressionAttributeValues: {
':device': deviceData,
':customerEmailDomain': customerEmailDomain,
':friendlyName': friendlyName,
':created': (new Date()).getTime(),
':updated': (new Date()).getTime()
}
};
Sample device as Map:-
Alternate Approach:-
Add device id as sort key of the table
The attributes email and device id forms the unique combination for an item on DynamoDB
You can accomplish the update/delete easily with this data model
Related
I want to get data form dynamoDB, shorted by timestamp. Anyone can help? My code is given below.
const AWS = require("aws-sdk");
const dynamoDbClient = new AWS.DynamoDB.DocumentClient();
const USERS_TABLE = process.env.USERS_TABLE;
const getNews = async (req, res) => {
try {
//dynamodb params
const params = {
TableName: USERS_TABLE,
FilterExpression: "PK = :this",
ExpressionAttributeValues: { ":this": "newsTable" },
};
//get dynamodb data
const data = await dynamoDbClient.scan(params).promise();
res.status(200).send({ data: data });
} catch (e) {
return res.status(400).send({ message: e.message });
}
};
module.exports = { getNews };
Option 1: Keep Scan; Sort client-side
Works for small tables only. Single Scan call will scan only the first 1 MB of data in the table.
If you're doing scan operation as in your code example, it's impossible to get results sorted from DynamoDB. The only way to sort them is on client-side after you download all your data from database.
Replace:
res.status(200).send({ data: data });
With:
res.status(200).send({data: data.sort((a, b) => b.date - a.date)});
However, this is not recommended, since Scan operation without pagination will scan only 1st MB of data in your table. So you could get partial results. Possible solutions are:
Option 2: (recommended) Don't use Scan; Use Query; Sort by secondary key
This will work if you have your timestamp in the secondary key of the table
Don't use Scan; Use Query -- that way you can sort your data by SK (secondary key) by passing the ScanIndexForward: false to get the most recent results first.
Assuming you have such a table schema, where a timestamp is in the secondary key:
PK
SK
email
newsTable
2022-01-01
some-1#example.com
newsTable
2022-02-01
some-2#example.com
newsTable
2022-03-01
some-3#example.com
You can change your code from:
const params = {
TableName: USERS_TABLE,
FilterExpression: 'PK = :this',
ExpressionAttributeValues: {':this': 'newsTable'},
};
//get dynamodb data
const data = await dynamoDbClient.scan(params).promise();
To:
const params = {
TableName: USERS_TABLE,
KeyConditionExpression: 'PK = :this',
ExpressionAttributeValues: {':this': 'newsTable'},
ScanIndexForward: false,
};
//get dynamodb data
const data = await dynamoDbClient.query(params).promise();
And it will return results sorted from database already.
If you don't have a timestamp in your secondary key, and you cannot add it, you can add Local Secondary Index or Global Secondary Index.
Option 3: Keep Scan, but paginate; Sort client-side
Works if you cannot change DB schema and cannot switch your code to the Query operation.
Beware, it will be much more expensive, much slower. The larger the table, the slower it gets.
If you absolutely need to use Scan, you need to paginate through all the pages of the Scan operation, and then sort results in the JS code, like I described before. I've developed a handy library that makes scanning in parallel and supports pagination.
I try to update an item in dynamodb by adding a condition, without passing the key in the parameters.
And as soon as my condition is true update. Is it possible to do this?
Below an example of an item:
{
"id" : "bcc2f32e-305e-4469-88e2-463724b5c6a9",
"name" : "toto",
"email" : "toto#titi.com"
}
Where email is unique for items.
I tested this code and it works :
const name= "updateName";
const params = {
TableName: MY_TABLE,
Key: {
id
},
UpdateExpression: 'set #name = :name',
ExpressionAttributeNames: { '#name': 'name' },
ExpressionAttributeValues: { ':name': name },
ReturnValues: "ALL_NEW"
}
dynamoDb.update(params, (error, result) => {
if (error) {
res.status(400).json({ error: 'Could not update Item' });
}
res.json(result.Attributes);
})
But i want to do something like this (replace the Key by conditionExpression):
const params = {
TableName: MY_TABLE,
UpdateExpression: 'set #name = :name',
ConditionExpression: '#email = :email',
ExpressionAttributeNames: {
'#name': 'name',
'#email': 'email'
},
ExpressionAttributeValues: {
':name': name,
':email': email
},
ReturnValues: "ALL_NEW"
}
dynamoDb.update(params, (error, result) => {
if (error) {
res.status(400).json({ error: 'Could not update User' });
}
res.json(result.Attributes);
})
But this code doesn't work.
Any ideas?
You cannot update an item in DynamoDB without using the entire primary key (partition key, and sort key if present). This is because you must specify exactly one record for the update. See the documentation here.
If you want to find an item using a field that is not the primary key, then you can search using a scan (potentially slow and expensive) or by using a Global Secondary Index (GSI) on that field. Either of these methods requires that you do a separate request to find the item in question, and then use its primary key to perform the update.
It sounds like you want to do an update that waits for a condition. That's not how DynamoDb works; it cannot wait for anything (except consistency, I suppose, but that's somewhat different). What you can do is make a request with a condition, and if it fails the condition (returning immediately), make the request again later. If you do this you'll need to be careful to backoff appropriately, or you might end up making a lot of requests very quickly.
The key is a required parameter when doing updates; the condition expression can be used in addition to providing the key, but can't be used instead of the key.
Also, I am not sure you fully understand what the conditionExpression is for - its not like the 'where' clause in an SQL update statement (i.e. update mytable set name='test' where email='myemail.com'.
Instead, logically the conditionExpression in an update would be more like:
update mytable set name='test' where key='12345' but only if quantity >0 - for example,
i.e. you are telling dynamodb the exact key of the record you want updated, and once it finds it it uses the condition expression to determine if the update should proceed - i.e. find the record with id=12345, and change the name to 'test', only of the quantity is greater than 0.
It does not use the conditionExpression to find records to update.
Running a Node.js serverless backend through AWS.
Main objective: to filter and list all LOCAL jobs (table items) that included the available services and zip codes provided to the filter.
Im passing in multiple zip codes, and multiple available services.
data.radius would be an array of zip codes = to something like this:[ '93901', '93902', '93905', '93906', '93907', '93912', '93933', '93942', '93944', '93950', '95377', '95378', '95385', '95387', '95391' ]
data.availableServices would also be an array = to something like this ['Snow removal', 'Ice Removal', 'Salting', 'Same Day Response']
I am trying to make an API call that returns only items that have a matching zipCode from the array of zip codes provided by data.radius, and the packageSelected has a match of the array data.availableServices provided.
API CALL
import * as dynamoDbLib from "./libs/dynamodb-lib";
import { success, failure } from "./libs/response-lib";
export async function main(event, context) {
const data = JSON.parse(event.body);
const params = {
TableName: "jobs",
FilterExpression: "zipCode = :radius, packageSelected = :availableServices",
ExpressionAttributeValues: {
":radius": data.radius,
":availableServices": data.availableServices
}
};
try {
const result = await dynamoDbLib.call("query", params);
// Return the matching list of items in response body
return success(result.Items);
} catch (e) {
return failure({ status: false });
}
Do I need to map the array of zip codes and available services first for this to work?
Should I be using comparison operators?
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/LegacyConditionalParameters.QueryFilter.html
Is a sort key value or partition key required to query and filter? (the table has a sort key and partition key but i would like to avoid using them in this call)
Im not 100% sure on how to go about this so if anyone could point me in the right direction that would be wonderful and greatly appreciated!!
I'm not sure what your dynamodb-lib refers to but here's an example of how you can scan for attribute1 in a given set of values and attribute2 in a different set of values. This uses the standard AWS JavaScript SDK, and specifically the high-level document client.
Note that you cannot use an equality (==) test here, you have to use an inclusion (IN) test. And you cannot use query, but must use scan.
const AWS = require('aws-sdk');
let dc = new AWS.DynamoDB.DocumentClient({'region': 'us-east-1'});
const data = {
radius: [ '93901', '93902', '93905', '93906', '93907', '93912', '93933', '93942', '93944', '93950', '95377', '95378', '95385', '95387', '95391' ],
availableServices: ['Snow removal', 'Ice Removal', 'Salting', 'Same Day Response'],
};
// These hold ExpressionAttributeValues
const zipcodes = {};
const services = {};
data.radius.forEach((zipcode, i) => {
zipcodes[`:zipcode${i}`] = zipcode;
})
data.availableServices.forEach((service, i) => {
services[`:services${i}`] = service;
})
// These hold FilterExpression attribute aliases
const zipcodex = Object.keys(zipcodes).toString();
const servicex = Object.keys(services).toString();
const params = {
TableName: "jobs",
FilterExpression: `zipCode IN (${zipcodex}) AND packageSelected IN (${servicex})`,
ExpressionAttributeValues : {...zipcodes, ...services},
};
dc.scan(params, (err, data) => {
if (err) {
console.log('Error', err);
} else {
for (const item of data.Items) {
console.log('item:', item);
}
}
});
I'm trying to retrieve all items from a DynamoDB table that match a FilterExpression, and although all of the items are scanned and half do match, the expected items aren't returned.
I have the following in an AWS Lambda function running on Node.js 6.10:
var AWS = require("aws-sdk"),
documentClient = new AWS.DynamoDB.DocumentClient();
function fetchQuotes(category) {
let params = {
"TableName": "quotient-quotes",
"FilterExpression": "category = :cat",
"ExpressionAttributeValues": {":cat": {"S": category}}
};
console.log(`params=${JSON.stringify(params)}`);
documentClient.scan(params, function(err, data) {
if (err) {
console.error(JSON.stringify(err));
} else {
console.log(JSON.stringify(data));
}
});
}
There are 10 items in the table, one of which is:
{
"category": "ChuckNorris",
"quote": "Chuck Norris does not sleep. He waits.",
"uuid": "844a0af7-71e9-41b0-9ca7-d090bb71fdb8"
}
When testing with category "ChuckNorris", the log shows:
params={"TableName":"quotient-quotes","FilterExpression":"category = :cat","ExpressionAttributeValues":{":cat":{"S":"ChuckNorris"}}}
{"Items":[],"Count":0,"ScannedCount":10}
The scan call returns all 10 items when I only specify TableName:
params={"TableName":"quotient-quotes"}
{"Items":[<snip>,{"category":"ChuckNorris","uuid":"844a0af7-71e9-41b0-9ca7-d090bb71fdb8","CamelCase":"thevalue","quote":"Chuck Norris does not sleep. He waits."},<snip>],"Count":10,"ScannedCount":10}
You do not need to specify the type ("S") in your ExpressionAttributeValues because you are using the DynamoDB DocumentClient. Per the documentation:
The document client simplifies working with items in Amazon DynamoDB by abstracting away the notion of attribute values. This abstraction annotates native JavaScript types supplied as input parameters, as well as converts annotated response data to native JavaScript types.
It's only when you're using the raw DynamoDB object via new AWS.DynamoDB() that you need to specify the attribute types (i.e., the simple objects keyed on "S", "N", and so on).
With DocumentClient, you should be able to use params like this:
const params = {
TableName: 'quotient-quotes',
FilterExpression: '#cat = :cat',
ExpressionAttributeNames: {
'#cat': 'category',
},
ExpressionAttributeValues: {
':cat': category,
},
};
Note that I also moved the field name into an ExpressionAttributeNames value just for consistency and safety. It's a good practice because certain field names may break your requests if you do not.
I was looking for a solution that combined KeyConditionExpression with FilterExpression and eventually I worked this out.
Where aws is the uuid. Id is an assigned unique number preceded with the text 'form' so I can tell I have form data, optinSite is so I can find enquiries from a particular site. Other data is stored, this is all I need to get the packet.
Maybe this can be of help to you:
let optinSite = 'https://theDomainIWantedTFilterFor.com/';
let aws = 'eu-west-4:EXAMPLE-aaa1-4bd8-9ean-1768882l1f90';
let item = {
TableName: 'Table',
KeyConditionExpression: "aws = :Aw and begins_with(Id, :form)",
FilterExpression: "optinSite = :Os",
ExpressionAttributeValues: {
":Aw" : { S: aws },
":form" : { S: 'form' },
":Os" : { S: optinSite }
}
};
I need to store user's info in DynamoDB and send a mail to the same user if it doesn't already exist in DynamoDB table. I am doing this in for loop. The list contains only 2 records. The issue is only the second record gets inserted in table and the mail is sent twice to the same user. Here is the code:
module.exports.AddUser = function(req, res, usersList, departmentId) {
var _emailId = "";
var _userName = "";
var _departmentId = departmentId;
for (var i = 0; i < usersList.length; i++) {
_emailId = usersList[i].emailId;
_userName = usersList[i].userName;
var params = {
TableName: "UsersTable",
Key: {
"emailId": _emailId,
"departmentId": _departmentId
}
};
docClient.get(params, function(err, data) {
if (!err) {
if (!data.items)
AddUserAndSendEmail("UsersTable", _emailId, _userName);
//The above function is being called twice but for the same user.
//It has a check so not inserting the same record twice but
//sending two mails to the same user.
}
});
}
res.end("success");
}
function AddUserAndSendEmail(tableName, emailId, _userName) {
var params = {
TableName: tableName,
Item: {
"emailId": emailId,
"departmentId": 101//Default Department
}
};
docClient.put(params, function(err, data) {
if (!err) {
//Send Email Code Here
} else {
console.log("error");
}
});
}
What could be the reason for this strange behavior? Really frustrated, I am about to give up on this.
1) Please note that DynamoDB is eventually consistent. If you insert the item and check whether the item exists immediately, it may not always find the item in the database.
This means the second iteration of the loop may not always find the first item inserted into the table.
2) If the item already exists in the table, the Put api will update the item and give successful response.
This means the Put will be successful for the same email id and department id in the second iteration because it updates the record if it is already present.
GetItem – The GetItem operation returns a set of Attributes for an
item that matches the primary key. The GetItem operation provides an
eventually consistent read by default. If eventually consistent reads
are not acceptable for your application, use ConsistentRead.
PutItem – Creates a new item, or replaces an old item with a new item
(including all the attributes). If an item already exists in the
specified table with the same primary key, the new item completely
replaces the existing item. You can also use conditional operators to
replace an item only if its attribute values match certain conditions,
or to insert a new item only if that item doesn’t already exist.
Based on the above points, there is a possibility to get two emails if you have same email id and department id in the array.