I have this table:
DomainId string HashKey
EmailId string RangeKey
I was wondering if it's possible query this table with HashKey only, like this:
var AWS = require("aws-sdk");
var client = new AWS.DynamoDB.DocumentClient();
var dm = 'infodinamica.cl';
//Set params
var params = {
TableName : 'table-name',
KeyConditionExpression: "DomainId = :dm",
ExpressionAttributeValues: {
":dm": dm
},
Select: 'COUNT'
};
client.query(params, (err, data) => {
if(err)
console.log(JSON.stringify(err, null, 2));
else
console.log(JSON.stringify(data, null, 2));
}
ps: note that this table has HashKey and RangeKey.
Yes, it is possible to query the data using Hash Key only using query API.
Use the KeyConditionExpression parameter to provide a specific value
for the partition key. The Query operation will return all of the
items from the table or index with that partition key value. You can
optionally narrow the scope of the Query operation by specifying a
sort key value and a comparison operator in KeyConditionExpression.
You can use the ScanIndexForward parameter to get results in forward
or reverse order, by sort key.
Related
I want to get data form dynamoDB, shorted by timestamp. Anyone can help? My code is given below.
const AWS = require("aws-sdk");
const dynamoDbClient = new AWS.DynamoDB.DocumentClient();
const USERS_TABLE = process.env.USERS_TABLE;
const getNews = async (req, res) => {
try {
//dynamodb params
const params = {
TableName: USERS_TABLE,
FilterExpression: "PK = :this",
ExpressionAttributeValues: { ":this": "newsTable" },
};
//get dynamodb data
const data = await dynamoDbClient.scan(params).promise();
res.status(200).send({ data: data });
} catch (e) {
return res.status(400).send({ message: e.message });
}
};
module.exports = { getNews };
Option 1: Keep Scan; Sort client-side
Works for small tables only. Single Scan call will scan only the first 1 MB of data in the table.
If you're doing scan operation as in your code example, it's impossible to get results sorted from DynamoDB. The only way to sort them is on client-side after you download all your data from database.
Replace:
res.status(200).send({ data: data });
With:
res.status(200).send({data: data.sort((a, b) => b.date - a.date)});
However, this is not recommended, since Scan operation without pagination will scan only 1st MB of data in your table. So you could get partial results. Possible solutions are:
Option 2: (recommended) Don't use Scan; Use Query; Sort by secondary key
This will work if you have your timestamp in the secondary key of the table
Don't use Scan; Use Query -- that way you can sort your data by SK (secondary key) by passing the ScanIndexForward: false to get the most recent results first.
Assuming you have such a table schema, where a timestamp is in the secondary key:
PK
SK
email
newsTable
2022-01-01
some-1#example.com
newsTable
2022-02-01
some-2#example.com
newsTable
2022-03-01
some-3#example.com
You can change your code from:
const params = {
TableName: USERS_TABLE,
FilterExpression: 'PK = :this',
ExpressionAttributeValues: {':this': 'newsTable'},
};
//get dynamodb data
const data = await dynamoDbClient.scan(params).promise();
To:
const params = {
TableName: USERS_TABLE,
KeyConditionExpression: 'PK = :this',
ExpressionAttributeValues: {':this': 'newsTable'},
ScanIndexForward: false,
};
//get dynamodb data
const data = await dynamoDbClient.query(params).promise();
And it will return results sorted from database already.
If you don't have a timestamp in your secondary key, and you cannot add it, you can add Local Secondary Index or Global Secondary Index.
Option 3: Keep Scan, but paginate; Sort client-side
Works if you cannot change DB schema and cannot switch your code to the Query operation.
Beware, it will be much more expensive, much slower. The larger the table, the slower it gets.
If you absolutely need to use Scan, you need to paginate through all the pages of the Scan operation, and then sort results in the JS code, like I described before. I've developed a handy library that makes scanning in parallel and supports pagination.
I am trying to return data from a DynamoDB table with results ordered numerically by the Primary Sort Key. I am using a Lambda scan function to return the data but it is not returning in numerical order.
The Primary Sort Key is 'time', how can i achieve this?
const AWS = require('aws-sdk');
const dynamodb = new AWS.DynamoDB({region: 'eu-west-2', apiVersion: '2012-08-10'});
exports.handler = (event, context, callback) => {
const params = {
TableName: "finalTrickstar",
};
dynamodb.scan(params, function(err, data){
if (err) {
console.log(err);
callback(err);
} else {
console.log(data);
const items = data.Items.map(
(dataField) => {
return {time: dataField.time.S, day: dataField.day.S, show: dataField.show.S, showID: dataField.showID.S};
}
);
callback(null, items);
}
});
};
I thought having a Primary Sort Key would return results ordered by the key but instead they are seemingly not ordered at all.
I don't know what you mean by "Primary Sort Key" but DynamoDB supports two types of primary keys:
partition key
partition key + sort key (composite primary key)
If your table has a primary key composed of a partition key and a sort key then the data will be kept and retrieved sorted by the sort key. If the data type of the sort key is Number, the results are returned in numeric order; otherwise, the results are returned in order of UTF-8 bytes. By default, the sort order is ascending. To reverse the order, set the ScanIndexForward parameter to false.
Make sure your table is configured correctly and that you're not expecting the data to be sorted in any other way than by the UTF-8 bytes of the sort key if it's not a number.
Below is the code of my Lambda function. I'm having trouble querying rows based on the timestamps. My plan is to get all the rows from 5 seconds before the current time to the current time in milliseconds. TimeMillis(Number) stores the current time in miiliseconds and it is the primary key and the range key is PhoneId(String). Please help me with the solution or is there any way to overcome the problem?
I'm not able to get the output, it is throwing error.
'use strict';
var AWS = require("aws-sdk");
AWS.config.update({
region: "us-east-1",
});
var docClient = new AWS.DynamoDB.DocumentClient();
exports.handler = function(event, context, callback) {
var timemillis = new Date().getTime();
var timemillis1 = timemillis - 5000;
var params = {
TableName: 'Readings',
KeyConditionExpression: "TimeMillis = :tm and TimeMillis BETWEEN :from AND :to",
ExpressionAttributeValues: {
":tm" : "TimMillis",
":from" : timemillis1,
":to" : timemillis
}
};
docClient.query(params, function(err, data) {
if(err){
callback(err, null);
}
else{
callback(null, data);
}
});
};
Here is my DynamoDB table image.
You cannot have multiple conditions inside a KeyConditionExpression. What you can do is use a FilterExpression with KeyConditionExpression to narrow down the result set.
Quoting from the documentation,
Use the KeyConditionExpression parameter to provide a specific value
for the partition key. The Query operation will return all of the
items from the table or index with that partition key value. You can
optionally narrow the scope of the Query operation by specifying a
sort key value and a comparison operator in KeyConditionExpression. To
further refine the Query results, you can optionally provide a
FilterExpression. A FilterExpression determines which items within the
results should be returned to you. All of the other results are
discarded.
Also for the test, only supported test for partition key is equality. Other conditions can be applied to sort key.
partitionKeyName = :partitionkeyval AND sortKeyName = :sortkeyval
Another way is to create a GSI which support further querying. By the way, traditional RDBMS thinking would not work best with DynamoDB. You can read about best practices here.
I understand that I can create "Lists" only from primitive data types, so look at my (Node.js using AWS Document Client) code as pseudo code. My objective is to attach a JSON array to an item so that I can later retrieve/update/delete the device (and corresponding data) from the customer's record. I understand I may be able to use Maps to do this, but I'm a beginner and the documentation regarding how to do that using document client is unclear to me.
This is what I am trying to do:
var deviceData = {
'deviceID': deviceID,
'attributes': [
{'firmwareVersion': firmwareVersion},
{'productID': productID},
{'knickName': 'New Device'},
{'dateAdded': (new Date()).getTime()}
]
};
var newCustomerData = {
TableName: process.env.customerMasterFile,
Key: {
'email': email
},
ReturnValues: 'UPDATED_NEW',
UpdateExpression: 'ADD #device :device SET #customerEmailDomain = :customerEmailDomain, #friendlyName = :friendlyName, #created = :created, #updated = :updated',
ExpressionAttributeNames: {
'#device': 'deviceList',
'#customerEmailDomain': 'customerEmaiDomain',
'#friendlyName': 'friendlyName',
'#created': 'createAccountTime',
'#updated': 'updateAccountTime',
},
ExpressionAttributeValues: {
':device': docClient.createSet([deviceData]), // I know this is incorrect...
':customerEmailDomain': customerEmailDomain,
':friendlyName': friendlyName,
':created': (new Date()).getTime(),
':updated': (new Date()).getTime()
}
};
docClient.update(newCustomerData, function(err, data) {
if (err) console.log(err);
else console.log(data);
});
Normally, JSON data will be persisted as Map on DynamoDB. If you store JSON array on DynamoDB, it will be stored as "List of Map" data type on DynamoDB which will make it difficult to update, delete, retrieve without knowing the index of the List data type (i.e. device). It is not recommended to use "List of Map" if you need to accomplish update/delete without knowing the index of list (i.e. index of an array).
1) Changed to SET for all attributes including device
To store single JSON object as Map which will allow to update/delete without knowing the index of an array:-
var params = {
TableName: process.env.customerMasterFile,
Key: {
'email': email
},
ReturnValues: 'UPDATED_NEW',
UpdateExpression: 'SET #device = :device, #customerEmailDomain = :customerEmailDomain ,#friendlyName = :friendlyName, #created = :created, #updated = :updated',
ExpressionAttributeNames: {
'#device': 'deviceList',
'#customerEmailDomain': 'customerEmaiDomain',
'#friendlyName': 'friendlyName',
'#created': 'createAccountTime',
'#updated': 'updateAccountTime',
},
ExpressionAttributeValues: {
':device': deviceData,
':customerEmailDomain': customerEmailDomain,
':friendlyName': friendlyName,
':created': (new Date()).getTime(),
':updated': (new Date()).getTime()
}
};
Sample device as Map:-
Alternate Approach:-
Add device id as sort key of the table
The attributes email and device id forms the unique combination for an item on DynamoDB
You can accomplish the update/delete easily with this data model
I have a user table with a field username. I need to write something equivalent to this in dynamo db: Select * from user where username in('a','b','c');
Adding more from code prosepective i have usernames in an array say var arr=['a','b','c'];
I so far tried this which is giving me zero result
this.dynamo.client.scanAsync({
TableName: this.dynamo.table('users'),
FilterExpression: 'username IN (:list)',
ExpressionAttributeValues: {
':list': arr.toString()
}
}).then((response) => {
console.log(response);
return {
userFriends: result.Item.friends
};
});
When I pass one element in array it give me result searching passed single element in user table but its not working with more than one element in array.
The individual users should be given as comma separated String variables. JavaScript array is equivalent to List in AWS DynamoDB data type. The DynamoDB can't compare the String data type in database with List attribute (i.e. Array in JavaScript).
var params = {
TableName : "Users",
FilterExpression : "username IN (:user1, :user2)",
ExpressionAttributeValues : {
":user1" : "john",
":user2" : "mike"
}
};
Construct the object from array for FilterExpression:-
Please refer the below code for forming the object dynamically based on Array value.
var titleValues = ["The Big New Movie 2012", "The Big New Movie"];
var titleObject = {};
var index = 0;
titleValues.forEach(function(value) {
index++;
var titleKey = ":titlevalue"+index;
titleObject[titleKey.toString()] = value;
});
var params = {
TableName : "Movies",
FilterExpression : "title IN ("+Object.keys(titleObject).toString()+ ")",
ExpressionAttributeValues : titleObject
};
Note:-
I don't think IN clause with 1000s of usernames is a good idea in terms of performance.