DynamoDB FilterExpressions with BETWEEN - node.js

I am performing a scan operation on my DynamoDB table then filtering the result to have items between two dates in NODE.js
The DynamoDB table formats data like this:
{
TableName: tableName,
Item: {
"visitorID": visitorIDq, // Primary Key
"dateID": dateTime, // What I am filtering the scan for
"visitorName": "END OF Q",
"employeeName": "END OF Q",
"comments": "END OF Q"
}
Current code:
var date1 = String(threeMonths); // milliseconds since epoch - 3 months in milliseconds
var date2 = String(dateTime); // milliseconds since epoch aka now
var params2 = {
TableName: tableName,
FilterExpression: "dateID BETWEEN :date1 and :date2",
ExpressionAttributeValues: {
":date1": { "S": date1},
":date2": { "S": date2}
}
};
I am getting this error:
ERROR Unable to scan the table. Error JSON: { "message": "Invalid
FilterExpression: Incorrect operand type for operator or function;
operator or function: BETWEEN, operand type: M", "code":
"ValidationException", "time": "2019-07-06T02:00:44.569Z",
"requestId": "REQUESTID1294743204701HHH443", "statusCode": 400,
"retryable": false, "retryDelay": 26.7865000058784 }
After some internet searching I am kind of stuck, not really find anybody else with this issue in node. Any ideas?
For some reason Stack Overflow is telling me this is mostly code. I thought I did a pretty good job of explaining it and giving details. This is placeholder text because I am tryna post. It is still happening. Details. Details. Details.

The correct resolution:
var dateID = String("\""+ threeMonths + "\""); // var for time in milliseconds, stringifying
var params2 = {
ExpressionAttributeValues: {
':msID' : dateID
},
FilterExpression: "dateID >= :msID", // dateID column in DDB is greater than epoch - 3 months
TableName: "Visitor"
};
JSON.stringify(params2); //params are stringified here for the scan
Specifically:
ExpressionAttributeValues: {
':msID' : "15666767546459" // or whatever value you are comparing, the above resolution has dateID there because its a var getting current time
}

Related

Filter in DynamoDB by non PartitionKey and non sort key error - Query condition missed key schema element

I want to filter results inside DynamoDB by data that is not PartitionKey or Sort KEY (name = RowKey).
I can limit the result by PartitionKey.
So I tried it in according to the documentation to make a query (without Sort Key) but I get this error:
Unable to query. Error: {
"message": "Query condition missed key schema element: RowKey",
"code": "ValidationException",
"time": "2020-04-03T13:09:17.635Z",
"requestId": "ECD4VQBGFB0IUM6G9TUSPKA3LRVV4KQNSO5AEMVJF66Q9ASUAAJG",
"statusCode": 400,
"retryable": false,
"retryDelay": 43.542922337630074
}
The following image is the configuration of the table
And this is the code that I used:
async function FilterUsersAwsQUERY (_tenant_id,_filter) {
//debug('Enter in function');
var params = {
TableName : "zzzuser",
ProjectionExpression:"#pk, comapnyrole",
KeyConditionExpression: "#pk = :pk and #cr = :cr",
ExpressionAttributeNames:{
"#pk": "PartitionKey",
"#cr": "companyrole"
},
ExpressionAttributeValues: {
":pk": _tenant_id,
":cr": _filter
}
};
return new Promise (function(resolve, reject) {
//debug('Start query');
docClient.query(params, function(err, data) {
if (err) {
console.error("Unable to query. Error:", JSON.stringify(err, null, 2));
reject(err);
} else {
//console.log("AWS ALL Query succeeded.");
resolve(data.Items);
//data.Items.forEach(function(item) {
// console.log(" -", item.year + ": " + item.title);
//});
}
});
});
};
PS. I know that there is a .scan options but it takes over 10 seconds to do it because I have 10.000 records inside it and every PartitionKey is about 400 records.
There is any way to do it efficently?
Thanks to all
The key condition expression has to be the partition key or the partition key and sort key. Then you can narrow it down with a FilterExpression. That said, refining with a filter expression can be expensive if you have to do it often AND the result set to filter is large.
This is a time where a LSI or GSI might be a better bet if this is an access pattern you have to do often and with high performance. if it is not something you do often, it might be cheaper just to take the hit with the FilterExpression as that would be cheaper. You have not given enough info for me to know one way or the other.

DynamoDB Scan with FilterExpression in nodejs

I'm trying to retrieve all items from a DynamoDB table that match a FilterExpression, and although all of the items are scanned and half do match, the expected items aren't returned.
I have the following in an AWS Lambda function running on Node.js 6.10:
var AWS = require("aws-sdk"),
documentClient = new AWS.DynamoDB.DocumentClient();
function fetchQuotes(category) {
let params = {
"TableName": "quotient-quotes",
"FilterExpression": "category = :cat",
"ExpressionAttributeValues": {":cat": {"S": category}}
};
console.log(`params=${JSON.stringify(params)}`);
documentClient.scan(params, function(err, data) {
if (err) {
console.error(JSON.stringify(err));
} else {
console.log(JSON.stringify(data));
}
});
}
There are 10 items in the table, one of which is:
{
"category": "ChuckNorris",
"quote": "Chuck Norris does not sleep. He waits.",
"uuid": "844a0af7-71e9-41b0-9ca7-d090bb71fdb8"
}
When testing with category "ChuckNorris", the log shows:
params={"TableName":"quotient-quotes","FilterExpression":"category = :cat","ExpressionAttributeValues":{":cat":{"S":"ChuckNorris"}}}
{"Items":[],"Count":0,"ScannedCount":10}
The scan call returns all 10 items when I only specify TableName:
params={"TableName":"quotient-quotes"}
{"Items":[<snip>,{"category":"ChuckNorris","uuid":"844a0af7-71e9-41b0-9ca7-d090bb71fdb8","CamelCase":"thevalue","quote":"Chuck Norris does not sleep. He waits."},<snip>],"Count":10,"ScannedCount":10}
You do not need to specify the type ("S") in your ExpressionAttributeValues because you are using the DynamoDB DocumentClient. Per the documentation:
The document client simplifies working with items in Amazon DynamoDB by abstracting away the notion of attribute values. This abstraction annotates native JavaScript types supplied as input parameters, as well as converts annotated response data to native JavaScript types.
It's only when you're using the raw DynamoDB object via new AWS.DynamoDB() that you need to specify the attribute types (i.e., the simple objects keyed on "S", "N", and so on).
With DocumentClient, you should be able to use params like this:
const params = {
TableName: 'quotient-quotes',
FilterExpression: '#cat = :cat',
ExpressionAttributeNames: {
'#cat': 'category',
},
ExpressionAttributeValues: {
':cat': category,
},
};
Note that I also moved the field name into an ExpressionAttributeNames value just for consistency and safety. It's a good practice because certain field names may break your requests if you do not.
I was looking for a solution that combined KeyConditionExpression with FilterExpression and eventually I worked this out.
Where aws is the uuid. Id is an assigned unique number preceded with the text 'form' so I can tell I have form data, optinSite is so I can find enquiries from a particular site. Other data is stored, this is all I need to get the packet.
Maybe this can be of help to you:
let optinSite = 'https://theDomainIWantedTFilterFor.com/';
let aws = 'eu-west-4:EXAMPLE-aaa1-4bd8-9ean-1768882l1f90';
let item = {
TableName: 'Table',
KeyConditionExpression: "aws = :Aw and begins_with(Id, :form)",
FilterExpression: "optinSite = :Os",
ExpressionAttributeValues: {
":Aw" : { S: aws },
":form" : { S: 'form' },
":Os" : { S: optinSite }
}
};

Dynamo db query using contains operator

My table items are of the form of
function addDoc(movie,cb){
var params = {
TableName: "Movies",
Item: {
"year": movie.year,
"title": movie.title,
"info": movie.info,
"genres" : movie.info.genres || []
}
};
docClient.put(params, function(err, data) {
bar.tick(1)
i++;
cb(err);
});
}
async.eachLimit(allMovies,50,addDoc,function (err) {
console.log(err)
console.log("done inserting " + i + " movies");
});
I'm running this code :
var params = {
TableName : "Movies",
//ProjectionExpression:"#yr, title, genres, info.actors[0]",
KeyConditionExpression: "#yr = :yyyy and contains(genres, :g1)",
ExpressionAttributeNames:{
"#yr": "year"
},
ExpressionAttributeValues: {
":yyyy":1992,
":g1" : "Drama"
},
//Select : "COUNT"
};
var start = Date.now()
docClient.query(params, function(err, data) {
if (err) {
console.error("Unable to query. Error:", JSON.stringify(err, null, 2));
} else {
console.log("time elapsed :",Date.now()-start);
console.log("Query succeeded.");
console.log(data)
}
});
and I'm getting this error
"Invalid operator used in KeyConditionExpression: contains"
any idea?
There are few things that need to be clarified here.
1) The Key attributes of DynamoDB has to be scalar data type. So, I believe the attribute genres can't be defined as SET or LIST data type
2) KeyConditionExpression - can refer to Hash and Sort key only. So, I presume the attribute genres is defined as SORT key of the table
3) contains can be used on FilterExpression on data types STRING, SET or LIST. It can't be used on KeyConditionExpression
Conclusion - Refer point 3 for straight forward answer
Based on the SDK documentation, KeyConditionExpression supports the following expressions:
a = b — true if the attribute a is equal to the value b
a < b — true if a is less than b
a <= b — true if a is less than or equal to b
a > b — true if a is greater than b
a >= b — true if a is greater than or equal to b
a BETWEEN b AND c — true if a is greater than or equal to b, and less than or equal to c.
The following function is also supported:
begins_with (a, substr)— true if the value of attribute a begins with a particular substring.
see documentation page:
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Query.html

Trouble incrementing value stored in dynamodb table via node.js

I am using a dynamodb database and attempting to update a value via node.js. I have created a table called counter in order to determine the current count of students stored in a separate table. I have had luck updating string values, but have had nothing but trouble trying to update this counter value. I have tried many different configurations of code over the last several hours to no avail. Any help would be appreciated.
function updateCount(operand) {
var params = {
TableName : "counter",
Key:{
"type" : {
"S" : "student"
}
},
UpdateExpression : "SET pos = pos + :o",
ExpressionAttributeValues:{
":o": Number(operand)
},
ReturnValues:"UPDATED_NEW"
}
docClient.update(params, function(err, data) {
if (err)
console.log(err);
else {
console.log(data);
res.send(JSON.stringify(data, null, 2));
}
});
}
updateCount(1);
The current error shows this but I've had many more:
C:\Users\Ko_Kor\Desktop\CS 496\assignment#3>node server.js
Express started on http://localhost:8081; press Ctrl-C to terminate.
{ [ValidationException: The provided key element does not match the schema]
message: 'The provided key element does not match the schema',
code: 'ValidationException',
time: Fri Oct 21 2016 20:36:43 GMT-0700 (Pacific Daylight Time),
requestId: 'T8G9V8KF587THVK5OTGQ95TH9VVV4KQNSO5AEMVJF66Q9ASUAAJG',
statusCode: 400,
retryable: false,
retryDelay: 0 }
I have also tried using this format:
ExpressionAttributeValues:{
":o": {"N": operand}
},
DynamoDB doesn't provide this kind of feature.
You can try to play with DynamoDB atomic counters, but I will suggest to try another approach like string UUID.
You could use ADD
[...]
UpdateExpression: "ADD counter :o SET updatedAt = :updatedAt",
ExpressionAttributeValues: {
':o': valueToIncrementWith,
':updatedAt': Date.now()
}
[...]

AWS DynamoDB Node.js scan- certain number of results

I try to get first 10 items which satisfy condition from DynamoDB using lambda AWS. I was trying to use Limit parameter but it is (basis on that website)
https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/DynamoDB.html#scan-property
"maximum number of items to evaluate (not necessarily the number of matching items)".
How to get 10 first items which satisfy my condition?
var AWS = require('aws-sdk');
var db = new AWS.DynamoDB();
exports.handler = function(event, context) {
var params = {
TableName: "Events", //"StreamsLambdaTable",
ProjectionExpression: "ID, description, endDate, imagePath, locationLat, locationLon, #nm, startDate, #tp, userLimit", //specifies the attributes you want in the scan result.
FilterExpression: "locationLon between :lower_lon and :higher_lon and locationLat between :lower_lat and :higher_lat",
ExpressionAttributeNames: {
"#nm": "name",
"#tp": "type",
},
ExpressionAttributeValues: {
":lower_lon": {"N": event.low_lon},
":higher_lon": {"N": event.high_lon}, //event.high_lon}
":lower_lat": {"N": event.low_lat},
":higher_lat": {"N": event.high_lat}
}
};
db.scan(params, function(err, data) {
if (err) {
console.log(err); // an error occurred
}
else {
data.Items.forEach(function(record) {
console.log(
record.name.S + "");
});
context.succeed(data.Items);
}
});
};
I think you already know the reason behind this: the distinction that DynamoDB makes between ScannedCount and Count. As per this,
ScannedCount — the number of items that were queried or scanned,
before any filter expression was applied to the results.
Count — the
number of items that were returned in the response.
The fix for that is documented right above this:
For either a Query or Scan operation, DynamoDB might return a LastEvaluatedKey value if the operation did not return all matching items in the table. To get the full count of items that match, take the LastEvaluatedKey value from the previous request and use it as the ExclusiveStartKey value in the next request. Repeat this until DynamoDB no longer returns a LastEvaluatedKey value.
So, the answer to your question is: use the LastEvaluatedKey from DynamoDB response and Scan again.

Resources