Problem in counting elements in DynamoDB with Nodejs using scan - node.js

Question

I have a NodeJS function that scan a table in DynamoDB (without primary sort key) and return the number of elements of the column sync that are null.
My table:
var params = {
AttributeDefinitions: [
{
AttributeName: "barname",
AttributeType: "S"
},
{
AttributeName: "timestamp",
AttributeType: "S"
}
],
KeySchema: [
{
AttributeName: "barname",
KeyType: "HASH"
},
{
AttributeName: "timestamp",
KeyType: "RANGE"
}
],
ProvisionedThroughput: {
ReadCapacityUnits: 1,
WriteCapacityUnits: 1
},
TableName: tableName
};
The function that count when sync==false
var dynamodb = new AWS.DynamoDB({apiVersion: '2012-08-10'});
async function getCountNoSync(type){
console.log(type)
var params = {
TableName: tableName,
FilterExpression: 'sync = :sync and billing = :billing',
ExpressionAttributeValues: {
':billing' : {S: type},
':sync' : {BOOL: false}
},
};
var count = 0;
await dynamodb.scan(params).promise()
.then(function(data){
count = data.Count;
})
.catch(function(err) {
count = 0;
console.log(err);
});
return count;
}
The function works fine If a have few elements in my table (eg. less than 150). If the number of elements are higher, the count variable is always 0. It loooks like the scan do not find all elements.
Any ideia?
Best regards

Answer 1

The reason that you do not find all the items where attribute sync == null is that the scan operation is only reading part of your table.
As the documentation states:
If the total number of scanned items exceeds the maximum dataset size limit of 1 MB, the scan stops and results are returned to the user as a LastEvaluatedKey value to continue the scan in a subsequent operation.
So if your table is several hundred of megabytes big, you need to call scan() multiple times and provide the LastEvaluatedKey to read the next "page" of your table. This process is also called "pagination".
But this will take a lot of time and the time this needs will just increase with your table size. The proper way of doing this would be to create an index of the sync field and then do a query() on that index.
You can read more about that in the AWS documentation:
Querying and Scanning a DynamoDB Table
Reference documentation for scan()
Paginating the Results