How to do capacity control in this case? - node.js

Question

my app reads data from DynamoDB, which has a pre-configured read capacity, which limits the read throughput. I'd like to control my query to not reach the limit, here is how I'm doing this now:
const READ_CAPACITY = 80
async function query(params) {
const consumed = await getConsumedReadCapacity()
if (consumed > READ_CAPACITY) {
await sleep((consumed-READ_CAPACITY)*1000/READ_CAPACITY)
}
const result = await dynamoDB.query(params).promise()
await addConsumedReadCapacity(result.foo.bar.CapacityUnits)
return result.Items
}
async function getConsumedReadCapacity() {
return redis.get(`read-capacity:${Math.floor(Date.now() / 1000)}`)
}
async function addConsumedReadCapacity(n) {
return redis.incrby(`read-capacity:${Math.floor(Date.now() / 1000)}`, n)
}
as you can see, a query will first check current consumed read capacity, if it does'nt exceed READ_CAPACITY, do the query, and add up the consumed read capacity.
the problem is the code is running on several servers, so there are race conditions, where the consumed > READ_CAPACITY check passed, and before it executes dynamoDB.query, dynamodb readed capacity limit by queries from other processes on other servers. How can I improve this?

Answer 1

Some things to try instead of avoiding hitting capacity limits...
Try, then back-off
From DyanmoDB error handling:
ProvisionedThroughputExceededException: The AWS SDKs for DynamoDB automatically retry requests that receive this exception. Your request is eventually successful, unless your retry queue is too large to finish. Reduce the frequency of requests, using Error Retries and Exponential Backoff.
Burst
From Best Practices for Tables:
DynamoDB provides some flexibility in the per-partition throughput provisioning. When you are not fully utilizing a partition's throughput, DynamoDB retains a portion of your unused capacity for later bursts of throughput usage. DynamoDB currently retains up to five minutes (300 seconds) of unused read and write capacity. During an occasional burst of read or write activity, these extra capacity units can be consumed very quickly—even faster than the per-second provisioned throughput capacity that you've defined for your table.
DynamoDB Auto Scaling
From Managing Throughput Capacity Automatically with DynamoDB Auto Scaling:
DynamoDB auto scaling uses the AWS Application Auto Scaling service to dynamically adjust provisioned throughput capacity on your behalf, in response to actual traffic patterns. This enables a table or a global secondary index to increase its provisioned read and write capacity to handle sudden increases in traffic, without throttling. When the workload decreases, Application Auto Scaling decreases the throughput so that you don't pay for unused provisioned capacity.
Cache in SQS
Some AWS customers have implemented a system where, if throughput is exceeded, they store the data in an Amazon SQS queue. They then have a process that retrieves the data from the queue and inserts into the table later, when there is less demand on throughput. This allows the DynamoDB table to be provisioned based on average throughput rather than peak throughput.