I've created dynamic sqs standard queues which are used as an event source for my lambda function invocation. Whenever any message is pushed into the queues the lambda function is invoked. Now, i want to add some throttling on my lambda function like a single event source can have only one active invocation of lambda at a time. There are some answers but they only work on throttling overall lambda concurrency.
From Managing Concurrency for a Lambda Function:
When a function has reserved concurrency, no other function can use that concurrency. Reserved concurrency also limits the maximum concurrency for the function, and applies to the function as a whole, including versions and aliases.
Therefore, reserved concurrency can be used to limit the number of concurrent executions of a specific AWS Lambda function.
AWS Lambda functions can also be triggered from an Amazon SQS FIFO queue.
From New for AWS Lambda – SQS FIFO as an event source | AWS Compute Blog:
In SQS FIFO queues, using more than one MessageGroupId enables Lambda to scale up and process more items in the queue using a greater concurrency limit. Total concurrency is equal to or less than the number of unique MessageGroupIds in the SQS FIFO queue.
So, it seems that if you specify all messages with the same MessageGroupId and a batch size of 1, then it will only process one message at a time.
Short answer is yes it can be done but only in a roundabout way.
When you have a Lambda function set as a triggered function on an SQS queue, the Lambda service polls the queue and handles the receiving and deletion of a message from the queue. The only control you have over how many messages the Lambda service reads and how many instances of your function the Lambda service invokes is (a) batch size, and (b) function concurrency.
Neither of these will help you when applied directly to your function, because setting the batch size to a small number (e.g. 1) will result in more instances being started (takes longer to process 1 message at a time), and setting it to a high number may not be desirable in your case, and if it is then it still won't help if the number of messages is higher than the batch size or they are received frequently and your function is already busy processing the previous batch. And you already said function concurrency is a no go because you only want to limit the concurrency from a source, not overall.
So here's a way it can be accomplished: create another function with a concurrency limit of 1, set it as the triggered function instead of your function. That function will receive messages, and it in turn will invoke your function with said message(s). It will wait for your function to return before returning itself. Only when the new function returns can it receive another message/batch from the Lambda service, and invoke your function again. So your "real" function has no overall concurrency limit, but there is only ever one instance invoked/running at a time from your SQS source (via the new function).
Related
We have a distributed architecture and there is a native system which needs to be called. The challenge is the capacity of the system which is not scalable and cannot take on more load of requests at same time. We have implemented Service Bus queues, where there is a Message handler listening to this queue and makes a call to the native system. The current challenge is whenever a message posted in the queue, the message handler is immediately processing the request. However, We wanted to have a scenario to only process two requests at a time. Pick the two, process it and then move on to the next two. Does Service Bus Queue provide inbuilt option to control this or should we only be able to do with custom logic?
var options = new MessageHandlerOptions()
{
MaxConcurrentCalls = 1,
AutoComplete = false
};
client.RegisterMessageHandler(
async (message, cancellationToken) =>
{
try
{
//Handler to process
await client.CompleteAsync(message.SystemProperties.LockToken);
}
catch
{
await client.AbandonAsync(message.SystemProperties.LockToken);
}
}, options);
Message Handler API is designed for concurrency. If you'd like to process two messages at any given point in time then the Handler API with maximum concurrency of two will be your answer. In case you need to process a batch of two messages at any given point in time, this API is not what you need. Rather, fall back to building your own message pump using a lower level API outlined in the answer provided by Mikolaj.
Careful with re-locking messages though. It's not a guaranteed operation as it's a client-side operation and if there's a communication network, currently, the broker will reset the lock and the message will be processed again by another competing consumer if you scale out. That is why scaling-out in your scenario is probably going to be a challenge.
Additional point is about lower level API of the MessageReceiver when it comes to receiving more than a single message - ReceiveAsync(n) does not guarantee n messages will be retrieved. If you absolutely have to have n messages, you'll need to loop to ensure there are n and no less.
And the last point about the management client and getting a queue message count - strongly suggest not to do that. The management client is not intended for frequent use at run-time. Rather, it's uses for occasional calls as these calls are very slow. Given you might end up with a single processing endpoint constrained to only two messages at a time (not even per second), these calls will add to the overall time to process.
From the top of my head I don't think anything like that is supported out of the box, so your best bet is to do it yourself.
I would suggest you look at the ReceiveAsync() method, which allows you to receive specific amount of messages (NOTE: I don't think it guarantees that if you specify that you want to retrieve 2 message it will always get you two. For instance, if there's just one message in the queue then it will probably return that one, even though you asked for two)
You could potentially use the ReceiveAsync() method in combination with PeekAsync() method where you can also provide a number of messages you want to peek. If the peeked number of messages is 2 than you can call ReceiveAsync() with better chances of getting desired two messages.
Another way would be to have a look at the ManagementClient and the GetQueueRuntimeInfoAsync() method of the queue, which will give you the information about the number of messages in the queue. With that info you could then call the ReceiveAsync() mentioned earlier.
However, be aware that if you have multiple receivers listening to the same queue then there's no guarantees that anything from above will work, as there's no way to determine if these messages were received by another process or not.
It might be that you will need to go with a more sophisticated way of handling this and receive one message, then keep it alive (renew lock etc.) until you get another message and then process them together.
I don't think I helped too much but maybe at least it will give you some ideas.
I'm considering to use such a solution when Function is triggered by Queue on Java. I'm trying to understand how to configure batchSize and newBatchThreshold more efficiently. I would like to mention below what I managed to find out about it. Please correct me as soon as you find a mistake in my reasoning:
Function is executed on 1 CPU-core environment;
Function polls messages from Queue in batches with size 16 by default and executes them in parallel (right from the documentation);
so I make a conclusion that:
if messages need CPU-intensive tasks - they are executed sequentially;
so I make a conclusion that:
since processing of messages starts at the same time (when batch arrived) then processing of more last messages takes longer and longer (confirmed experimentally);
all these longer and longer processings are billable (despite Function's body execution lasts 10 times less);
so I make a conclusion that:
One should set both batchSize and newBatchThreshold to 1 for CPU-intensive tasks and can vary only for non-CPU intensive tasks (looks like only IO-intensive tasks).
Does it make sense?
I am trying to replace AWS SQS with Kue.js in a node application.
My concern is, if there are n no of items added in a queue. How many items in the queue will be processed concurrently?
Is it like.... processor will process the each item one at a time or in bulk?
Also if the queue.process() is called for the same queue from two or more different places. how will the processing be done??
I have a nodejs function on AWS Lambda that runs multiple setTimeouts within Async Parallel. Some are instant and some could be in 30min+ from now. The problem I am running into is that It will never get to the 30min timeout because it is going idle and then dies. Is there anyway to keep the lambda function alive while it is waiting to fire off the other timeout functions.
The lifetime of a Lambda is maximum of 300 seconds.
See: AWS Lambda Limits
There is no way to increase it beyond 300 seconds. When Lambda was introduced, the maximum execution time was 60 seconds. It was later increased to 300 seconds.
You need to revisit your design and check if Lambda is the correct solution. Running an on-demand EC2 instance that matches Lambda specifications could be a solution. Or state your problem, and we can propose a solution.
In addition to the fact that you can’t do that, see hellov’s answer, I would say that this is an incorrect design choice anyway. If you needed a long lived service, using an ec2 instance directly would be a better choice.
If you just need to do something once 30 minutes later, then I would see about generating a AWS Lambda event at that time outside of the lambda code itself. In other words, Lambda is meant for pure calculations, waiting for anything inside it seems the Wrong Approach.
As others have mentioned, there is a hard-limit of 300 seconds for the maximum execution time for a Lambda function. Based on the quick overview of your problem, I don't think Lambda is the correct solution.
If you need to handle these long-running asynchronous tasks then you will need to add some type of "connector" between these different tasks. One possible solution is to use SQS Queues.
Component A --> SQS 1 --> Component B
Your Lambda function does some parallel tasks. The best way to do this in Lambda is to split each task into a separate Lambda and then coordinate those tasks in a way that best makes sense to your application.
This can be done in several different ways (the best approach depends on your application):
Step Functions
AWS Lambda + SNS
AWS Lambda + SNS/SQS
AWS Lambda + Kinesis
AWS Lambda + DynamoDB Streams
Do the settings in host.json apply to each function individually, or apply to all of the functions as a whole?
For example, I've two functions in the same Project that both get messages from Azure ServiceBus Queues.
If I set maxConcurrentCalls to 10 in host.json, does that mean that as a whole only 10 concurrent calls to ServiceBus will be made, or that it is 10 per function, so there will be 20 concurrent calls?
Thanks in advance.
host.json file is shared for all functions of a FunctionApp. That's to say that maxConcurrentCalls value will apply to all functions of the app, as any other setting will.
The effect of maxConcurrentCalls will be independent for each function. In your example, each function will have up to 10 messages processed concurrently. If you set it to 1, there will be 1 thread working per function.
Note that maxConcurrentCalls applies per instance. If you have multiple instances running, the max concurrency increases proportionally.