I'm working on queue trigger based Azure function which is under consumption plan. Currently I have millions of messages in my queue which should be processed. Based on Azure Functions documentation it should add more servers to be able to process my messages. However it's running only 145 servers and more than 2 millions messages are stuck on the queue for last 24 hours. Why Azure is not adding more servers to speed up the message processing? Is there any limit for the servers can be allocated for each Function App?
Related
We have a solution where we use an Azure Storage Queue to process messages that take approx 6 minutes.
I've read that the maximum batchSize of Queue messages concurrently processed are 32 per VM.
If the function app scales out to multiple VMs, each VM could run one instance of each queue-triggered function.
https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-storage-queue?tabs=in-process%2Cextensionv5%2Cextensionv3&pivots=programming-language-csharp#host-json
How does that translate to Azure Functions Premium plan?
Lets say we want to be able to process 64 messages at once using Azure Functions Premium plan with Always ready instances. If we have 2 ready instances, can they process 2 * 32 concurrent messages? Or do they underwater really need to be on seperate VM's and 2 instances will not do anything different?
In the Premium plan, you can have your app always ready on a specified number of instances. The maximum number of always ready instances is 20. When events begin to trigger the app, they are first routed to the always ready instances.
https://learn.microsoft.com/en-us/azure/azure-functions/functions-premium-plan?tabs=portal#always-ready-instances
Yes. In Azure Functions premium plan, if you have pre-warmed instance, then that is given a dedicated VM instance. So, if you had 2 VM instances running your function app, then they can process 2*(batchSize + newBatchThreshold) concurrent Queue messages!
The Azure platform scales the function app onto new VM as the existing instances gets more busy.
I have an application that sends messages in batches to the Azure Service Premium queue. I am noticing that if I am sending 1000 messages in total (20 batches with 50 messages each, per-message size is 10 KB), I am noticing an extra count of messages in the queue. The extra count varies for every run. There are strictly no other senders. Is there any reason why this could happen? Or, if my logic needs to be checked thoroughly? I just want to eliminate any reasons inherent to the operations of the Azure Service bus.
I am using Azure.Messaging.ServiceBus latest version in .net 4.8
Per this documentation, I am using the Azure Function consumption plan and am trying to limit the parallelism of one of the queue-triggered functions so that only one instance runs at a time:
{
"queues": {
"batchSize": 1
}
}
The queue is part of a Microsoft Storage Account, and isn't a service bus.
However, my problem is that the function is still being run in parallel if there are multiple items on the queue at once. I read in the fine print of the documentation above:
If you want to avoid parallel execution for messages received on one queue, you can set batchSize to 1. However, this setting eliminates concurrency only so long as your function app runs on a single virtual machine (VM). If the function app scales out to multiple VMs, each VM could run one instance of each queue-triggered function.
Since I am using a consumption plan, how do I know if the function app is running on a single VM or on multiple? How can I successfully limit this function's batch size to one?
In consumption plan, a single function app only scales up to a maximum of 200 instances. A single instance may process more than one message or request at a time though, so there isn't a set limit on number of concurrent executions.
Also, when you're using a Consumption plan, instances of the Azure Functions host are dynamically added and removed based on the number of incoming events.
As you want to limit the parallelism of one of the queue-triggered functions, so I suggest that you could use Azure App Service Plan to achieve it.
For more details, you could refer to this article.
The Problem:
So we are building a newsletter system for our app that must have a capacity to send 20k-40k emails up to several times a day.
Tools of Preference:
Amazon SES - for pricing and scalability
Azure Functions - for serverless compute to send emails
Limitations of Amazon SES:
Amazon SES Throttling having Max Send Rate - Amazon SES throttles sending via their services by imposing a max send rate. Right now, being out of the sandbox SES environment, our capacity is 14 emails/sec with 50K daily emails cap. But this limit can be increased via a support ticket.
Limitations of Azure Functions:
On a Consumption Plan, there's no way to limit the scale as to how many instances of your Azure Function execute. Currently the scaling is handled internally by Azure, and thus the function can execute between just a few to hundreds of instances.
From reading other post on Azure Functions, there seems to be "warm-up" period for Azure Functions, meaning the function may not execute as soon as it is triggered via one of the documented triggers.
Limitations of Azure Functions with SES:
The obvious problem would be Amazon SES throttling sending emails from Azure functions because the scaled execution of Azure Function that sends out an email will be much higher than allowed send rate for SES.
Due to "warm-up" period of Azure Function messages may end up being piled up in a queue before Azure Function actually starts processing them at a scale and sending out the email, thus there's a very high probability of hitting that send/rate limit.
Question:
How can we utilize sending emails via Azure Functions while still being under X emails/second limit of SES? Is there a way to limit how many times an Azure Function can execute per time frame? So like let's says we don't want more than 30 instances of Azure Function running per/second?
Other thoughts:
Amazon SES might not like continuous throttling of SES for a customer if the customer's implementation is constantly hitting up that throttling limit. Amazon SES folks, can you please comment?
Azure Functions - as per documentation, the scaling of Azure Functions on a Consumption Plan is handled internally. But isn't there a way to put a manual "cap" on scaling? This seems like such a common requirement from a customer's point of view. The problem is not that Azure Functions can't handle the load, the problem is that other components of the system that interface with Azure Functions can't handle the load at the massive scale at which Azure Functions can handle it.
Thank you for your help.
If I understand your problem correctly, the easiest method is a custom queue throttling solution.
Basically your AF just retrieve the calls for all the mailing requests, and then queue them into a queue system (say ServiceBus / EventHub / IoTHub) then you can have another Azure function that runs by x minute interval, which will pull a maximum of y messages and push it to SES. Your control point becomes that clock function, and since the queue system will help you to ensure you know your message delivery status (has sent to SES yet) and you can pop the queue once done, it would allow you to ensure the job is eventually processed.
You should be able to set the maxConcurrentCalls to 1 within the host.json file for the function; this will ensure that only 1 function execution is occurring at any given time and should throttle your processing rate to something more agreeable from AWS perspective in terms of sends per second:
host.json
{
// The unique ID for this job host. Can be a lower case GUID
// with dashes removed. When running in Azure Functions, the id can be omitted, and one gets generated automatically.
"id": "9f4ea53c5136457d883d685e57164f08",
// Configuration settings for 'serviceBus' triggers. (Optional)
"serviceBus": {
// The maximum number of concurrent calls to the callback the message
// pump should initiate. The default is 16.
"maxConcurrentCalls": 1,
...
so my environment is set in cloud service with 2 instances of worker role which process messages from service bus queue.I have also set up autoscaling block to increase instances when an instance has more than 10 messages to handle.
here are steps i take.
I push messages to a queue about 1000
current all my messages are unprocessed as my instances are not up.
i publish the worker role with 2 instances, and when they are up,they start reading messages correctly.
then i configure autoscaling in above stated rule for queues and 10 messages per instance.
What I excpected was since the instances already have more than they can handle, azure should start spinning up new instance.but this doesnt happen untill at least 10-15 minutes after my first two instances are up.
What could be the reason behind this and any algorithm on microsoft side?