We are currently running load tests on our Azure Function App but the throughput is not what we expected.
There are multiple functions in the Function App but the ones with the most traffic are one with an Event Hub Trigger and one with a Service Bus Trigger consuming messages from a Session-Enabled Queue.
When the system is under load, Messages in the Session-Enabled Queue wait for up to 10 Minutes in the queue until they get processed by the consuming Function.
I know there are some settings in host.json to tune this behavior but it's still far from what we expect.
This is our host.json
{
"version": "2.0",
"logging": {
"applicationInsights": {
"samplingSettings": {
"isEnabled": true,
"excludedTypes": "Request"
}
}
},
"extensions": {
"serviceBus": {
"prefetchCount": 100,
"sessionHandlerOptions": {
"autoComplete": true,
"messageWaitTimeout": "00:00:30",
"maxAutoRenewDuration": "00:55:00",
"maxConcurrentSessions": 200
},
"batchOptions": {
"maxMessageCount": 1000,
"operationTimeout": "00:01:00",
"autoComplete": true
}
}
}
}
So i would expect the Function App to process up to 200 Sessions concurrently but in fact, although the Function Runtime provisions lots of instances, most of them seem to sit there and idle out. So to me it seems there is still another setting limiting the througput of the Function App.
I know it would improve performance if we would split the functions to separate Function Apps but as the load on the both functions is quite similar my plan was to postpone this step to a later stage and still get acceptable throughput with a single Function App.
We are using Azure Functions 3 on .NET Core 3.1 with
Microsoft.Azure.Functions.Extensions 1.1.0
Microsoft.Azure.WebJobs.Extensions.ServiceBus 5.0.0
Microsoft.Azure.WebJobs.Extensions.EventHubs 5.0.0
on a Windows Consumption Plan.
Thank you for any hints on how to achieve acceptable throughput.
I figured out that handling Batch Messages (receiving ServiceBusMessage[] instead of single instances) in the ServiceBus-Triggered Function along with enabled Sessions has a massively negative impact on scalability.
After changing this to single instances, the behavior of the system was as expected and the sessionHandlerOptions in host.json were respected.
I am wondering though what's the reason for this. I guess it could be related to the circumstance that Azure Function Instances lease a number of sessions from Service Bus to process but could not find anything in the documentation on that.
Related
I have an Azure Function durable task that will spread into 12 smaller tasks. I am using dedicated plan, my maxConcurrentActivityFunctions is currently set to 4, and I have total of 3 instances (P3v2 - 4 cores) in the app service plan.
What I understand is, I should be able to process 12 concurrent tasks, and each instance should use all of its CPU to process the job, because the job is CPU oriented.
But in reality, scaling doesn't improve the performance, all of the task go to a single instance. Other 2 instances just stay idle, despite the fact that the main instance is being totally tortured and CPU usage always sit at 100% percent.
I am sure they go to the same instance because I can read that information from the log analytics. Every log has the same host instance id. If I filter out that host instance id, no logs will even get returned.
I also tested making 3 separated call, with 4 tasks in each. It also doesn't seem to use 3 instances too. The metric for the app service plan seem like, there can only be 1 instance online at a time, depite having 3 instances available. The dashline seems to mean "offline". Because when I filter by instance, it just show at 0.
Here is the host.json file
{
"version": "2.0",
"functionTimeout": "01:00:00",
"logging": {
"logLevel": {
"default": "Information"
},
"console": {
"isEnabled": "false"
},
"applicationInsights": {
"samplingSettings": {
"ExcludedTypes": "Request",
"isEnabled": true
}
}
},
"extensions": {
"durableTask": {
"hubName": "blah",
"maxConcurrentActivityFunctions": 4,
"maxConcurrentOrchestratorFunctions": 20
}
}
}
My expection is. 12 tasks should immediately begin. And 3 instances should all be busy processing the data, instead of only 1 instance with 4 concurrent task.
Am I doing anything wrong. Or am I misunderstand something here?
As far as I know and as per the Microsoft documentation Multiple applications in the same app service plan will share all the instances you have in your premium plan.
For example if you have if the app service plan is configured to run multiple VM instances, then all the apps in the plan will run on multiple instances.
In your case, the application you have is only one but that application has many sub units (functions). So the application is using only one instance.
if you want to use all the instances then try deploying multiple function apps into the same app service plan.
Also, you can use Scaling functionalities or you can set by default Auto Scaling for the app service plan
I have written a blob-triggered function that uploads data on a CosmosDB database using the Gremlin API, using Azure Functions version 2.0. Whenever the function is triggered, it is going to read the blob, extract relevant information, and then queries the database to upload the data on it.
However, when all files are uploaded on the blob storage at the same time, the Function is going to process all files at the same time, which results in too many requests for the database to handle. To avoid this, I ensured that the Azure Function would only process one file at a time, by setting the batchSize to 1 in the host.json file :
{
"extensions": {
"queues": {
"batchSize": 1,
"maxDequeueCount": 1,
"newBatchThreshold": 0
}
},
"logging": {
"applicationInsights": {
"samplingSettings": {
"isEnabled": true,
"excludedTypes": "Request"
}
}
},
"version": "2.0"
}
This worked perfectly fine for 20 files at a time.
Now, we are trying to process 300 files at a time, and this feature doesn't seem to work anymore, the Function processes all the files at the same time again, which results in the database not being able to handle all the requests.
What am I missing here ? Is there some scaling issue I'm not aware of ?
From here:
If you want to avoid parallel execution for messages received on one queue, you can set batchSize to 1. However, this setting eliminates concurrency as long as your function app runs only on a single virtual machine (VM). If the function app scales out to multiple VMs, each VM could run one instance of each queue-triggered function.
You need to combine this with the app setting WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT when you run in Consumption plan.
Or, according to the docs, the better way would be through the Function property functionAppScaleLimit: https://learn.microsoft.com/en-us/azure/azure-functions/event-driven-scaling#limit-scale-out
WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT would work of course.
You can also scale to multiple Function App instances within one Host then you can have less hosts and more FUNCTIONS_WORKER_PROCESS_COUNT per host. Cost implications would depend on your plan.
Note that all workers within a Host would share resources, so this is recommended for more IO bound workload.
I am working on an Azure Function that triggers from a session-based Service Bus topic and sends data to a CRM system via API requests. The CRM's API has usage and load-protection limits, and we may recieve a huge amount of messages in a short timeframe, so I am looking for the best way to throttle the function.
I'm thinking the maxConcurrentSessions setting in the host.json might help me, I could limit it to around 20 messages at a time. I could also look at the "Enforce Scale Out Limit" in the Function App, but I guess that wouldn't necessarily limit the number of messages processed at one time. Would maxConcurrentSessions be the best setting to go for, or are there other considerations?
Another approach (probably in addition to the above) would be to "pause" or stop the function from taking any more messages for a period of time after I reach an API limit. Does anyone know if it's possible to programatically pause the execution of an Azure Function in real-time, or to prevent it from taking any more messages?
I'm thinking the maxConcurrentSessions setting in the host.json might
help me, I could limit it to around 20 messages at a time. I could
also look at the "Enforce Scale Out Limit" in the Function App, but I
guess that wouldn't necessarily limit the number of messages processed
at one time. Would maxConcurrentSessions be the best setting to go
for, or are there other considerations?
You can use maxConcurrentCalls to limit, the maximum number of concurrent calls to the callback that the message pump should initiate. By default, the Functions runtime processes multiple messages concurrently. If you set to 1, it will process only a single queue or topic message at a time.
The structure is like below:
{
"version": "2.0",
"extensions": {
"serviceBus": {
"prefetchCount": 100,
"messageHandlerOptions": {
"autoComplete": true,
"maxConcurrentCalls": 1,
"maxAutoRenewDuration": "00:05:00"
},
"sessionHandlerOptions": {
"autoComplete": true,
"messageWaitTimeout": "00:00:30",
"maxAutoRenewDuration": "00:55:00",
"maxConcurrentSessions": 1
}
}
}
}
At the same time, you can set the Maximum Scale Out Limit to a smaller value to prevent the function from expanding under high load.
I could also look at the "Enforce Scale Out Limit" in the Function App, but I guess that wouldn't necessarily limit the number of messages processed at one time.
I would also suggest looking at the Singleton attribute as a way to control scale-out concurrency (see the Microsoft docs). It will provide control over concurrency down to the function level. The documents also discuss your scenario of using it with maxConcurrentSessions/Calls.
I have thsese in my host.json but every time i run the function it runs in parallel runing much more threads then 1 ( so much as there are messages in queue)
{
"version": "2.0",
"extensions": {
"serviceBus": {
"prefetchCount": 1,
"messageHandlerOptions": {
"maxConcurrentCalls": 1
}
}
}
}
my function
[FunctionName(nameof(ResourceEventProcessorFunction))]
public async Task Run([ServiceBusTrigger("%TopicName%", "%SubscriptionName%", Connection = "ServiceBusConnection", IsSessionsEnabled = true)]Message message, IMessageSession messageSession, ILogger log)
Leveraging sessions
Since you are using sessions, you can use the same sessionId for all messages, and they will be processed in order by a single instance, regardless of the settings in your host.json.
https://learn.microsoft.com/en-us/azure/service-bus-messaging/message-sessions
Using Singleton attribute
If you can't use the sessionId for your purpose, you should try the [Singleton] attribute on your function. This will ensure that only one instance across all of your function instances will process the request.
We have this working successfully for WebJobs in production, and it should work just the same for Azure Functions. If you have dedicated app service plans, using this attribute should be enough. This is not recommended for a consumption plan.
[Singleton] does work on functions. The Azure Function host will create or wait for a lock in the Azure Storage account. The lock is the host ID which should be the same for all hosts of an app across all instances - so all instances share this lock and will only allow one execution to occur at a time.
To test this I put 1000 queue messages at once on a function with [Singleton]. The function would wake up, emit the invocation ID, sleep, and then emit the invocation ID. After processing all 1000 I looked at logs and never saw invocation IDs overlap. Only one invocation would happen globally at a time.
https://github.com/Azure/azure-functions-host/issues/912#issuecomment-419608830
[Singleton]
[FunctionName(nameof(ResourceEventProcessorFunction))]
public async Task Run([ServiceBusTrigger("%TopicName%", "%SubscriptionName%", Connection = "ServiceBusConnection", IsSessionsEnabled = true)]Message message, IMessageSession messageSession, ILogger log)
In a consumption plan
Continuing the quote above:
With that said I think the recommendation is: [Singleton] isn't recommended for consumption hosted function plans. If you have a dedicated app service plan it's fine (as you are paying for the instance anyway). If you want to enforce [Singleton] like behavior in a consumption plan you are likely best to:
Set WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT to 1 so you never scale to more than one instance
Set the host.json file to only allow 1 concurrent execution at a time for that trigger (for instance a batch size of 1 for Azure Queues).
https://github.com/Azure/azure-functions-host/issues/912#issuecomment-419608830
{
"version": "2.0",
"extensions": {
"serviceBus": {
"prefetchCount": 1,
"messageHandlerOptions": {
"maxConcurrentCalls": 1
}
}
}
}
Maybe you can set WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT to 1 to make the function run only one instance at a time.
If you develop locally, you can set it in local.settings.json, if you develop in Azure portal, you can set it in Configuration -> Application settings.
Noteļ¼
1. If you set WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT to 1, your function will not scale out and can only run in one instance.
2. In addition to setting WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT, you still need to set maxConcurrentCalls to 1
3. This setting is in preview. An app property for function max scale out has been added and is the recommended way to limit scale out.
For more details, you can refer to this official document.
so the problem was that every message had a differnet sessionId.
Disabling sessionId on subscription in azure solved this problem.
In details below for bounty :D
azure docs doesnt exactly specify how to limit thread number, but I looked a bit dipper.
there is MessageRecievePump
and
SessionRecievePump
one uses MaxConcurrentCalls the other one MaxConcurrentSessions and MaxConcurrentAcceptSessionCalls
be aware of this if you include session in your subscription (MaxConcurrentCalls doesnt work) it works only when session id is the same.
when session is differnt try to use MaxConcurrentSessions or MaxConcurrentAcceptSessionCalls but be aware there are no docs about this....
This is my first time using Azure functions and service bus.
I'm trying to build a function app in Visual Studio and I am able to connect to the queue topic (which I don't control). The problem is that every time I enable a breakpoint, 10s of messages are processed at once in VS which is making local testing very difficult (not to mention the problems arising from database pools).
How do I ensure that only 1 message gets processed at once until I hit complete?
public static void Run([ServiceBusTrigger("xxx", "yyy", AccessRights.Manage)]BrokeredMessage msg, TraceWriter log)
{
// do something here for one message at a time.
}
Set maxConcurrentCalls to 1 in the host.json. You also could get answer from host.json reference for Azure Functions
maxConcurrentCalls 16 The maximum number of concurrent calls to the callback that the message pump should initiate. By default, the Functions runtime processes multiple messages concurrently. To direct the runtime to process only a single queue or topic message at a time, set maxConcurrentCalls to 1
For function apps 2.0 you need to to update the host.json like this:
{
"version": "2.0",
...
"extensions": {
"serviceBus": {
"messageHandlerOptions": {
"maxConcurrentCalls": 1
}
}
}
}
Since the host.json file is usually published to Azure along with the function itself, it is preferable not to modify it for debugging and development purposes. Any change to host.json can be made to local.settings.json instead.
Here is how to set maxConcurrentCalls to 1:
{
"IsEncrypted": false,
"Values": {
...
"AzureFunctionsJobHost:Extensions:ServiceBus:MessageHandlerOptions:MaxConcurrentCalls": 1
}
}
This override functionality is described here: https://learn.microsoft.com/en-us/azure/azure-functions/functions-host-json#override-hostjson-values