I have an azure function with a servicebus trigger. I only want x numbers og function instances to run concurrently. This is done with the maxConcurrentCalls=x in the host file. Can this also be achieved with Azure Storage Queues?
Make sure you have installed latest nuget package(e.g Microsoft.Azure.WebJobs.Extensions.Storage) and try following settings.
If the function is on Consumption plan, in Application settings, set WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT to 1 to avoid adding host instances. For dedicated App service plan, we could fix instance count to 1.
In host.json, configure queue batch size according to runtime version(Find in portal, Platform features> Function app settings).
Runtime ~1
{
"queues": {
"batchSize": x,
"newBatchThreshold": 0
}
}
Runtime ~2
{
"version":"2.0",
"extensions": {
"queues": {
"batchSize": x,
"newBatchThreshold": 0
}
}
}
Related
Below is from Azure Storage Queue documentation https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-storage-queue. It seems to me all Queue triggers in an Azure Function app get the same settings (the settings in the "queues" block) and there's no way to have different settings for different Queue triggers. Am I correct? If that is true, any recommendation on potential workarounds?
{
"version": "2.0",
"extensions": {
"queues": {
"maxPollingInterval": "00:00:02",
"visibilityTimeout" : "00:00:30",
"batchSize": 16,
"maxDequeueCount": 5,
"newBatchThreshold": 8,
"messageEncoding": "base64"
}
}
}
As far as I know, in an Azure function app, all queue triggers share the same setting.
I'm afraid you can only create multiple Azure function apps and then create different settings.
I need to have only one instance of an Azure Function App in place and put the following json code in host.json.
But when a function gets triggered by a servicebus queue, I can clearly see in Live Metrics Stream in Application Insights that several servers get provisioned to serve the load. What am I missing to limit the running servers to only one?
{
"version": "2.0",
"WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT": 1,
"maxConcurrentCalls": 2,
"extensions": {
"serviceBus": {
"prefetchCount": 1,
"messageHandlerOptions": {
"maxConcurrentCalls": 2
}
}
}
}
Why do you want to do that?
Remember that each instance can process multiple messages at the same time, so 1 instance does not means one message at at time
Anyways, you can go to your app settings and add the following:
WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT = 1
WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT is an application setting, not a host.json setting. I would also point out that it's somewhat less than 100% reliable, so if you need a hard guarantee of only 1 instance then you should use either a Premium plan or a standard App Service plan rather than a Consumption plan.
I have this setup on Azure.
1 Azure Service Bus
1 Sql Azure Database
1 Dynamic App Service Plan
1 Azure function
I'm writing messages in the service bus, my function is triggered when a message is received and writes to the database.
I have a huge number of messages to process and I have this exception:
The request limit for the database is 90 and has been reached
I dig here on SO and in the docs and I found this answer from Paul Battum: https://stackoverflow.com/a/50769314/1026105
You can use the configuration settings in host.json to control the level of concurrency your functions execute at per instance and the max scaleout setting to control how many instances you scale out to. This will let you control the total amount of load put on your database.
What is the strategy to limit the function, since it can be limited on:
the level of concurrency your functions execute at per instance
the number of instances
Thanks guys!
Look in to using the Durable Extensions for Azure Functions where you can control the number of orchestrator and activity functions. You will need to change your design a little but will then get far better control over concurrency.
{
"version": "2.0",
"durableTask": {
"HubName": "MyFunctionHub",
"maxConcurrentActivityFunctions": 10,
"maxConcurrentOrchestratorFunctions": 10
},
"functionTimeout": "00:10:00"
}
I am using session queue on Azure and when I push some data on queue,I write one Azure function to trigger.
Please note that I have created statefull/session based queue.
The problem is when I push data to queue at that moment I got error like
The listener for function 'xxx' was unable to start.
Microsoft.ServiceBus: It is not possible for an entity that requires
sessions to create a non-sessionful message receiver
So my question is am I not able to use function with queue/topic with session?
Update 2020:
Set isSessionsEnabled property in your function.json.
This is a common ask, but currently Web Jobs SDK, and thus Azure Functions, don't support Service Bus sessions. See WebJobs SDK issue; unfortunately there's no progress 3 years after it was created. Add a +1 in Azure Functions issue.
I think its actually possible now, using the beta package Microsoft.Azure.WebJobs.Extensions.ServiceBus/3.1.0-beta2.
public static void Run([ServiceBusTrigger("core-test-queue1-sessions",
Connection = "AzureWebJobsServiceBus",
IsSessionsEnabled = true)]string myQueueItem,
IMessageSession messageSession,
ILogger log)
Also you can specify new SessionHandlerOptions section in host.json:
{
"version": "2.0",
"extensions": {
"serviceBus": {
"SessionHandlerOptions":
{
"MaxAutoRenewDuration": "00:01:00",
"MessageWaitTimeout": "00:05:00",
"MaxConcurrentSessions": 16,
"AutoComplete": true,
}
}
}
}
https://github.com/azure/azure-webjobs-sdk/issues/529#issuecomment-491113458
You need to specify in function.json of consumer AF a property "IsSessionsEnabled": true
** Problem Background **
As we know, Azure WebJob SDK, has no way of defining a retention policy for logs. That means the execution or dashboard Blob storage can grow and impose problems including slowing down or crash the kudu Dashboard – which could compromise the stability of the other apps in the App Service plan.
The problem stated here:
https://github.com/Azure/azure-webjobs-sdk/issues/560
https://github.com/Azure/azure-webjobs-sdk/issues/1050
https://github.com/Azure/azure-webjobs-sdk/issues/107
My web job functions are extensively logging and they are running more than 100,000 times a day. That means I have a huge amount of log files piled up in my storage.
** The Workaround approach that I am planning: **
I am planning to add a time trigger Functions to my WebJob code that purges log entries older than 30 days.
We have the following blob containers created or used by the WebJobs SDK:
1.Storage Connection: AzureWebJobsDashboard
1.1. azure-webjobs-dashboard
1.2. azure-jobs-host-archive
1.3. Duplicates with AzureWebJobsStorage
1.3.1 azure-jobs-host-output
1.3.2 azure-webjobs-host
2.Storage AzureWebJobsStorage
2.1. azure-jobs-host-output
2.2. azure-webjobs-host
2.2.1 Heartbeats
2.2.2 Ids
2.2.3 Output-logs
I am thinking to create a process that deletes every file older than 30 days from above containers. But I am concern that some of the blobs might be required by the running WebJobs.
** Question **
Which of the above blob containers do I need to purge, to prevent blob file pile-up problem without interfering running WebJobs ?
As far as I know, AzureWebJobsDashboard connection string account is used to store logs from the WebJobs Dashboard. This connection string is optional.
It will generate two container 'azure-webjobs-dashboard'and 'azure-jobs-host-archive'.
Azure-webjobs-dashboard: WebJob dashboard to store host and execution endpoint (function) details
Azure-jobs-host-archive: This is used as an archive for execution logs.
So both of these containers could be deleted without interfering running WebJobs.
azure-jobs-host-output is the key for troubleshooting web jobs. This container hosts logs created by the WebJob runtime during initialization and termination of every execution. If you don't want this log , you could delete it.
Azure-webjobs-host container in-turn hosts three directories:
Heartbeats – Containing 0 byte blogs for every heartbeat check performed on the service. If you don't want it, you could delete the old file.
Ids – Containing the directory with a single blog holding a unique identifier for this service.I don't suggest you delete this container's file.
Output-logs – Hosts the output of the explicit logs for each run. Explicit logs being logs introduced by WebJob developers within the execution code. You could delete the old log.
We've just implemented Storeage Lifecycle Management and are testing this:
{
"version": "0.5",
"rules": [
{
"name": "DeleteOldLogs",
"type": "Lifecycle",
"definition": {
"actions": {
"baseBlob": {
"delete": {
"daysAfterModificationGreaterThan": 30
}
}
},
"filters": {
"blobTypes": [
"blockBlob"
],
"prefixMatch": [
"azure-webjobs-host/output-logs",
"azure-webjobs-dashboard/functions/recent",
"azure-webjobs-dashboard/functions/instances",
"azure-jobs-host-output",
"azure-jobs-host-archive"
]
}
}
}
]
}