I need to have only one instance of an Azure Function App in place and put the following json code in host.json.
But when a function gets triggered by a servicebus queue, I can clearly see in Live Metrics Stream in Application Insights that several servers get provisioned to serve the load. What am I missing to limit the running servers to only one?
{
"version": "2.0",
"WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT": 1,
"maxConcurrentCalls": 2,
"extensions": {
"serviceBus": {
"prefetchCount": 1,
"messageHandlerOptions": {
"maxConcurrentCalls": 2
}
}
}
}
Why do you want to do that?
Remember that each instance can process multiple messages at the same time, so 1 instance does not means one message at at time
Anyways, you can go to your app settings and add the following:
WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT = 1
WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT is an application setting, not a host.json setting. I would also point out that it's somewhat less than 100% reliable, so if you need a hard guarantee of only 1 instance then you should use either a Premium plan or a standard App Service plan rather than a Consumption plan.
Related
I have created a Azure service bus topic subscription which receives a message form another http trigger function app and insert information into a database.
Every message I receive has an ID and based on the ID, I decide to either add a new record or updating existing one.
Problem is happening when 2 messages with the same ID are received at the same time and ends up creating 2 database records.
Is there any way to configure topic subscription function to run only 1 instance at a time? I don't have control on the function app which send
Is there any way to configure topic subscription function to run only 1 instance at a time?
You can configure everything like number of instances to be running, number of messages to be processing, number of calls and sessions, etc., in host.json file. All the attributes of host.json related to Azure Functions Service bus trigger bindings are available in this MS Doc reference.
For your requirements such as processing 1 message at a time, you can define the following attributes to 1 in the host.json file:
"maxConcurrentSessions": 1,
"maxMessageBatchSize": 1,
"maxConcurrentCalls": 1,
"messageHandlerOptions": {
"maxConcurrentCalls": 1
},
Normally, Azure Functions will process the messages in multiple and in parallel. So, the attributes like maxConcurrentSessions, maxConcurrentCalls plays the major role to define number of sessions and calls to be processed for every instance.
"batchOptions": {
"maxMessageCount": 1
}
Above complete configuration of host.json will do processing of 2nd message only after the execution of 1st message in the Service bus topic subscription azure function.
While answering Retrieve quota for Microsoft Azure App Service Storage, I stumbled upon the FileSystemUsage metric for Microsoft.Web/sites resource type. As per the documentation, this metric should return Percentage of filesystem quota consumed by the app..
However when I execute Metrics - List REST API operation (and also in the Metrics blade in Azure Portal) for my web app, the value is always returned as zero. I checked it against a number of web apps in my Azure Subscriptions and for all of them the result was zero. I am curious to know the reason for that.
In contrast, if I execute App Service Plans - List Usages REST API operation, it returns me the correct value. For example, if my App Service Plan is S2, I get following response back:
{
"unit": "Bytes",
"nextResetTime": "9999-12-31T23:59:59.9999999Z",
"currentValue": 815899648,
"limit": 536870912000,//500 GB (50 GB/instance x max 10 instances)
"name": {
"value": "FileSystemStorage",
"localizedValue": "File System Storage"
}
},
Did I misunderstand FileSystemUsage for Web Apps? Would appreciate if someone can explain the purpose of this metric? If it is indeed what is documented, then why the API is returning zero value?
This should be the default behavior, please check this doc Understand metrics:
Note
File System Usage is a new metric being rolled out globally, no data
is expected unless your app is hosted in an App Service Environment.
So currently this metric File System Usage should only be working on ASE.
I have an azure function with a servicebus trigger. I only want x numbers og function instances to run concurrently. This is done with the maxConcurrentCalls=x in the host file. Can this also be achieved with Azure Storage Queues?
Make sure you have installed latest nuget package(e.g Microsoft.Azure.WebJobs.Extensions.Storage) and try following settings.
If the function is on Consumption plan, in Application settings, set WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT to 1 to avoid adding host instances. For dedicated App service plan, we could fix instance count to 1.
In host.json, configure queue batch size according to runtime version(Find in portal, Platform features> Function app settings).
Runtime ~1
{
"queues": {
"batchSize": x,
"newBatchThreshold": 0
}
}
Runtime ~2
{
"version":"2.0",
"extensions": {
"queues": {
"batchSize": x,
"newBatchThreshold": 0
}
}
}
I have this setup on Azure.
1 Azure Service Bus
1 Sql Azure Database
1 Dynamic App Service Plan
1 Azure function
I'm writing messages in the service bus, my function is triggered when a message is received and writes to the database.
I have a huge number of messages to process and I have this exception:
The request limit for the database is 90 and has been reached
I dig here on SO and in the docs and I found this answer from Paul Battum: https://stackoverflow.com/a/50769314/1026105
You can use the configuration settings in host.json to control the level of concurrency your functions execute at per instance and the max scaleout setting to control how many instances you scale out to. This will let you control the total amount of load put on your database.
What is the strategy to limit the function, since it can be limited on:
the level of concurrency your functions execute at per instance
the number of instances
Thanks guys!
Look in to using the Durable Extensions for Azure Functions where you can control the number of orchestrator and activity functions. You will need to change your design a little but will then get far better control over concurrency.
{
"version": "2.0",
"durableTask": {
"HubName": "MyFunctionHub",
"maxConcurrentActivityFunctions": 10,
"maxConcurrentOrchestratorFunctions": 10
},
"functionTimeout": "00:10:00"
}
** Problem Background **
As we know, Azure WebJob SDK, has no way of defining a retention policy for logs. That means the execution or dashboard Blob storage can grow and impose problems including slowing down or crash the kudu Dashboard – which could compromise the stability of the other apps in the App Service plan.
The problem stated here:
https://github.com/Azure/azure-webjobs-sdk/issues/560
https://github.com/Azure/azure-webjobs-sdk/issues/1050
https://github.com/Azure/azure-webjobs-sdk/issues/107
My web job functions are extensively logging and they are running more than 100,000 times a day. That means I have a huge amount of log files piled up in my storage.
** The Workaround approach that I am planning: **
I am planning to add a time trigger Functions to my WebJob code that purges log entries older than 30 days.
We have the following blob containers created or used by the WebJobs SDK:
1.Storage Connection: AzureWebJobsDashboard
1.1. azure-webjobs-dashboard
1.2. azure-jobs-host-archive
1.3. Duplicates with AzureWebJobsStorage
1.3.1 azure-jobs-host-output
1.3.2 azure-webjobs-host
2.Storage AzureWebJobsStorage
2.1. azure-jobs-host-output
2.2. azure-webjobs-host
2.2.1 Heartbeats
2.2.2 Ids
2.2.3 Output-logs
I am thinking to create a process that deletes every file older than 30 days from above containers. But I am concern that some of the blobs might be required by the running WebJobs.
** Question **
Which of the above blob containers do I need to purge, to prevent blob file pile-up problem without interfering running WebJobs ?
As far as I know, AzureWebJobsDashboard connection string account is used to store logs from the WebJobs Dashboard. This connection string is optional.
It will generate two container 'azure-webjobs-dashboard'and 'azure-jobs-host-archive'.
Azure-webjobs-dashboard: WebJob dashboard to store host and execution endpoint (function) details
Azure-jobs-host-archive: This is used as an archive for execution logs.
So both of these containers could be deleted without interfering running WebJobs.
azure-jobs-host-output is the key for troubleshooting web jobs. This container hosts logs created by the WebJob runtime during initialization and termination of every execution. If you don't want this log , you could delete it.
Azure-webjobs-host container in-turn hosts three directories:
Heartbeats – Containing 0 byte blogs for every heartbeat check performed on the service. If you don't want it, you could delete the old file.
Ids – Containing the directory with a single blog holding a unique identifier for this service.I don't suggest you delete this container's file.
Output-logs – Hosts the output of the explicit logs for each run. Explicit logs being logs introduced by WebJob developers within the execution code. You could delete the old log.
We've just implemented Storeage Lifecycle Management and are testing this:
{
"version": "0.5",
"rules": [
{
"name": "DeleteOldLogs",
"type": "Lifecycle",
"definition": {
"actions": {
"baseBlob": {
"delete": {
"daysAfterModificationGreaterThan": 30
}
}
},
"filters": {
"blobTypes": [
"blockBlob"
],
"prefixMatch": [
"azure-webjobs-host/output-logs",
"azure-webjobs-dashboard/functions/recent",
"azure-webjobs-dashboard/functions/instances",
"azure-jobs-host-output",
"azure-jobs-host-archive"
]
}
}
}
]
}