I have this setup on Azure.
1 Azure Service Bus
1 Sql Azure Database
1 Dynamic App Service Plan
1 Azure function
I'm writing messages in the service bus, my function is triggered when a message is received and writes to the database.
I have a huge number of messages to process and I have this exception:
The request limit for the database is 90 and has been reached
I dig here on SO and in the docs and I found this answer from Paul Battum: https://stackoverflow.com/a/50769314/1026105
You can use the configuration settings in host.json to control the level of concurrency your functions execute at per instance and the max scaleout setting to control how many instances you scale out to. This will let you control the total amount of load put on your database.
What is the strategy to limit the function, since it can be limited on:
the level of concurrency your functions execute at per instance
the number of instances
Thanks guys!
Look in to using the Durable Extensions for Azure Functions where you can control the number of orchestrator and activity functions. You will need to change your design a little but will then get far better control over concurrency.
{
"version": "2.0",
"durableTask": {
"HubName": "MyFunctionHub",
"maxConcurrentActivityFunctions": 10,
"maxConcurrentOrchestratorFunctions": 10
},
"functionTimeout": "00:10:00"
}
Related
I have created a Azure service bus topic subscription which receives a message form another http trigger function app and insert information into a database.
Every message I receive has an ID and based on the ID, I decide to either add a new record or updating existing one.
Problem is happening when 2 messages with the same ID are received at the same time and ends up creating 2 database records.
Is there any way to configure topic subscription function to run only 1 instance at a time? I don't have control on the function app which send
Is there any way to configure topic subscription function to run only 1 instance at a time?
You can configure everything like number of instances to be running, number of messages to be processing, number of calls and sessions, etc., in host.json file. All the attributes of host.json related to Azure Functions Service bus trigger bindings are available in this MS Doc reference.
For your requirements such as processing 1 message at a time, you can define the following attributes to 1 in the host.json file:
"maxConcurrentSessions": 1,
"maxMessageBatchSize": 1,
"maxConcurrentCalls": 1,
"messageHandlerOptions": {
"maxConcurrentCalls": 1
},
Normally, Azure Functions will process the messages in multiple and in parallel. So, the attributes like maxConcurrentSessions, maxConcurrentCalls plays the major role to define number of sessions and calls to be processed for every instance.
"batchOptions": {
"maxMessageCount": 1
}
Above complete configuration of host.json will do processing of 2nd message only after the execution of 1st message in the Service bus topic subscription azure function.
While answering Retrieve quota for Microsoft Azure App Service Storage, I stumbled upon the FileSystemUsage metric for Microsoft.Web/sites resource type. As per the documentation, this metric should return Percentage of filesystem quota consumed by the app..
However when I execute Metrics - List REST API operation (and also in the Metrics blade in Azure Portal) for my web app, the value is always returned as zero. I checked it against a number of web apps in my Azure Subscriptions and for all of them the result was zero. I am curious to know the reason for that.
In contrast, if I execute App Service Plans - List Usages REST API operation, it returns me the correct value. For example, if my App Service Plan is S2, I get following response back:
{
"unit": "Bytes",
"nextResetTime": "9999-12-31T23:59:59.9999999Z",
"currentValue": 815899648,
"limit": 536870912000,//500 GB (50 GB/instance x max 10 instances)
"name": {
"value": "FileSystemStorage",
"localizedValue": "File System Storage"
}
},
Did I misunderstand FileSystemUsage for Web Apps? Would appreciate if someone can explain the purpose of this metric? If it is indeed what is documented, then why the API is returning zero value?
This should be the default behavior, please check this doc Understand metrics:
Note
File System Usage is a new metric being rolled out globally, no data
is expected unless your app is hosted in an App Service Environment.
So currently this metric File System Usage should only be working on ASE.
I need to have only one instance of an Azure Function App in place and put the following json code in host.json.
But when a function gets triggered by a servicebus queue, I can clearly see in Live Metrics Stream in Application Insights that several servers get provisioned to serve the load. What am I missing to limit the running servers to only one?
{
"version": "2.0",
"WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT": 1,
"maxConcurrentCalls": 2,
"extensions": {
"serviceBus": {
"prefetchCount": 1,
"messageHandlerOptions": {
"maxConcurrentCalls": 2
}
}
}
}
Why do you want to do that?
Remember that each instance can process multiple messages at the same time, so 1 instance does not means one message at at time
Anyways, you can go to your app settings and add the following:
WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT = 1
WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT is an application setting, not a host.json setting. I would also point out that it's somewhat less than 100% reliable, so if you need a hard guarantee of only 1 instance then you should use either a Premium plan or a standard App Service plan rather than a Consumption plan.
I am using Azure Logic App with Azure BLOB Storage trigger.
When a blob is updated or modified in Azure Storage, I pull the content of blob created or modified from Storage, do some transformations on data and push it back to Azure Storage as new blob content using Create Content - Azure Blob Storage action of LogicApp.
With large number of blobs inserted (for example 10000 files) or updated into blob storage, Logic App gets triggered multiple runs as expected for these inserted blobs, but the further Azure Blob Actions fail with following error:
{
"statusCode": 429,
"message": "Rate limit is exceeded. Try again in 16 seconds."
}
Did someone face similar issue in Logic App? If yes, can you suggest what could be the possible reason and probable fix.
Thanks
Seems like you are hitting the rate limits on the Azure Blob Managed API.
Please refer to Jörgen Bergström's blog about this: http://techstuff.bergstrom.nu/429-rate-limit-exceeded-in-logic-apps/
Essentially he says you can setup multiple API connections that do the same thing and then randomize the connection in the logic app code view to randomly use one of those connection which will eliminate the rate exceeding issue.
An example of this (I was using SQL connectors) is see below API connections I setup for my logic app. You can do the same with a blob storage connection and use a similar naming convention e.g. blob_1, blob_2, blob_3, ... and so on. You can create as many as you would like, I created 10 for mine:
You would then in your logic app code view replace all your current blob connections e.g.
#parameters('$connections')['blob']['connectionId']
Where "blob" is your current blob api connection with the following:
#parameters('$connections')[concat('blob_',rand(1,10))]['connectionId']
And then make sure to add all your "blob_" connections at the end of your code:
"blob_1": {
"connectionId": "/subscriptions/.../resourceGroups/.../providers/Microsoft.Web/connections/blob-1",
"connectionName": "blob-1",
"id": "/subscriptions/.../providers/Microsoft.Web/locations/.../managedApis/blob"
},
"blob_2": {
"connectionId": "/subscriptions/.../resourceGroups/.../providers/Microsoft.Web/connections/blob-2",
"connectionName": "blob-2",
"id": "/subscriptions/.../providers/Microsoft.Web/locations/.../managedApis/blob"
},
...
The logic app would then randomize which connection to use during the run eliminating the 429 rate limit error.
Please check this doc: https://learn.microsoft.com/en-us/azure/azure-resource-manager/resource-manager-request-limits
For each Azure subscription and tenant, Resource Manager allows up to
12,000 read requests per hour and 1,200 write requests per hour.
you can check the usage by:
response.Headers.GetValues("x-ms-ratelimit-remaining-subscription-reads").GetValue(0)
or
response.Headers.GetValues("x-ms-ratelimit-remaining-subscription-writes").GetValue(0)
Initially we had deployed database elastic pool in Azure though ARM template. The pool is in Standard edition and had 50 EDTUs in total. This is happening when deploying the app from VSTS through release management.
At some point the databases grew in size so we had to increase the EDTUs of the pool to get some additional space. We did this directly from the portal and we didn't deploy through ARM templates. We increase the EDTUs to 100.
The problem happens now when we want to redeploy the app through VSTS and use the ARM template. We update the value in ARM template to reflect the one we configured in the portal (100) but we are getting the following error.
The DTUs or storage limit for the elastic pool 'pool-name' cannot be decreased since that would not provide sufficient storage space for its databases. "
Our ARM template for the pool is like the following
{
"comments": "The elastic pool that hosts all the databases",
"apiVersion": "2014-04-01-preview",
"type": "elasticPools",
"location": "[resourceGroup().location]",
"dependsOn": ["[concat('Microsoft.Sql/Servers/', variables('sqlServerName'))]"],
"name": "[variables('elasticPoolName')]",
"properties": {
"edition": "Standard",
"dtu": "100",
"databaseDtuMin": "0",
"databaseDtuMax": "10",
}
}
The message is descriptive but we don't get why it tries to decrease the size even if we have provided an appropriate size through EDTUs value.
We partially identified why the problem is happening.
As it's mentioned here and especially the documentation of StorageMB optional argument is better not to provide this and let Azure calculate the size.
Specifies the storage limit, in megabytes, for the elastic pool. You cannot specify a value for this parameter for the Premium edition.
If you do not specify this parameter, this cmdlet calculates a value that depends on the value of the Dtu parameter. We recommend that you do not specify the StorageMB parameter.
As noted in the initial post we didn't specify the StorageMB option in the ARM template and this was set by Azure. What is not mentioned and it was not clear is that this happens only the first time.
So when we deployed for first time with 50 EDTUs the size of the pool was set to 50 GB. When we deployed again and set the EDTUs to 100 then the size remains on 50GB which is confusing. So the solution and probably a safer way is to always specify the StorageMB option for the pool to have a better view and control of what is happening.
My guess is that the current size of the databases in the pool may be greater than the included data storage that comes with a Standard 100 eDTU pool. The included storage amount for that size is 100 GB. The amount of storage is a meter that can be adjusted separately so that you have pools with fewer eDTU but higher amounts of storage. The current max of storage on an Standard 100 eDTU pool is 750 GB.
I wonder if someone went into the portal and also adjusted the max data storage size for the pool. If this is the case, and the databases within the pool now exceed the 100 GB mark, then this error you are seeing makes sense. Since the template doesn't specify the larger data storage amount then my guess is the system is defaulting it to the included amount of 100 GB and attempting to apply that, which may be too small now.
I'd suggest checking the portal for the total size of storage currently being used by the databases in the pool. If it exceeds the 100 GB then you'll want to update the template to also include the additional setting for the max size you are using.
If it doesn't exceed the 100 GB total now I'm not sure what it's complaining about.