I have an Azure Functions app that is running on a timer trigger that I don't expect to exceed the 10 minute timeout limit, but I would like to receive an alert in the unlikely event that the application runs longer than 10 minutes. Is this possible to do in Application Insights? I didn't see an alert trigger for this use case. In Application Insights there is a "Long dependency duration" in the Smart Detection settings where I can add an email and there's also a "Failure Anomalies" alert rule already set up. Will either of these alert me if a function is running longer than 10 minutes?
I'd also like an alert if an individual function instance encounters any type of exception. I can set this up myself in the Python code by wrapping my code in a try except block and emailing if an exception is caught, but it would be easier if this was possible in Application Insights.
You can go to your host.json of your function app,
and change it like this:
{
"version": "2.0",
"extensionBundle": {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
"version": "[1.*, 2.0.0)"
},
"functionTimeout": "00:10:00"
}
Then, if your instance is running more than 10 minutes, it will throw out an error.
So you can go to the Application Insight of your function and and select like below:
You can set the action type as Email/EMS and give it your email address, then you will receive an email when the timetrigger is timeout.
Related
We are currently running load tests on our Azure Function App but the throughput is not what we expected.
There are multiple functions in the Function App but the ones with the most traffic are one with an Event Hub Trigger and one with a Service Bus Trigger consuming messages from a Session-Enabled Queue.
When the system is under load, Messages in the Session-Enabled Queue wait for up to 10 Minutes in the queue until they get processed by the consuming Function.
I know there are some settings in host.json to tune this behavior but it's still far from what we expect.
This is our host.json
{
"version": "2.0",
"logging": {
"applicationInsights": {
"samplingSettings": {
"isEnabled": true,
"excludedTypes": "Request"
}
}
},
"extensions": {
"serviceBus": {
"prefetchCount": 100,
"sessionHandlerOptions": {
"autoComplete": true,
"messageWaitTimeout": "00:00:30",
"maxAutoRenewDuration": "00:55:00",
"maxConcurrentSessions": 200
},
"batchOptions": {
"maxMessageCount": 1000,
"operationTimeout": "00:01:00",
"autoComplete": true
}
}
}
}
So i would expect the Function App to process up to 200 Sessions concurrently but in fact, although the Function Runtime provisions lots of instances, most of them seem to sit there and idle out. So to me it seems there is still another setting limiting the througput of the Function App.
I know it would improve performance if we would split the functions to separate Function Apps but as the load on the both functions is quite similar my plan was to postpone this step to a later stage and still get acceptable throughput with a single Function App.
We are using Azure Functions 3 on .NET Core 3.1 with
Microsoft.Azure.Functions.Extensions 1.1.0
Microsoft.Azure.WebJobs.Extensions.ServiceBus 5.0.0
Microsoft.Azure.WebJobs.Extensions.EventHubs 5.0.0
on a Windows Consumption Plan.
Thank you for any hints on how to achieve acceptable throughput.
I figured out that handling Batch Messages (receiving ServiceBusMessage[] instead of single instances) in the ServiceBus-Triggered Function along with enabled Sessions has a massively negative impact on scalability.
After changing this to single instances, the behavior of the system was as expected and the sessionHandlerOptions in host.json were respected.
I am wondering though what's the reason for this. I guess it could be related to the circumstance that Azure Function Instances lease a number of sessions from Service Bus to process but could not find anything in the documentation on that.
We have an Azure function v3 running Node, consumption plan, with an input trigger connected to a cosmos database. The function.json looks like this:
{
"disabled": false,
"bindings": [
{
"type": "cosmosDBTrigger",
"name": "productDocuments",
"collectionName": "products",
"direction": "in",
"connectionStringSetting": "DB_CONNECTION_STRING",
"databaseName": "product-management",
"createLeaseCollectionIfNotExists": true,
"maxItemsPerInvocation": 1
},
{
"name": "productDocument",
"type": "cosmosDB",
"databaseName": "product-management",
"collectionName": "products",
"createIfNotExists": true,
"connectionStringSetting": "_DB_CONNECTION_STRING",
"direction": "out"
}
],
"scriptFile": "dist/nameOfFunction.js"
}
But this trigger is working really, really slow and unreliable. If we add an item to the DB it sometimes triggers straight away, sometimes it seems to take hours and sometimes not at all. I am manually monitoring the cosmos db so I can see that items are added.
I am looking at this page, and most of the time nothing happens. I don't know how else to debug this
Should it really take hours for an invocation to show up here? Or is it the trigger that's unreliable?
General guidance is in this doc: https://learn.microsoft.com/azure/cosmos-db/troubleshoot-changefeed-functions#my-changes-take-too-long-to-be-received
What happens on Consumption Plan is that, after a period of inactivity, instances are deprovisioned. When a new instance is provisioned, it hits a cold start.
The key part here is that, when your instances are deprovisioned, they are not checking the Change Feed for events, so how does Functions know when to "wake them up"?
There is a periodic check done by an external component that checks to see if there are new changes, if there are new, then it would provision new instances of your Function to start consuming them.
This external component in your case, could be having an issue or delays in this checks.
If you have no Function logs for an hour even though you are making changes to the monitored collection, I would try to contact Azure Support to understand why is your Function not "waking up".
One of the known issues I've heard about was related to where the Cosmos DB Connection Strings were stored. Apparently this component at some point (maybe it's already fixed) had a problem where it could not access the Connection String if it was saved in "Connection Strings" section of the Function configuration, but was looking for it only on the "App Settings". In this cases, it could not wake up the Function and the Function only woke up if someone opened it on the Azure Portal. My recommendation would be to check where are you storing your connection string and see if you can move it to "App Settings" and see how it behaves.
Our problem with this was that we had two separate functions that both had a CosmosDBTrigger on the same collectionm but used the same lease, and apparently you can't do that. So it was solved by setting two separate leases (we used the leaseCollectionPrefix in the function.json.)
Using Python 3.8 for an Azure functions app in which all the functions are HTTP triggers. We have HTTP 2 enabled ...
Below is our host.json file
{
"version": "2.0",
"extensionBundle": {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
"version": "[1.*, 2.0.0)"
}
}
We are sending 30 requests at the same time from the client (Angular 9) application to the server (15 are OPTIONS requests and the other 15 are GETs) and are noticing that 20 of those are handled relatively quickly but then the rest take a noticeably longer time to process. Below are two of the requests side-by-side
For the longer requests, I have verified through curl and Postman that individually they return in a much quicker period of time, which leads me to believe there is some concurrency setting on the srever I can adjust but I can't figure out where.
Edit: Here's a little more information. My anonymous function begins like the below ...
def main(req: func.HttpRequest) -> func.HttpResponse:
"""."""
logging.info("received request")
but note the times reported in the Azure log for that function when the function responds slowly ...
2020-11-17 14:29:24.094 Executing 'Functions.download-image' (Reason='This function was programmatically called via the host APIs.', Id=xxx-xxx)
Information
2020-11-17 14:29:32.143 received request
There is an 8 second delay between when I'm told the function was invoked and the first logging statement from the function. Below is what my "Scale Out" looks like ...
For this problem, you can check if the scale out tab of your function app.
You can scale out the instance manually, or you can also define custom autoscale according to your requirement.
As you mentioned the first 20 requests are handled relatively quickly, so I guess you choose P1 app service plan(as below screenshot show) because P1 plan has maximum 20 instances. If your current app service plan just has maximum 20 instances, you need to scale up your plan to a higher pricing tier.
By the way, if you want the requests be handled quickly, you'd better enable "Always on" in your first screenshot. Otherwise your function app will idle if it it hasn't received a request for a long time.
This is my first time using Azure functions and service bus.
I'm trying to build a function app in Visual Studio and I am able to connect to the queue topic (which I don't control). The problem is that every time I enable a breakpoint, 10s of messages are processed at once in VS which is making local testing very difficult (not to mention the problems arising from database pools).
How do I ensure that only 1 message gets processed at once until I hit complete?
public static void Run([ServiceBusTrigger("xxx", "yyy", AccessRights.Manage)]BrokeredMessage msg, TraceWriter log)
{
// do something here for one message at a time.
}
Set maxConcurrentCalls to 1 in the host.json. You also could get answer from host.json reference for Azure Functions
maxConcurrentCalls 16 The maximum number of concurrent calls to the callback that the message pump should initiate. By default, the Functions runtime processes multiple messages concurrently. To direct the runtime to process only a single queue or topic message at a time, set maxConcurrentCalls to 1
For function apps 2.0 you need to to update the host.json like this:
{
"version": "2.0",
...
"extensions": {
"serviceBus": {
"messageHandlerOptions": {
"maxConcurrentCalls": 1
}
}
}
}
Since the host.json file is usually published to Azure along with the function itself, it is preferable not to modify it for debugging and development purposes. Any change to host.json can be made to local.settings.json instead.
Here is how to set maxConcurrentCalls to 1:
{
"IsEncrypted": false,
"Values": {
...
"AzureFunctionsJobHost:Extensions:ServiceBus:MessageHandlerOptions:MaxConcurrentCalls": 1
}
}
This override functionality is described here: https://learn.microsoft.com/en-us/azure/azure-functions/functions-host-json#override-hostjson-values
Note: Even though it may seem duplicate, My issue is different and I request you to read complete description before hastily marking the question down just by reading Question title.
I opened "Azure function is not triggering on scheduled time " issue on official "azure-webjobs-sdk-script" git repository on May 24, 2017-but there is no reply yet. So I am re-asking this here.
I am using azure function in consumption plan, and have scheduled it to execute at every 4.00 am utc by setting following cron expression in function.json:
{
"bindings": [
{
"name": "myTimer",
"type": "timerTrigger",
"direction": "in",
"schedule": "0 0 4 * * *"
}
],
"disabled": false
}
Azure function does get execute on time only if I am logged into portal or clicks on function blade. It does not invoke when I am logged out from portal (suggesting either system listener or function go to sleep after some interval).
Their official documentation states that functions in consumption plan do not require any other settings (like Always on) to keep the function alive, instances of the Azure Functions host are dynamically added and removed based on the number of incoming events.. So as per documentation, no other settings I have to configure to execute function in consumption plan.
What I have tried?
From "Timer triggered azure function not getting triggered" question on SO, I re-checked and ensured my plan (consumption plan) and time zone. (I want it to run in utc, so no explicit setting is required)
From "#1445: Azure function timer trigger not firing" git issue, I checked whether it was just logs that are not appearing. But I am certainly sure, that its not the logs but the actual function does not get triggered unless I am having my portal on or trigger it manually.
I tried to check whether this behavior exists if I change the schedule to a more closer recursive invocations--I scheduled function to get executed at every 2 hours, and this schedule perfectly worked even when I logged out or did not awake function manually. This means, there is some issue when schedule is set to run on larger set of intervals (in my case each day)
As discussed here in #1534, deleting and redeploying the function completely in new function app did not reproduce the issue. So deleting the function and/or function app-and redeploying the same should make things working. Meanwhile, Azure team has announced to add internal logging which will help future diagnosis.