We have a Azure function behind private end point. The Application Insights associated with this Azure function is unable to capture logs.The transaction search in the App Insights gives the below error message
AI: Diagnostic message: backoff logic was enabled. Backoff internal exceeded 60 min. Did anyone face similar issue.
TIA.
I understand that some of the logs data from your Azure Functions is not captured by the Application Insights and showing the backoff internal exceeded 60 min in the diagnostic message.
There is a retry mechanism in the Azure Functions - Application Insights SDK for specifically solving the issues of data loss, Events missed and Application health monitoring.
Some of the Azure Functions provides with the built-in retry mechanism, retry policies with that individual function runtime.
Whatever the Runtime Stack of Azure Functions integrated with the Application Insights, there are retry mechanisms provided by Microsoft for resolving the production issues of data (logs) missing from the Event based triggers.
Reference: MS Doc on Retry Mechanism Implementation.
Related
I keep getting an error 503 on my health checks for my azure functions, it says error 503. Functions host is not running. It's very inconsistent and only happens once every few days, I'm on the consumption plan but looking at whether a premium plan would fix the issue.
In Azure Functions, 503 service unavailable causes for the reasons like:
Function host is down/restarting
Platform issue due to the backend server not running/ allocated
Memory leak/issue from the code causing the backend server to return 503
To get some insights about the function host related issues, take a look into the "Diagnose and solve problems" blade in the Function app and select the "Function app down or reporting" detector. This detector will show all the diagnostic information about the function app and its infrastructure.
503 service unavailable comes sometimes when the function takes more than 5 minutes to return an HTTP response in consumption plan. Regardless of the function app timeout setting, 230 seconds is the maximum amount of time that an HTTP triggered function can take to respond to a request.
For longer processing times, use Azure Durable Functions async pattern. Refer to this link.
We should not change the app settings frequently in the production environment. If you update the app settings, the app will be restarted. In this cases, you will get 503 error. In order to avoid this, you can use the slot feature
Function host is not running
This issue happens due to invalid host.json. To diagnose, it's best to look at the function host logs from the log stream in Azure Portal.
Few errors and resolutions of this kind of error are:
If you have any startup.cs class, check if any error available where the errors were logged in Application Insights.
One of the reason is a missing app setting. Ensure you publish local settings as well.
If it didn't help then one of the reason could be platform issue and to confirm this we need to look into the backend logs on what was happing during that time resulting in 503 errors.
You can create the support ticket with Microsoft to assist you further.
According to this thread, one possible cause of 503 service-unavailable responses is when the service consumes more memory than what is available under the consumption (serverless) plan, causing the service to be evicted. Switching to a dedicated hosting plan can fix this issue. According to Microsoft's documentation, it appears that the function is allowed a maximum of 1,536MB of memory at one time. Of course, it could also be the case that your function is exceeding any of the other service limits associated with that plan, so my advice would be to add instrumentation and code defensively.
Got 503 after redeploying an Azure function.
Turned out the Python version had defaulted to 3.6, updated to 3.9 and started working.
I was using terraform and github actions
I am using Azure Functions V2 with Azure service bus trigger which is setup to be triggered when there is a message on a ASB subscription. I am trying to see if there is any metrics available on the message receive latency so that i can plot it in my dashboard. I am using ASB standard SKU so when a noisy neighbor issue happens, at least i can identify it as so.
I suspect the code under trigger attribute takes care of retrieving the message. So is there a way i can instrument this to visualize the avg latency? Basically the function provides metrics just for the execution. What i am looking for is end to end latency metrics.
If you are using application insights for logging, the function runtime logs a line like below for each message:
2019-08-04 21:09:06.026 Trigger Details: MessageId: <Guid>, DeliveryCount: 1, EnqueuedTime: 8/4/2019 9:09:05 PM, LockedUntil: 8/4/2019 9:09:35 PM
If you parse this log line for the EnqueuedTime, and also the logging time (the first timestamp), then we can calculate the send-to-receive latency.
I know this is not the ideal way. I wish function runtime would log events that can easily be looked at in Azure Monitor. Or if it passed in the EnqueuedTime with the input trigger, then we can log it ourselves as a custom event in Application Insights.
I'm experiencing intermittent 503 Service Unavailable from azure storage.
Webjobs runner hosted as a top shelf service. Due to the fact that I used JobHost.Start() instead of JobHost.RunAndBlock() every time I get 503 from azure storage, service ends up in a corrupted stopping state.
When I switched to JobHost.RunAndBlock() using instead service is running continuously now but after 503 exception queue trigger stops polling queues.
I use standard azure queue trigger bindings. No manual setup.
Anyone experienced similar behaviour? How to recover from such connectivity errors?
Assuming you are using C# for your web job, I think using something like Polly or Enterprise Library's Transient Fault Handling Application Block to implement retry logic for when an occasional error occurs while using an Azure service, as you might be hitting a throttling thresholds (resource limit per your selected service tier).
Hope it helps!
I want to create logging for API and azure functions.
As thinking to use, "service bus" to create logging.
Logging needs for each request, response and error.
Would it be correct approach to do logging for api management and azure functions through service Bus. and appreciate if any example to create service bus and call from azure or api management to log the requests/responses.
Note: Regarding Application Insights, as found its hit the performance and its more for performance monitoring then logging. https://blogs.msdn.microsoft.com/apimanagement/2018/01/12/application-insights-integration/
I would still use Application Insight for that. You want to track requests and errors which Applicatin Insight will offer out of the box and also provides you a query language to query your logs or to build dashboards. Regarding your performance concern, you should just test the impact on your system - it most likely isn't that relevant.
I am working on Azure web API which uses Log4Net Application Insight appender to track traces. We are planning to use the Application Insight SDK directly (TrackTrace(), TrackEvent()) in order to use built-in alerting features.
However it does seem like, Azure is not supporting trace or event alerts but only metric alerts. So we got some issue there.
If I go one step back, web API is invoked by number of Logic Apps runs in x time intervals. Logic App simply calls web API (business logic is here) and it log all information/managed exceptions.
The main requirement is to be proactive when an exception happens like sending a mail to technical inbox. Secondary requirement is notify sources, if it got any data issues.
Any suggestion on our approach please, what we can do more to fulfill our requirements?
Please refer to this App Insights Exception Alerts. It is possible to set alerts for rates of exceptions in a defined time period.
Set up Exception alerts
You can also invoke webhooks to perform additional actions when the alert is fired.
Thanks Sreejit for your suggestion.
To conclude the answer, what's the preferred engine to process alerts please, through application insight or log analytics?
application insight only support metric alerts only, so the application exceptions need to be flagged as custom metrics as opposed to traces.
[TelemetryClient --> TrackTrace() vs TrackMetric()]
If we go for log analytics, we can use custom events. Then use alert management solution in OMS workspace. Perhaps even use of a separate logic app to built the logic of sending alerts.
[TelemetryClient --> TrackEvent()]