When I look at request tracing in azure application insights I sometimes see this thing:
This question holds 2 parts:
Is it correct that the app service icon part is the outgoing dependency and the app service plan icon is the actual execution of "my" code in the app?
The second part: if that is true; why does it sometimes look like it does on the right side of the image (with a much large dependency duration then actual execution, 1.7 mins to 309 ms).
I have checked my performance and diagnostics and the cpu is not
Application insights clarification and what is my app service doing?
I understand that the dependencies time also tracked in the Request calls as per this MS Doc of Azure Monitor Transaction diagnostics telemetry details.
Above image taken from the same reference. it is showing the levels of tracking such as: Request > Dependency > Exception - Here, every level is a component.
Some are expandible components, and some are non-collapsible components.
This question holds 2 parts: Is it correct that the app service icon part is the outgoing dependency and the app service plan icon is the actual execution of “my” code in the app?
In your case, App Service is the main component and App Service Plan is the expandible component which is a part of App Service Request and consists of exceptions and external dependencies.
The second part: if that is true; why does it sometimes look like it does on the right side of the image (with a much large dependency duration then actual execution, 1.7 mins to 309 ms).
I believe the sum of the execution of dependency components duration and exception duration is the main component execution time.
Related
In some 1-5% of our requests, we are seeing slow communication between APIs (REST API requests). Both APIs are developed by us and hosted on Azure, each app service on its own app service plan in the same region, P1v2 tier.
What we are seeing on application insights is that POST or GET requests on origin API can take a few seconds to execute, while real execution time on destination API is only a few milliseconds.
Examples (first line POST request on origin, second execution time on destination API): slow req 1, slow req 2
Our best guess is that the time difference is lost in communication between components. We don't have an explanation for it since the payload is really small and in most cases, communication takes less than 5 milliseconds.
We dismiss the possible explanation it could be due to component cold start since it happens during constant load and no horizontal scaling was performed.
Do you have any idea what might cause it or how to do additional analysis in order to discover it?
If you're running multiple sites on the App Service Plan, then enable the "Always On" setting for your web app > All Settings > Application Settings > Click on Always On
See here for details: https://azure.microsoft.com/en-us/documentation/articles/web-sites-configure/
When Always On is off, the site is shut down after 20 minutes of inactivity to free up resources for any additional websites that might be using the same App Service Plan.
The amount of information it needs to collect, process and then present itself requires some time, and involve internal calls as well, that is why considering the server load and usage, it takes around 6 to 7 seconds sometimes even more.
To Troubleshoot that latency, try this steps, provided by Microsoft.
I have following controller class and a PUT method in an azure app service (.NET C#)
MyController {
[MyApiRoute("anapi")]
// PUT: anapi/items
[HttpPut("items")]
public async Task<ActionResult<AnObject>> ManageLineItems()
{
//some code here....
}
}
I am trying to measure performance of making a call to this REST URL using azure insights. It shows First line with AJAX call as 7.7 sec and second line with controller method as 3.9 seconds as shown in diagram. My understanding is that first line in azure insights would correspond to call to this URL and second line corresponds to the time the method in controller took to execute. If that is correct, were typically on azure app service would 7.7 sec - 3.9 sec would be spent? This doesn't appear to be time spent by my code.
If my understanding about these two lines in azure insights is incorrect, can any experts here explain me what this means? Also, were could 2.8 seconds which is difference between total execution for route and method would have been spent?
Please let me know if i need to provide more details about the problem.
Application Insights is a full-scale application performance monitoring tool which does much more than only logging. It is Application Performance Management (APM) service for collecting and monitoring the application log data
In the above picture in your question, the first line represents the outgoing dependency form your web service and the following line represents the incoming request. And the duration column you see is the cumulative request duration at each level.
If Service A calls service B then from point of view of Service A , call to Service B is an ‘Outgoing dependency’ and when this request is received at Service B then from its point of view the call is an ‘Incoming request’ from service A. That's the reason App insights captures both of them .
This is called end to end correlation, which works by default when you enable application insight.
I would suggest to read this End to end correlation using Azure application insights for asp.net mvc apps document to understand the entire to understand hierarchical view of the request flow telling which service or API calls happened along with where and at what point exception occurred along with all the logs / traces that happened during that request.
I have deployed a .Net Core API to Azure as an App Service.
I have set the Always on feature to true.
When I log the requests, I see that Azure Always on requests are coming every 5 minutes.
My usage with API is HTTPS but Always on requests are sending with HTTP. I don't know if this is the case
For the first request, it is sometimes 10 seconds, but after the first request, it is around 100ms.
What is missing here?
I have logged the durations:
There are quite a few reasons why this might be the case:
You're connecting to resources that take time connecting to the first time
Some information is being cached and needs to be read the first time
There is initialization code present
Lazy instantiation of (static/singleton) instances
... other ...
Add some logging to your application, maybe enable Application Insights if you haven't done so already and go try to find the culprit.
I have an Azure Functions application which once in a while "freezes" and stops processing messages and timed events.
When this happens I do not see anything in the logs (AppInsight), neither exceptions nor any kind of unfamiliar traces.
The application has following functions:
One processing messages from a Service Bus topic subscription (belonging to another application)
One processing from an internal storage queue
One timer based function triggered every half hour
Four HTTP endpoints
Our production app runs fine. This is due to an internal dashboard (on big screen in the office), which polls one of the HTTP endpoints every 5 minutes, there by keeping it alive.
Our test, stage and preproduction apps stop after a while, stopping to process messages and timer events.
This question is more or less the same as my previous question, but the without error message that was in focus then. Much fewer error messages now, as our deployment has been fixed.
A more detailed analysis can be found in the GitHub issue.
On a consumption plan, all triggers are registered in the host, so that these can be handled, leading to my functions being called at the right time. This part of the host also handles scalability.
I had two bugs:
Wrong deployment. Do zip based deployment as described in the Docs.
Malformed host.json. Comments in JSON are not right, although it does work in most circumstances in Azure Functions. But not all.
The sites now works as expected, both concerning availability and scalability.
Thanks to the people in the Azure Functions team (Ling Toh, Fabio Cavalcante, David Ebbo) for helping me out with this.
I'm running an Azure Function app on Consumption Plan and I want to monitor the amount of instances currently running. Using REST API endpoint of format
https://management.azure.com/subscriptions/{subscr}/resourceGroups/{rg}
/providers/Microsoft.Web/sites/{appname}/instances?api-version=2015-08-01
I'm able to retrieve the instances. However, the result doesn't match the information that I see in Application Insights / Live Metrics Stream.
For example, right now App Insights shows 4 servers online, while API call returns just one (the GUID of this 1 instance is also among App Insights guids).
Who can I trust? Is there a better way to get instance count (e.g. from App Insights)?
UPDATE: It looks like data from REST API are wrong.
I was sending 10000 messages to the queue, logging each function call with respective instance ID which processed the request.
While messages keep coming in and the backlog grows, instance count from REST API seems to be correct (scaled from 1 to 12). After sending stops, the reported instance count rapidly goes down (eventually back to 1, while processors are still busy).
But based on the speed and the execution logs I can tell that the actual instance count kept growing and ended up at 15 instances at the moment of last message processed.
UPDATE2: It looks like SDK refuses to report more than 20 servers. The metric flats out at 20, while App Insights kept steady growth and is already showing 41.
Who can I trust? Is there a better way to get instance count (e.g. from App Insights)?
Based on my understanding we need to use Rest API endpoint to retrieve the instance, App Insights could be configured for multiple WebApps, so the number of servers online in the App Insights may be for multiple WebApps.
Updated:
Based on my test, the number of the application insight may be not real time.
During my test if the WebApp Function scale out then I could get multiple instances with Rest API, and I also can check the number of servers online in the App Insights.
https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourcegroup}/providers/Microsoft.Web/sites/{functionname}/instances?api-version=2016-08-01
But after I finished the test, I could get the number of the instance with Rest API is 1, based on my understanding, it is right result.
At the same time I check it in the Application Insight the number of the servers online is the max number during my test.
And after a while, the number of server online in the application insight also became 1.
So If we want to get the number of intance for Azure function, my suggestion is that using REST API to do that.
Update2：
According to the DavidEbbo mentioned that the REST API is not always reliable.
Unfortunately, the REST API is not always reliable. Specifically, when a Function App scales across multiple scale units, only the instances from the 'home' scale unit are reflected. You probably will not see this in a smallish test, but likely will if you start scaling out widely (say over 20 instances).