I noticed a difference in AVG and MAX response time between Insights (Performance / View in Logs / Response Time) and the response time that I see in the (Overview / Response Time) of the Web App.
The values on OverView are higher (seems that there are some requests that take 20 seconds and mess up the average.
I have no traces of these requests looking in the Insights response time or trying to search the requests that take more time.
Insights values seem more correct because seems that the web app is working properly. What could be that difference?
There is a similar question with yours: What is the difference between Azure Monitor 'Response time' and AppInsights 'Duration'?
Response time in overview page is the Azure Monitor metric, just like the reply described:
Azure Monitoring is gathering statistings on the web server as a
whole. That's why it is reporting on CPU and memory usage in addition
to response times.
Response time in performance is the Application Insights metric
Application Insights calculate average request duration for all
requests, without considering if calculated before request hits the
load balancer.
Related
We have a service which pings our EP1 Premium service and yesterday we received 3 client side timeout errors after 2 minutes of waiting. When opening the trace in App insights, these requests which time out are not even logged and have no trace of ever being received Azure side, and therefore stay unanswered. By looking at the metrics provided in the Azure Functions app, I found out that 1-2 minutes after the request has been sent, the app loses all its ability to work as its Total App Domains falls to 0 as well as all connections, threads and so on and this state lasts until the next request is received, therefore "skipping" the request that happened beforehand. This is a big issue as I need to make sure requests get answered in a timely manner.
The client service sent HTTP requests to the Azure Functions app expecting an answer, only to time out while the Azure-side doesn't have any record of ever receiving the request.
I believe this issues is related to Consumption Plan of Azure Functions called Cold Start behaviour. The "skipping" mechanism is explained below:
Apps may scale to zero when idle, meaning some requests may have additional latency at startup. The consumption plan does have some optimizations to help decrease cold start time, including pulling from pre-warmed placeholder functions that already have the function host and language processes running.https://learn.microsoft.com/en-us/azure/azure-functions/functions-scale#cold-start-behavior
Please also consider of having look on this article, which explains the behaviour. https://azure.microsoft.com/en-us/blog/understanding-serverless-cold-start/
Context: During performance evaluation we have noticed that there is a discrepancy in the response time between Postman and Application Insight.
During debugging we found that the response time given obtained from Postman Vs Application Insight are different for the same transaction. We noticed is for different transactions. Why it is so?
Current Configuration: We have application insight for an application which is running within an app service and "Adaptive Sampling" is enabled for Application insight.
Note: We are tracing the transaction with tracing ID.
Comparisions Chart:
Smaller Evaluation:
Medium Evaluation:
When the system is under stress the response time goes to min on postman, but in application insight the response time are change drastically.
May be the difference in the response time is lesser but when the number of transactions is more and when we see the 95th (refer the example below) and 99th percentile the game completely changed. Any suggestion on this? Am I missing something?
For example:
Postman Result:
Application Insight:
Posting the suggestion provided by Peter-Bonsas an answer so that it will be helpful for other community members.
From the below screenshot application insights won't take the actual latency of sending the request to the api and getting the response back. The time which we will be seeing in app insights is total execution time on the azure side
In some 1-5% of our requests, we are seeing slow communication between APIs (REST API requests). Both APIs are developed by us and hosted on Azure, each app service on its own app service plan in the same region, P1v2 tier.
What we are seeing on application insights is that POST or GET requests on origin API can take a few seconds to execute, while real execution time on destination API is only a few milliseconds.
Examples (first line POST request on origin, second execution time on destination API): slow req 1, slow req 2
Our best guess is that the time difference is lost in communication between components. We don't have an explanation for it since the payload is really small and in most cases, communication takes less than 5 milliseconds.
We dismiss the possible explanation it could be due to component cold start since it happens during constant load and no horizontal scaling was performed.
Do you have any idea what might cause it or how to do additional analysis in order to discover it?
If you're running multiple sites on the App Service Plan, then enable the "Always On" setting for your web app > All Settings > Application Settings > Click on Always On
See here for details: https://azure.microsoft.com/en-us/documentation/articles/web-sites-configure/
When Always On is off, the site is shut down after 20 minutes of inactivity to free up resources for any additional websites that might be using the same App Service Plan.
The amount of information it needs to collect, process and then present itself requires some time, and involve internal calls as well, that is why considering the server load and usage, it takes around 6 to 7 seconds sometimes even more.
To Troubleshoot that latency, try this steps, provided by Microsoft.
I have a Web API application hosted under an on-prem (not Azure) IIS which logs to Insights and is showing Request durations very similar to the times reported in the IIS logs.
However, every so often, there is an entry in the IIS logs with a TimeTakenMS duration which is much longer than the Insights Request duration.
For example, a relatively simple request to read a small amount of data from the DB is logged with a total duration by Insights as 984.126ms but IIS is logging it with TimeTakenMS as 43718.
I have conclusively linked the two requests (they show a unique URL). I have eliminated application start up/recycle times (the application is clearly already started and serving other requests and the recycle boundary is hours away from the logged time stamps) and I have eliminated exceptions (Insights is not logging any exceptions at this time).
I also have a StopWatch set up in the controller for every web method and TrackTrace the elapsed milliseconds to Insights in a finally block, just before the method returns.
What other factors could cause an IIS hosted application to fail to log actual execution times to Insights but cause IIS to log much longer times?
I've considered network + processing time, but 42 seconds seems an unreasonably long duration (even for the start up time of this particular application.)
After doing some load testing with a an azure function on the consumption plan that scales I ran into an interesting scenario. Right now I have a function app that is expecting an http post behind an api management instance. The function app does some processing and returns a response to the caller. The API manager has no caching for the time being since we are expecting the process to just pass through the body through for processing.
When I call the function directly, the app will scale appropriately and I see SOME cold start behavior from the function app, but the average response time is sub 200 ms from a local request to azure for a sustained load.
When I call the function through the api manager and have a sustained load on the instance I start seeing 30-60 second response times around the 5 minute mark of the load test.
The load is 2 requests per second and fully asynchronous so there is no blocking. Additionally, there are no database calls, this is solely a compute function.
Has anyone else seen this behavior?
It seems to be a setting or LB problem. What you can do is reduce the functionTimeout setting in your host.json (e.g. 20 seconds) and then, add a retry policy on API Management.
https://learn.microsoft.com/en-us/azure/api-management/api-management-advanced-policies#example-7