I am doing performance testing for my API hosted in Azure App Service. My API response time is increasing whenever there is a spike in the Working Memory Set graph. But my app service plan is showing 50-55% of memory with only one instance running.
Can you clarify to me, why API response time is increasing every time there is a spike in the 'Working memory Set' even though my app service plan memory is only around 50%?
Response Time Graph
Working Memory Set Graph
This is tough to answer without accessing your site logs but in general your api is consuming resources when it's called and then resting again. Similar to how your heartrate would increase when you start running but should return to normal when you stop again.
Is the response time not within your allowed time range? What is the experience from the customer side? Performance testing on cloud environments can be a slippery slope.
I would suggest using the built in Diagnose and Solve blade of your web app, which has the same troubleshooting tools a support engineer would use to assist you in a paid technical support ticket. This should help tell you if there are any issues with your site that might be impacting performance.
Also, please note that if you're running on the free or shared tier that perf testing is not really applicable as we do not support running production apps on those tiers.
Related
we had a problem at 24th of February between 12:35 and 12:50 (pm by utc). Our app starts to respond slowly and it led to failures, but our app didn’t overload Availability and Performance.
Could the system memory and CPU change cause slowdown our app respond? If yes, then how can avoid this problem next time?
There is a screenshot of our app slowdown chart: Web App Slow
In this time we detect change in physical memory and CPU Memory Analysis, High CPU Analysis
Availability and Performance tools detected top 5 slow request execution. It is requests from our app service to external services. Could these requests the overall performance of the app service or app service plan?
Yes. Low compute resource can lead to slow performance (application using high memory/CPU).
Other reasons for performance issue at application level issues, network requests taking a long time, application code or database queries being inefficient, or application crashing due to an exception. To isolate and avoid such issues in future you may try these steps.
Firstly, review the service heath for any reported issues during that time-frame:
You can track the health of the service on the Azure portal
From the screenshots you have shared, it looks like there 2 5xx errors and 48 4xx errors, you may review the logs to fetch more details on the issue.
-Access Kudu - https://.scm.azurewebsites.net/.
To analyze logs and collect diagnostic dumps as required.
Enable diagnostics logging for apps in Azure App Service
Typically, in Azure App Service, for increased performance and throughput, you can adjust the scale at which you are running your application. I'm unsure on what App Service Plan (ASP) you're using.
If you have multiple Apps under a single ASP, the compute resources are shared by all of those running apps. Based on your requirement and usage you may either consider changing your App Service plan to a higher pricing tier or scale-out instances.
We have web api hosted on azure in net framework 461 as target framework and libraries of .net core 2.1.
We are actually calling a 3rd party api and getting data in GBs, and again we need to process this data.
Is there any object memory limitations?
It depends on the underlying app service plan. You are limited to 1024 MB for the Free and Shared plans. The higher plans are limited by the VM's they run on, you can use the pricing calculator to investigate the instance size. Keep in mind there will be some overhead on the VM's so you will likely only be able to leverage roughly ~75% of the capacity reliably. Please find the Azure App Service plan limits here.
You should also consider you application design carefully, can you implement your client with streaming the data received from the API and processing it as it "passes through" This way you can you process huge files without running in to memory limitations. I googled around and found an example of what I am talking about here.
I have an MVC based web app running on Azure. The CPU performance of it has been very predictable over the past five months. However, over the past 24 hours, and most recently, from 1:00 pm to 1:30 pm Eastern time, today, in the USA, I have had CPU spikes nearing 100%. The image below, which is for the past 7 days shows this.
This CPU spike is not coming from my app or my users. There has not been an abnormal increase in users, user activity or queries. I also checked Google Analytics to see if perhaps my site was getting hammered by random users etc. It showed nothing out of the ordinary.
There also was a corresponding huge jump in data going out of my site, which is highly unusual. The second image shows data egress for the past week. However, as I said, I checked my Azure SQL Database Query Store and it shows absolutely nothing out of the ordinary. Furthermore, my DTU percentage never even neared 100% during this time, which it certainly would have if this much data was pulled from the database.
I have basically ruled out anything amiss on my end. Is there some way I can check to see if there were issues with Azure causing this?
If you are suspecting an underlying Azure platform issue, both Azure Service Health and Azure Resource Health are useful resources to determine if you are being impacted by platform issue.
Azure Service Health provides personalized service health information when Azure platform issues impact your resources.
https://learn.microsoft.com/en-us/azure/service-health/service-health-overview
Azure Resource Health provides visibility into whether your Azure resources are healthy or unhealthy.
https://learn.microsoft.com/en-us/azure/service-health/resource-health-overview
For a list of supported Azure resources, you can refer to this article which also describes the set of health checks being performed.
https://learn.microsoft.com/en-us/azure/service-health/resource-health-checks-resource-types
How does one diagnose performance (response time) bottlenecks in Azure?
I've got a .NET Core website on Azure that consists of a web app service and one SQL database.
I've set up load test and deployed it via the cloud to hit the website. Configuration properties for the load test agents are
4 cores
start at 10 simultaneous users, + 10 every 20 seconds, up to 150 users
5 second think time between requests
Web app resource allocation is as follows
2 instances of
4 Core, 7GB RAM (S3 Standard)
This image shows the hardware utilization during the load test (2 tests shown, around 1pm and 1:30pm)
Seems reasonable, except my response times are, in my opinion, too slow, considering the hardware isn't stressed at all. For instance, at 10 users, my response time starts at 20ms, but at 150 users (at the end of the test), I'm seeing 5 second response times.
For the last portion of the test, My requests per second was at about 50.
Database performance, at 100DTUs, doesn't seem to be a factor:
What else can I do to diagnose slow response times? If the web server hardware isn't pegged, and the database isn't even sneezing, what other knobs can I turn on Azure?
The long response time bottleneck can be caused by various reasons, for example, bandwidth restrictions, source limited, bad application design, dependency of tightly coupled component etc.More information about how to troubleshoot slow web app performance issues , please refer to the document. There are some snipped from the document.
Enable diagnostics logging for your web app.
WebApp provides diagnostic functionality for logging information from both the web server and the web application.
We can enable Detailed Error Logging, Failed Request Tracing, Web Server Logging for web server diagnostic
Use Kudu Debug console (https://. scm.azurewebsites.net/)
Kudu provides environment settings for your application, log stream, diagnostic dump
We also can use Azure Application Insights to monitor the usage and performance of our app, then we can get more detail info about request, more detail exception info, response time and so on.
If we get more detail info about application exception, request failed, server logs and application log, it will be more helpful for us to diagnose.
There are also some related articles about how to diagnostic Web App and how to use Application insights:
Enable diagnostics logging for web apps in Azure App Service
Monitor performance in web applications
Diagnose exceptions in your web apps with Application Insights
Using Search in Application Insights
Try using new relic extensions
It provides great insight to response time and lot more with the free account
You can also enable application insights on the web app. It will provide you with details on response time and other details
My cloud service app deployed on windows azure seems to be taking a whopping 2GB memory!! That is the w3wp.exe process. I have checked this by remote desktop. This is under hardly any load!
I am using co-located caching with 20% of the allocated ram going to the cache. Its a medium VM with 3.5GB ram.
What could be causing this? Any pointers where to start? Its an asp.net app talking to sql azure using Entity framework.
Sql azure is in the same data center as the cloud service, I also use the storage service which is in the same data center too.
Thanks
A normal Azure WebRole does not encounter this issue, so this is something specific to your app and I don't think anybody on here can just guess at what it might be without lots of details about your application. I think your best bet is to figure out what objects are taking up all the memory and where they are rooted. It is likely one simple thing that can be easily fixed in your code once you know what object is consuming the memory. Check out the following 3 blog posts for some really good walkthroughs on how to do this:
http://blogs.msdn.com/b/tess/archive/2005/11/25/496973.aspx
http://www.codeproject.com/Articles/19490/Memory-Leak-Detection-in-NET
http://blogs.msdn.com/b/delay/archive/2009/03/11/where-s-your-leak-at-using-windbg-sos-and-gcroot-to-diagnose-a-net-memory-leak.aspx