I've been looking at my app's performance and I noticed relatively high Send Request Times.
If I got it right the Send Request time includes the network trip of the payload from the client to the server + the server's processing time (AppInsights).
Looking at the page load time I'm confused because the ajax call is taking ~300ms while, but the page loads in 4.27s. AppInsights is missing the server response graph for some reason (I have two websites in the same web app and for one I see server side telemetry, but for this one it's missing).
I don't see high resource consumption on the web app so I'm not really getting what's taking the almost 4 seconds in the request.
Is there a way to dig further?
To find out which part of the application is causing slowness, you can collect .Net profiling traces during the time of issue, which will help to get the complete traces of application and its thread which are running on the app service plan instance during the time of issue, by analyzing the report generated by traces you can easily identify what is causing the issue.
Following are the steps to collect .Net Profiler traces:
Login to Azure portal.
Select the Azure Web App that you intend to troubleshoot
Click on Diagnose and solve problems > Diagnostic Tools > Profiler.
Once Collect .NET Profiler Trace tab is opened kindly click on
Collect Profiler Trace button to collect profiling traces. As shown
below.
After Collecting the Profiling traces, it will be generating the report as shown below which will help you in understanding cause slowness.
Further Clicking on Details Button it will be opening call stack with details.
Please find below link for more information:
https://blogs.msdn.microsoft.com/appserviceteam/2018/09/24/announcing-the-new-app-service-diagnostics-experience/
Related
Currently my app service in azure performs well in staging environments but when it comes to production i find an unusual spike in the response times in application and demands for app-service restart in a few scenarios.
I am trying to analyse this issue and was trying to generate a thread dump using the kudu lite but the container is crashing when we try this and i am currently working with Microsoft on this.
Meanwhile what are the best practices or approach to understand this . I have tried to dig in the application insights logs but there was no much info about the worker threads that are hung up or if the thread pool is exhausted .
Please advise me on this situation on how to analyse and reverse engineer to get to the bottom of this problem .
Thanks in Advance !
In your app service plan check the Diagnosed and solve problems” analyze CPU usage of your App on all instances and see a breakdown of usage of all apps on your server. And check the CPU utilization on each instance serving your app and identify the app and the corresponding process causing High CPU in percentage. Check the Troubleshoot performance degradation of our service
You can use the Kudu console to download the Diagnostic dump in a KUDU -> Tools -> Diagnostic dump
Once you download the diagnostic dump you have log files and a Deployment directory.
You can check the log files to know the spike details
Refer here for more information
My .net app hosted in Azure AppService unexpectedly crashed 4 times yesterday and I'm struggling to get details on why it went down. The report "Diagnose and solve problems\Application Crashes", indicates that Stackoverflow exceptions was the cause of the crashes, but I'm looking to get more details like (uri or stack dump). Here are the things I've tried and come up empty:
EventLog: I used the kudu app to get the eventLog(/api/vfs/LogFiles/eventlog.xml) and there are no details on the Stackover exceptions. In fact there are no matches on "stack", "overflow", or "recursion"
Nlog Files: The nlog files just abruptly terminate when these crashes occurred so no details are captured.
Azure AppIngishts: This too has no exceptions logged during the outage windows. There are some exceptions before and after but nothing about the details of the stack overflow.
AppSerive Utilization: The memory and CPU utilization were running in normal limits (40-70%) before the crashes.
Lastly, the app hasnt been updated for weeks so the likelyhood of new functionally causing this is low. In any case, I would need to know where to look as its a fairly complex app.
Any tips to disgonotise this issue would be really appreciated.
Thanks
You could isolate the issue by running the app locally with the latest changes.
You may capture a memory dump to identify if a line in the code is causing the crash (typically array size/recursive loop). Kindly take a look at the blog for the steps.
Kindly let us know the status with more details on the issue, we would be very glad to assist you further.
I deployed an Azure web app back in July and it's been running flawlessly up until about three weeks ago. At that time, I would notice my CPU utilization constantly between 80% to 100%, with no corresponding increase in traffic. The first time I saw this, after concluding it wasn't my app, or increased traffic, causing this, I restarted the web app service and the CPU utilization returned to its normal 5% to 15%. Then after a couple days it started to do it again. And, again, a restart solved the issue.
My question is this. Is this normal to have to restart the web service every day or so? And, if so, why?
Assuming no changes have been made to your code and you have not seen a corresponding increase in traffic, it is not normal. An Azure Web App with no app deployed should almost always stay at 0% CPU utilization. I say "almost always" because Microsoft does run diagnostic and monitoring tools in the background that can cause some very temporary spikes. See here for a thread on that particular issue.
My recommendations are:
When CPU pegs and stays pegged, log into your SCM site. Check the Process Explorer and confirm that it's your w3wp.exe (Note there's a separate w3wp.exe for your SCM site.) that's pegged the CPU.
Ensure that you don't have any Site Extensions or WebJobs that are losing their mind. You can check your installed Site Extensions on the SCM site under the Site Extensions -> Installed tab. Any WebJobs will show up on your SCM process explorer as separate processes from step #1.
Log into the Azure Portal and browse to your Web App's management blade. Go to the Diagnose and Solve Problems blade. From here, you can try "Metrics per Instance" and go through all of the Perf Counters to see if it gives you a clue as to what's wrong. For example, I had SignalR go nuts once and only found it by seeing that my thread count was out of control.
On the Diagnose and Solve Problems blade, you can also check Application Events.
You may have some light shed on this by installing Application Insights on your web application. It has a free tier that will likely have enough space to troubleshoot for a few days. If this is something going bananas with your code, you may get some insight here.
I'm including failed request tracing logs here for completeness. But these would likely show up in Application Insights.
If you've exhausted all of these possibilities, file a support ticket with Microsoft. As the above link shows, they have access to diagnostic tools that we don't and can eliminate the possibility of a runaway diagnostics or infrastructure process. I don't know how much help they can be if the CPU spike is due to your own w3wp.exe that's spiking the CPU.
Of course, if your app is seriously easy to redeploy and it's not a ridiculous hassle, you can just re-provision it and see if you see the same behavior.
I have a Azure Web Site running for 6 months and on Friday 1st April 2016 at 09:50pm the CPU was very high and this had a impact on the performance of the web site. Stopping and restarting the web service solved the problem but it came back at 13:00pm. Since then the CPU has stayed high and making the web site un-useable
I've tried all monitoring tools, Daas, Event Logs, checked for Open Connections and ensure my software is closing or disposing objects correctly.
But the CPU is still high. Only way to resolve is to restart the web service but I dont want to keep doing this.
Has anyone else experience a similar problem and what was the solutions.
The only thing from the event logs that look an issues is the odd "A network-related or instance-specific error occurred while establishing a connection to SQL Server", which could be because the SQL Aure is not available.
Please help
Hmmm, high cpu means that your web site is executing code, perhaps a wrong loop on some not frequent code path.
The brute force way to identify what code is being executed, would be to add tracing to your solution by System.Diagnostics.Trace.WriteLine("I am here") and then check the Azure Application Log.
Another way would be to attach the Visual Studio Debugger during high cpu and check what is being executed
The other way would be to take a dump or minidump from kudu site and analyze it with WinDbg:
1)What thread is conuming cpu:
!runaway
2) What is this thread doing:
!clrstack
hth,
Aldo
I am relatively new to Azure. I have a website that has been running for a couple of months with not too much traffic...when users are on the system, the various dashboard monitors go up and then flat line the rest of the time. This week, the CPU time when way up when there were no requests and data going in or out of the site. Is there a way to determine the cause of this CPU activity when the site is not active? It doesn't make sense to me that I should have CPU activity being assigned to my site when there is to site activity.
If your website has significant processing at application start, it is possible your VM got rebooted or your app pool recycled and your onstart handler got executed again (which would cause CPU to spike without any request).
You can analyze this by adding application logs to your Application_Start event (but after initializing trace). There is another comment detailing how to enable logging, but you can also consult this link.
You need to collect data to understand what's going on. So first thing I would say is:
1. Go to Azure management portal -> your website (assuming you are using Azure websites) -> dashboard -> operation logs. Try to see whether there is any suspicious activity going on.
download the logs for your site using any ftp client and analyze what's happening. If there is not much data, I would suggest adding more logging in your application to see what is happening or which module is spinning.
A great way to detect CPU spikes and even determine slow running areas of your application is to use a profiler like New Relic. It's a free add on for Azure that collects data and provides you with a dashboard of data. You might find it useful to determine the exact cause of the CPU spike.
We regularly use it to monitor the performance of our applications. I would recommend it.