I have upgraded my payment application from .Net 5.0 to .Net 6.0 without any single line of code changes. I performed stress test using jmeter on both pre migration and post migration releases.
The stress results of post migration are showing degradation in 2 respects, when users were 100-200 and duration was 5 mins:
lower TPS (Transaction per second)
High CPU usage
Am I missing something? Do i need to do some server level configurations to make my application give best results with .net6.0?
UPDATES:
The high CPU usage issue is resolved, which was due to a costly database call in one of the application. But still I cannot see any difference in the TPS, infact TPS is a bit dropped in .net 6.0.
I have set the below variables:
set DOTNET_ReadyToRun=0
set DOTNET_TieredPGO=1
set DOTNET_TC_QuickJitForLoops=1
Still no difference in performance can be seen. Pls. suggest.
It shouldn't be the case, looking at Performance Improvements in .NET 6 article your app should work faster.
If you're absolutely sure that the lower TPS is caused by higher CPU usage you need to use a profiler tool like YourKit or an APM tool to see what exactly causes the CPU usage or slowdown.
You also can try setting the following options:
set DOTNET_ReadyToRun=0
set DOTNET_TieredPGO=1
set DOTNET_TC_QuickJitForLoops=1
Related
I have been looking into performance, specifically calls to an ASP.NET Core 3.1 Web API project that is running on Azure.
Note: yes, we should be moving to a later version of .NET Core, and that's in the pipeline, but it's not something I can just switch to without a bit of effort.
We are targeting netcoreapp3.1 for our libraries, and are referencing Entity Framework Core v3.1.5.
Looking at a typical end-to-end trace in Application Insights, we see this:
If I'm reading this correctly, we're spending a grand total of 135ms in the database executing queries, but between the last 2 queries we appear to stall for ~12 seconds!
When I dig into the profiler trace for this request, I see this:
Again, if I read this right, that means during the second DB call (from our end to end transaction above), we spend ~12.4 seconds inside the call to EntityFrameworkQueryableExtensions.ToListAsync() doing some jit compilation.
That looks excessive to me.
This appears to be a pattern that I see through out the day, even though the application is set to be Always On and there are no restarts of the application between occurrences of this.
The questions I have around this are:
is this to be typically expected?
if so, should it really be taking this long?
is there a way to reduce the need to jit as often as we appear to be doing?
will a move to .NET 6 (and future framework versions) help us here?
On average, the API operates pretty well, and does have a typical avg response time in the < 1 second range. However, when these do occur, they are noticeable and are causing me a headache.
Had exactly the same problem. After emailing azure with the issue, back and forth, it turns out the solution (that worked for us) was to turn off the profiler. https://github.com/dotnet/runtime/issues/66649
Off option - same in .net / .net core
After turning the option off we went from having over 1,500 requests a day taking over 10 seconds, to none taking over 5.
Thanks,
Dave
We have ASP.Net Core 2.1 Web API hosted in AppService (S1) that talks to Azure SQL DB (S1-20DTUs). Both are in same region. During load testing we found that some API instances are taking too much time to return the result.
We tried to troubleshoot the performance issue and below are our observations.
API responds within 0.5 secs most of the time.
API methods are all async methods.
Sometimes it takes around 50 secs to over a minute.
CPU & Memory utilization are below 60%
Database has 20 DTU capacity, out of which 6 DTUs are used during load testing.
In the below example snapshot from Application Insights, we see total duration of the request was 27.4 secs. But the database dependency duration was just 97ms. There is no activity till the database was called. Please refer below example.
Can someone please help me to understand what was happening in this 27 secs of wait time. What could be the reason for this?
I would recommend checking the Application Map on Application Insights resource as shown below to double check the dependencies.
Verify the CPU and Memory metrics by going to the "Diagnose and solve problems" link on App service as shown below and run the Availability and Performance report to find out if there were any issues during your load testing.
Use Async methods on your API to maximize the CPU usage. It may be that the worker process threads are hitting the limits and your app is the bottleneck. You should get some insights when you run the report mentioned in point 2 above.
The S1 tier will support no more than 900 concurrent session. If you request per second (RPS rate) during the load test is very high you may face issues.
Also S3 and above are recommended for intensive workloads. Checking if all the connections are closed properly also helps
You can find details about different pricing tiers and their capabilities in the below link
https://learn.microsoft.com/en-us/azure/sql-database/sql-database-dtu-resource-limits-single-databases
I have performed some performance tests on WSO2 APIM on both WebServices (WSDL) and Gateway interfaces. Everything went good on the gateway one, however I am facing an odd behavior when using the WebServices one.
Basically I created a test that add, change password and delete a user and run a test plan using 64 threads. At the very beggining my throughput increases a lot up until reach all 64 threads (throughput peak was 1600 req/seg). However, after that the throughput start to decrease with no reason.
All 64 threads are still active and running, and the machine hosting the wso2am reduce CPU usage. It seems that APIM is given up of handling the request even though it has threads and processors for that.
The picture below shows the vmstat result for processor (user, system and idle) and the context switch and interruptions. It is possible to cpu/context switch follows the throughput.
And the next picture illustrate the jmeter test result after at the end (after decrease throughput).
Basically what I need is a clue on what may be the reason for such behavior. I have already tried to increase the pool of threads on both wso2am and tomcat, however it has no effect. It is like the requests were not arriving at all. Even though jmeter is full of power and had already send a bigger throughput before.
I would bet that a simple configuration on tomcat or wso2 is the answer for that. Any help is appreciate.
Thanks and Regards
It may be due to JMeter not being able to send the requests fast enough, try the following steps:
Upgrade JMeter to the latest version (3.1 as of now), you can get the most recent JMeter distribution from JMeter download page
Run your test in command-line non-GUI mode. JMeter GUI can be used for tests development and/or debugging only, it is not designed for running load tests.
Remove (or disable) all the listeners during test execution. Later on you can open JMeter GUI, add the listener of your choice, load .jtl results file and perform analysis or create an HTML Reporting Dashboard out of results file
See 9 Easy Solutions for a JMeter Load Test “Out of Memory” Failure article for above points explained in details and few more tips on configuring JMeter for maximum performance and throughput
I have been trying to understand what is the cause of high memory usage from processes in the windows server I have. I installed that tool DebugDiag 1.2 to try to find the problem.
Here is what runs in my server:
I have the IIS server which has a decent number of pool applications (68 pool applications). For each pool application there are at least 4 applications.
Recently, I have faced problems related to high memory usage, causing the server to work at 97% of memory usage or higher.
It was working fine when I took this printscreen below. However, the memory usage will easily get higher.
Task Manager:
With that being said, I have been trying to understand how to use the tool "DebugDiag1.2" from microsoft to find something (part of the source code, an sql procedure) that might help me locate what is causing the problem.
I read that we can't limit the memory for each IIS pool application, so I guess the solution would be trying to optmize the application. But first I need to know where to start.
I hope someone can help me out.
I have a web site and I am using iis as my web server. I noticed that on production server, the cpu reaches 95% usage pretty fast with very little users. this behaviour I don't see on my developement server. I am using visual studio to develop and iis as my local web server as well.
How much big traffic you have on production comparing to development server? How their parameters compare? Before starting a deep analysis of the application itself, I would identify all the infrastructure and environmental differences. Sometime such problems happens because of some other software, like antivirus software running in the background...
Nevertheless, because it sounds rather as a application problem, I would first check Event Viewer for errors. Then I would start from monitoring a few Performance Counters to correlate % Processor Time counter with Current Connections, Available Memory, # of Exceps Thrown / sec, % Time in GC and so on. This kind of behavior usually has a reason from the list:
excessive loops usage due to some logic error, like calling the same service again and again, trying to load or parse malfunctioned file etc. This can be analyzed with dump analysis (look below).
high CPU usage due to Garbage Collector - when memory usage is extensive (or there is a memory leak even) GC may start to consume more and more CPU fighting with the memory shortage. You will see this with memory-related performance counters.
a considerable amount of exceptions thrown (for example due to some environmental problems like network unavailability, production data difference) can also consume a lot of CPU. Event Viewer and exception-related performance counters (as they can be handled silently by your application) should be a indicator here.
To further analyze your application, I suggest to make a full memory dump during high CPU usage. You can do that with Debug Diag tool. Please refer this IIS troubleshooting guide for details.