I have been trying to understand what is the cause of high memory usage from processes in the windows server I have. I installed that tool DebugDiag 1.2 to try to find the problem.
Here is what runs in my server:
I have the IIS server which has a decent number of pool applications (68 pool applications). For each pool application there are at least 4 applications.
Recently, I have faced problems related to high memory usage, causing the server to work at 97% of memory usage or higher.
It was working fine when I took this printscreen below. However, the memory usage will easily get higher.
Task Manager:
With that being said, I have been trying to understand how to use the tool "DebugDiag1.2" from microsoft to find something (part of the source code, an sql procedure) that might help me locate what is causing the problem.
I read that we can't limit the memory for each IIS pool application, so I guess the solution would be trying to optmize the application. But first I need to know where to start.
I hope someone can help me out.
Related
I'm currently working on a Node.js project and my server keeps running out of memory. It has happened 4 times in the last 2 weeks, usually after about 10,000 requests. This project is live and has real users.
I am using
NodeJS 16
Google Cloud Platform's App Engine (instances have 2048mb of memory)
Express as my server framework
TypeORM as database ORM (database is postgres hosted on separate GCP SQL instance)
I have installed the GCP profiling tools and have captured the app running out of memory, but I'm not quite sure how to use the results. It almost looks like there is a memory leak in the _handleDataRow function within the pg client library. I am currently using version 8.8.0 of the library (8.9.0 was just released a few weeks ago and doesn't mention fixing any memory leaks in the release notes).
I'm a bit stuck with what I should do at this point.
Any suggestions or advice would be greatly appreciated! Thanks.
Update: I have also cross-posted to reddit and someone there helped me determine that issue is related to large queries with many joins. I was able to reproduce the issue, and will report back here once I am able to solve it.
When using App Engine, a great place to start looking for "why" a problem occurred in your app is through the Logs Explorer. Particularly, if you know the time-frame of when the issues started escalating or when the crash occurred.
Although based on your Memory Usage graph, it's a slow leak. So a top-to-bottom approach of your back-end is really necessary to try and pin-point the culprit. I would go through the whole stack and look for things like Globals that are set and not cleaned up, promises that are not being returned, large result-sets from the database that are bottle-necking the server, perhaps from a scheduled task.
Looking at the 2pm - 2:45pm range on the right-hand of the graph, I would narrow the Logs Explorer down to that exact time-frame. Then I would look for the processes or endpoints that are being utilized most frequently in that time-frame as well as the ones that are taking the most memory to get a good starting point.
I have big enterprise JAVA application , running on several machines under tomcat7.
There are different performance problems such slow response , server hangs etc.
I want to try to play with different params like maxThread , maxConnection ,acceptCount and so on .
But before change them, how can I check that I run out of connections for example and I need to increase it ? Or everything else , like acceptCount that should be increased ?
Typically, Apache Tomcat performance issues are with the underlying JavaVM configuration, in my experience those are mainly with the size of the permGen, and other memory settings. I have been able to troubleshoot quite a few of them using VisualVM, which visualizes a lot of the JVM memory ops. Would also highly recommend JMeter.
IMHO maxThread and other Tomcat-specific parameters have rarely been the source of application performance issues, but it's the JVM settings where most issues are.
Start with minimum of these settings:
-Xms1024M -Xmx2048m -XX:MaxPermSize=1024m
I would recommend to find the problem before starting "fixing" things.
There are several applications to monitor your servers and check where the problems are. You can try appdynamics, newrelic, ruxit, or any other application monitoring product. (Some have free version offers that comes handy)
Then you search for your bottlenecks, they can be anywhere, server, database, network, jvm, ... depending on your application and your architecture.
And once you find the problem, you can start fixing it.
Good luck!
I have a web application that is eventually running out of memory when it runs on IIS 7 Windows Server 2008. When I attempt to run a memory profiler against the application to determine the leak, it is not reproducible on my development workstation...Windows Vista.
The GC collection cycles are not consistent between the server and the workstation and it appears the server's collection is not reclaiming all of its memory and is eventually running out. The server becomes non responsive and throws out of memory exceptions.
We have tried setting objects that are surviving too many generations to null...Some improvement was noticed.
Any assistance/recommendations would be greatly appreciated
Tess Ferrandez's blog has some great information on debugging memory leaks using Windbg.
By taking a dump of the running application and then analysing it in Windbg, you should be able to find the source of the leaks you are seeing.
The following entries are probably a good starting point:
Setup (including links to configuring Windbg
Memory Leak Lab 1
Memory Leak Lab 2
Good luck!
Every night my IIS hangs with 3 w3wp.exe processen in the task manager list. It is not possible to kill the w3wp task using the most memory. Not even iisreset helps. I have to reboot the machince to get rid of the w3wp prosess.
Any suggestions?
Added: I have reduced the maxmemory of the default app pool to 120, but still the process goes above 200mb.
In the app-pool, set a limit on memory use, that will restart the app. when it uses too much memory. Might help.
From the screen shot, I cannot see problems such as high CPU utilization or huge memory usage.
You can learn how to use Debug Diagnostics to capture a hang dump and then analyse the dump with Debug Diagnostics. Its report should provide you some hints.
http://www.microsoft.com/downloadS/details.aspx?FamilyID=28bd5941-c458-46f1-b24d-f60151d875a3&displaylang=en
If you cannot do that yourself, I suggest you contact our Microsoft Support team.
I did not find a permament solution to my problem, but when i removed some web sites from my vps and reduced the max memory allowed for the default app pool the situation improved. I also removed the admin website which ran under its own appPool.
I have a webserver that is pegged and I've been able to isolate it to a particular website instance. I'd like to dig deeper and isolate the particular page/process that is causing the issue.. Any tips?
You can take a memory dump of the process and poke around with windbg.
There are posts on this issue from Tess Ferrandez blog. Just do as she say.
Which version of IIS are you using? Some of the higher ones allow for a separation of which process gets used to handle requests such as a worker process that you could isolate a bit more that way. I'd also suggest reading through the IIS logs to see what requests were being handled, how long they took, etc.
There are many different quirks to each IIS version. The really low ones just had a start/stop functionality, but the newer ones have really given administrators much more control and power, IMO.
You should try using a profiler to identify what is using up the most resources. I've used dotTrace Profiler, although that can be expensive if you're on a tight budget.
It allows you to see exactly what processes and method calls use of the most processing time of a request really well so you can isolate the most resource intensive operations.
You should really be able to use any profiler to do this, not just dotTrace. I just happen to only have experience with this one in particular.
Change your web garden setting to 10 or greater. Then watch your CPU and memory utilization on the web server.
Continue to increase the web garden setting until either the app is completely responsive with less than 5% average utilization OR you have actually maxed your web server's memory.
UPDATE
It's not about diagnosing, it's about properly configuring the IIS server. Web Gardens are one of the top misunderstood features of IIS. By increasing the available threads to handle new requests you remove the appearance of contention at the web server level and place it squarely where it belongs. In this case at your database. Instead of masking a problem it actually highlights exactly where the problem is.
This turned out to be a SQL problem (sql 2005). The solution was found by using SQL activity monitor to identify a suspended process with a Async_network_io wait type. We then ran SQL profiler to narrow it down to two massive queries which were returning an over abundance of results.