I'm new to IIS. I have several questions about recycling application pool:
Private memory limit and Vitual memory limit are all 0 by default. I read the official document of IIS 7.0 (We are using IIS 8.0 and Windows Server 2012 but I think they should be the same in this respect). So there is really no limit for the memory usage? It's just waiting 1740 minutes (by default) for the application to recycle? It won't recycle even if the total memory usage is very high until 1740 minutes later? I searched very hard for the answers but couldn't find any...
I read an article which said application pool should never be recycled. So what is the mechanism of memory management for IIS? When would the memory not being used be released? Is it similar to Java? I think no one said that full GC in Java is no good practice...
Thanks.
Related
I have a websocket server that hoards memory during days, till the point that Kubernetes eventually kills it. We monitor it using prometheous-net.
# dotnet --info
Host (useful for support):
Version: 2.1.6
Commit: 3f4f8eebd8
.NET Core SDKs installed:
No SDKs were found.
.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.6 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.6 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.1.6 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
But when I connect remotely and take a memory dump (using createdump), suddently the memory drops... without the service stopping, restarting or loosing any connected user. See the green line in the picture.
I can see in the graphs, that GC is collecting regularly in all generations.
GC Server is disabled using:
<PropertyGroup>
<ServerGarbageCollection>false</ServerGarbageCollection>
</PropertyGroup>
Before disabling GC Server, the service used to grow memory way faster. Now it takes two weeks to get into 512Mb.
Other services using ASP.NET Core on request/response fashion do not show this problem. This uses Websockets, where each connection last usually around 10 minutes... so I guess everything related with the connection survives till Gen 2 easily.
Note that there are two pods, showing the same behaviour, and then one (the green) drops suddenly in memory ussage due the taking of the memory dump.
The pods did not restart during the taking of the memory dump:
No connection was lost or restarted.
Heap:
(lldb) eeheap -gc
Number of GC Heaps: 1
generation 0 starts at 0x00007F8481C8D0B0
generation 1 starts at 0x00007F8481C7E820
generation 2 starts at 0x00007F852A1D7000
ephemeral segment allocation context: none
segment begin allocated size
00007F852A1D6000 00007F852A1D7000 00007F853A1D5E90 0xfffee90(268430992)
00007F84807D0000 00007F84807D1000 00007F8482278000 0x1aa7000(27947008)
Large object heap starts at 0x00007F853A1D7000
segment begin allocated size
00007F853A1D6000 00007F853A1D7000 00007F853A7C60F8 0x5ef0f8(6222072)
Total Size: Size: 0x12094f88 (302600072) bytes.
------------------------------
GC Heap Size: Size: 0x12094f88 (302600072) bytes.
(lldb)
Free objects:
(lldb) dumpheap -type Free -stat
Statistics:
MT Count TotalSize Class Name
00000000010c52b0 219774 10740482 Free
Total 219774 objects
Is there any explanation to this behaviour?
The problem was the connection to RabbitMQ. Because we were using sort lived channels, the "auto-reconnect" feature of the RabbitMQ.Client was keeping a lot of state about dead channels. We switched off this configuration, since we do not need the "perks" of the "auto-reconnect" feature, and everything start working normally. It was a pain, but we basically had to setup a Windows deploy and do the usual memory analysis process with Windows tools (Jetbrains dotMemory in this case). Using lldb is not productive at all.
Disclaimer: I am no .NET Wizard.
But you should do two things to go with Kubernetes best practices:
Define sensible resource limits for your app. If the app does not need more than 200MB memory define a resource limit to prevent the app from consuming all available host memory. But be aware that the Unix API to get available memory is not capable of processing the cgroup the process has and always outputs the host memory no matter what your cgroup says.
Tell your app what this resource limit is. It seems like your app does not "feel the need" to free memory as there is plenty. Almost all applications, and frameworks as well, have a switch to define the maximum memory to be consumed. Tell your app this limit, and it will "see" memory pressure and perform a full GC (what I guess could be the problem here)
We have an issue on Live site (MVC .Net).
Max users 25
After every 2-3 hours IIS memory usage peak to Max memory
And All users get a exception saying "System Out Memory"
We then need to restart IIS and All work Well.
Let me know if any solution to fix this.
You should check your application if there is no memory leaks. Also IIS has this behavioir during DOS attack.
Also you can set automatic IIS reset after memory usage increased some amount
I have a web application that is eventually running out of memory when it runs on IIS 7 Windows Server 2008. When I attempt to run a memory profiler against the application to determine the leak, it is not reproducible on my development workstation...Windows Vista.
The GC collection cycles are not consistent between the server and the workstation and it appears the server's collection is not reclaiming all of its memory and is eventually running out. The server becomes non responsive and throws out of memory exceptions.
We have tried setting objects that are surviving too many generations to null...Some improvement was noticed.
Any assistance/recommendations would be greatly appreciated
Tess Ferrandez's blog has some great information on debugging memory leaks using Windbg.
By taking a dump of the running application and then analysing it in Windbg, you should be able to find the source of the leaks you are seeing.
The following entries are probably a good starting point:
Setup (including links to configuring Windbg
Memory Leak Lab 1
Memory Leak Lab 2
Good luck!
Every night my IIS hangs with 3 w3wp.exe processen in the task manager list. It is not possible to kill the w3wp task using the most memory. Not even iisreset helps. I have to reboot the machince to get rid of the w3wp prosess.
Any suggestions?
Added: I have reduced the maxmemory of the default app pool to 120, but still the process goes above 200mb.
In the app-pool, set a limit on memory use, that will restart the app. when it uses too much memory. Might help.
From the screen shot, I cannot see problems such as high CPU utilization or huge memory usage.
You can learn how to use Debug Diagnostics to capture a hang dump and then analyse the dump with Debug Diagnostics. Its report should provide you some hints.
http://www.microsoft.com/downloadS/details.aspx?FamilyID=28bd5941-c458-46f1-b24d-f60151d875a3&displaylang=en
If you cannot do that yourself, I suggest you contact our Microsoft Support team.
I did not find a permament solution to my problem, but when i removed some web sites from my vps and reduced the max memory allowed for the default app pool the situation improved. I also removed the admin website which ran under its own appPool.
I wanted to know what people used as a best practice for limiting memory on IIS [5/6/7]. I'm running on 32bit web servers with 4GB of physical memory, and no /3GB switch. I'm currently limiting my app pools to 1GB used memory. Is this too low? any thoughts?
All the limits in the application pool are for bad behaving apps. And more specifically:
To prevent the bad app from disturbing to good apps.
To try and keep the bad app running as much as possible.
In that light, the answer is of course: It depends.
If your application is leaking then without a limit it will crash around 1.2 - 1.6 Gb (if memory serves). So 1 Gb is sensible. If during normal operation your application consumes not more than 100 Mb and you have many app pools on the server, than you should set the limit lower to prevent one app from damaging other apps.
To conclude: 1 Gb is sensible. Hitting the limits should be treated as an application crash and should be debugged and fixed.
David Wang blog is a good resource on those issues.
There's a great writeup from a MS Field Engineer about this subject.