I have an application pool configured to be have maximum 1 worker process.
Other settings on the pool:
.NET : v4.0
32bit : True
StartMode : AlwaysRunning
Identity : domain\account
Idle Time-out : 0
Disable Overlapped Recycle : False
Specific Time Recycle : 02h00
This morning I noticed that the CPU of the server is on 100% constantly.
2 weeks ago the same problem happened.
I killed the process ID that was still on running, and a bit later there was only 1 pool left.
If I look to the worker processes, I see that there is one on state "running".
It looks like the new worker was started after recycle. And it is now waiting for the old one to shutdown.
But it's stuck somewhere since 24/02 ~ 25/02.
In Process Explorer on the process start timestamps. I see that the second and third worker started on the 25th.
The weird is here that the 2nd is 3 seconds before the recycle config value. And the 3rd is exact on the recycle time.
And since then each day happens the same. At this moment there are 29 exe's instead of 1 for this pool.
In the EventViewer I have found no warnings/errors related to this application pool.
When I do a manual Recycle, the issue does not happen.
I have two web servers with a loadbalancer in front. And on both the servers the same issue happened.
I did some googling, but I have not found anything similar to this.
In the pools there are multiple WCF services running.
This is installed on customers servers in their own network.
But we have this at 50+ customers, and we only had this issue at this customer.
Windows Server 2016 Standard, 1607 (14393.3443)
IIS Version 10.0.14393.0
Related
I stopped the World Wide Web Publishing Service - normally this service won't stop until all of the w3wp.exe processes shut down. But I am now seeing a strange behavior where the service actually stops quickly -- but the w3wp.exe is still there, and those processes end after a number of seconds (10-15 seconds?) Did something change in the behavior of IIS?
The default ShutdownTimeLimit value is 90 seconds. application pool needs time to fully shut down, as any requests currently processing when the shutdown is initiated and needs to be given a certain amount of time to complete.
if it takes too long time to shut down then you could troubleshoot the application pool by capturing a memory dump of the w3wp.exe process in which the application pool is running, when the shutdown problem is occurring.
For more information on capturing memory dumps of IIS processes, see Debug Diagnostics Tool v1.2 is now available.
Goal
Determine the cause of the sporadic lock ups of our web application running on IIS.
Problem
An application we are running on IIS sporadically locks up throughout the day. When it locks up it will lock up on all workers and on all load balanced instance.
Environment and Application
The application is running on 4 different Windows Server 2016 machines. The machines are load balanced using ha-proxy using a round robin load balancing scheme. The IIS application pools this website is hosted in are configured to have 4 workers each and the application it hosts is a 32-bit application. The IIS instances are not using a shared configuration file but the application pools for this application are all configured the same.
This application is the only application in the IIS application pool. The application is an ASP.NET web API and is using .NET 4.6.1. The application is not creating threads of its own.
Theory
My theory for why this is happening is that we have requests that are coming in that are taking ~5-30 minutes to complete. Every machine gets tied up servicing these requests so they look "locked up". The company rolled their own logging mechanism and from that I can tell we have requests that are taking ~5-30 minutes to complete. The team responsible for the application has cleaned up many of these but I am still seeing ~5 minute requests in the log.
I do not have access to the machines personally so our systems team has gotten memory dumps of the application when this happens. In the dumps I generally will see ~50 threads running and all of them are in our code. These threads will be all over our application and do not seem to be stopped on any common piece of code. When the application is running correctly the dumps have 3-4 threads running. Also I have looked at performance counters like the ASP.NET\Requests Queued but it never seems to have any requests queued. During these times the CPU, Memory, Disk and Network usage look normal. Using windbg none of the threads seem to have a high CPU time other than the finalizer thread which as far as I know should live the entire time.
Conclusion
I am looking for a means to prove or disprove my theory as to why we are locking up as well as any metrics or tools I should look at.
So this issue came down to our application using query in stitch on a table with 2,000,000 records in it to another table. Memory would become so fragmented that the Garbage Collector was spending more time trying to find places to put objects and moving them around than it was running our code. This is why it appeared that our application was still working and why their was no exceptions. Oddly IIS would time out the requests but would continue processing the threads.
I have a website in IIS 8.5.9600.16384, we communicate with thousands of mobile devices through cyclic synchronisation and through SignalR 2.3.0.
This morning we had an application pool reset during working hours, which caused the SignalR to call "OnReconnect" of all our mobile devices at the same time.
I though that IIS started new processes first and then killed the old, not having downtime.
Can somebody tell me exactly what happens when IIS recycles it's application pool on the SignalR side? And in which cases can there be a connection downtime? (ex : if the server is busy?)
Edited : The application pool was recycled by IIS because of the "time limit". The IT team will change this setting so that the application pools reset every day at night time when it will have a lower impact on our applications.
A worker process with process id of '8720' serving application pool 'DefaultAppPool' has requested a recycle because the worker process reached its allowed processing time limit.
Also confirmed that disallowOverlappingRotation is not set to True. Any hint would help.
A few years later, I'm still getting some problems with the application pool recycle and SignalR. We are occasionally seeing thousands of re-connections of SignalR while the application pool recycle occurs, opening more than 60k TCPIP ports and causing a crash in IIS.
We managed to have it run "okay" for quite some time but it still crashes. Any hint would help. thanks
I'd first identify how IIS was reset. If you experienced a crash or performed an IISReset, the processes would be down before a new one stood back up. If on the other hand you configured AppPool recycling, then the overlapping processes should occur as you mention. I would check the System Event Log for recycling messages. Note that not all recycle reasons are logged by default.
You may also check to make sure disallowOverlappingRotation is not set to True.
Specifies whether the WWW Service should start another worker process to replace the existing worker process while that process is shutting down. The value of this property should be set to true if the worker process loads any application code that does not support multiple worker processes.
https://learn.microsoft.com/en-us/iis/configuration/system.applicationhost/applicationpools/add/recycling/
We have ASP.NET Core application used internally which are used during office hours and a batch that should be processed 3 AM every morning which is scheduled by HangFire like this:
RecurringJob.AddOrUpdate(
() => MyBatch(),
"0 0 3 1/1 *");
The problem is that the Application Pool goes to sleep and the batch isn’t processed if the site isn’t manually started (by going to the website usually).
I have searched SO and tampered with these settings in the Application Pool but with no success:
Some sources that I used to modify the settings:
How to prevent/extend idleTimeout in IIS 7?
https://serverfault.com/questions/443065/how-do-i-prevent-iis-8-from-stopping-idle-asp-net-applications
IIS seems like to sleep until the next request
The Application Pool is used by a total of 7 applications (all being inactive during night when the batch should be processed). The used Application Pool uses .NET CLR Version 2.0.
I'm using IIS version 10.0.17134.1.
How can I make the Application Pool stay active so the batch can be invoked regularly every morning?
I ran into the same issue where my ASP.NET core application goes into idle even with "AlwaysRunning" as start mode for the app pool, "Preload Enabled" set to true for the site, and idle timeout set to 0.
I got it to work by installing the Application Initialization module and setting the .NET CLR version to v4.0. Don't use the "No Managed Code" as that would prevent the Always Running from triggering the app start.
I wrote a blog post on this explaining in more details the steps I took to get the app to run continuously.
They got documentation on how to set up service to run without stopping.
http://docs.hangfire.io/en/latest/deployment-to-production/making-aspnet-app-always-running.html#enabling-service-auto-start
My experience (with older IIS versions 7.5, 8.0) is that it works, but not for app pool recycle/domain unload.
Workaround for me was to send init request on the application_end event.
As above - you need to enable the Service Autostart - in addition to this, if you hit multiple exceptions, I have found that the Rapid Fail Protection has shut down Application pools in the past when using HangFire. So it's also worth disabling (or increasing to reasonable limits) this on the application pool.
I'd suggest you put in your process a single call to the HTTP address first, just like a ping, that would be enought to trigger the site startup if it isn't running for some reason.
One other thing is that, by microsoft's description at MSDN the "AlwaysRunning" option would be:
"Specifies that the Windows Process Activation Service (WAS) will
always start the application pool. This behavior allows an application
to load the operating environment before any serving any HTTP
requests, which reduces the start-up processing for initial HTTP
requests for the application."
That may be, to produce the compilation of web pages that is done on the first call to be done before any request coming, but may not actually run the application at all times.
I am on a Shared IIS Hosting with no access to most settings. What I did is add a Recurring Job that would be triggered in minute interval less than the IIS Timeout/Idle.
RecurringJob.AddOrUpdate<IMyKeepAliveService>("KeepHangFireAlive", svc => svc.KeepHangFireAlive(URL_TO_SELF), "*/4 * * * *");
The above CRON is enough to prevent IIS App pool from going to sleep.
I use RestSharp to make a tiny ping/GET request to "Self".
I have a web application hosted under IIS. It is a data warehouse, and during its startup process it requires instantiating a large set of items in memory (takes roughly ~20 minutes to fully set up). Because this website is critical to our company, this system must be online 100% during the daytime, and only can be restarted during off-work hours.
For some reason, this web application seems to be "offline" when there is no usage for some time. I know this because the cache is not fully instantiated when the website is visited. This is unacceptable.
It is not clear to me why the website is shut off. The application pool is only set to recycle daily at 4:00 AM (it is 11 AM now).
Are there other settings which I'm not aware of on the IIS part that causes it to shut down the website automatically?
Addl Note: The website does not shut off automatically when running in IISExpress in Visual Studio. Only the production version hosted on IIS shuts off.
Here's a screen of the Advanced Settings for the Application Pool the web site is running under. (Not sure if it's useful.)
I'm on IIS 7.5, Server 2008 R2. It is an ASP.NET 5 Web App.
Check Idle Time-out settings under process model in screenshot. That setting is causing app pool shutting down when remain idle for 20 mins. You can set it to 0 to keep it running all time even when its idle i.e. not processing any requests.
Note: Keeping app pool running all time will consume server's precious memory. It may become critical especially if application is leaking memory.