We are facing a strange issue with our IIS Deployments.
ApplicationPools sometimes fail to start properly but do not throw errors when doing so.
The only containing site within the Application Pool is not responsive (not even returning 500 or the like, just times out after some time).
The ApplicationPool and Sites are up and running (not stopped) as far as IIS is concerned.
Restarting the Site or the ApplicationPool does not fix the issue.
However, removing the site and ApplicationPool and recreating it with identical Properties does fix it.
Once any ApplicationPool has reached this state, the only way to solve this (as far as we know) is recreating the entire ApplicationPool.
We would gladly do so in an automated way, but there is no error to catch and handle respectively.
Some background data:
We are using IIS Version 10
The ApplicationPool appears to start correctly. EventLog states that Application '<OUR_APP>' started successfully.
We suspect that the problem might be multiple ApplicationPool-starts happening simultaniously (as they are automatically triggered by our CI/CD Pipeline).
Now, I am by no means an IIS Expert, so my questions are:
Would it be possible, that many app pool starts (circa 20-60) happening at roughly the same time cause such behaviour?
What could I do to investigate this further?
Would it be possible, that many app pool starts (circa 20-60)
happening at roughly the same time cause such behaviour?
Difficult to say. An app pool is just an empty container, mostly what takes the time and places limits on this number is what your application code and dependencies are doing at startup and runtime with a little dotnet precompilation overhead.
What could I do to investigate this further?
Check the HTTPERR logs in the Windows folder - might provide a clue if your not seeing the request logged elsewhere.
monitor the w3wp.exe processes themselves - those are your apppools(AKA "app domains"). Its possible for them to get stuck and not "properly" crash which sounds like your case.
Assuming all your apps normally work and you just want a way to recover random failures, try this...
When you have a broken app pool, run the following on your server from PowerShell or ISE (as an Administrator) to view the running IIS worker processes:
Get-WmiObject Win32_Process -Filter "name = 'w3wp.exe'" | Select-Object ProcessId,CommandLine
Above outputs the worker processes ID's, and the arguments used to start them. Among the arguments you can see the sites name - use the correct ProcessId with the command Stop-Process -Force -Id X (replacing X with the ProcessId number) to forcibly kill the process. Does the app successfully start once you try and access it after killing the process?
If you know the name of the app pool to kill you can use this code to terminate the process:
$AppPoolName = 'NAMEOFMYAPPPOOL';
Stop-Process -Force -id (Get-WmiObject Win32_Process -Filter "name = 'w3wp.exe' AND CommandLine like '%-in%$($AppPoolName)%'").ProcessId
(substitute NAMEOFMYAPPPOOL for the name of the app pool, and run as Administrator)
If killing the stalled process is sufficient to let it restart successfully it would be fairly easy to script a simple health check. I would read the bindings of each site, make an HTTP request to each binding and confirm the app pool really is running/responsive and returns a 200 OK response. If the request fails after some reasonable timeout, try terminating the process and re-requesting the HTTP request to restart the app pool. Add some retry logic and maybe add a delay between attempts so it doesnt get stuck in a loop.
Just a thought - try giving each app pool its own temp folder - configured in web.config per site:
<system.web>
<compilation tempDirectory="D:\tempfiles\apppoolname" />
Cross talk in here during startup is a possible source of weirdness.
The problem seemed to be caused by our Deployment Scripts not waiting for Application-Pools to actually be in Stopped state, before continuing to remove old application files and replacing them with the new ones and immediately Starting the ApplicationPools again.
We noticed issues related to this earlier this year when files could not be deleted because the were still being used, even after stopping the ApplicationPool (which we "solved" by implementing a retry mechanism)...
Solution
Calling the following code after stopping the ApplicatonPool seems to solve the issue....
$stopWaitCount = 0;
while ((Get-WebAppPoolState -Name $appPool).Value -ne "Stopped" -and $stopWaitCount -lt 12)
{
$stopWaitCount++
Write-Log "Waiting for Application-Pool '$appPool' to stop..."
Start-Sleep -Seconds $stopWaitCount
}
We implemented this 2 days ago and the problem didn't occur in 100+ deployments since.
Related
We have a long time response for first time request call to Web API !
in spite we have changed the following Settings of WEB API's pool -and then we noticed a noticeable progress for the number of times for waiting time is decreased - :
StartMode To AlwaysRunning
Idle-Time-Out(minutes ) TO 0
But still having long time response for first time of the request, specially on the morning (before working time) or during weekend, what could cause this? any missing Settings you advice ?are there any other settings we need to change, to Run the pool automatically ? -snapshot of advanced settings below-
As it is look to me some how the IIS or Pool has been recycled periodically ! however I was thinking changing StartMode to "AlwaysRunning" will run automatically the Pool and keep it up and running even after recycle or shutdown !
BTW: the setting of Recycling /Regular time intervale is 1470
!
It might relate to your ORM framework, such as Entity framework.
See the below discussion.
How to "warm-up" Entity Framework? When does it get "cold"?
We need to install the Application Initialization Windows feature in order to make the Start Mode configuration take effect.
https://serverfault.com/questions/683017/first-request-after-a-while-always-slow-asp-net-mvc-iis-8-5-windows-server
Also, Precompiling website might be a good option to speed up first access.
Please check the below links.
Fixing slow initial load for IIS
I have an app running in its own app pool that writes often to a text log file. When the 29-hour fixed recycle hits, I usually run into problems because the new process is launched while the old process is still there. This causes "can't open the file because it's in use by another process."
First I simply tried to set Disable Overlapped Cycle to True, but that takes my service off the air for 90 seconds or more, which is not good. If I reduce the Shutdown Time Limit I suppose it would help, but I'd prefer not to kill the process immediately.
I tried setting up an Application_End handler in Global.asax, but I find it's almost never getting called because (I think) my process is still running a thread somewhere (possibly the logging process).
I tried Application_Disposed but so far it does not get called when IIS kills the process after the shutdown time expires, and it's probably too late by then if I go back to overlapped recycle, which I think is preferable.
What I need is a signal from IIS that says "Hey, I want you to stop," that lets me close the log file immediately, but that does not seem to exist.
Any suggestions would be appreciated.
The Problem is in your App Itself and not in IIS.
Just Disposed the object after you calling or writing into text file to avoid this issue.
I've been encountered that same way before. I also writing to a text file for a log then error occurs "can't open the file because it's in use by another process." but when i disposed the object not just the word disposed, you have to check it carefully for a proper disposal.
Its all gone and run the app normally.
Hope it helps
Recently we want to cater the slow loading problem of IIS for first request, after I did some research, I've found that IIS7.5+ has a feature named "Application Initialization" which maybe what I need.
However I have to understand the mechanism before I try to apply it and here is my understanding:
With default IIS setting:
The application pool idle after 20 minutes
The corresponding worker process is killed
First request comes in
IIS starts to create a new worker process
IIS starts to load the application
The client can see after application is loaded
And step 4, 5 makes first request not so responsive.
With Application Initialization set:
The application pool idle after 20 minutes
The corresponding worker process is killed
IIS starts to create a new worker process
IIS starts to load the application through a "fake" request
First request comes in
The client can see after application is loaded
Now the first request is responsive as indeed it is not the first request to the server, sometimes before there was a "fake" request which kicks loading of the application.
What I would like to know is that:
Is my understanding correct?
When application initialization is set, the worker process is still being killed, but a new one is created right after it, is it the case?
That's pretty much how it works. Without Application Initialization, as you mentioned, once the worker process is killed, it is not restarted until a request is sent to it. Upon the first request, a new worker process (W3WP.exe) is started and it starts to load the application. And this cold start of the application is what typically makes the first request less responsive. For eg. if it's an ASP.NET application, the first request triggers the recompilation of the temporary ASP.NET files and this can take several seconds in a moderately large enterprise application.
If you look at the setup of Application Initialization, you will see that there are two main parts to it:
You need to set the startMode of the application pool associated with the website to AlwaysRunning
You need to set preloadEnabled to true on some path (path to the website) on the ApplicationPool
Step 1 is what tells IIS to automatically restart the IIS worker process whenever there is a reboot or IISReset. (You can easily see this in action in TaskManager - do only step 1 and do an IISReset, you should be seeing the existing W3WP.exe process getting removed and a new one is getting created)
Step 2 is what tells IIS to make the initial fake/dummy request that will do all the required initialisation of your web application. For eg. for an ASP.NET application, this essentially will trigger the compilation of all the ASP.NET files, so that the next request - the actual first request to the page does not experience the long delays associated with app initialisation.
While it is true that a traditional approach of keeping using a script to poll the app to prevent it from going idle can do the job, the ApplicationInitalization module makes the job much easier. You can even have IIS issue the dummy request to a custom warmup script that does much more than a simple page load - preloading a cache of several webpages, ahead of time generate/do any task that might otherwise take longer etc.
Official documentations here:
IIS 7.5
IIS 8.0
Your understanding is correct based on my experiences. I first ran into this capability in a performance testing scenario way back in 2014. I was custom coding the ping portion of this into monitoring jobs :O
"The Application Initialization Module basically allows you to turn on
Preloading on the Application Pool and the Site/IIS App, which
essentially fires a request through the IIS pipeline as soon as the
Application Pool has been launched. This means that effectively your
ASP.NET app becomes active immediately, Application_Start is fired
making sure your app stays up and running at all times." - Rick Strahl
Official detailed docs are on the MSDN site, from what I see not much has changed between IIS 7.5 and 8.0 in the way of config.
We're using robocopy in monitor mode to copy files between web servers in Azure in a daisy-chain fashion. The command is invoked as an Administrator at startup.
The process works beautifully, but every now and then it will stop without error. We recently enabled history in Task Scheduler to get more insights into what was happening. But it simply reveals two steps: Action Completed and Task Completed.
The output of Action Completed:
Task Scheduler successfully completed task "\Robocopy Content From VM1" , instance "{some-vm-guid}" , action "C:\Windows\system32\robocopy.EXE" with return code 2147942408.
The output of Task Completed:
Task Scheduler successfully finished "{some-vm-guid}" instance of the "\Robocopy Content From VM1" task for user "HCVMHOST-2\SomeAdminUser".
The robocopy command:
robocopy "\\vm-1\c$\Content\SomeFolder" "c:\Content\SomeFolder" /mon:1 /FFT /MIR /s /zb /copyall /mt:4 /ns /nc /nfl /xd OutputCache Temp /LOG:"d:\RobocopyLogs.txt" /b
I suspect it may have something to do with either latency or a VM rebooting. But I'm not convinced this is the reason, and I want to make sure that my command isn't missing a param or has a param it shouldn't. As far as I can tell, it works fantastically, until it doesn't.
Anyone experience this before?
After a bunch of digging, and some random luck (my past attempts to Google the return code were fruitless because of a simple formatting different, 2,147,942,408 versus 2147942408). It turns out that the return code actually has significance, indicating that:
Not enough storage is available to process this command.
It is a curious response though, given each of the VMs has plenty of remaining disk space on all drives. I will raise a support ticket with Azure, and post back if any relevant info comes up.
I have a intranet SiteCore website set up on IIS 7 which randomly throws the following error message
HTTP Error 503.2 - Service Unavailable
The serverRuntime#appConcurrentRequestLimit setting is being exceeded.
To fix this issue, I have made following changes
Increased the Queue Length of application pool myrjetAppPool from 1000 to 65535.
Modified Machine.Config to increase requestQueueLimit property of ProcessModel element to 100000
Increased appConcurrentRequestLimit to 10000 by running
C:\Windows\System32\inetsrv\appcmd.exe set config /section:serverRuntime /appConcurrentRequestLimit:100000
But I'm still getting the same error. ANy help is greatly appreaciated.
You might check to see where all your threads are going. We had occurrences where threads for Media Library assets were hanging and blocking up the queue.
In IIS Manager, select the server node from the tree, then the "Worker Processes" feature icon, then right-click the application pool of interest and select "View current requests". You might find something is getting stuck. I sometimes hit F5 on this screen a few dozen times in very quick succession to see the rate the requests are going through (of course Performance Monitor is better for viewing metrics but it won't tell you what URLs are being processed).
Investigate references in the linked url to 'MaxConcurrentReqeustsPerCPU' which you may need to set by creating a new registry key, depending on your OS and framework.
https://learn.microsoft.com/en-us/archive/blogs/tmarq/asp-net-thread-usage-on-iis-7-5-iis-7-0-and-iis-6-0
As already commented - check the actual concurrent request count using performance counters to determine which limit you're hitting i.e. it could be a limit of 5000 or maybe 12 (per cpu).
Edit: I realise this may look like I'm talking about a different setting entirely, but I believe there is overlap here.
We got this problem after an installation of an IIS plugin. After long investigating we saw that the config-file C:\Windows\System32\inetsrv\config\applicationHost.config had an extra location tag for the site with the problem. After removing the extra entry and an iisreset, the site/server worked normally againg. So something must went wrong during the installation....