I am running a web service in IIS which calls a third party vendor application. The web service is called from yet another third party application to which the IIS hosted web service replies to. So basically My App -> IIS App -> Third Party App -> IIS App-> My App.
Occasionally, there are app pool recycles which the Third party App is executing, meaning the IIS app is waiting for the reply. This causes the My App to indefinitely wait on the Third Party app because the singnalling chain is now broken.
Is it by design that IIS will force kill the child process running in it during recycles? Is there a way to make the recycle wait till the child is completed?
Did you set Disable Overlapped Reycle
If you set it to true, then IIS recycle will terminate the old process instance and create a new one immediately.
How if you were setting it to false, Then IIS Recycling will not terminate old process and requests immediately. You can see there are two instance side by side if the old process haven't finish all requests.
It will create a new application pool as well. The difference is IIS will leave the old process until all requests in old worker process being finished or shut down time exceed the ShutDown Time limit. Its default value is 90
So if your old request didn't hang in the old worker process and it require more than 90s to finish the request, please just expand the value of ShutDown Time Limit. It will give your old process more time to handle current requests.
Related
I am building a Blazor Server intranet application for my customer. One of the requirements is that they can stay logged in indefinitely. If they starting inputting some data on a Friday afternoon, they should be able to return on Monday morning and continue working without interruption.
I came to observe that the client-side was getting disconnected from the server about once per day. When this happened I would see the dreaded Blazor error “Reconnection failed. Try reloading the page if you’re unable to reconnect.”. If I click the link to Reload, it immediately reconnects to my server, but any work in process would be lost.
I found the root cause: by default, IIS is recycling the application pool every 29 hours. When this happens, the Blazor SignalR connection is getting interrupted, and hence the code running in the browser times out and disconnects.
I am able to work around this issue by disabling application pool recycling altogether. So far, it looks like that works fine (I could keep connectivity for the past 3 days). But I am worried this may not be safe long term, since application pool recycling helps deal with issues such as memory leaks, fragmentation, etc.
SO, my question is: is it possible to configure IIS in way that I can recycle the application pool AND also keep my blazor server connection available during that recycle period?
When you recycle an application pool, HTTP.SYS holds onto the client connection in kernel mode while the user mode worker process recycles. After the process recycle, HTTP.SYS transparently routes the new requests to the new worker process. Thus, the client never "loses all connectivity" to the server - the TCP connection is never lost - and never notices the process recycle.
I believe your problem is with the applications running in your application pool that store state within the process, such as whether a user is logged in or not. Everytime the process recycles, that state is automatically lost... which is by-design since that is what a process recycle accomplishes. As a result, your users "lose all connectivity" and "have to log back into their applications" to re-establish that lost state. The only way to fix this is for your applications to store its state outside of the IIS worker process such that it is friendly to being recycled.
The following blog entry talks more about what is going on:
https://learn.microsoft.com/en-us/archive/blogs/david.wang/why-do-i-lose-asp-session-state-on-iis6
Recently we want to cater the slow loading problem of IIS for first request, after I did some research, I've found that IIS7.5+ has a feature named "Application Initialization" which maybe what I need.
However I have to understand the mechanism before I try to apply it and here is my understanding:
With default IIS setting:
The application pool idle after 20 minutes
The corresponding worker process is killed
First request comes in
IIS starts to create a new worker process
IIS starts to load the application
The client can see after application is loaded
And step 4, 5 makes first request not so responsive.
With Application Initialization set:
The application pool idle after 20 minutes
The corresponding worker process is killed
IIS starts to create a new worker process
IIS starts to load the application through a "fake" request
First request comes in
The client can see after application is loaded
Now the first request is responsive as indeed it is not the first request to the server, sometimes before there was a "fake" request which kicks loading of the application.
What I would like to know is that:
Is my understanding correct?
When application initialization is set, the worker process is still being killed, but a new one is created right after it, is it the case?
That's pretty much how it works. Without Application Initialization, as you mentioned, once the worker process is killed, it is not restarted until a request is sent to it. Upon the first request, a new worker process (W3WP.exe) is started and it starts to load the application. And this cold start of the application is what typically makes the first request less responsive. For eg. if it's an ASP.NET application, the first request triggers the recompilation of the temporary ASP.NET files and this can take several seconds in a moderately large enterprise application.
If you look at the setup of Application Initialization, you will see that there are two main parts to it:
You need to set the startMode of the application pool associated with the website to AlwaysRunning
You need to set preloadEnabled to true on some path (path to the website) on the ApplicationPool
Step 1 is what tells IIS to automatically restart the IIS worker process whenever there is a reboot or IISReset. (You can easily see this in action in TaskManager - do only step 1 and do an IISReset, you should be seeing the existing W3WP.exe process getting removed and a new one is getting created)
Step 2 is what tells IIS to make the initial fake/dummy request that will do all the required initialisation of your web application. For eg. for an ASP.NET application, this essentially will trigger the compilation of all the ASP.NET files, so that the next request - the actual first request to the page does not experience the long delays associated with app initialisation.
While it is true that a traditional approach of keeping using a script to poll the app to prevent it from going idle can do the job, the ApplicationInitalization module makes the job much easier. You can even have IIS issue the dummy request to a custom warmup script that does much more than a simple page load - preloading a cache of several webpages, ahead of time generate/do any task that might otherwise take longer etc.
Official documentations here:
IIS 7.5
IIS 8.0
Your understanding is correct based on my experiences. I first ran into this capability in a performance testing scenario way back in 2014. I was custom coding the ping portion of this into monitoring jobs :O
"The Application Initialization Module basically allows you to turn on
Preloading on the Application Pool and the Site/IIS App, which
essentially fires a request through the IIS pipeline as soon as the
Application Pool has been launched. This means that effectively your
ASP.NET app becomes active immediately, Application_Start is fired
making sure your app stays up and running at all times." - Rick Strahl
Official detailed docs are on the MSDN site, from what I see not much has changed between IIS 7.5 and 8.0 in the way of config.
I have some questions about IIS application pool configuration.
I know that idleTimeout specifies how long a worker process should run idle if no new requests are received and the worker process is not processing requests. After the allocated time passes, the worker process should request that it be shut down by the WWW service (taken from here). My question is when and how the worker process starts when a request arrives after it shut down?
Other question is what is the difference between application pool recycling and between shut down? Does it better to recycke instead of shut down and then start again?
When and How worker process starts? When a new request comes in IIS spins a new process for the app pool your website is running under.
The difference between shutting down and recycle is that during a recycle a new process is started for the app pool in parallel, while the existing process is still serving an existing request. Any new requests that come in are handled by the new process. So in effect you do not loose any request that come in and any existing requests are not prematurely dropped. In a shutdown, you will loose the requests while the app is down.
To see this live in action - open task manager, select processes tab and see IIS spin up the app pool processes and notice the behavior between a shutdown and recycle. During a recycle for a brief moment you will see two processes for the same app pool.
See this video for a better understanding : http://dotnetslackers.com/articles/iis/IIS-Overlapping-App-Pools-Week-16.aspx
Is Recycling an App Pool programatically the same as starting and stopping the app pool in IIS.
Kind of, the difference being that during a recycle http.sys keeps the client connection alive whilst the worker process recycles and transparently routes new requests to the new worker process once it's started.
Also IIS (6 and 7) defaults to using "Overlapped Recycling". What this means is that IIS will start a new worker process before it shuts down the old worker process. Once the new process is started all new requests are directed to the new process.
If you stop and start the application pool you will dump all of your client connections and users may receive a Service Unavailable message whilst the new pool is being started.
Additional differences are:
You can do recycle to your application pool only, while stop & start influences all application pools.
When you are asking to recycle you have time to finalize your work (like saving your data etc.), which is not in restart.
You can schedule the IIS to run the recycle job depends on specific time, after period of time, depends on your memory consumption etc..
Sometimes when IIS restarts the app pool it will start a new instance of my application before the previous instance is shut down completely. This causes me alot of problem so i wonder what i can do about it.
The course of action goes something like this. (spanning about 20 seconds)
Application is running, let's call this instance A.
Restart initializes
A new instance is started, let's call this B (Logged by Application_Start)
Incomming request is processed by instance B, this invalidates all data A has cached.
Timer on instance A is triggered, assumes its cache is valid and writes something invalid into the persistant storage.
Instance A is shut down (logged by Application_End)
Preferable i would like to disable the above behavior completely, IIS should only allow one instance. If not possible, can i in my code detect if other instances is alread running and then wait for it to quit inside application_start? If not possible, what is the best way to work around this?
Disable overlapped recycling:
"In an overlapped recycling scenario,
the process targeted for a recycle
continues to process all remaining
requests while a replacement worker
process is created simultaneously. The
new process is started before the old
worker process stops, and requests are
then directed to the new process. This
design prevents delays in service,
since the old process continues to
accept requests until the new process
has initialized successfully, and is
instructed to shut down only after the
new process is ready to handle
requests."
http://msdn.microsoft.com/en-us/library/ms525803(v=vs.90).aspx