I need to determine WHY the application pool is recycling. (its for no obvious reason)
Is there any way to determine this inside of the application_end sub in the global.asax file?
I have put some basic logging in there, so I know WHEN its shutting down, but I cannot tell why.
(and its nothing obvious... it just seems like every couple of requests certain operations cause the application to end. I have turned off every normal reason for recycling such as time outs, memory checks, etc, etc, etc. Same code is working fine on a different server, so I am sure its something wrong with this setup, but what?...)
You don't have to incur overhead to add custom logging, ASP.NET 2.0 health monitoring does the job for you. You can add the following configuration which will log events in the eventlogs with information why Application pool is restarted.
To turn ASP.NET health monitoring ON, you can edit the "master" web.config file, normally found in %systemroot%\microsoft.net\framework\v2.0.50727\config.
First, look for <healthMonitoring> in the master web.config
Inside the healthMonitoring node, find the <rules> node
Inside rules, add the following:
<add name="Application Lifetime Events Default"
eventName="Application Lifetime Events"
provider="EventLogProvider"
profile="Default"
minInstances="1"
maxLimit="Infinite"
minInterval="00:01:00"
custom="" />
Reproduce the issue and look in the Application event log fpr a source of ASP.NET 2.0. This should log why application pool is recycled.
Try looking in the EventLog. When the app pool recycles, there is an entry written to the log along with the reason.
The following link describes the errors codes you'll see in the eventlog for IIS 7.5
http://technet.microsoft.com/en-us/library/dd349270(WS.10).aspx
If there are not already log entries in your troubled machine's event viewer, you can modify IIS to log all app pool recycles.
Article from Microsoft on how to do it is found below
http://support.microsoft.com/kb/332088
Related
Using IIS 7 with a deployed dotnetcore 2.1 or 3.1 web API alone in an application pool, we discovered while looking at Process Monitor on the server, the w3wp.exe workers were logging many errors where they were apparently looking for a web.config. They checked every route in the api's route. The expected behavior was that the w3wp.exe (an IIS worker) would "hand off" the request to the dotnetcore application's routing, which would find the endpoint, but instead, it appeared to be also checking for a web.config. The process monitor revealed w3wp.exe QueryOpen NAME NOT FOUND and PATH NOT FOUND errors.
I looked at a few articles and concluded it was a problem with web.config inheritance, and there must be some setting in IIS or a dotnetcore configuration that was dictating the behavior of checking each API route path as if it were a virtual directory folder system that might contain a new web.config. The benefit would be that you could have a different web.config in a sub-application, but we didn't want that benefit and we didn't want these IIS workers blowing up the logs with thousands of these errors throughout the day. We found an insanely simple solution that an IIS admin might say "duh" but will hopefully save someone out there some time.
We found the answer on an old blog.iis.net post about web.config inheritance (https://blogs.iis.net/steveschofield/control-web-config-inheritance-with-iis-7-asp-net-options). There is a configuration called allowsubdirconfig that directs the w3wp.exe worker to check subdirectories for a web.config file. Here's how you change it in IIS applicationhost.config that can be found through IIS Manager:
Go to configuration editor
Go to system.applicationHost => sites => virtual directory defaults
Set allowSubDirConfig to False
We also discovered that Microsoft recommends you use this setting for hosting dotnetcore applications on IIS
Skipping the additional file operations can significantly improve
performance of websites that have a very large set of randomly
accessed static content.
https://learn.microsoft.com/en-us/previous-versions//dn529134(v=vs.85)?redirectedfrom=MSDN
Keep in mind, if you use this setting, you'll need to come up with a solution to separate applications that use or don't use the setting.
Related issue with MVC:
ASP.NET MVC security and IIS allowSubDirConfig configuration
We have application insights running in our application (on premise and hosted in azure) and we are sending telemetry without issues, different resources, regular data, pageViews, exceptions, traces etc, recently I was asked to increase the telemetry data by adding Windows Event logs (from event viewer), but, to be honest, I am quite new with azure or application insights and all the documentation I find it a bit confusing, since all I find talks about azure monitor, log analytics workspaces configuration, but nothing clear enough (at least to me) that points me to get this data logged into application insights resources specifically. Is this possible to achieve? Something like adding a nuget package and configure applicationInsights.config?
Update... I've followed your suggestions, and added the nuget package for EtwCollectorTelemetryModule, and modified the applicationInsights.config file.
This is how it looks now:
<Add Type="Microsoft.ApplicationInsights.EtwCollector.EtwCollectorTelemetryModule, Microsoft.ApplicationInsights.EtwCollector"> <Sources <Add ProviderName="Microsoft-Windows-Eventlog" Level="Warning" /> </Sources>
But, I'm still not able to see any kind of logs in the traces table (if i understand correctly, logs will be sent to that table.) Do I need to initialize this module in order to start tracing these logs?
Or am I doing something wrong?
I agree with #Peter Bons here. When you need to add Windows Event logs to azure application insights, you need to add ETW module in the config file.
And pls note here, the EtwCollectorTelemetryModule is an asp.net module so that it suitable for asp.net application.
You can refer to this tutorial to modify the configuration if your program written in asp.net. Pls note, Add Application Insights Telemetry mentioned in the doc is the operation 'right click the project and click Configure Application Insights'
[I'm posting this to record what I actually found out after hours of painful trial-and-error.]
I have a website that I need to be "always running" (because in this case it has a Hangfire job that's responsible for kicking off a scheduled task every 5 minutes), and by default, websites are only started up when the first request is received.
So, how can I ensure that the website is started automatically? And, how can I configure this via the TFS release management tool?
[This answer isn't specific to Hangfire, but see the Hangfire documentation's discussion of this issue for details of how it affects Hangfire, but note that the recommended work-around is somewhat involved, and much more complex than the solution below. See also a separate and quite comprehensive discussion on the Hangfire support forum that gives several alternative solutions.]
In IIS, each website is associated with an Application Pool (App Pool). You can configure your App Pool to start automatically via IIS Manager by changing the "Start Mode" to AlwaysRunning in "Advanced Settings" for the App Pool:
However, starting the App Pool doesn't start the website (or websites) associated with it. The website does not get loaded until the first request is received.
In IIS8 (or IIS7.5 with an extension), a new setting was added that allows us to work around this. You can ensure that the website gets sent a request as soon as the App Pool starts by setting "Preload Enabled" to True in "Advanced Settings" for the website:
The combination of these settings ensure that the website will automatically start up when IIS starts, and immediately after the App Pool is recycled, etc.
But, how can you get those settings applied automatically as part of a TFS release pipeline, rather than having to remember to set them manually?
In your release definition, you presumably have an "IIS Web App Management" task, which sets up the App Pool and the website. In the configuration panel for this step, there should be an "Advanced" box with an "Additional AppCmd.exe Commands" entry field. You can use AppCmd to apply the settings described above.
AppCmd has the most confusing command-line syntax I've yet seen outside of code-golf competitions, but here's the incantation that worked for me:
set config /section:applicationPools -[name='myAppPoolName'].startMode:AlwaysRunning
set app "mySiteName/" /preloadEnabled:true
Note that if you have configuration variables defined for your App Pool name and website name, then you can use those rather than hard-coding the name, such as:
set config /section:applicationPools -[name='$(appPoolName)'].startMode:AlwaysRunning
I hope this helps somebody... Thanks for listening :-)
I have two servers sitting behind a loadbalancer in my service tier. Both of them should be identical - IIS setup the same, AppFabric (to keep two services warmed up), app pools running under either a service account or the app pool identity. On one server, everything works. On the other server, three of my app pools (the two that AppFabric is warming up, under the service accounts, and one that's just a standard app pool with no changes made from default settings) stop running almost as soon as I start them up (sometimes on the first request).
I get five of the following error in the Application log each time I try to start one of the app pools:
There was an error during processing of the managed application service auto-start for configuration path: 'MACHINE/WEBROOT/APPHOST/Site/App'. The error message returned is: ''. The worker process will be marked unhealthy and be shutdown. The data field contains the error code.
The error code referenced is 80070005.
This is actually for the same Site/App regardless of the app pool being started (though it may change after recreating the app pools).
In the System log, I get the following warning five times before it errors (Application pool 'AppPool' is being automatically disabled due to a series of failures in the process(es) serving that application pool.):
A process serving application pool 'AppPool' reported a failure during application preloading or service loading. The process id was '2396'. Please ensure that all application preload or service settings in the application pool are configured properly. The data field contains the error number.
The error code referenced is 80004005.
The AppPool here is the one being started.
I've tried recreating; I've tried uninstalling AppFabric (but we need it, so reinstalled and still no go). I'm out of ideas. Any suggestions?
EDIT: I tried copying the applicationHost.config over from the working server, but that didn't work either..
EDIT2: One of the app pools works when running under a real user account but doesn't when running under the ApplicationPoolIdentity....
(Also, we had an issue where the site was running under 2.0 and the apps were running under 4.0. That may have resolved the ones that are running as the service accounts.)
I was just wrestling with this same problem for a few hours and found a different culprit.
I had added a new configuration section to my Web.config in a recent commit. I also added this section to a separate ERB file used by Puppet to generate a custom Web.config at the point of deployment. In this template file, I added the new section but forgot to include its declaration in <configSections>.
Once I added the declaration to the template, our app's test VMs were able to start up again and this error went away.
While the app pools for the applications were 4.0, the app pool for the site itself was 2.0, causing some of the issues. We also had inetpub on a different drive, and we had to grant access to SERVER\Users.
I know one can set the session timeout. But, if the application itself has received no requests for a given period of time, IIS shuts down the application.
This behavior is configurable in the IIS management console, and I know how to do this. Still, I wonder if it is possible to configure this in web.config.
Not in IIS 6. In IIS 6, Application Pools are controlled by Worker Processes, which map to a Request Queue handled by HTTP.sys. HTTP.sys handles the communication with the WWW Server to determine when to start and stop Worker Processes.
Since IIS 6 was created before .Net, there's no communication hooks between .Net and the low-level http handlers.
ASP.net is implimented as an ISAPI filter, which is loaded by the Worker Process itself. You have a chicken-before-the-egg issue if you are looking at the web.config controlling a worker process. This is primarily why MS did the major re-write of IIS 7 which integrates .Net through the entire request life-cycle, not just the ISAPI filter portion.
You can edit these settings, but not in web.config. If you have IIS7, the setting is in applicationHost.config, and the key attribute is the shutdownTimeLimit.
You can google for it, to find out how to use appcmd and other tools to set or change it.
Example
Also you can directly modify the shutdownTimeLimit by editing the applicationHost.config file, which is in the \inetsrv\config directory.
The schema for the applicationHost.config file is in the \inetsrv\config\schema\IIS_schema.xml file.
So open it in your favorite schema-aware XML editor and you'll get intellisense, etc.
You can do it with setting IdleTimeout.
BUT changes to the processModel element take effect only when the worker process is restarted—not immediately after the setting is changed, as with other configuration elements.
You can see it here:
processModel Element
You also must change machene.config to can edit machine configuration from other archive.