I have two servers sitting behind a loadbalancer in my service tier. Both of them should be identical - IIS setup the same, AppFabric (to keep two services warmed up), app pools running under either a service account or the app pool identity. On one server, everything works. On the other server, three of my app pools (the two that AppFabric is warming up, under the service accounts, and one that's just a standard app pool with no changes made from default settings) stop running almost as soon as I start them up (sometimes on the first request).
I get five of the following error in the Application log each time I try to start one of the app pools:
There was an error during processing of the managed application service auto-start for configuration path: 'MACHINE/WEBROOT/APPHOST/Site/App'. The error message returned is: ''. The worker process will be marked unhealthy and be shutdown. The data field contains the error code.
The error code referenced is 80070005.
This is actually for the same Site/App regardless of the app pool being started (though it may change after recreating the app pools).
In the System log, I get the following warning five times before it errors (Application pool 'AppPool' is being automatically disabled due to a series of failures in the process(es) serving that application pool.):
A process serving application pool 'AppPool' reported a failure during application preloading or service loading. The process id was '2396'. Please ensure that all application preload or service settings in the application pool are configured properly. The data field contains the error number.
The error code referenced is 80004005.
The AppPool here is the one being started.
I've tried recreating; I've tried uninstalling AppFabric (but we need it, so reinstalled and still no go). I'm out of ideas. Any suggestions?
EDIT: I tried copying the applicationHost.config over from the working server, but that didn't work either..
EDIT2: One of the app pools works when running under a real user account but doesn't when running under the ApplicationPoolIdentity....
(Also, we had an issue where the site was running under 2.0 and the apps were running under 4.0. That may have resolved the ones that are running as the service accounts.)
I was just wrestling with this same problem for a few hours and found a different culprit.
I had added a new configuration section to my Web.config in a recent commit. I also added this section to a separate ERB file used by Puppet to generate a custom Web.config at the point of deployment. In this template file, I added the new section but forgot to include its declaration in <configSections>.
Once I added the declaration to the template, our app's test VMs were able to start up again and this error went away.
While the app pools for the applications were 4.0, the app pool for the site itself was 2.0, causing some of the issues. We also had inetpub on a different drive, and we had to grant access to SERVER\Users.
Related
I'm currently in the process of migrating an ASP platform from Windows 2003 R2 IIS 6 web servers to Windows 2012 R2 IIS 8.5 web servers. I'm at the stage where I've migrated a number of sites across to two separate 2012 web servers, all looked great, clients and developers are happy... However the following error has presented itself after a few days hosting on one of the new servers.
Active Server Pages error 'ASP 0223'
TypeLib Not Found
/jobboard/conf/constants.vbs.inc, line 1
METADATA tag contains a Type Library specification that does not match any Registry entry.
The METADATA tag is below:
<!--METADATA TYPE="typelib" NAME="Microsoft ActiveX Data Objects 2.8 Library" UUID="{2A75196C-D9EB-4129-B803-931327F72D5C}" VERSION="2.8"-->
Restarting IIS on this server resolved the issue (albeit temporarily).
Subsequently the other 2012 web server in production presented the same error a couple of days later, again, restarted IIS and works for now.
I've checked the registry and the relevant tag exists with the right UUID and correct permissions.
It doesn't affect all sites on the server, only all sites in a particular application pool.
The application pools use a domain user identity and sites are split up into a number of shared pools.
I've now determined what was causing the above problem...
Our sites on IIS run in a number of shared application pools running as a domain user. We also have a Windows scheduler job which runs a number of scripts over night which also run as the same domain user.
It seems there are cases when this scheduler job runs it interferes with the IIS worker processes. When it completes and ends its user session it unloads the registry file in memory, which the w3wp.exe processes could also using.
This error is presented in the Event log...
Windows detected your registry file is still in use by other
applications or services. The file will be unloaded now. The
applications or services that hold your registry file may not function
properly afterwards. No user action is required.
Along with references to the w3wp.exe processes currently running.
It was replicated when I terminal serviced in as the domain user and logged out again after a period of time. The event log presented the error and the sites all bombed shortly afterwards.
Running the scheduled job as a different user has fixed this issue for us.
I remember having an include file for ADOVBS.inc with all the ADO constants inside and including it as a standard ASP include inside my global include file which is included on every page on the site.
This was before I used the META way of including the file.
So maybe a last resort is to revert to that method of loading in the ADO constants.
It seems like some sort of threshold is being hit, CPU/Memory?, which then prevents IIS caching/loading the file in from the registry. This then causes the error and a recycle of the pool. As no redirect is being done to the 500.100.asp error handler page which hides the error details from the user. It would suggest the error is in IIS and related to the server.
Thanks
I am trying to deploy a Cloud Service with 1 Web Role to Azure.
When I do so, I get this message:
Your role instances have recycled a number of times during an update or upgrade operation. This indicates that the new version of your service or the configuration settings you provided when configuring the service prevent the role instances from running. Verify your code does not throw unhandled exceptions and that your configuration settings are correct and then start another update or upgrade operation.
The project runs just fine locally, and I'm having a hard time figuring out how to start debugging this issue. Are there any common problems that cause this message or steps to figure out what is causing it?
See https://learn.microsoft.com/en-us/archive/blogs/kwill/windows-azure-paas-compute-diagnostics-data. This will walk through all of the diagnostic data available as well as how to troubleshoot the most common issues.
We also had this annoying problem and in our case:
We use local storage, but it wasn't defined in service definition (or Worker Role's properties)
Our worker role project has reference to a service project which has reference to data layer project. But, the worker role project doesn't have reference to the data layer project. As soon as we added reference to data layer project in worker role project, it deploys successfully.
Problem #1 can be easily noticed if you first run the project in your local machine. Exception will be thrown.
Problem #2, however, is more difficult, mainly because it runs just fine in local machine. After 5 days of trouble shooting, we finally found the problem. So, check all references and try to add sub-reference projects, those that are referenced by other references.
We had similar problem, and it was due to some DDLs failed to load. (due to different version from the one MS have deployed to the VM)
Try to set CopyLocal to "true" for all the References in the project, and re-deploy.
I would either remote desktop to the cloud instance and review the Windows Event Logs for exceptions or redeploy with IntelliTrace Enabled. If you choose the later, you can download the IntelliTrace logs from Visual Studio and debug
http://msdn.microsoft.com/en-us/library/windowsazure/ff683671.aspx
One way to find out the actual error is to click on the " 1 instance" at the top of Dashboard after trying to deploy your web role. It will tell you the status of the role instance. The status should include more information about the type of error which blocks your deployment.
It depends on what your case is. For me, the status claimed that I had an unhandled Security exception. After some investigation, it turned out that under my role's OnStart(), I tried to create a event source. However, Azure service doesn't have the permission to create an event source.
For more possible issues, check http://blogs.msdn.com/b/kwill/archive/2013/09/06/troubleshooting-scenario-3-role-stuck-in-busy.aspx
For me, the issue was with my SQL Azure DB firewall rules. My Azure SQL Database servers are not set to "Allow Access to Azure services", so I have to explicitly list IPs that are allowed.
I discovered this after wrapping my code in a try/catch that swallowed all exceptions, refactoring my OnStart() and RunAsync() methods, and setting all my references to Copy Local = True. None of that worked, then I saw that I had this line in my RunAsync() method:
log4net.Config.XmlConfigurator.Configure();
I am using the AdoNetAdapter for log4net and connecting to an Azure SQL DB for logging, so that led me to check the firewall rules.
For me, I had some differing version of nuget packages in my various projects. Once I consolidated everything to the same version(s), it worked fine.
With the release of Windows Azure SDK version 2.2 for Visual Studio 2012 and 2013, Now you can Remote Debug Cloud Resources within Visual Studio.
Once your cloud service is published and running live in the cloud, you can simply set a breakpoint in your local source code. This may help you in digging out what's going wrong!
I am facing a problem while starting the websphere message engine for one of the application deployed on websphere. This application is getting deployed automatically as a part of the installation of Websphere Lombardi 7.2 express edition. It's using websphere 7 internally to deploy it. When I try to start the message engine from the administrative console of websphere I am getting following error:
The messaging engine ProcessCenter01.twperfsvr-twperfsvr_bus cannot be started as there is no runtime initialized for it yet, retry the operation once it has initialized. For the runtime to successfully initialize the hosting server must be started, have its 'SIB service' already enabled, and dynamic configuration reload enabled. If this is a newly configured messaging engine and it is the first messaging engine to be hosted on this server, then it is most likely the 'SIB service' was not previously enabled and thus the server will need to be restarted. The messaging engine runtime might not be initializing because of an error while trying to start, examine the SystemOut.log of the hosting server to check for error messages indicating the problem.
After restarting the server, the same error shows. Can anyone help me to to find what gets loaded as a part of "initialization of runtime"? Are there any config files etc. that I need to check to solve this issue? I am suspecting some missing configuration causing error to load the runtime for this particular application.
I too faced this issue today had to delete all the files under the message store
check the directory-file path mentioned in
Application servers > server1 > Messaging engines > XXX.server1-primaryBus > File store
Just Enable the SIB Services For the particular Server.
Example:Server-->Application Server-->click on Server Name-->right hand side we can see SIB services-->Check box the Enable services.
This will solve your problem
Recently I have faced the same issue when I rebuilt the jvms in UAT envt. After searching on web I found that because of the old messages saved in the message store(flat files in my envt) the messaging engines was not getting initiated. After deleting the old messaging store and restarting the servers it got initialized.
I have struggled with this problem too.
In our situation the problem was that the file message store location was used that was already created for a different (or old) message-engine.
If you add a busmember to the service-bus and use a file store implementation, then you need to supply the path for the store and log folder. Make sure these locations don't exist yet, other wise you will run in the problem above. The message-engine for this member will use these folders.
If you have a script for creating the message-bus infrastructure, make sure that when you delete the bus or remove message-engines, that you remove the file store/log folders for these, before you re-run your script.
Another possibility is that you are using a external database as a data store, and the user that is used for the connection is not allowed to create a database. You might find a ffdc entry like this:
DB2 SQL Error: SQLCODE=-552, SQLSTATE=42502, SQLERRMC=DB2ADMIN;CREATE
SCHEMA, DRIVER=3.61.65
Then you have to go to your database administration tool and give DB2ADMIN the proper privileges. Then restart the server or cluster.
Finally this issue is been resolved. I did not create the schema in SQL Server with same name as that of the username I gave to connect SQL Server during the installation of WLE 7.2
Please find details about this at below link:
http://www.ibm.com/developerworks/forums/message.jspa?messageID=14795282
I've recently ported to a different hosting company (VDS), and on the new server, installed IIS 7.5 Express, latest versions of PHP and MySQL.
I then copied the files from the old server to new server (NOTHING beyond the content of the folders of my websites, so no config files or anything except the web.configs in the sites themselves), set-up websites. I've tried to visit a website, and I got a 503 Service Unavailable message, tried both remotely and locally at the server, no change.
Tried on every site, even on Default Web Site, and no change: still 503.
I've checked the permissions to the folders of the websites (and all the children), and both my account, Network Service, Administrators have effectively full control of all the contents.
All the apps run in DefaultAppPool, which runs under ApplicationPoolIdentity. Tried creating a new app pool, moving sites there, no luck, tried setting "Load User Profile" to False in pool settings (read it here HTTP Error 503 on IIS 7.5 after SP Install), no change.
The event viewer in an extremely detailed way says:
"A process serving application pool 'DefaultAppPool' terminated unexpectedly. The process id was '2356'. The process exit code was '0xfffffffe'."
and nothing else. Five times of this error, and the pool is shut down due to rapid protection. I restart the pool, try again, nothing changes. No matter I'm trying to get an .aspx, .php, or static content. Here is the most weird part: when I set the app pool to run under LocalSystem, it works. No problem. But everything else fails.
I've checked the website folders about ten times, changed, erased, re-added rights etc tried everything but no change. It's as if there is another file that I'm forgetting to check that the pool's user is unable to access. But the event viewer does not help me. This started with the new server and I tried after installing everything, so I can't tell after what this started happening. I obviously won't be running under LocalSystem. Even my own user, Administrator, doesn't work.
The only identity that works is LocalSystem. My user, LocalService, NetworkService, ApplicationPoolIdentity all fail the same way. I'm going crazy, I'm 99% sure this is a user-rights issue. But all the website files are accessible, and I haven't changed anything in the system32 inetconfig or anywhere.
Ok, found my own answer again.
The process w3wp.exe runs under the DefaultAppPool user, not NETWORK SERVICE. I don't know why it doesn't run under NETWORK SERVICE, but after some research it turned out that the process needed access to C:\Windows\System32\inetsrv\config\schema and giving NETWORK SERVICE access didn't change anything. I've by coincidence found somewhere about the DefaultAppPool user, gave read permissions, and by starting the pool, all my websites started working perfectly. Quite weird, I always thought that the IIS processes ran under NETWORK SERVICE by default, and I'm sure that I haven't changed any setting.
Try running below commands with some changes
appcmd set apppool /apppool.name: <YourAppPoolHavingIssuesHere> /managedRuntieVersion:v<.net Framework version here>
I need to determine WHY the application pool is recycling. (its for no obvious reason)
Is there any way to determine this inside of the application_end sub in the global.asax file?
I have put some basic logging in there, so I know WHEN its shutting down, but I cannot tell why.
(and its nothing obvious... it just seems like every couple of requests certain operations cause the application to end. I have turned off every normal reason for recycling such as time outs, memory checks, etc, etc, etc. Same code is working fine on a different server, so I am sure its something wrong with this setup, but what?...)
You don't have to incur overhead to add custom logging, ASP.NET 2.0 health monitoring does the job for you. You can add the following configuration which will log events in the eventlogs with information why Application pool is restarted.
To turn ASP.NET health monitoring ON, you can edit the "master" web.config file, normally found in %systemroot%\microsoft.net\framework\v2.0.50727\config.
First, look for <healthMonitoring> in the master web.config
Inside the healthMonitoring node, find the <rules> node
Inside rules, add the following:
<add name="Application Lifetime Events Default"
eventName="Application Lifetime Events"
provider="EventLogProvider"
profile="Default"
minInstances="1"
maxLimit="Infinite"
minInterval="00:01:00"
custom="" />
Reproduce the issue and look in the Application event log fpr a source of ASP.NET 2.0. This should log why application pool is recycled.
Try looking in the EventLog. When the app pool recycles, there is an entry written to the log along with the reason.
The following link describes the errors codes you'll see in the eventlog for IIS 7.5
http://technet.microsoft.com/en-us/library/dd349270(WS.10).aspx
If there are not already log entries in your troubled machine's event viewer, you can modify IIS to log all app pool recycles.
Article from Microsoft on how to do it is found below
http://support.microsoft.com/kb/332088