We have WFF 2.5 installed, and have used it to successfully configure a farm and provision a secondary server in our test environment.
Our environment (controller, primary, secondary servers) is Windows 2008 Server Web Edition R2, running IIS 7.5, with WFF 2.5 installed.
We have ongoing issues with a .tmp file in an app pools related directory being locked. Process Monitor indicates that it is the worker process (w3wp.exe) locking the file.
The exact error message is Failed to run operation "ProvisionApplications". Failed to run method "Microsoft.Web.Farm.SyncApplicationsRemoteMethod" on server "abc". Exception in response stream. An error was encountered when processing operation "Delete File" on "ABC85DA.tmp". The error code was 0x80070020. The process cannot access "C:\inetpub\temp\appPools\ABC85DA.tmp" because it is being used by another process
If I shut down the Windows Process Activation Service which AFAIK hosts the worker process, the error is disappears.
Obviously, however , to bring the server online, we need to start the service, and as soon as we do, the automated provisioning step fails, and WFF marks the server as unhealthy, and takes it out of the farm.
I have tried to turn Application Provisioning off by unchecking "Enable Application Provisioning" under the Application Provisioning Module, but the operation still seems to fire every 30 seconds.
So - two problems really:
How to solve the file locking issue on the App pool temp file.
How to turn off automated application provisioning operation on
secondary servers? (this is really a second prize workaround in case
there is no solution to problem 1)
TIA
Rebooting the ARR server caused the server to stop attempting to repeatedly provision the secondary servers (in other words applied the "Enable Application Provisioning" option, which I had turned off)
Otherwise, I think the locking issue would still occur. This may not be an issue, since you can turn off the Windows Process Activations Service while initially provisioning the service, and during any subsequent application provisioning intended to synch the servers.
Automated provisioning on a schedule will still be an issue, I suspect.
Related
We are using Azure DevOps Self hosted agents to build and release our application. Often we are seeing
below error and recovering automatically. Does anyone know what is this error ,how to tackle this and where to exactly check logs about the error ?
We stopped hearing from agent <agent name>. Verify the agent machine is running and has a healthy network connection. Anything that terminates an agent process, starves it for CPU, or blocks its network access can cause this error. For more information, see: https://go.microsoft.com/fwlink?Linkid=846610
This seems to be a known issue with both self-hosted and Microsoft-hosted agents that many people have been reporting.
Quoting the reply from #zachariahcox from the Azure Pipelines Product Group:
To provide some context, the azure pipelines agent is composed of two
processes: agent.listener and agent.worker (one of these per
step in the job). The listener is responsible for reporting that
workers are still making progress. If the agent.listener is unable
to communicate with the server for 10 minutes (we attempt to
communicate every minute), we assume something has Gone Wrong and
abandon the job.
So, if you're running a private machine, anything that can interfere
with the listener's ability to communicate with our server is going to
be a problem.
Among the issues i've seen are anti-virus programs identifying it as a
threat, local proxies acting up in various ways, the physical machine
running out of memory or disk space (quite common), the machine
rebooting unexpectedly, someone ctrl+c'ing the whole listener process,
the work payload being run at a way higher priority than the listener
(thus "starving" the listener out), unit tests shutting down network
adapters (quite common), having too many agents at normal priority on
the same machine so they starve each other out, etc.
If you think you're seeing an issue that cannot be explained by any of
the above (and nothing jumps out at you from the _diag logs folder),
please file an issue at
https://azure.microsoft.com/en-us/support/devops/
If everything seems to be perfectly alright with your agent and none of the steps mentioned in the Pipeline troubleshooting guide help, please report it on Developer Community where the Azure DevOps Team and DevOps community are actively answering questions.
We have ASP.NET Core application used internally which are used during office hours and a batch that should be processed 3 AM every morning which is scheduled by HangFire like this:
RecurringJob.AddOrUpdate(
() => MyBatch(),
"0 0 3 1/1 *");
The problem is that the Application Pool goes to sleep and the batch isn’t processed if the site isn’t manually started (by going to the website usually).
I have searched SO and tampered with these settings in the Application Pool but with no success:
Some sources that I used to modify the settings:
How to prevent/extend idleTimeout in IIS 7?
https://serverfault.com/questions/443065/how-do-i-prevent-iis-8-from-stopping-idle-asp-net-applications
IIS seems like to sleep until the next request
The Application Pool is used by a total of 7 applications (all being inactive during night when the batch should be processed). The used Application Pool uses .NET CLR Version 2.0.
I'm using IIS version 10.0.17134.1.
How can I make the Application Pool stay active so the batch can be invoked regularly every morning?
I ran into the same issue where my ASP.NET core application goes into idle even with "AlwaysRunning" as start mode for the app pool, "Preload Enabled" set to true for the site, and idle timeout set to 0.
I got it to work by installing the Application Initialization module and setting the .NET CLR version to v4.0. Don't use the "No Managed Code" as that would prevent the Always Running from triggering the app start.
I wrote a blog post on this explaining in more details the steps I took to get the app to run continuously.
They got documentation on how to set up service to run without stopping.
http://docs.hangfire.io/en/latest/deployment-to-production/making-aspnet-app-always-running.html#enabling-service-auto-start
My experience (with older IIS versions 7.5, 8.0) is that it works, but not for app pool recycle/domain unload.
Workaround for me was to send init request on the application_end event.
As above - you need to enable the Service Autostart - in addition to this, if you hit multiple exceptions, I have found that the Rapid Fail Protection has shut down Application pools in the past when using HangFire. So it's also worth disabling (or increasing to reasonable limits) this on the application pool.
I'd suggest you put in your process a single call to the HTTP address first, just like a ping, that would be enought to trigger the site startup if it isn't running for some reason.
One other thing is that, by microsoft's description at MSDN the "AlwaysRunning" option would be:
"Specifies that the Windows Process Activation Service (WAS) will
always start the application pool. This behavior allows an application
to load the operating environment before any serving any HTTP
requests, which reduces the start-up processing for initial HTTP
requests for the application."
That may be, to produce the compilation of web pages that is done on the first call to be done before any request coming, but may not actually run the application at all times.
I am on a Shared IIS Hosting with no access to most settings. What I did is add a Recurring Job that would be triggered in minute interval less than the IIS Timeout/Idle.
RecurringJob.AddOrUpdate<IMyKeepAliveService>("KeepHangFireAlive", svc => svc.KeepHangFireAlive(URL_TO_SELF), "*/4 * * * *");
The above CRON is enough to prevent IIS App pool from going to sleep.
I use RestSharp to make a tiny ping/GET request to "Self".
I noticed that for some reason one of the web role stopped and restarted itself. Could someone help me to understand on what scenarios does the webrole restarts itself?
And, is there any way to find why the webrole restarted itself?
That happens once in a while when Azure performs guest OS upgrades - it stops instances honoring upgrade domains and then starts them shortly thereafter. This is the most frequent scenario, the same could happen if the server hosting the VM was diagnosed faulty, but that happens quite rarely.
You should be ready for such restarts - they are normal - and your code should be designed to be able to continue working after such restart.
Here's a post with more details on the upgrade process.
Using a large instance of a virtual machine on Windows Azure. The instance runs Microsoft SQL 2012 with light usage, on Windows Server 2012 + all up to date. No user is logged in at time of failures.
However, several (between none and three) times a day (appears random), the VM halts and shuts down. It does not come back online until someone logs back into the Management Portal and starts the VM again. There is no memory dump created. So I am guessing the host halts the running VM, rather than some configuration instance within the guest OS causes the halt. The subscription has billable funds. Other VMs in the subscription are also affected.
Only event logs generated:
Kernel-Power logged:
The system has rebooted without cleanly shutting down first. This
error could be caused if the system stopped responding, crashed, or
lost power unexpectedly.
Kernel-Boot logged:
The last shutdown's success status was false. The last boot's success
status was true.
How can this be resolved? There is no way to initiate a support request within Azure.
The first point I would do is install some monitoring software like newrelic or foglight and see if you can see if you are running out memory or a process is pushing the CPU into a spin.
This will give you some visibility of the activity on the box over time and give you some evidence should you need it to open a support request.
Azure now has paid support only
http://www.windowsazure.com/en-us/support/plans/
We use developer for exactly this type of situation where you are bit lost to figuring out a situation the cost of $30 dollars compared to running a SQL Server 2012 VM per month makes it worth having. The support under Microsoft are generally very good and they will have more diagnostic information and will be able to give you the heads up if this is because of Azure failure or something else.
Getting diagnostic going though would be first port of call then you can see what is going on and get some evidence together and help you track down the problem.
I have a classic ASP application that has been stable for years and now we're having all kinds of problems with it. After moving the app between machines and wiping the original so we could have a fresh install of windows, we've come to the following "symptom". The app pools do not appear to allow for multiple simultaneous requests. Here's what we are seeing:
The app runs normally for most people, but when someone within one of the app pools accesses a long-running script (usually one with lots of DB access) all of the other users in the pool must wait for that script to complete. Once the script completes, everyone else's requests run. This initially made us suspect the DB connection string or something.
UNTIL... we noticed also that large file uploads into our system also cause the app pool to stop responding. What's interesting about this is that we're using the SAFileup COM+ object to do our uploads, which has a progress display in a pop-up window. When you go to upload the file, the progress display comes up, but then never refreshes to show upload progress. If you wait it out, however, the file will eventually upload and the other pending requests will process as normal.
Our app pools are in the default configuration, using the IWAM account to launch. I checked to ensure that the IWAM account has all the appropriate permissions. It does.
We've tried a variety of DB connection strings, none solved the problem (though I'm thinking it's not the DB connection string). Just in case someone thinks it is, here's our connection string: "Provider=SQLNCLI;Trusted_Connection=yes;Server=(local);Database=demo;". It couldn't be simpler. This string was previously not a problem.
I fussed with the web gardens thing and it does, indeed, make the system respond to multiple requests, but each worker thread in the garden has its own session state which causes our users to get booted when their request gets randomly assigned to a new worker thread. Only having a single worker process in the garden was never an issue before anyway.
I've used SQL Profiler and sp_who2 to see if during the long-running scripts there are any deadlocks or blocks on the SQL Server. There are not.
The issues initially started after we had installed some patches from Microsoft. We wiped a machine clean and installed Win2k3 server, then SP2, and then didn't patch anymore after that. The problem remained, so it doesn't appear to have been a patch.
I'm pretty much at a loss now... does anyone have any experience with similar issues? If so, how were they fixed?
Check that you don't have ASP debugging enabled on the server. This will force the ASP script engine to run on a single thread.
Sounds like an limit on the number of concurrent incoming requests to the IIS or the Windows Server..
Check out http://blogs.msdn.com/b/david.wang/archive/2006/04/12/howto-maximize-the-number-of-concurrent-connections-to-iis6.aspx and http://forums.iis.net/p/1152112/1880908.aspx#1880908 on how to tweak the settings.