I have a web site hosted on Azure as a cloud service. It has a web role with two instances. The instances belong to one resource group and are assigned to different update- and fault domains, which in theory should prevent them from simultaneous reboot.
Some time ago we experienced a ten minutes long downtime and it seems that our both web role instances were rebooted at the same time.
Azure management logs do not say anything relevant, the reboot logs are completely empty.
Is this actually possible that both instances were restarted at the same time?
Why do "Reboot Logs" contain no information?
How can I check what really happened (any other logs somewhere)?
How can I prevent both instances from being rebooted at the same time?
Related
I have a few different containerized web apps running on Azure Container Instance (ACI). I recently noticed that some of these containers just restart with no apparent reason once in a month or so. Since the restarts are on different apps/containers each time, I have no reason to suspect that the apps are crashing.
The restart policy on all of them are set to "Always".
Is it normal or expected for the containers to restart even when there is no app crash? Perhaps when Azure does maintenance on the host machines or maybe a noisy neighbor on the same host causing a pod movement to another host?
(I am in the process of adding a log analytics workspace so that I can view the logs before the restart. Since the restarts are so infrequent, I wouldn't have any logs to look at for quite some time.)
Same here
I've contacted MS support and got the response that per design ACI maintenance can restart the hosts so it can't be expected to run ACI for weeks uninterrupted
Recommendation is to
adapt your app to be resilient (so you don't care about restarts)
use AKS to gain full control over lifecycle
use VM as host for your app with appropriate policies (no updates / restarts...)
For me this was a deal-breaker since I couldn't find this info anywhere. I've ended up with VM.
Is it possible to use the cloud role (classic) LoadBalancerProbe with HTTPS Health probe to automatically restart a failed role instance?
I understand that this situation should be investigated, but if there's some spontaneous issue and all it needs is a reboot, and nobody is immediately aware of the issue (i.e. after hours or in a crisis), then it could resolve the situation.
Or at worst, can we tell it to move the existing connection to a health instance? Or will that happen when the user hits refresh on their browser?
Service Healing feature should help in this case. It is already provided service in Azure. For both web and worker roles, Azure injects a guest agent into the VM to monitor your role. This guest agent also performs a health check every 15 seconds of the role instance in the virtual machine. Again, if a predefined number of consecutive health check failures or a signal from the load balancer causes a role to become unhealthy and a recovery action to be initiated which is to restart the role instance.
Ref : https://azure.microsoft.com/en-us/blog/service-healing-auto-recovery-of-virtual-machines/
What is happening behind the scenes when an App Service is restarted?
I'm trying to troubleshoot a slow restart for my app (ASP.Net and Sql published from Visual Studio) and I feel like understanding this would help me know what the issue is. My app starts within a few seconds on my dev machine but takes 90 seconds on Azure.
From my research, it sounds like a new service instance is provisioned, application files are copied from the shared storage to the instance and it is started. Is this correct? Is there a way to monitor the startup process to see what is slow?
Edit:
It's a tier S1 service plan. The app isn't slow, just the restart. I was hoping to understand the process so that I could understand whether the slow startup is due to my code or just the nature of the way the restart works. I've noticed that my app runs for about 10 seconds after the restart (refreshing the page), then I get a service unavailable for about 20 seconds, then the page is loading until for about 60 seconds.
It all depends which app service plan you are using, different plans have a different memory, network bandwidth, IO etc, App Service runs customer apps in a multi-tenant hosting environment. Apps deployed in the Free and Shared tiers run in worker processes on shared virtual machines, while apps deployed in the Standard and Premium tiers run on the virtual machine(s) dedicated specifically for the apps associated with a single customer.
Refer to this link for a guide on Troubleshooting slow WebApp in Azure.
Sorry guys if my question does not make sense. Let us say I have a website running in cloud. It is running under one instance.
I have a Scale out app service plan such that when CPU usage is more than 80% it creates one more instance. On day 3 let us say CPU usage gets to 85% and one more instance is created. My question is
1) Is it going to put a load balancer in front of these two instances.
2) Are these two Virtual machines going to be in different fault domain?
3) when I have two instances and I deploy from Visual studio, is it going to deploy to both instances?
1) Is it going to put a load balancer in front of these two
instances.
The answer is yes, after you enable autoscale settings, when your service receives additional traffic, you will get more instances without doing anything.
2) Are these two Virtual machines going to be in different fault
domain?
When multiple Cloud Service role instances are deployed, Azure deploys these instances to different fault domains.
More information about cloud service instances and fault domains, please refer to this blob.
3) when I have two instances and I deploy from Visual studio, is it
going to deploy to both instances?
No, after we deploy a new package, it will be applied in all instances of your WebApp.
More information about how to deploy APP to Azure app service, please refer to the link.
1-) Yes
2-) Probably, however I can't say this with 100% sure, because WebApps are PaaS, you delegate the management to Microsoft.
3-) When you deploy a new package it will be applied in all instances of your WebApp.
What is the correct way to way to warm up Cloud Service instances before performing a VIP Swap?
We're running two Web Role instances in our cloud service, and have already performed the following optimizations:
IIS 8.0 Application Initialization module in a Windows Azure Web Role
Controlling Application Pool Idle Timeouts in Windows Azure
Set AutoStart = true
Despite all these changes our site still takes around 10-20 seconds to start up after having performed a VIP Swap.
Is doing something like using a WebRole to hit the endpoints still the best way to go?