I have an App Service Plan, and in this plan I have deployed 5 components of my solution as Web Apps. I use 'Release Management' in Azure DevOps to deploy code to these apps.
To minimise downtime during deployment, I deploy to staging slots first, and then swop the staging slots into production slots to complete deployment.
I have configured App Service Warmup (as detailed here) to call an endpoint that will 'warm up' the application during the slot swapping process.
This seems to work, but I have two issues:
Even though the warmup has run, the first request made to the app after slot swapping takes a long time. I suspect that this is due to the production slot having a 'sticky / slot settings', which as I understand it necessitates an app restart. To test this, I removed the slot settings, but the delay is still there.
The applications are dependent on each other, and the slot swapping (even though kicked off in parallel in Azure DevOps), is not guaranteed to complete at the same time, which means that it is possible for newer code to be interacting with old code. While we can engineer around this, this is not optimal.
From my investigations so far, the only way I can think of to work around these issues is to spin up a second app service plan, and configure traffic manager to sit in front of the two service plans. When deploying, I will prioritise one of the service plans while I deploy to the other service plan, and once that is complete divert traffic to the newly deployed service plan while upgrading the other, and then balancing the traffic between the two again once both are on the same code level.
What is the current 'best practice' for zero downtime deployments when using WebApps in Azure?
Is the duplicated service plan with traffic manager a viable option, and if not, what would you suggest?
Follow these more best practice recommendation.
SWAP BASED ON THE STATUS CODE
During the swap operation the site in the staging slot is warmed up by making an HTTP request to its root directory. More detailed explanation of that process is available at How to warm up Azure Web App during deployment slots swap.
By default the swap will proceed as long as the site responds with any status code. However, if you prefer the swap to not proceed if the application fails to warm up then you can configure it by using these app settings:
WEBSITE_SWAP_WARMUP_PING_PATH: The path to make the warm up request
to. Set this to a URL path that begins with a slash as the value. For
example, “/warmup.php”. The default value is /.
WEBSITE_SWAP_WARMUP_PING_STATUSES:Expected HTTP response codes for
the warm-up operation. Set this to a comma-separated list of HTTP
status codes. For example: “200,202” . If the returned status code is
not in the list, the swap operation will not complete. By default,
all response codes are valid.
MINIMIZE RANDOM COLD STARTS
WEBSITE_ADD_SITENAME_BINDINGS_IN_APPHOST_CONFIG: setting this to “1”
will prevent web app’s worker process and app domain from recycling
when the App Service’s storage infrastructure gets reconfigured.
https://ruslany.net/2019/06/azure-app-service-deployment-slots-tips-and-tricks/#prevent-cold-start
CONTROL SLOT-STICKY CONFIGURATION
If however for any reason you need to revert to the old behavior of swapping these settings then you can add the app setting WEBSITE_OVERRIDE_PRESERVE_DEFAULT_STICKY_SLOT_SETTINGS to every slot of the app and set its value to “0” or “false”.
https://ruslany.net/2019/06/azure-app-service-deployment-slots-tips-and-tricks/#slot-sticky-config
I would suggest to use Local cache in conjunction with Deployment Slots to prevent any downtime.
Add the sticky app setting WEBSITE_LOCAL_CACHE_OPTION with the value Always to your Production slot. If you're using WEBSITE_LOCAL_CACHE_SIZEINMB, also add it as a sticky setting to your Production slot.
• Create a Staging slot and publish to your Staging slot. You typically don't set the staging slot to use Local Cache to enable a seamless build-deploy-test life-cycle for staging if you get the benefits of Local Cache for the production slot.
• Test your site against your Staging slot.
• When you are ready, issue a swap operation between your Staging and Production slots.
• Sticky settings include name and sticky to a slot. So when the Staging slot gets swapped into Production, it inherits the Local Cache app settings. The newly swapped Production slot will run against the local cache after a few minutes and will be warmed up as part of slot warmup after swap. So when the slot swap is complete, your Production slot is running against the local cache.
Refer the Azure Best Practice Document:
https://learn.microsoft.com/en-us/azure/app-service/deploy-best-practices
https://learn.microsoft.com/en-us/azure/app-service/overview-local-cache
I utilise the "Swap with Preview" feature before warming the sites.
The main problem with swapping slots is that the worker processes are restarted when you swap. This means the sites need to be re-warmed.
When you use Swap with Preview the worker processes are restarted but the swap does not complete, meaning you can warm the sites accordingly. Once you are happy with your testing and performance you simply "Complete Swap" and the sites will respond the same.
Related
I have an Azure webapp (ASP Core MVC) and normally use 2 slots:
Production
Staging
I deploy to Staging, test to make sure things are running properly, and swap with Production. No problems except for the 1-2 min downtime during swap.
However, I have a new piece of functionality that I want to test out with a certain % of traffic. So I added a slot called "TestProduction". I deployed the new functionality to that slot and allocated 50% of my traffic to that slot (in Deployment Slots screen).
However, I can't seem to get any traffic to go there. At least traffic from my computer/tablet all seems to go to the regular Production slot. I bumped TestProduction all the way up to 95% and still I'm seeing the Production slot code. Is there something else I'm missing here to test out some traffic to the TestProduction slot?
Thx!
I have tried to repro the same in my lab environment and got positive results after following the below steps.
Step 1: Create an azure webapp service with two deployment slots.
Step 2: Deploy two different codes into the slots.
Now whenever I tried to access the webpage through the browser, I used to see the code only in the specific slot. The reason is when we accessed the web page for the first time it will store cookies on the browser. Refer to this link for more information.
So, to test the slots, open the browser in private mode or incognito mode and enter the URL. You must do this for every new hit.
Our company website will soon be hosted in an App Service in Azure. The website communicates with an API layer that also hosted in Azure and links to our internal systems and databases. The architecture at this level cannot be changed at this time and has quite a bit of background history, etc.
We are looking at implementing always on deployments using Deployment Slots in the App Service in Azure. The API layer will have non-breaking changes for each deployment and deploying the APIs will be the first part of any release, with the website following.
Is have a clear separation between our environments and the release will be tested in Dev, Test and Pre-Prod environments before the production deployment begins. Overall the whole process is fairly simple until it comes to post-implementation (PI) testing that is currently this is mandatory in our company.
We need to be able to test the production deployment prior to the customers using the site. Currently we feature toggle the site into maintenance mode unless its being accessed from a select IP address list. We now need to perform the PI testing on the new version of the site whilst the customer continues to use the older version of the site. I wasn't sure of the best way of achieving this.
One idea I did have is having a subdomain that links directly to the websites _staging deployment slot bypassing the deployment slot settings. In turn some logic in here could go direct to the API _staging deployment slot. This would give the option to post implement the change just prior to clicking the 'Swap' button to swap over the deployment slots.
I know the overall process isn't ideal, but at the moment this can't be changed. Does anyone have any thoughts or other suggestions on the above please?
Azure makes it easy to create deployment slots for App Services. It’s available in the Standard or Premium App Service plan mode. Deployment slots are actually live apps with their own hostnames. App content and configuration elements can be swapped between two deployment slots, including the production slot.
Azure customers can easily perform the following steps
- Deploy the web application to an online deployment slot.
- Run the tests on a deployment slot, within the live environment that potential testers are going to use. Testing environment and production environment exist side-by-side and provide the similar environment.
- Perform an internal swapping of the IP addresses of both slots (via load balancing and traffic management for both the nodes — slots)
- Update applications with zero downtime
- Swap back to a previous version of your app instantly, with zero downtime for users.
References
https://learn.microsoft.com/en-us/azure/app-service/deploy-staging-slots
The overall reason to have deployment slots enabled is that it helps your team to run live testing on the production environment, and in case there are some problems on the production slot, it lets you roll back the swap without having to take your application down for maintenance.
I heard that there will be no session time out happens when the deployment switch happens in Azure from staging to production. Is my understanding correct?. If so how Azure handles internally this switch?.
The answer depends on what switch form staging to production you're talking about. Because you could use Deployment Slots like this, but it is not recommended to have a full-blown Staging environment as a slot on your App Service. Since these slots run on the same App Service Plan as Production, heavy load on Staging could hurt Production performance.
I tend to think of it more as a 'Pre-Production' environment using a Deployment Slot where you can do some last checks (smoke test) before you release the new version of your application into the wild.
I would think sessions are managed internally since the two slots run on the same App Service Plan, making this a relative simple scenario.
Documentation
Deploying an app to a slot first and swapping it into production ensures that all instances of the slot are warmed up before being swapped into production. This eliminates downtime when you deploy your app. The traffic redirection is seamless, and no requests are dropped as a result of swap operations.
Find more information about using slots here: Set up staging environments in Azure App Service
I have one Node.js web app running on Azure with 2 configured instances. When I publish my changes, the web app is not available until it recycles approximately for 1 minute.
The question is - how to make one instance available with the older source code, while updating the other instance with the new code? And when one instance is successfully updated, do the same thing for the other one.
your best bet would be to use Azure Deployment slots to stage your deployments and then swap to production.
More information can be found here:
https://azure.microsoft.com/en-us/documentation/articles/web-sites-staged-publishing/
Directly from the article
Deploying a web app to a slot first and swapping it into production
ensures that all instances of the slot are warmed up before being
swapped into production. This eliminates downtime when you deploy your
web app. The traffic redirection is seamless, and no requests are
dropped as a result of swap operations. This entire workflow can be
automated by configuring Auto Swap when pre-swap validation is not
needed.
I've a .net 4.5 website running on Azure Websites. I've a staging deployment set up. I warm up both sites and then i run the swap process (both from azure portal and from Azure powershell). No matter what I try the swapping process restarts my websites (both) and they take about 3 minutes to boot up.
I was under the impression the deployment swapping was supposed to keep the websites "warm" and have no down time? My websites are unusuable for about 3 minutes!
As far as I can tell both production and staging slots have the same application settings etc.
Any advice or pointers would be greatly appreciated.
Jeff is right. When there are sticky app settings or connection strings defined for the slot then the website's worker process will be restarted during the swap. The sequence of actions that happen in that case is this:
Apply the sticky settings from production slot to the site's worker process that is currently in staging slot. That causes the restart of worker process;
Warm up the site's worker process in the staging slot by making http requests to its root URL on every VM instances where site should be running;
Swap the worker processes between slots. Now the warmed up worker process is in production slot and is accepting production traffic;
Apply the sticky settings from staging slot to the worker process that is currently in staging slot. At this point this is the "used to be in production" worker process. This config change causes restart of that worker process, but at that point it does not take any production traffic.
I've run into this when I am using the sticky app settings options. My assumption is that it does the swap but then has to change the setting to the sticky value and therefore it is like you making a change in web.config or changing the value in the portal yourself which needs to restart the app.
I haven't found any official documentation on this but if you have sticky slots setup I believe it is the culprit. And I'm not sure if there is anything that can be done about it if that is the case.