I am updating my Azure App Service from Azure DevOps. Currently, my release is like this:
Stop the App Service,
Update the App Service, and
Start the App Service.
My question is whether it reasonable to stop the App Service during the update? When I select a release template from Azure DevOps for Azure App Service, there are't any stop/start steps, only the update step. So I am wondering if the stop/start is even needed?
What we have done mostly is:
Stop staging slot
Deploy to slot
Start slot
Swap staging to production
Stop staging slot
Martin's suggestion on Take app offline is also a good one!
We prefer to deploy to slots and then swap so we incur minimal impact to production and can also rollback easily.
Stopping/taking app offline can prevent file locking issues.
It probably depends on your app. If you don't have any issues when you just update your app (such as the a file is in use issue) you can consider to use the Take App Offline flag which will place an app_offline.htm file in the root directory of the App Service during the update (then it will be removed). This way user will recognize that something is happening with the app.
However, I often ended up doing the same like you: Stop, Update, Start 😉
There are (5) options for safe-deployment (atomic updates) to Azure Web Apps. Here is my preferred order ranked by priority and feature richness:
Run-from-Package + ZipDeploy (makes site read-only)
ZipDeploy (using kudu REST api - automatically takes site offline)
Azure CLI (az webapp)
msdeploy (-enableRule:AppOffline, or stop/start site to enforce atomicity)
FTP (using publish profile, make sure to upload appoffline.htm)
There are numerous other deployment options like cloud sync, github continuous, local git, etc - but they are all built upon Kudu APIs (as is Azure CLI).
Note: If you're using Azure DevOps - it's supports nearly all these options - leverage the Azure App Service Deploy task
Agree with both Martin and juunas. If you want to deploy without impacting users then you need to use the slot swap approach. juunas brings up the great point of easily rolling back too. Our approach includes another slot we call "hotfix". This adds a few benefits:
Having an environment with production configs so that you can optionally do additional testing before actually doing the swap.
Roll back in prod even when devs have already deployed into a staging environment.
Allows you to test bugs in the current and previous versions of the code. Helpful when someone says "well it worked before this deployment"...
This is what it looks like.
Related
Our company website will soon be hosted in an App Service in Azure. The website communicates with an API layer that also hosted in Azure and links to our internal systems and databases. The architecture at this level cannot be changed at this time and has quite a bit of background history, etc.
We are looking at implementing always on deployments using Deployment Slots in the App Service in Azure. The API layer will have non-breaking changes for each deployment and deploying the APIs will be the first part of any release, with the website following.
Is have a clear separation between our environments and the release will be tested in Dev, Test and Pre-Prod environments before the production deployment begins. Overall the whole process is fairly simple until it comes to post-implementation (PI) testing that is currently this is mandatory in our company.
We need to be able to test the production deployment prior to the customers using the site. Currently we feature toggle the site into maintenance mode unless its being accessed from a select IP address list. We now need to perform the PI testing on the new version of the site whilst the customer continues to use the older version of the site. I wasn't sure of the best way of achieving this.
One idea I did have is having a subdomain that links directly to the websites _staging deployment slot bypassing the deployment slot settings. In turn some logic in here could go direct to the API _staging deployment slot. This would give the option to post implement the change just prior to clicking the 'Swap' button to swap over the deployment slots.
I know the overall process isn't ideal, but at the moment this can't be changed. Does anyone have any thoughts or other suggestions on the above please?
Azure makes it easy to create deployment slots for App Services. It’s available in the Standard or Premium App Service plan mode. Deployment slots are actually live apps with their own hostnames. App content and configuration elements can be swapped between two deployment slots, including the production slot.
Azure customers can easily perform the following steps
- Deploy the web application to an online deployment slot.
- Run the tests on a deployment slot, within the live environment that potential testers are going to use. Testing environment and production environment exist side-by-side and provide the similar environment.
- Perform an internal swapping of the IP addresses of both slots (via load balancing and traffic management for both the nodes — slots)
- Update applications with zero downtime
- Swap back to a previous version of your app instantly, with zero downtime for users.
References
https://learn.microsoft.com/en-us/azure/app-service/deploy-staging-slots
The overall reason to have deployment slots enabled is that it helps your team to run live testing on the production environment, and in case there are some problems on the production slot, it lets you roll back the swap without having to take your application down for maintenance.
We are deploying into azure using Octopus deploy. We are using it since more than a year, and suddenly we started (about 3 weeks ago) to get errors on few deployments.
Microsoft.Web.Deployment.DeploymentDetailedClientServerException: Web Deploy cannot modify the file 'msvcr120.dll' on the destination because it is locked by an external process. In order to allow the publish operation to succeed, you may need to either restart your application to release the lock, or use the AppOffline rule handler for .Net applications on your next publish attempt. Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_FILE_IN_USE
We have the webapp running and always on and we have the app setting 'MSDEPLOY_RENAME_LOCKED_FILES' to 1 that in theory prevents this.
Does anyone knows if something was changed in azure or octopus?
There are a number of reasons files may be locked during deployment. You should be able to get an idea of what may be locking files by using the kudu process explorer, which you can access using the url {yoursite}.scm.azurewebsites.net.
In order to avoid the locking issue altogether, you could make use of slots to achieve a zero downtime deployment if that's an option for you. In this case you could stop the site or enable App Offline which should unlock any files and allow the deployment to succeed after which a slot swap will make the deployment live. App Offline is preferred over using MSDEPLOY_RENAME_LOCKED_FILES, but will take the application offline during the deployment. Octopus also has support for this as an option on the Deploy an Azure Web App step itself, so may be worth a try even without slots.
You can use custom pre/post deployment scripts as part of your Deploy an Azure Web App to make use of the Stop-AzureRmWebAppSlot, Start-AzureRmWebAppSlot and Switch-AzureRmWebAppSlot Powershell commands Azure commandlets to achieve the above.
An alternative may be to use zip deployments, however, the Deploy an Azure Web App Octopus step doesn't have first class support for this quite yet. It can still be achieved using a Run an Azure PowerShell Script along with a package references if this is what you are wanting to do.
I have one Node.js web app running on Azure with 2 configured instances. When I publish my changes, the web app is not available until it recycles approximately for 1 minute.
The question is - how to make one instance available with the older source code, while updating the other instance with the new code? And when one instance is successfully updated, do the same thing for the other one.
your best bet would be to use Azure Deployment slots to stage your deployments and then swap to production.
More information can be found here:
https://azure.microsoft.com/en-us/documentation/articles/web-sites-staged-publishing/
Directly from the article
Deploying a web app to a slot first and swapping it into production
ensures that all instances of the slot are warmed up before being
swapped into production. This eliminates downtime when you deploy your
web app. The traffic redirection is seamless, and no requests are
dropped as a result of swap operations. This entire workflow can be
automated by configuring Auto Swap when pre-swap validation is not
needed.
What I want to accomplish:
I want to deploy an Azure Cloud Service via Release Management. I managed to get this working by following the steps outlined in this post. In the post the Azure publishsettings file is added to the project and used in Release Management to deploy the Azure package to a Cloud Service. So far so good.
What is the issue:
The Azure publishsettings file will also contain information about the production environment. I don't want that information to be available to all the developers and therefor I would like to have a more secure alternative.
What did I try:
I created a custom action which takes 3 arguments: subscription id, subscription name and certificate key. This way the Azure information stays in Release Management and can be passed to a script. This didn't work because the action is not shown in the Release Template Toolbox.
What is my question:
What is the best way to pass Azure credentials to a deployment script via Release Management on a secure manner?
We have a solution for Build today that will work for RM in the future.
Publish Settings file is an important one with which anybody can get access to certain activities. And once how ever the way you pass on the publish settings file, it can be misused (if tried).
So along with the publish settings file, you need to add a bit of process to the deployment like -
Inactive or remove the management certificate which will in turn invalidate the given publish settings and anyone should request for a new set of publish settings file before they actually start any release procedures.
Even though it adds a rough edge to your smooth flow of deployment process, as it is a live or production system, it is always better to tight the process and make it idiot proof.
Just getting used to VS2012 publishing of Cloud Services. At present I have a one instance webrole which contains a MVC3 application. I can publish it to Azure without issue, and it creates the Cloud Service>Web Role>VMs. Fine. Takes a little while.
However when I do a little code change how can I migrate just this code change without replacing all the VMs that implement the WebRole etc.
It seems that Code and infrastructure are inseparable, or have I misunderstood. Is there a way to just update the code bit?
Thanks.
When you roll out an update, you upload an entire package containing not only your code files, but also the configuration for the VM, such as # of instances, ports to open on the firewall, local resources to allocate, etc. These configuration settings are part of the code package - so there is more going on than just updating code files.
However, there are a couple of methods you can use to have more granular control over updates.
Use Web Deploy. One thing to keep in mind, is that any automatic service updates will restore your website to the last fully-deployed package, which may not be as up-to-date. You would only want to use this in staging, then do a full package update for production rollout.
Use an Azure Web Site instead, which allows continuous integration with your source control provider, and direct updates to the code.
Use an Iaas VM instead. These are basically the same as running your own custom server in the Azure cloud, and you have full control over the OS. However, you also have full responsibility for keeping the OS updated and secure.
You can also enable RDP to your Azure Web Role VM's. You will find all your code files there and IIS, but I wouldn't recommend updating your code this way for the same reasons listed in #1.
The code and infrastructure, in a cloud service, are actually separate. All you upload is a deployment package containing just your code and supporting libraries / files. You don't upload a vhd. Azure provides that for you, spinning up a vhd, and then accessing your code on a file folder on that vhd. Same process happens each time you scale out to more instances.
when you make a code change, you build a new deployment package and deploy that. If you do it as an in-place update (vs delete+redeploy), each role is updated on each instance (when you have multiple instances of a role, they're not all updated at the same time). You can even specify that you only want a single role within the deployment to be updated (helpful if, say, you have a worker role in addition to your web role, and want to leave all the worker role instances running).
when the code update happens, the VMs aren't replaced, but they are recycled, and when they start back up, they are running the updated code.
You can use WebDeploy with Cloud Services in production across multiple servers using the AzureWebFarm project (disclaimer: I maintain it).
Alternatively, you can also use the excellent Octopus Deploy deployment technology in conjunction with the AzureWebFarm.OctopusDeploy project (disclaimer: I maintain this one too).
To be honest though, if you just have a simple web app then I wouldn't both with cloud services - I'd just use Web Sites. Feel free to check out my blog post to see the situations which might force you to use cloud services though.
If you enable WebDeploy on the cloud service, you can use web deploy to publish the MVC application.
See http://msdn.microsoft.com/en-us/library/windowsazure/ff683672.aspx for details.
All of the above answers are correct and if you are trying to change your code for a production service you definitely want to do an in place upgrade as described. However, frequently during the dev/test phase or troubleshooting I want to make one small change and test it out quickly. To do this check out http://blogs.msdn.com/b/kwill/archive/2013/09/05/how-to-modify-a-running-azure-service.aspx which describes how to modify the code via RDP to the Azure VM.