I'm currently running a webApp with several deployment slots (e.g. dev, staging, production). Every Slot is connected to a database (db_dev, db_staging, db_production). I want to deploy to the staging slot and then switch with production. How do database migrations fit in here?
I mean, if i deploy a new build with db migrations to staging the db_staging gets updated. What happens if I switch the slots? Do the migrations get applied to db_production? What about downtimes?
From my understanding only the URLs are switched, so after the switch the app in the staging slot would point to the db_production? That does not make sense.
I could deploy to the staging slot and point to db_production (with migrations), but then the db would be updated and could possibly break the app in the live slot.
I have been pondering this too, and as far as I can see the only sensible process is as follows:
Stop Prelive stites
Clone live DB back to a new staging DB
Run scripts to make data safe (clear information which may contact real users etc)
Change staging slot sticky connection string to point at this database
Run DBUP aganst staging DB (now an upgraded version of live-ish)
Deploy to staging slot
Restart prelive sites
Test staging until satisfied
Backup live DB
Run DBUP against live (if they are non-breaking changes, site can
stay up)
Swap live and prelive slots
Check live
If you can keep your db updates non-breaking, then rollback can be simple as swapping the slots back. If not, you are back in the familiar pain of rollback scripts or restoring snapshots.
Instead of hard coding the connecting string in your source code, put them under the connecting strings section of your app service settings and access it as an environment variable. It's not only safer as it will allow you to have just one code for any environment and by checking the setting as a "slot setting" no matter if you swap or not, for that slot, the configuration remains fixed.
More info here:
https://azure.microsoft.com/en-us/documentation/articles/web-sites-configure/
Update:
In the case of a database update, i.e, necessary scripts that need to be run in order to update the database schema for the new app version, you can use the applicationInitialization section of your web.config. Usually used to warm-up the app, but should work for your case as well.
<system.webServer>
<applicationInitialization >
<add initializationPage="/init-script.php" hostName="xxxxxx.azurewebsites.net"/>
</applicationInitialization>
<system.webServer>
The AppInit module will wait until this code completes before finishing the swap process which is basically allowing production traffic to the app. Basic logic would be checking if the database is running the expected version and if not some other logic would be executed in sequence.
Related
Basically I am looking for a way to deploy azure fabrics where services are updated based on nothing but the version number.
I am working on deploying a service fabric application via azure devops. I have written a script that does a diff and updates the version numbers on ApplicationManifest.xml and ServiceManifest.xml. This script has been tested and it updates the versions correctly for the services that have been changed.
Now, when I try to deploy, I get the following error message:
##[error]The content in CodePackage Name:Code and Version:1.0.111 in Service Manifest 'MyMicroServicePkg' has changed, but the version
number is the same.
This error message keeps showing for one service after another until I have updated the version on every single package. Basically it is forcing me to update every single package.
Here is the publish profile I am using:
<?xml version="1.0" encoding="utf-8"?>
<PublishProfile xmlns="http://schemas.microsoft.com/2015/05/fabrictools">
<ClusterConnectionParameters .... />
<ApplicationParameterFile Path="..\ApplicationParameters\PublishProfName.xml" />
<CopyPackageParameters CompressPackage="true" />
<UpgradeDeployment Mode="Monitored" Enabled="true">
<Parameters FailureAction="Rollback" Force="True" />
</UpgradeDeployment>
</PublishProfile>
Here is the devops task on yaml:
- task: ServiceFabricDeploy#1
inputs:
publishProfilePath: $(publishProfilePath)
applicationPackagePath: $(applicationPackagePath)
serviceConnectionName: ${{ parameters.connection }}
overrideApplicationParameter: true
upgradeMode: Monitored
FailureAction: Rollback
# 30 mins timeout
UpgradeTimeoutSec: 3600
I have looked up this issue online. Generally people talk about how to make sure all services with code changes have versions updated. In my case, I am positive the versions are updated for the changed services.
How do I configure the deployment such that it does not do any code comparison and updates only the services that have an updated version?
The gist of the answer here, based on my experience, is that Service Fabric only deploys the entire application as a single versioned collection of each of the internal packages, but leaves specifics of service deployment accessible to you for customization. So even if you haven't made changes to a single service within that application, ultimately it's still going to be bundled up in the application deployed to the cluster and you're still going to have to make a decision about whether to deploy the service or not (or use the provided Deplooy-FabricInstaller.ps1 script to handle such deployments).
My suggestion (based off my own pipeline) is that if you really trust that your script is detecting changed versions properly and you really want to avoid upgrades for services you haven't made changes to (presumably to cut down on deployment time) is to shift around your approach and instead don't fight the application deployment and instead optimize service installation.
Build the solution as you normally would, and utilize the ServiceFabricUpdateManifests#2 task to let it replace all the versions of each service automatically.
At this point, I generally diverge from the standard pipeline.
I've got a script that removes the DefaultServices from ApplicationManifest.xml to remove automatic service deployments
I run a separate script that checks to see if the application already exists or not (to inform whether it's a fresh deployment or an upgrade)
Based on it being an install/upgrade I take a different deployment task route and capture information about the existing services in the cluster
Once the application has been fully deployed, I wrote a custom script that's similar to the included Deploy-FabricApplication.ps1 script that iterates through each of the services and itself installs, upgrades or removes them based on their state in the new deployment utilizing ServiceTemplates to populate configuration values.
Because I control the whole of the actual service install process then, I can infer based on data I collect and pass through the build pipeline what should and shouldn't actually be deployed independent of application deployment.
You might find the customization process a bit easier going this route compared to the approach you're taking.
I have an Azure website with several deployment slots. I use Visual Studio to deploy to a Staging slot and then swap this into Production.
My Visual Studio solution contains one project for my web application and another for a continuous WebJob. I used (Right-click project) > Add > Existing Project as Azure WebJob to configure the WebJob to be deployed to the site along with the web application.
My problem is that the WebJob sometimes continues to run old code after a swap. Sometimes it runs the new code on one invocation and then switches back to the old code on the next invocation. I can't seem to figure out a rhyme or reason to this.
This morning, in order to try to find the cause of this, I did the following experiment:
Delete WebJob in Staging.
Delete WebJob in Production.
Deploy web app to Staging.
Verify that WebJob is running new code in Staging.
Swap Staging into Production.
Verify that WebJob no longer appears in Staging and that it now appears in Production.
What I found was that, after step 4, all the WebJob functions started afresh in Staging and were all running the new code. But after I swapped Staging into Production, there were no new invocations of the WebJob in Production, and when I manually invoked it, the old code ran!
What on earth is going on here? How can I totally clear out the old code and ensure the new WebApp code will run in Production?
I did see another thread saying that checking the Exclude files from the App_Data folder box in publishing settings could cause this issue, but I do not have it checked.
I feel like I'm taking crazy pills here. Any pointers would be sincerely appreciated.
Update 1: As described in my comment below, I have given up and switched to Azure Functions, but I just thought of something that might help anyone else who is encountering this. I recall that at least once or twice, I deployed my WebJob by manually uploading a zip file as opposed to deploying via Visual Studio. I'm starting to wonder if perhaps, under the covers, there is a different code path in use that could give rise to this behavior. Perhaps that will be a useful pointer for someone.
Update 2: I've figured out what happened. I'm describing this in the hopes that it might help someone in the future.
We have several deployment slots: Staging, Production, and a few others. One such slot is called Replica. Its purpose is to be a replica of Production (same code, same configuration settings). The idea is that, for those weird one-off problems that only repro in Production, you can upload a debug build to the Replica slot and attach a debugger without disrupting Production.
This is useful once in a while, but we didn't use it very frequently. The result was that the code running in the Replica slot, including our WebJob code, gradually got further and further out of date compared to what was running in Production. However, meanwhile, the Replica WebJobs kept running the old code, and since the Replica slot has all the same configuration settings as Production, it was operating on the same data and writing to the same logs as our Production WebJobs. So when I would go look at my Production WebJob logs, I would see output that I knew should no longer be there, and I couldn't figure out why. The rogue code was actually running in another slot that I had forgotten about.
So, mystery solved. This probably means we didn't need to cut everything over to Azure Functions. But I'm still glad that we did. I like them better. I like how we have the option to run them in the same app service as our site or a separate one, and the logging system is better. It feels more modern overall.
Anyway, I hope this helps someone out there.
Usually it is not possible run old code after swap successfully and restart your web app. So you can check the content under site->wwwroot in kudu: https://YourWebApp.scm.azurewebsites.net/DebugConsole to make sure your webjob is correct.
I'm looking for some advices concerning release management in azure
I've found a lot of ressources, but I still have some questions
I have an asp.net 4 solution (to make it simple : one asp.net project, one database project, one test project)
I'm using GIT in visual studio online
At this moment I have one app service and one sql server database in azure.
I have a build that download nuget packages, build, execute a dacpac for the database, executes the tests from the project (I have integration tests that uses the database) and finaly deploys the app on an azure app services
What I want to do seems a "normal stuff" :
I want to cerate the build, then deploy it on a "dev" environnement in azure, then in a "qa" environnement, then in a "stagging" and in "prod"
In my web project, I created different web.config transformations (one for each environnement)
I've seen the releases in visual studio online and I get that its for the deployment part in different environnements
What I have questions with :
In Azure :
Do I create 1 app service by environnement ? or do I create a single app service and use slots ?
Do I create 1 sql server for each environnement or is it best to use a single sql server and to have one database for each environnement ?
In visual studio online :
How do I do the tasks ?
In the build part, what configuration do I use ? Which environnement do I select ?
In the build, how do I manage the database project ? I thing the deployment part should be in the release part, but I don't see how to configure the connexionstring ?
For the tests -> Do I execute them in the release part? How do I change the connexionstrings and appsettings? there is no transformations for the app.config
I've seen that there is test-plans as well, but I don't really get it
Can somebody help me to see a little bit better in all of that ?
I cannot answer all of these, but I can answer a few.
I would create separate Web App instances for your separate environments. With slots, your slots exist on the same Web App and share computing resources. If something goes horribly wrong (your staging code pegs CPU to 100% or eats all of your RAM), this will cause problems for your production slot. I see slots as part of A/B testing or to aid in deployment.
You will likely need to create a separate database per environment as well. This is almost always required if you will be upgrading your database schema at any point in the future and introduce breaking changes to your database schema. For example, what happens if your production code requires a specific field in a database table, but your next version of the database removes that field?
You mentioned you're using web.config transforms, but I want to throw out another option that we've found to be easier and have fewer moving parts and sources for error. If you're just changing connection strings and AppSettings, consider using the Web App's application settings per environment. These override whatever is in your web.config. Doing so means you can forget about web.config transforms and not have one more thing that could possibly go wrong in a deployment.
Since you're using a Database project, to deploy your database, check out the VSTS Azure SQL Database Deployment task. It'll use your database project to create a DACPAC, and then deploy that to your target server.
Consider the following deployment workflow, where staging and product use the same database:
Deploy application to Azure Website staging slot
Run EF code first migrations in staging (updating the shared database)
Test application in staging
Swap endpoints so that what was deployed to staging is now in the production slot
This process worked for a few releases, but today it didn't. The issue is around migrations ran in the Staging slot against the same database as the Production slot. We're every careful to not rename or remove any columns that would break production between steps 2 and 4. Today's migrations included a new foreign key and a script to create some tables for a forum control (there are no entities for these new tables, it's managed entirely by a third party control - we only added the migration so that we didn't have to manually run the install script).
We got the standard AutomaticMigrationsDisabledException below in the production slot after step 2.
System.Data.Entity.Migrations.Infrastructure.AutomaticMigrationsDisabledException:
Unable to update database to match the current model because there are
pending changes and automatic migration is disabled. Either write the
pending model changes to a code-based migration or enable automatic
migration. Set DbMigrationsConfiguration.AutomaticMigrationsEnabled to
true to enable automatic migration.
Is there a list of known changes that cause this exception? Or are migrations just not designed to be used this way?
We've also considered something like what SO does, but we'd rather not recreate the wheel if we don't have to.
We are trying to work out a good continuous deployment setup using TFS, Visual Studio and Azure. At our company, each developer has their own Azure subscription that we use for testing, as well as shared QA1/QA2/PROD subscriptions that we can deploy to. We have matching TFS XAML build definitions for each of these, running Powershell scripts with parameters and PublishSettings files.
This all gives us a set of .cspkg and .cspkg files, and in theory we can deploy the right cspkg with the correct cspkg to any Azure system.
The problem we are encountering now though is that we want to start using the Redis Cache service. Installing the nuget package writes subscription-specific settings into the web.config, to point at the cache. This means that the cspkg is now complied specifically for the Azure subscription.
We could use SlowCheetah to merge web.config files on build, but this means that we would have to compile the package for each build definition, and as the number of developers increases this is obviously going to become unsustainable.
I am looking for a way to keep our old generic packages and still use the Redis Cache. We can connect to the cache in code during app_start, but then we can't use it to store IIS session state. I understand that the Azure Load Balancer is meant to keep users on the same server, but I'm unsure how that will work as we swap servers in/out.
It feels like we are approaching the problem wrong and there should be a simple solution that we are overlooking.
We are using Azure Tools 2.6, Visual Studio 2013, TFS 2015r2.
I think there are always 3 ways of doing this.
1st one is config during build, which is building one thing for one thing you described, which is not desired in most of scenario.
2nd is config during deployment, which means you open the cspkg file, change config, then put it back before upload without re-compile.
3nd is config after deployment, have a configuration management tool adjust the config file for you on the fly.
We use octopus deploy to archive #2 above, our CI tool feed octopus with cspkg and cscfg, octopus handles the rest. I would definitely not going after #1 but consider #3 is a valid option too.
As of today we store all our connection settings in .cscfg files. Even if for security reasons, we avoid storing any production connection strings in source control, only QA. And we have CI for QA, but not production. This way it works well for us, we just maintain different .cscfg for different environments (subscriptions)
However, in near future I think we will move to Key Vault for this.