We have 4 Azure environments for the stages of our development process; Dev, QA, UAT, Production. As you would expect, settings and options need to differ between environments, e.g., "apiurl": "http://dev-api.ourdomain.com" in dev needs to become "apiurl": "https://uat-api.ourdomain.com" for UAT.
At the moment we manually set these in the App Service Configuration page in the Azure Portal. There are problems with this method we are trying to overcome:
It cannot be timed to happen with releases, it has to be manually done
It's prone to human error
Previous values are lost
We cannot easily compare values between environments
We cannot easily see which settings are no longer used
We would like to setup an appsettings.json with environment transforms for the differences. This addresses the last 3 issues as it will be stored in our code source control (if not secret), but this is useless if we cannot deploy that same file to set the Azure configuration. Pipeline steps might solve issues 1 and 2, but reintroduces issues 3 and 5.
Surely there is a simple way to do this that I am missing?
I'd recommend creating the appsettings.json file in your source code, and use a pipeline to deploy the app service with it included; those settings will take effect. The pipeline can either adjust the file contents immediately before deployment, or upload the file as-is but also deploy app settings to override the file's contents.
This fits your requirements because:
using a pipeline for automation gives you timing control (req 1) and reduces the risk of human error (req 2)
keeping some of the settings in source control gives you a (partial) history of previous values (req 3)
some of the settings may be the same across all environments; those can be left untouched in the appsettings.json, but those which differ can be overridden by the pipeline and adjusted correctly
you could override settings by using a script task such as Powershell or a FileTransform task to actually change the appsettings.json
or, you could override settings by automated configuration of the azure app service configuration, using the Azure CLI or an ARM Template for example.
For comparison of settings across the different environments (req 5), I'd recommend:
organising your pipeline variables into different groups by environment, so the pipeline code makes it clear what the differences will be when deployed
using azure cli or powershell commands to interrogate the actual deployed app services.
Related
I'm working on an asp.net core project and I'm trying to figure out how to keep my source and my pipelines 100% secret free.
I've got a VM running the azure agent and an azure dev ops pipelines for build and release.
If i delete the site on the VM, the release pipeline will auto-magically recreate it for me and deploy the latest build.
Super cool.
Now I read up on best practices for configuring a .Net core app and I found this article: https://www.humankode.com/asp-net-core/asp-net-core-configuration-best-practices-for-keeping-secrets-out-of-source-control
So its a bad idea to keep secrets in code, that makes perfect sense.
But if i apply the same security principals to Yaml, then surely I shouldn't place secrets in my pipelines either.
But I need the pipelines to be able to just recreate the site from scratch and it should just work. Somehow the site needs to know where its default sql connection is, or it needs to have a key to the azure app config service. I shouldn't have to log onto the VM and create an appsettings.json manually after every release!
So whatever the site needs to operate needs to be included in the pipeline, therefore some artifact, or included in the code.
I've googled for days, but I can't seem to find any info on how to fully automate this.
I've considered creating a custom configuration provider that reads from the actual VM registry, but that feels wrong too.
I basically need a config option that is NOT hosted in the site itself. So i set it up once on the VM and never again.
The approach that Lex Li lists in the comments is the Microsoft recommended way of securing "secrets" in pipelines.
Ben Smith's answer in my opinion is just as good, maybe slightly less secure.
I use this approach in our organization. All of our release pipelines do the final configuration transformation with the appropriate settings based on the environment they are being deployed to.
i.e db connections are transformed at the dev, test and UAT and production deployment stages.
I keep the relevant secrets in the pipeline variables as protected secrets. I do this for 2 reasons:
Only a select number of trusted personnel have access to the release pipeline definitions.
Even if someone does have access to those definitions - you cannot see a secured variable. Even you you "undo the padlock" on the variable tab - you cannot see what the setting is.
Our actual secrets are then stored in our enterprise secret vault.
Using the Azure Key Vault is definitely a good approach. However we already have a centralized place to keep our stuff; I don't want it in 3 spots.
I would be remiss to not include Variable Groups as part of the pipeline process. Same concept as the build / release variables - the difference is you can now share them in one spot.
These are all opinions of course. This is just one way of doing this; which I feel is a pretty good balance of security and flexibility.
In addition to the suggestions in your questions comments, you can also store secrets in the pipeline "Variables" section.
In here you can add variables and then mark them as secret by selecting "Keep this value secret". Once you've saved a secret its value is then obfuscated i.e. you can make use of it but you can no long see its original value within Azure Devops (which admittedly can be rather frustrating if you want to revisit the variable to check it!).
You can then reference the secret variable in your pipeline YAML using the syntax:
$(variable-name)
So this approach keeps secrets safe within Azure Devops until they need to be resolved by the pipeline YAML script.
I have created multiple slots(test, stage and prod) in my azure app service. Similarly I have created respective web.config files for each environment. I am deploying my application through octopus deployment tool in test environment slot, so initially it's picking web.test.config file and it's working fine.
But, I want to swap complete transformation section of web.config file when I swap it to stage or Prod slot while doing swapping through azure portal. Is there any way to do ?
using application setting and connection string of configuration setting, I am able to segregate the setting of each slot. But I am not sure how can I do it for other section like system.identityModel,system.web system.identityModel.services, etc. Therefore I want to replace complete transformation section according to environment while doing swapping.
When I talked with the app service team, they said slots are not meant for this purpose. The main purpose of slots is to allow deployment of new versions with little or no downtime. Or to test new features with a small percentage of the traffic. Not really for different environments, you should use have separate app services for that, to which you deploy separately.
I would like to start using the deployment slots in my Azure App Services for a staging to production task. The problem that I am running into is I can't seem to find a way to do transformations on the web.config outside of the appsetting and connectionstrings. I seem to be missing something, but I need to be able to adjust other configurations, from logging levels, to other integrated config sections.
We currently have been using direct deployments from our build server with msdeploy and Parameters.xml file to do much of this work, however that won't work with deployment slots.
When you create the additional deployment slot, you can clone all settings from the existing instance, or dont copy anything. As per documentation:
"You can clone configuration from any existing slot. Settings that can
be cloned include app settings, connection strings, language framework
versions, web sockets, HTTP version, and platform bitness."
In the new deployment slot, under Settings > Configuration, you can specify any settings here to be specific to the new deployment slot by selecting "Deployment slot setting".
I have used this documentation to setup continuous deployment on Azure functions with git-hub. Now I have successfully setup git-hub repository in Azure function for continuous deployment.
I am able to see APP settings in local.settings.json file. But I want to know how to configure Connection strings (SQL Connection strings) in continuous deployment.
You want to keep your sensitive data out of your source code repository. That being said, app.settings.json of any kind is not an option.
You have several options there. Two mostly used are either Azure Vault or Azure AppSettings where you can set key/value pairs which are then available to your app either through settings (merged with other settings) or through environment variables.
I use second approach with env variables access to AppSettings.
This link might help:
https://www.hanselman.com/blog/BestPracticesForPrivateConfigDataAndConnectionStringsInConfigurationInASPNETAndAzure.aspx
The documentation that you've referred to is sort of a "quick and easy" continuous deployment option for Functions that gets you up and running quickly. It's not ideal for a production environment though, as you've discovered there is no way to parameterize your app settings. In order to build a proper build/release pipeline you'll need to move away from the direct Git integration and use something like VSTS which you can configure properly against multiple environments including the ability to set AppSettings and ConnectionStrings properly. Based on the approach you've taken currently, your only option is to go into the Azure Portal and manually configure AppSetting and ConnectionStrings in your function app directly.
This blog post covers a lot on the topic and shows one approach for accomplishing what you're after:
https://blogs.msdn.microsoft.com/visualstudioalmrangers/2017/10/04/azure-function-ci-cd-devops-pipeline/?utm_source=vs_developer_news&utm_medium=referral
Configuring a complete CI/CD pipeline is a bit involved (as you'll be able to see from the link) so there's no easy answer that can fit right into a StackOverflow post. I highly recommend that you read through that and do some more research on Google around the VSTS and Functions integrations.
I'm quite used to publishing websites to IIS and Azure using the publish profiles feature of Visual Studio 2012. A big feature for me is the ability to specify a configuration transform file per environment, such that I can apply both the "Release" config transform and the "Production" config transform to produce a single result which is a release build tailored for he the production environment.
Now I'm trying to build an Azure worker role, which supports the idea of multiple discrete configurations for the role and will apply the "Release" or "Debug" build configuration transforms depending on the packaging settings.
Is it possible to also apply a target environment config transform to the web.config in a similar fashion to how publishing works?
To clarify, I want to apply multiple config transforms to the web.config of the web application hosted in the Web Role based on the build configuration and target environment.
Have a look at my answer here: Azure Web Role configuration settings across environments
The best solution I've found is to keep your web.config transforms for (Debug/Release).
Then separately, to keep individual Cloud Deployment Projects for each target environment you are going to deploy to (e.g. LocalEmulator, QA, Production).
It is possible to have CloudDeployment level config transforms, (i.e. on the ServiceConfiguration.cscfg) but you can only have a single Service Definition per Cloud Deployment. (see note below). So if you end up in a situation where you need different scaling, instance or endpoint options for different environments, this option falls down. The best solution I've found is one cloud deployment project per environment.
note It might/would be possible to build a config transform system for the csdef file as well using something CTT.exe or SlowCheetah but honestly this was more work than was required for my needs.