Referencing Azure Key Vault secrets from CI/CD YAML - azure

We have a multi-stage YAML pipeline that does CI/CD to an existing set of Azure Resources
The stages are
Build
Deploy to Development and Run Tests
If Previous succeeded - Deploy to Production and Run Tests
We use the AzureRmWebAppDeployment task during the deployment stages and we use the AppSettings argument to that task to specify environment-specific settings. For example
- task: AzureRmWebAppDeployment#4
displayName: 'Deploy Azure App Service'
inputs:
azureSubscription: '$(azureSubscriptionEndpoint)'
appType: '$(webAppKind)'
WebAppName: 'EXISTING__AZURE_RESOURCENAME-DEV'
Package: '$(Pipeline.Workspace)/**/*.zip'
AppSettings: >
-AzureAd:CallbackPath /signin-oidc
-AzureAd:ClientId [GUID was here]
-AzureAd:Domain [domain was here]
-AzureAd:Instance https://login.microsoftonline.com/
-AzureAd:TenantId [Id was here]
-EmailServer:SMTPPassword SECRETPASSWORD
-EmailServer:SMTPUsername SECRETUSERNAME
There are two settings in that set, EmailServer: SMTPUsername and EmailServer: SMTPPassword that I want to pull from an Azure KeyVault. I know how to reference the KV secret from Azure Portal using the syntax
#Microsoft.KeyVault(SecretUri=https://our.vault.azure.net/secrets/SendGridPassword/ReferenceGuidHere)
but how do I reference the value from the YAML pipeline so it is set in Azure?

As pointed out by Thomas in this comment, Referencing Azure Key Vault secrets from CI/CD YAML
I can explicitly set the value in the YAML file like this:
-EmailServer:SMTPPassword #Microsoft.KeyVault(SecretUri=https://our.vault.azure.net/secrets/SendGridPassword/ReferenceGuidHere)

You need to set an AzureKeyVault#1 task with RunAsPreJob to true, this will make your key vault values available as CI/CD jobs environment variables so you can use it as $(KEY-OF-SECRET-VALUE) on the rest of your stages in the job.
The following piece of yaml file is a working example.
We set for python unittest a set of env variable provided from Azure key-vault
trigger:
batch: true # disable concurrent build for pipeline
branches:
include:
- '*' # CI start for all branches
pool:
vmImage: ubuntu-16.04
stages:
- stage: Test
jobs:
- job: sample_test_stage
steps:
- task: AzureKeyVault#1
inputs:
azureSubscription: 'YOUR SUBSCRIPTION HERE'
KeyVaultName: 'THE-KEY-VAULT-NAME'
SecretsFilter: '*'
RunAsPreJob: true
- task: UsePythonVersion#0
inputs:
versionSpec: '3.7'
- script : python -m unittest discover -v -s tests
displayName: 'Execute python unittest'
env: { MY-ENV-VAL-1: $(SECRET-VALUE-1), MY-ENV-VAL-2: $(SECRET-VALUE-2)}
Note that sometimes you need to approve connection beetween AzureDevops and another Azure service like KeyVault

Related

How to declare variable groups once in template and use in multiple repos in Azure yaml pipeline

How to authorize variable in a yaml template in another repo to be used in a different repo. IOW, how to declare variables in a template once and use in multiple repos in azure devops
I am trying to migrate from classic pipelines to yaml in azure devops. So i am trying to set up a repo to host all yaml templates so it can be referenced and reused by multiple repos for builds, etc.
I wrote this yaml pipeline to sample prototyping it:
`name: FirstPL
trigger:
- my_test_branch
pool: my-agent
resources:
repositories:
- repository: blah
type: git
name: foo/bar
ref: refs/heads/poc
variables:
- template: pipeline_vars.yml#blah
steps:
- script: echo $(variable_from_pipeline_vars)
`
However when i run this i get the follwoing error:
An error occurred while loading the YAML build pipeline. Variable group was not found or is not authorized for use. For authorization details, refer to https://aka.ms/yamlauthz.
How can i declare my variables and variables groups once in a template in a repo that is dedicated to host those templates and then use them over and again in multiple repos using the resourcs syntax above? Also, I tried to find a way authorize the variables template but couldn't find anything to enable this.
How to authorize variable in a yaml template in another repo to be
used in a different repo. IOW, how to declare variables in a template
once and use in multiple repos in azure devops. However when I run
this i get the follwoing error:
An error occurred while loading the YAML build pipeline. Variable group was not found or is not authorized for use. For authorization
details, refer to
https://aka.ms/yamlauthz.
You can directly add the variable group in your azure DevOps project in the Library tab and save all your variables from pipeline_vars.yml in the variable group like below:-
Now, You can access this variable group in your YAML pipeline of multiple repos like the below:-
# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml
pool:
vmImage: ubuntu-latest
workspace:
clean: all
resources:
repositories:
- repository: repo_a
type: git
name: InternalProjects/repo_a
trigger:
- main
- release
- repository: repo_b
type: git
name: InternalProjects/repo_b
trigger:
- main
variables:
- group: SharedVariables
steps:
- checkout: repo_a
- checkout: repo_b
- script: |
echo $(databaseServerName)
- task: AzureCLI#2
inputs:
azureSubscription: 'xxx subscription(xxxxxxxxx-f598-44d6-b4fd-xxxxxxxxxxxx)'
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: 'az resource list --location uksouth'
Output:-
It asks for approving permission for the Variable group to run in the pipeline like below:-
Console:-
Tried the same with another repo repo_b in the project and it asks to approve access for repositories and variable groups like the below:-
Output:-
If you want this variable to be accessed in multiple stages/repos/pipelines within the project without authorization prompt. You can click on Security on top and allow it:-
I created one variables template and referenced it in the YAML pipeline to use across multiple repos by checking out another repo like below:-
# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml
pool:
vmImage: ubuntu-latest
workspace:
clean: all
resources:
repositories:
- repository: repo_a
type: git
name: InternalProjects/repo_a
trigger:
- main
- release
- repository: repo_b
type: git
name: InternalProjects/repo_b
trigger:
- main
variables:
- template: pipeline_vars.yml
steps:
- checkout: repo_a
- checkout: repo_b
- script: |
echo $(environmentName)
- task: AzureCLI#2
inputs:
azureSubscription: 'xxx subscription(xxxxxxxx-f598-44d6-b4fd-e2b6e97xxxxxx)'
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: 'az resource list --location uksouth'
Output:-
I tried to reference the same template in another repo where it does not exist it could not read the pipeline_vars.yml file as it does not exist in the repo.
You can make use of variable groups like the above to reference the variables in this pipeline.
One of the possible reasons for this is that the project that hosts the repository with the variables does not allow access to it's repositories from yaml pipelines.
To verify, go to your project's settings -> Pipelines -> Settings -> Verify "Protect access to repositories in YAML pipelines" . This setting is enabled by default. You could set it to off or add a checkout step to your pipeline yaml. See here for more information.

Create a pull request environment - Azure

I would want to create a pull request environment in Azure which gets deployed when a pull request is opened. Users can play around in that environment and find bugs. Once the bugs are fixed, and the PR is closed, I would want to delete that environment.
I am following through Sam Learns Azure and I was able to go through most steps, but its all .NET related.
Does anyone have an idea to do the same for a react-app?
I am also in favor of creating a new slot inside the app-service.
This is my modified code:
jobs:
- deployment: DeployWebServiceApp
displayName: "Deploy webservice app"
environment: ${{parameters.environment}}
pool:
vmImage: ubuntu-latest
strategy:
runOnce:
deploy:
steps:
- task: DownloadPipelineArtifact#2
displayName: 'Download Pipeline Artifacts'
inputs:
artifactName: "$(Build.BuildId)"
buildType: 'current'
- task: AzureCLI#2
displayName: 'Deploy infrastructure with ARM templates'
inputs:
azureSubscription: "${{parameters.serviceConnection}}"
scriptType: bash
scriptLocation: inlineScript
inlineScript: az webapp deployment slot create --name ui-dev-$(prLC)
--resource-group rg-dev
--slot $(prLC)
--configuration-source app-dev
- task: AzureRmWebAppDeployment#3
displayName: 'Azure App Service Deploy: web service'
inputs:
azureSubscription: "${{parameters.serviceConnection}}"
appName: "${{parameters.appServiceName}}"
DeployToSlotFlag: true
ResourceGroupName: '${{parameters.resourceGroupName}}'
package: '$(Pipeline.Workspace)/drop/$(Build.BuildId).zip'
RemoveAdditionalFilesFlag: true
TakeAppOfflineFlag: true
RenameFilesFlag: true
Why Not. You can do that in below steps-
Create the Build Pipe which build your react app
In the build pipe set a VSTS var only when it is a PR. eg varPR = x
Create a Release pipe which deploys the app
Add one more task at the end of build pipe to call the release using webhook based on condition varPR = x.
Ref - Trigger azure pipeline via webhook?
The link you have shared is also cool that does the job same job in a single build pipe.

Deploy packages to multiple webapp service by azure pipeline single stage

I have more than 100 webapp service in azure. I want to deploy packages in 100 webapps by azure pipeline with one pipeline yml file. But I couldn't find any documentation like this. I got one microsoft documentation and they prefer to increase pipeline steps. If I have 100 webapps service, then have to add 100 steps for each deployment. This is not an efficient way and its time consuming. I want just like this step.
- task: AzureWebApp#1
displayName: 'Azure Web App Deploy'
inputs:
azureSubscription: '$(Parameters.connectedServiceName)'
appType: webApp
ResourceGroupName: $(group)
appName: 'JustGoTestAgain, justgotesttwo, webapp123, webapp555, webapp777 and so on ........'
package: '$(build.artifactstagingdirectory)/**/*.zip'
This yaml file is showing error. I couldn't find any essential extensions to fix it. I also couldn't find any azure powershell deployment command regarding to this issue. How can I get the solution?
You will not be able to do this like this. However you can use Azure Cli task:
- task: AzureCLI#2
displayName: Azure CLI
inputs:
azureSubscription: '$(Parameters.connectedServiceName)'
scriptType: ps
scriptLocation: inlineScript
inlineScript: |
$apps= #('JustGoTestAgain, justgotesttwo, webapp123, webapp555, webapp777 and so on ........')
foreach ($app in $apps) {
az webapp deployment source config-zip -g $(group) -n $app --src '$(build.artifactstagingdirectory)/SOME_FOLDER/Artifact.zip'
}
And here you have more details about deployment itself
Annother approach with multiple task bu continuation if one fail is:
parameters:
- name: apps
type: object
default:
- JustGoTestAgain
- justgotesttwo
- and so on
steps:
- ${{ each app in parameters.apps}}:
- task: AzureWebApp#1
displayName: 'Azure Web App Deploy ${{ app }}'
continueOnError: true
inputs:
azureSubscription: '$(Parameters.connectedServiceName)'
appType: webApp
ResourceGroupName: $(group)
appName: ${{ app }}
package: '$(build.artifactstagingdirectory)/**/*.zip'
Thete was issue with space. Now is fine. Apart from that there is only one issue with connectedServiceName
Job Job: Step input azureSubscription references service connection $(Parameters.connectedServiceName) which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz. Job Job: Step input azureSubscription references service connection $(Parameters.connectedServiceName) which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz. Job Job: Step input azureSubscription references service connection $(Parameters.connectedServiceName) which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz.
Which I skipped here as you already have it on your solution.

How to set Pipeline 'Task version' in YAML in Azure Devops pipeline

How can I set this value in YAML? This is from Azure devops pipeline
EDIT: There is a powershell version below the same UI(Preferred PowerShell Version), the question is not about that.
You can just set it as,
steps:
- task: AzurePowerShell#4
displayName: 'Azure PowerShell script: FilePath'
inputs:
azureSubscription: 'Azure Demo'
preferredAzurePowerShellVersion: 1.0.0
However you should be able to see the relevant YAML code by clicking on the YAML
It is set in "#4"
- task: AzurePowerShell#4

Error when trying to use Azure Key Vault task in pipeline. No package found

I try to create my first CI/CD pipe with azure devops. My build, test and publish tasks works fine, but I have problem with fetching secrets from azure key vault:
##[error]Error: No package found with specified pattern: d:\a\1\s\**\*.zip
In my Azure Key Vault I had one secret named "testVault". I would like to print it in log console to check whether it works or not. This is my whole yaml file:
trigger:
- master
pool:
vmImage: 'vs2017-win2016'
variables:
buildConfiguration: 'Release'
steps:
- task: DotNetCoreCLI#2
displayName: Build
inputs:
command: build
projects: '**/*.csproj'
arguments: '--configuration $(buildConfiguration)'
--TEST TASK SKIPPED
- task: DotNetCoreCLI#2
inputs:
command: publish
publishWebProjects: True
arguments: '--configuration $(BuildConfiguration) --output
$(Build.ArtifactStagingDirectory)'
zipAfterPublish: True
- task: AzureRmWebAppDeployment#3
inputs:
azureSubscription: 'BudgetAppARMServiceConnection'
WebAppName: 'HomeBudgetApp'
- script: echo $(testVault)
displayName: 'testvault'
Unfortunately i received error:
Got connection details for Azure App Service:'HomeBudgetApp'
##[error]Error: No package found with specified pattern:
d:\a\1\s\**\*.zip
Successfully updated deployment History at
The error seems not about the Azure Key Vault. It is caused in task AzureRmWebAppDeployment.If you are using "Azure App Service Deploy" task, set the "Package or folder" path with your actual path which contians the app service contents generated by MSBuild or a compressed zip or war file. Please ensure the file or folder exists.
BTW, it's useful to turn on debug mode for your pipeline, define a variable system.debug with value true. You will get the detail log info when the job runs.

Resources