Azure Pipeline: Update variables according to parameter - azure

I have some very simple variables, which I would like to change according to the environment.
I have written the code below in very different ways (including indentation) but none was fruitful. Alternatives I see are
Use variable groups (trying to avoid to have too many of them)
Write a bash script which updates the variables (will work but I
think its not a super neat solution)
variables:
- group : secrets
- name: hello
value: world
${{ if eq(parameters.environment, 'dev') }}:
- name: RabbitMQ_replicaCount
value: 3
${{ if eq(parameters.environment, 'test') }}:
RabbitMQ_replicaCount: '1'
Any other ideas will be appriciated :)

I would rather go by a PS script/Bash script for this task. Why ? The logic part of build where manipulation is needed like setting or overriding var based on branch or env can be done in a better way in script rather than the build yaml itself. Also this part un-necessary elongates the yaml.
Step 1 : Define a var in the build pipe with default env name
and may be another var whose value you want to set based on condition
Step 2 : Add a yml file(lets name it BuildEnv.yml) in your repo which actually contains your PowerShell/Bash code:
steps:
- powershell: |
if($BuildEnv -ne "Test"){
Write-Host "##vso[task.setvariable variable=BuildEnv]Dev"
Write-Host "##vso[task.setvariable variable=RabbitMQ_replicaCount]11"
}
displayName: 'Override Build Env'
# MORE CODE HERE
Step 3: Plug your yml in the build pipe as a template-
trigger:
branches:
include:
- master
name: $(date:yyyy-MM-dd_HH.mm)_$(rev:.r)
stages:
- stage: Build_Stage
displayName: Build_Stage
jobs:
- job: Build_Job
pool:
name: ABC
steps:
- template: ..\BuildEnv.yml
#REST CODE
That's it. You are done.
Reference : Template usage in Azure DevOps build - https://learn.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops

Related

Accessing Variables in Azure pipeline

I am trying to access variables from Pipeline > Library like this
$(AndroidKeyAlias) but I can't see any output. I have tried a couple of other ways as well.
azure-pipelines.yml
trigger:
branches:
include:
- azure-develop
variables:
- group: Example Mobile App
pool:
vmImage: 'macos-latest'
stages:
- stage: initialise_environment
displayName: Initialise Environment
jobs:
- template: steps/azure-initialise-pipeline.yml
azure-initialise-pipeline.yml
jobs:
- job: initialise_node
displayName: initialise node
steps:
- checkout: self
persistCredentials: true
clean: true
- bash: |
FOO="some value"
echo "##vso[task.setvariable variable=FOO]$AndroidKeyAlias"
- bash: |
set -ex
mkdir -p packages
echo $AndroidKeyAlias > packages.json
cat packages.json
env:
CACHIX_AUTH_TOKEN: $(AndroidKeyAlias)
- bash: |
echo "$(FOO)"
As you can see I have tried getting variables values but I am not getting any output.
How do I get this value? I have tied fetching the values using the Azure CLI tool and I am getting following output
authorized: true
description: ''
id: 123
name: Example Mobile App
providerData: null
type: Vsts
variables:
AndroidKeyAlias:
isSecret: true
value: null
AndroidKeyAliasPassword:
isSecret: true
value: null
AndroidKeyStorePassword:
isSecret: true
value: null
What am I doing wrong?
Thanks in advance.
You're completely messing up the notation for bash environment variable substitution and the Azure Pirelines variable substitution. On top of that there are rules for when variables are available to a template and when not and also specific notation that might be needed there...
In general, when you want Azure pipelines to inline the value of a variable for you, use:
$(VariableName)
These variables will be substituted at the time that the task is invoked and will hold whatever value the variable has at that step in the job
Alternatively, use
${{ variables.Variablename }}
For 'compiletime' substitution. These will be expanded before the job even starts to run and will use the value that is available during template expansion and workflow initialisation.
If you are inside a bash script you can use any of the above mentioned syntaxis. You can also use an environment variable to pass the contents in
- bash: |
$VAR
env:
VAR: $(Variablename)
- pwsh: |
$env:VAR
env:
VAR: $(Variablename)
By passing in variables through the environment you don't have to mix the different variable substitution notations. It makes it easier to see what's going on.
When setting an environment variable in a bash step, the value won't be available in a subsequent step. The environment is cleaned up between steps for security and reasons.
For this reason, this won't work:
# Doesn't work
- bash: |
FOO="some value"
- bash: |
echo $FOO
There are special strings you can stream to the log that will instruct the agent to set a variable for the subsequent steps in the same job.
With templates things can become even harder, as you can also pass variables to a template where it's being called. You can do this with template parameters. The advantage of this syntax is that's it's very evident when you're passing in data:
jobs:
- template: templates/npm-with-params.yml # Template reference
parameters:
name: Linux
vmImage: 'ubuntu-latest'
With a template defined as:
parameters:
- name: name # defaults for any parameters that aren't specified
default: ''
- name: vmImage
default: ''
jobs:
- job: ${{ parameters.name }}
pool:
vmImage: ${{ parameters.vmImage }}
steps:
- script: npm install
- script: npm test
See:
templates
variables
stetting variables in scripts

Azure devops - use runtime variables in template

I have a pipeline that compares a feature branch to the latest common master.
The user provides a feature_hash which is then used to determine the common merge-base with the master branch (git_merge_base.merge_base).
For each - the feature and the master branch - I then proceed to check whether the binaries have already been built, and if not built & upload them.
My problem is that I can't seem to pass this "runtime decision" about the merge-base down to the template scope and have the variable evaluated at runtime.
I have read through the documentation but this left me more confused than before.
It looks somewhat like this:
stages:
- stage: determine_merge_base
dependsOn: []
jobs:
- template: ../job_templates/determine_merge_base.yml
parameters:
ref: ${{ parameters.feature_hash }}
- stage: build_master
dependsOn: determine_merge_base
jobs:
- template: ../job_templates/check_if_binary_release_exists.yml
parameters:
ref: "$[stageDependencies.determine_merge_base.DetermineMergeBase.outputs['git_merge_base.merge_base']]"
- template: ../job_templates/build_and_upload_binaries.yml
parameters:
ref: "$[stageDependencies.determine_merge_base.DetermineMergeBase.outputs['git_merge_base.merge_base']]"
- stage: build_feature
dependsOn: []
jobs:
- template: ../job_templates/check_if_binary_release_exists.yml
parameters:
ref: ${{ parameters.feature_hash }}
- template: ../job_templates/build_and_upload_binaries.yml
parameters:
ref: ${{ parameters.feature_hash }}
The ref parameter gets passed through 3 layers of template to be finally used within a step template like this:
- script: |
git_commit="${{parameters['ref']}}"
Where I end up with this error:
stageDependencies.determine_merge_base.DetermineMergeBase.outputs['git_merge_base.merge_base']: syntax error: invalid arithmetic operator (error token is ".determine_merge_base.DetermineMergeBase.outputs['git_merge_base.merge_base']")
Basically, the gist of it is you can't use parameters, as they are evaluated BEFORE runtime (see here).
So, you can only use variables, which you did with $[stageDepencies.<etc>], but you have to go all the way through, meaning at the execution time of your step as well.
You can use variables value for parameters, but only if the value of the variable is known "early enough" (ie pretty much when the whole pipeline starts), like BuildNumber and other similar ones, which is not your case.
So, in your case, I think this below is the way to do it. You can then package all that in your job templates, but the point is to use the variables directly in your templates, and NOT whatever you pass in as parameters. Typically, when I use parameters on my templates it is for default values, or for values known before the pipeline starts. Everything evaluated on-the-flight has to be consumed as variables :
- stage: MyCheckStage
jobs:
- job: MyCheckJob
steps:
- script: |
echo "##vso[task.setvariable variable=CheckValue;isOutput=true]MyValue"
name: MyCheckStep
- stage: MyDecisionStage
dependsOn: MyCheckStage
variables:
CheckValueFromPreviousStage: $[ stageDependencies.MyCheckStage.MyCheckJob.outputs['MyCheckStep.CheckValue'] ]
jobs:
- job: myJob
steps:
- script: |
echo $(CheckValueFromPreviousStage)

How can I use variable created inside of pipeline (from gradle.properties file) within an template as a task

I have an java project which have gradle.properties file. Im extracting variables defined in gradle.properties as
##vso[task.setvariable variable=myVariable;]`my script to extract it from gradle.properties`
Then im using template from another repository that needs that variable but I can't use it within task, but when I try use it within - script: echo $variable as a step instead of task it is working.
When i try to use it within task it sees variable as $variable not a value.
Maybe there is a better way to extract variables to azure pipeline instead of using this approach?
Check the error message:
We get the error before the pipeline run the bash task, Since it cannot create the variable parampass, we get the parameters value is $(parampass) instead of the variable value.
Check this doc:
In a pipeline, template expression variables ${{ variables.var }} get processed at compile time, before runtime starts. Macro syntax variables $(var) get processed during runtime before a task runs. Runtime expressions $[variables.var] also get processed during runtime but were designed for use with conditions and expressions.
As a workaround:
pipeline.yml
pool:
vmImage: ubuntu-20.04
resources:
repositories:
- repository: common
type: git
name: organisation/repo-name
variables:
- name: parampass
value: xxx
stages:
- stage: "Build"
jobs:
- job: "Build"
steps:
- template: templatename.yml#common
parameters:
par1: ${{ variables.parampass}}
Result:
Probably you do not provide variable to your template
Example execution of template with provided parameter
- template: my/path/to/myTemplate.yml#MyAnotherRepositoryResourceName
parameters:
projectsToBeTested: $(someVariable)
And example template accepting parameters
steps:
- task: DotNetCoreCLI#2
displayName: 'Just testing'
inputs:
command: test
projects: ${{ parameters.projectsToBeTested}}
Please provide more information if it does not help.
Code looks like this:
pipeline.yml
pool:
vmImage: ubuntu-20.04
resources:
repositories:
- repository: common
type: git
name: organisation/repo-name
stages:
- stage: "Build"
jobs:
- job: "Build"
steps:
- bash: |
echo "##vso[task.setvariable variable=parampass]anything"
- template: templatename.yml#common
parameters:
par1: $(parampass)
templatename.yml
parameters:
- name: par1
steps:
- task: SonarCloudPrepare#1
displayName: SonarCloud analysis prepare
inputs:
SonarCloud: ${{ parameters.par1}}
organization: 'orgname'
scannerMode: 'Other'
extraProperties: |
# Additional properties that will be passed to the scanner,
# Put one key=value per line, example:
# sonar.exclusions=**/*.bin
sonar.projectKey= # same param pass case
sonar.projectName= # same param pass case
Generally, it does not matter if i do have parameters passed or if I'm using the template as if it were part of the pipeline code within. Output is always $(parampass) could not be found or smth

Use variables in Azure DevOps Pipeline templates

We have a collection of Azure DevOps pipeline templates that we re-use across multiple repositories. Therefore we wanted to have a file that contains variables for all of our templates.
The repo structure looks like this
template repo
├── template-1.yml
├── template-2.yml
└── variables.yml
project repo
├── ...
└── azure-pipelines.yml
The variables.yml looks like this
...
variables:
foo: bar
In template-1.yml we are importing the variables.yml as described in here
variables:
- template: variables.yml
In the azure-pipelines.yml we are using the template like this
resources:
repositories:
- repository: build-scripts
type: git
name: project-name/build-scripts
steps:
...
- template: template-1.yml#build-scripts
When we now try to run the pipeline, we get the following error message:
template-1.yml#build-scripts (Line: 10, Col: 1): Unexpected value 'variables'
The issue is because you used variable template at steps scope. And variables simply doesn't exists at that level. This should work for you:
resources:
repositories:
- repository: build-scripts
type: git
name: project-name/build-scripts
variables:
- template: template-1.yml#build-scripts
steps:
...
this is available to use at any place where variables are possible to use. So for instance you can use this in that way:
jobs:
- job: myJob
timeoutInMinutes: 10
variables:
- template: template-1.yml # Template reference
pool:
vmImage: 'ubuntu-16.04'
steps:
- script: echo My favorite vegetable is ${{ variables.favoriteVeggie }}.
If your template file only has variables, you can refer to Krzysztof Madej's answer.
If your template file has both variables and steps as shown below, it can only be used by extends.
# File: template-1.yml
variables: ...
steps: ...
Or you can write them in a stage, as shown below.
# File: template-1.yml
stages:
- stage: {stage}
variables: ...
jobs:
- job: {job}
steps: ...
Then insert it as a separate stage.
# azure-pipelines.yml
stages:
- stage: ...
- template: template-1.yml
Replying as I think there was something missed in the original explanations:
First:
If you are referencing variables in a template then you must ensure that you are extending that template when you call it. Calling a template that defines variables that is not extended will result in a failed pipeline execution.
template-1.yml#build-scripts (Line: 10, Col: 1): Unexpected value 'variables'
Second: When referencing variables in multiple templates do not use:
${{ variables.foo }} #even though this is seen in templates examples.
but rather use the normal variable syntax
$(foo) #this works
So for the originally reference example the following should work.
Variables.yml:
variables:
foo: bar
template-1.yml:
variables:
- template: variables.yml
....
steps:
...
- task: CmdLine#2
displayName: Display Variable
inputs:
script: |
echo $(foo)
azure-pipelines.yml:
resources:
repositories:
- repository: build-scripts
type: git
name: project-name/build-scripts
extends:
template: template-1.yml#build-scripts
template-2.yml
steps:
...
- task: CmdLine#2
displayName: Display Variable
inputs:
script: |
echo $(foo)
This approach can help someone, so I decided to post it here.
One more kind of "workaround" for that case, maybe a bit "dirty" since you need to specify the parameters explicitly each time you execute template, which is not a good idea if you have a lot of parameters to pass. (actually, there is a way, read below improved version) But, if you really want or you have not that much parameters, this should work:
the logic is: you have all your parameters in variables template, like templates/vars.yml:
variables:
- name: myVar1
value: MyValue1
- name: myVar2
value: MyValue2
and since you have everything you need in the variables, probably there is no need to importing variables into template itself, because template will be executed in pipeline, which will have your variables imported and you can substitute it explicitly, like in the example below:
templates/my-template-setup-env.yml's content (without variables inside it):
steps:
- script: |
echo "$(myVar3)"
my-azure-pipeline.yml's content (with importing variables template):
name: my-cute-CI-name
trigger:
- main
pool:
vmImage: ubuntu-18.04
variables:
- template: templates/vars.yml # importing your variables from templates
stages:
- stage: RunTheStage
displayName: 'Run first stage'
jobs:
- job: RunTheJob
displayName: 'Run your job'
steps:
- template: templates/my-template-setup-env.yml # your template
parameters:
myVar3: $(myVar1) # substitute value from vars.yml, so myVar1 will be used in templated and printed
Improved version
But, if you have unique naming of your params and variables across all pipelines and templates, you are in safe to not specify it explicitly during template usage, that will work as well:
edited and shortened version of my-azure-pipeline.yml (in case you have the same name of your variable and parameter in template):
variables:
- template: templates/vars.yml
...
steps:
- template: templates/my-template-setup-env.yml # env preparation
# parameters:
# myVar2: $(myVar2) # you don't need to pass any parameters explicitly to the template since you have same name of variable
templates/my-template-setup-env.yml then should be like this:
steps:
- script: |
echo "$(myVar2)" # not myVar3, since myVar3 is not in initial variables file templates/vars.yml
or you need to add remaining variables (myVar3 in our first case) into templates/vars.yml file as well.

How to pass environment specific values to Azure pipeline?

I am deploying Service Fabric Application packages and I have several (~15) devtest environments, any one of which can be used to test a code fix. I can pass in the Service Connection so deploying the final package is not the issue. What I can't figure out is how to set the other environment specific variables based on the target environment.
I tried using the Service Connection name to pick one of several variable template files:
variables:
- name: envTemplateFileTest
${{ if eq( variables['DevConnection'], 'Environ01' ) }}:
value: ../Templates/DEV01-Variables-Template.yml
${{ if eq( variables['DevConnection'], 'Environ02' ) }}:
value: ../Templates/DEV02-Variables-Template.yml
... (snip) ...
variables:
- template: ${{ variables.envTemplateFile }}
But UI variables are not set at compile time. So the template expressions see blank values and fail.
I could use a pipeline variable but then QA would have to make a file change and check it in each time they want to deploy to a different environment than last time.
What I currently have is an empty variable template and a powershell script that sets the values based on different script names.
- task: PowerShell#2
inputs:
targetType: 'filePath'
filePath: '$(Build.ArtifactStagingDirectory)\drop\Deployment\Code\Scripts\Set-$(DevConnection)Variables.ps1'
#arguments: # Optional
displayName: Set environment variables
There has got to be a better way than this. Please.
There is not a direct way to achieve this, as the template expression is parsed at compile time.
However I have workaround which no need to write additional ps script and avoid making a file change and check it in to your repo each time.
Since all your devtest environments has the same deployment steps. Then you can create steps template yaml to hold the deployment steps.
Then you can modify your azure-pipelines.yml like below example:
jobs:
- job: A
pool:
vmImage: 'windows-latest'
steps:
- powershell: |
$con = "$(connection)"
if($con -eq "environ1"){echo "##vso[task.setvariable variable=variablegroup;isOutput=true]environ1"}
if($con -eq "environ2"){echo "##vso[task.setvariable variable=variablegroup;isOutput=true]environ2"}
name: setvarStep
- script: echo '$(setvarStep.variablegroup)'
- job: environ1
pool:
vmImage: 'windows-latest'
dependsOn: A
condition: eq(dependencies.A.outputs['setvarStep.variablegroup'], 'environ1')
variables:
- template: environ1.yaml
steps:
- template: deploy-jobs.yaml
- job: environ2
pool:
vmImage: 'windows-latest'
dependsOn: A
condition: eq(dependencies.A.outputs['setvarStep.variablegroup'], 'environ2')
variables:
- template: environ2.yml
steps:
- template: deploy-jobs.yaml
Above yml pipeline use depenpencies and condition. The first job A will output a variable according to the variable (eg.$(connection)) you specify when running the pipeline. In the following jobs, there are conditions to evaluate the output variable. If condition is satisfied then the job will be executed, the job will be skipped if failed on condition.
What we decided to do was add a Powershell script step that sets the variables based on a string passed in.
- task: PowerShell#2
inputs:
targetType: 'filePath'
filePath: $(Build.ArtifactStagingDirectory)\drop\Deployment\Code\Scripts\Set-DefaultValues.ps1
displayName: Set default pipeline variables
Then we load the appropriate file and loop through the variables, setting each in turn.
param(
[string]
$EnvironmentName
)
$environmentValues = #{}
switch ($EnvironmentName) {
'DEV98' { . '.\Dev98-Values.ps1'}
'DEV99' { . '.\Dev99-Values.ps1'}
}
foreach ($keyName in $environmentValues.Keys) {
Write-Output "##vso[task.setvariable variable=$($keyName)]$($environmentValues[$keyName])"
}
This allows us to put the environment specific variables in a plain PSCustom object file and dot import it.
$environmentValues = #{
currentYear = '2020';
has_multiple_nodetypes = 'false';
protocol = 'http';
endpoint = 'vm-dev98.cloudapp.com';
... snip ...
}
So QA has an easier time maintaining the different environment files.
Hope this helps others out there.

Resources