I defined a YAML build-pipeline in azure:
variables:
test: '${{ variables.Environment }}'
pool:
vmImage: 'ubuntu-latest'
steps:
- script: |
echo $(test)
displayName: 'Show test'
I queue that pipeline with the Environment variable defined as 'abc':
I expect it to echo abc but instead abc is replaced with nothing - variables.Environment seems to be undefined.
Later on I want to load a different Variable Group depending on the Environment variable, which is why I do not echo $(Environment) directly in the script. It's just a simplified example.
I expect it to echo abc but instead abc is replaced with nothing -
variables.Environment seems to be undefined.
According to this document:
Runtime happens after template expansion. Template variables are processed at compile time, and are replaced before runtime starts. Template variables silently coalesce to empty strings when a replacement value isn't found.
So in your case echo $(test) print out nothing but empty string. Cause the queue variables are used for runtime. For this, you can consider using macro or runtime expression which is for runtime. Both test: $(Environment) and test: $[variables.Environment] work well on my side.
Later on I want to load a different Variable Group depending on the
Environment variable, which is why I do not echo $(Environment)
directly in the script. It's just a simplified example.
As I know, linking different variable groups depending on the dynamic Environment variable is not supported yet, here's one discussion about that topic. And this is one good workaround in that scenario.
Nowadays Azure Devops Service is rolling out the new feature runtime parameters, I think it can meet most of your requirements. It could be a better choice for you, use runtime parameters instead of not supported dynamic Environment variable.
My simple test about this option:
1.Content in yaml:
parameters:
- name: group
displayName: Group Name
type: string
default: TestGroup
values:
- TestGroup
- Group2
- Group3
- Group4
variables:
- group: ${{ parameters.group }}
steps:
- script: |
echo $(Name)
displayName: 'Show group name'
2.My variable group TestGroup:
3.Click run pipeline:
4.The pipeline runs well and the displays the variable defined in variable group:
Hope it helps :)
A variation of this problem:
TL;DR:
Group variables do not seem to be available at template compile time, e.g. when conditionally setting a template to run, based on a variable from that group - only variables explicitly set in the pipeline seem to be available.
Long version:
I have a set of variable groups; 'Group-NonProd', 'Group-Prod', etc. Each contains variable 'identifier', with differing values ("dev", "prod", for the below example).
I have a master pipeline, 'main-pipeline.yml', and in that are several stages, each corresponding to the value of 'identifier', e.g.:
- stage: NonProd
variables:
- group: 'Group-NonProd' ## includes the variable 'identifier' with value "nonprod"
jobs:
- template: nonprod.yml
parameters:
identifier: $(identifier)
- script: echo "environment is $(identifier)"
- stage: Prod
variables:
- group: 'Group-Prod' ## includes the variable 'identifier' with value "prod"
jobs:
- template: prod.yml
parameters:
identifier: $(identifier)
- script: echo "environment is $(identifier)"
However, when I run the pipeline, the parameters 'identifier' do not expand to the value in the group variable 'identifier' - it is blank - I see that when I try using the variable in the below conditional logic (the aim being to use this logic to determine which template to call - see commented out lines - and pass that parameter down to them):
steps:
- ${{ if eq(parameters.identifier, 'nonprod') }}:
# - template: nonprod.yml
- script: echo "Using nonprod template, environment is ${{ parameters.identifier }}"
- ${{ if not(eq(parameters.identifier, 'prod')) }}:
# - template: prod.yml
- script: echo "Using prod template, environment is ${{ parameters.identifier }}"
The above script always resorts to the 2nd condition, because the result is always "Using prod template, environment is " (blank).
Here's the odd thing though - if I explicitly set the variable 'identifier' at each stage, it does work!
e.g. this works:
- stage: NonProd
variables:
- group: 'Group-NonProd'
- name: identifier
value: nonprod
jobs:
- template: nonprod.yml
parameters:
identifier: $(identifier)
- script: echo "environment is ${{ parameters.identifier }}"
Related
I am trying to access variables from Pipeline > Library like this
$(AndroidKeyAlias) but I can't see any output. I have tried a couple of other ways as well.
azure-pipelines.yml
trigger:
branches:
include:
- azure-develop
variables:
- group: Example Mobile App
pool:
vmImage: 'macos-latest'
stages:
- stage: initialise_environment
displayName: Initialise Environment
jobs:
- template: steps/azure-initialise-pipeline.yml
azure-initialise-pipeline.yml
jobs:
- job: initialise_node
displayName: initialise node
steps:
- checkout: self
persistCredentials: true
clean: true
- bash: |
FOO="some value"
echo "##vso[task.setvariable variable=FOO]$AndroidKeyAlias"
- bash: |
set -ex
mkdir -p packages
echo $AndroidKeyAlias > packages.json
cat packages.json
env:
CACHIX_AUTH_TOKEN: $(AndroidKeyAlias)
- bash: |
echo "$(FOO)"
As you can see I have tried getting variables values but I am not getting any output.
How do I get this value? I have tied fetching the values using the Azure CLI tool and I am getting following output
authorized: true
description: ''
id: 123
name: Example Mobile App
providerData: null
type: Vsts
variables:
AndroidKeyAlias:
isSecret: true
value: null
AndroidKeyAliasPassword:
isSecret: true
value: null
AndroidKeyStorePassword:
isSecret: true
value: null
What am I doing wrong?
Thanks in advance.
You're completely messing up the notation for bash environment variable substitution and the Azure Pirelines variable substitution. On top of that there are rules for when variables are available to a template and when not and also specific notation that might be needed there...
In general, when you want Azure pipelines to inline the value of a variable for you, use:
$(VariableName)
These variables will be substituted at the time that the task is invoked and will hold whatever value the variable has at that step in the job
Alternatively, use
${{ variables.Variablename }}
For 'compiletime' substitution. These will be expanded before the job even starts to run and will use the value that is available during template expansion and workflow initialisation.
If you are inside a bash script you can use any of the above mentioned syntaxis. You can also use an environment variable to pass the contents in
- bash: |
$VAR
env:
VAR: $(Variablename)
- pwsh: |
$env:VAR
env:
VAR: $(Variablename)
By passing in variables through the environment you don't have to mix the different variable substitution notations. It makes it easier to see what's going on.
When setting an environment variable in a bash step, the value won't be available in a subsequent step. The environment is cleaned up between steps for security and reasons.
For this reason, this won't work:
# Doesn't work
- bash: |
FOO="some value"
- bash: |
echo $FOO
There are special strings you can stream to the log that will instruct the agent to set a variable for the subsequent steps in the same job.
With templates things can become even harder, as you can also pass variables to a template where it's being called. You can do this with template parameters. The advantage of this syntax is that's it's very evident when you're passing in data:
jobs:
- template: templates/npm-with-params.yml # Template reference
parameters:
name: Linux
vmImage: 'ubuntu-latest'
With a template defined as:
parameters:
- name: name # defaults for any parameters that aren't specified
default: ''
- name: vmImage
default: ''
jobs:
- job: ${{ parameters.name }}
pool:
vmImage: ${{ parameters.vmImage }}
steps:
- script: npm install
- script: npm test
See:
templates
variables
stetting variables in scripts
I have some very simple variables, which I would like to change according to the environment.
I have written the code below in very different ways (including indentation) but none was fruitful. Alternatives I see are
Use variable groups (trying to avoid to have too many of them)
Write a bash script which updates the variables (will work but I
think its not a super neat solution)
variables:
- group : secrets
- name: hello
value: world
${{ if eq(parameters.environment, 'dev') }}:
- name: RabbitMQ_replicaCount
value: 3
${{ if eq(parameters.environment, 'test') }}:
RabbitMQ_replicaCount: '1'
Any other ideas will be appriciated :)
I would rather go by a PS script/Bash script for this task. Why ? The logic part of build where manipulation is needed like setting or overriding var based on branch or env can be done in a better way in script rather than the build yaml itself. Also this part un-necessary elongates the yaml.
Step 1 : Define a var in the build pipe with default env name
and may be another var whose value you want to set based on condition
Step 2 : Add a yml file(lets name it BuildEnv.yml) in your repo which actually contains your PowerShell/Bash code:
steps:
- powershell: |
if($BuildEnv -ne "Test"){
Write-Host "##vso[task.setvariable variable=BuildEnv]Dev"
Write-Host "##vso[task.setvariable variable=RabbitMQ_replicaCount]11"
}
displayName: 'Override Build Env'
# MORE CODE HERE
Step 3: Plug your yml in the build pipe as a template-
trigger:
branches:
include:
- master
name: $(date:yyyy-MM-dd_HH.mm)_$(rev:.r)
stages:
- stage: Build_Stage
displayName: Build_Stage
jobs:
- job: Build_Job
pool:
name: ABC
steps:
- template: ..\BuildEnv.yml
#REST CODE
That's it. You are done.
Reference : Template usage in Azure DevOps build - https://learn.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops
I have a pipeline that compares a feature branch to the latest common master.
The user provides a feature_hash which is then used to determine the common merge-base with the master branch (git_merge_base.merge_base).
For each - the feature and the master branch - I then proceed to check whether the binaries have already been built, and if not built & upload them.
My problem is that I can't seem to pass this "runtime decision" about the merge-base down to the template scope and have the variable evaluated at runtime.
I have read through the documentation but this left me more confused than before.
It looks somewhat like this:
stages:
- stage: determine_merge_base
dependsOn: []
jobs:
- template: ../job_templates/determine_merge_base.yml
parameters:
ref: ${{ parameters.feature_hash }}
- stage: build_master
dependsOn: determine_merge_base
jobs:
- template: ../job_templates/check_if_binary_release_exists.yml
parameters:
ref: "$[stageDependencies.determine_merge_base.DetermineMergeBase.outputs['git_merge_base.merge_base']]"
- template: ../job_templates/build_and_upload_binaries.yml
parameters:
ref: "$[stageDependencies.determine_merge_base.DetermineMergeBase.outputs['git_merge_base.merge_base']]"
- stage: build_feature
dependsOn: []
jobs:
- template: ../job_templates/check_if_binary_release_exists.yml
parameters:
ref: ${{ parameters.feature_hash }}
- template: ../job_templates/build_and_upload_binaries.yml
parameters:
ref: ${{ parameters.feature_hash }}
The ref parameter gets passed through 3 layers of template to be finally used within a step template like this:
- script: |
git_commit="${{parameters['ref']}}"
Where I end up with this error:
stageDependencies.determine_merge_base.DetermineMergeBase.outputs['git_merge_base.merge_base']: syntax error: invalid arithmetic operator (error token is ".determine_merge_base.DetermineMergeBase.outputs['git_merge_base.merge_base']")
Basically, the gist of it is you can't use parameters, as they are evaluated BEFORE runtime (see here).
So, you can only use variables, which you did with $[stageDepencies.<etc>], but you have to go all the way through, meaning at the execution time of your step as well.
You can use variables value for parameters, but only if the value of the variable is known "early enough" (ie pretty much when the whole pipeline starts), like BuildNumber and other similar ones, which is not your case.
So, in your case, I think this below is the way to do it. You can then package all that in your job templates, but the point is to use the variables directly in your templates, and NOT whatever you pass in as parameters. Typically, when I use parameters on my templates it is for default values, or for values known before the pipeline starts. Everything evaluated on-the-flight has to be consumed as variables :
- stage: MyCheckStage
jobs:
- job: MyCheckJob
steps:
- script: |
echo "##vso[task.setvariable variable=CheckValue;isOutput=true]MyValue"
name: MyCheckStep
- stage: MyDecisionStage
dependsOn: MyCheckStage
variables:
CheckValueFromPreviousStage: $[ stageDependencies.MyCheckStage.MyCheckJob.outputs['MyCheckStep.CheckValue'] ]
jobs:
- job: myJob
steps:
- script: |
echo $(CheckValueFromPreviousStage)
I have an java project which have gradle.properties file. Im extracting variables defined in gradle.properties as
##vso[task.setvariable variable=myVariable;]`my script to extract it from gradle.properties`
Then im using template from another repository that needs that variable but I can't use it within task, but when I try use it within - script: echo $variable as a step instead of task it is working.
When i try to use it within task it sees variable as $variable not a value.
Maybe there is a better way to extract variables to azure pipeline instead of using this approach?
Check the error message:
We get the error before the pipeline run the bash task, Since it cannot create the variable parampass, we get the parameters value is $(parampass) instead of the variable value.
Check this doc:
In a pipeline, template expression variables ${{ variables.var }} get processed at compile time, before runtime starts. Macro syntax variables $(var) get processed during runtime before a task runs. Runtime expressions $[variables.var] also get processed during runtime but were designed for use with conditions and expressions.
As a workaround:
pipeline.yml
pool:
vmImage: ubuntu-20.04
resources:
repositories:
- repository: common
type: git
name: organisation/repo-name
variables:
- name: parampass
value: xxx
stages:
- stage: "Build"
jobs:
- job: "Build"
steps:
- template: templatename.yml#common
parameters:
par1: ${{ variables.parampass}}
Result:
Probably you do not provide variable to your template
Example execution of template with provided parameter
- template: my/path/to/myTemplate.yml#MyAnotherRepositoryResourceName
parameters:
projectsToBeTested: $(someVariable)
And example template accepting parameters
steps:
- task: DotNetCoreCLI#2
displayName: 'Just testing'
inputs:
command: test
projects: ${{ parameters.projectsToBeTested}}
Please provide more information if it does not help.
Code looks like this:
pipeline.yml
pool:
vmImage: ubuntu-20.04
resources:
repositories:
- repository: common
type: git
name: organisation/repo-name
stages:
- stage: "Build"
jobs:
- job: "Build"
steps:
- bash: |
echo "##vso[task.setvariable variable=parampass]anything"
- template: templatename.yml#common
parameters:
par1: $(parampass)
templatename.yml
parameters:
- name: par1
steps:
- task: SonarCloudPrepare#1
displayName: SonarCloud analysis prepare
inputs:
SonarCloud: ${{ parameters.par1}}
organization: 'orgname'
scannerMode: 'Other'
extraProperties: |
# Additional properties that will be passed to the scanner,
# Put one key=value per line, example:
# sonar.exclusions=**/*.bin
sonar.projectKey= # same param pass case
sonar.projectName= # same param pass case
Generally, it does not matter if i do have parameters passed or if I'm using the template as if it were part of the pipeline code within. Output is always $(parampass) could not be found or smth
I am deploying Service Fabric Application packages and I have several (~15) devtest environments, any one of which can be used to test a code fix. I can pass in the Service Connection so deploying the final package is not the issue. What I can't figure out is how to set the other environment specific variables based on the target environment.
I tried using the Service Connection name to pick one of several variable template files:
variables:
- name: envTemplateFileTest
${{ if eq( variables['DevConnection'], 'Environ01' ) }}:
value: ../Templates/DEV01-Variables-Template.yml
${{ if eq( variables['DevConnection'], 'Environ02' ) }}:
value: ../Templates/DEV02-Variables-Template.yml
... (snip) ...
variables:
- template: ${{ variables.envTemplateFile }}
But UI variables are not set at compile time. So the template expressions see blank values and fail.
I could use a pipeline variable but then QA would have to make a file change and check it in each time they want to deploy to a different environment than last time.
What I currently have is an empty variable template and a powershell script that sets the values based on different script names.
- task: PowerShell#2
inputs:
targetType: 'filePath'
filePath: '$(Build.ArtifactStagingDirectory)\drop\Deployment\Code\Scripts\Set-$(DevConnection)Variables.ps1'
#arguments: # Optional
displayName: Set environment variables
There has got to be a better way than this. Please.
There is not a direct way to achieve this, as the template expression is parsed at compile time.
However I have workaround which no need to write additional ps script and avoid making a file change and check it in to your repo each time.
Since all your devtest environments has the same deployment steps. Then you can create steps template yaml to hold the deployment steps.
Then you can modify your azure-pipelines.yml like below example:
jobs:
- job: A
pool:
vmImage: 'windows-latest'
steps:
- powershell: |
$con = "$(connection)"
if($con -eq "environ1"){echo "##vso[task.setvariable variable=variablegroup;isOutput=true]environ1"}
if($con -eq "environ2"){echo "##vso[task.setvariable variable=variablegroup;isOutput=true]environ2"}
name: setvarStep
- script: echo '$(setvarStep.variablegroup)'
- job: environ1
pool:
vmImage: 'windows-latest'
dependsOn: A
condition: eq(dependencies.A.outputs['setvarStep.variablegroup'], 'environ1')
variables:
- template: environ1.yaml
steps:
- template: deploy-jobs.yaml
- job: environ2
pool:
vmImage: 'windows-latest'
dependsOn: A
condition: eq(dependencies.A.outputs['setvarStep.variablegroup'], 'environ2')
variables:
- template: environ2.yml
steps:
- template: deploy-jobs.yaml
Above yml pipeline use depenpencies and condition. The first job A will output a variable according to the variable (eg.$(connection)) you specify when running the pipeline. In the following jobs, there are conditions to evaluate the output variable. If condition is satisfied then the job will be executed, the job will be skipped if failed on condition.
What we decided to do was add a Powershell script step that sets the variables based on a string passed in.
- task: PowerShell#2
inputs:
targetType: 'filePath'
filePath: $(Build.ArtifactStagingDirectory)\drop\Deployment\Code\Scripts\Set-DefaultValues.ps1
displayName: Set default pipeline variables
Then we load the appropriate file and loop through the variables, setting each in turn.
param(
[string]
$EnvironmentName
)
$environmentValues = #{}
switch ($EnvironmentName) {
'DEV98' { . '.\Dev98-Values.ps1'}
'DEV99' { . '.\Dev99-Values.ps1'}
}
foreach ($keyName in $environmentValues.Keys) {
Write-Output "##vso[task.setvariable variable=$($keyName)]$($environmentValues[$keyName])"
}
This allows us to put the environment specific variables in a plain PSCustom object file and dot import it.
$environmentValues = #{
currentYear = '2020';
has_multiple_nodetypes = 'false';
protocol = 'http';
endpoint = 'vm-dev98.cloudapp.com';
... snip ...
}
So QA has an easier time maintaining the different environment files.
Hope this helps others out there.