I have a pipeline that compares a feature branch to the latest common master.
The user provides a feature_hash which is then used to determine the common merge-base with the master branch (git_merge_base.merge_base).
For each - the feature and the master branch - I then proceed to check whether the binaries have already been built, and if not built & upload them.
My problem is that I can't seem to pass this "runtime decision" about the merge-base down to the template scope and have the variable evaluated at runtime.
I have read through the documentation but this left me more confused than before.
It looks somewhat like this:
stages:
- stage: determine_merge_base
dependsOn: []
jobs:
- template: ../job_templates/determine_merge_base.yml
parameters:
ref: ${{ parameters.feature_hash }}
- stage: build_master
dependsOn: determine_merge_base
jobs:
- template: ../job_templates/check_if_binary_release_exists.yml
parameters:
ref: "$[stageDependencies.determine_merge_base.DetermineMergeBase.outputs['git_merge_base.merge_base']]"
- template: ../job_templates/build_and_upload_binaries.yml
parameters:
ref: "$[stageDependencies.determine_merge_base.DetermineMergeBase.outputs['git_merge_base.merge_base']]"
- stage: build_feature
dependsOn: []
jobs:
- template: ../job_templates/check_if_binary_release_exists.yml
parameters:
ref: ${{ parameters.feature_hash }}
- template: ../job_templates/build_and_upload_binaries.yml
parameters:
ref: ${{ parameters.feature_hash }}
The ref parameter gets passed through 3 layers of template to be finally used within a step template like this:
- script: |
git_commit="${{parameters['ref']}}"
Where I end up with this error:
stageDependencies.determine_merge_base.DetermineMergeBase.outputs['git_merge_base.merge_base']: syntax error: invalid arithmetic operator (error token is ".determine_merge_base.DetermineMergeBase.outputs['git_merge_base.merge_base']")
Basically, the gist of it is you can't use parameters, as they are evaluated BEFORE runtime (see here).
So, you can only use variables, which you did with $[stageDepencies.<etc>], but you have to go all the way through, meaning at the execution time of your step as well.
You can use variables value for parameters, but only if the value of the variable is known "early enough" (ie pretty much when the whole pipeline starts), like BuildNumber and other similar ones, which is not your case.
So, in your case, I think this below is the way to do it. You can then package all that in your job templates, but the point is to use the variables directly in your templates, and NOT whatever you pass in as parameters. Typically, when I use parameters on my templates it is for default values, or for values known before the pipeline starts. Everything evaluated on-the-flight has to be consumed as variables :
- stage: MyCheckStage
jobs:
- job: MyCheckJob
steps:
- script: |
echo "##vso[task.setvariable variable=CheckValue;isOutput=true]MyValue"
name: MyCheckStep
- stage: MyDecisionStage
dependsOn: MyCheckStage
variables:
CheckValueFromPreviousStage: $[ stageDependencies.MyCheckStage.MyCheckJob.outputs['MyCheckStep.CheckValue'] ]
jobs:
- job: myJob
steps:
- script: |
echo $(CheckValueFromPreviousStage)
Related
I have some very simple variables, which I would like to change according to the environment.
I have written the code below in very different ways (including indentation) but none was fruitful. Alternatives I see are
Use variable groups (trying to avoid to have too many of them)
Write a bash script which updates the variables (will work but I
think its not a super neat solution)
variables:
- group : secrets
- name: hello
value: world
${{ if eq(parameters.environment, 'dev') }}:
- name: RabbitMQ_replicaCount
value: 3
${{ if eq(parameters.environment, 'test') }}:
RabbitMQ_replicaCount: '1'
Any other ideas will be appriciated :)
I would rather go by a PS script/Bash script for this task. Why ? The logic part of build where manipulation is needed like setting or overriding var based on branch or env can be done in a better way in script rather than the build yaml itself. Also this part un-necessary elongates the yaml.
Step 1 : Define a var in the build pipe with default env name
and may be another var whose value you want to set based on condition
Step 2 : Add a yml file(lets name it BuildEnv.yml) in your repo which actually contains your PowerShell/Bash code:
steps:
- powershell: |
if($BuildEnv -ne "Test"){
Write-Host "##vso[task.setvariable variable=BuildEnv]Dev"
Write-Host "##vso[task.setvariable variable=RabbitMQ_replicaCount]11"
}
displayName: 'Override Build Env'
# MORE CODE HERE
Step 3: Plug your yml in the build pipe as a template-
trigger:
branches:
include:
- master
name: $(date:yyyy-MM-dd_HH.mm)_$(rev:.r)
stages:
- stage: Build_Stage
displayName: Build_Stage
jobs:
- job: Build_Job
pool:
name: ABC
steps:
- template: ..\BuildEnv.yml
#REST CODE
That's it. You are done.
Reference : Template usage in Azure DevOps build - https://learn.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops
I have an java project which have gradle.properties file. Im extracting variables defined in gradle.properties as
##vso[task.setvariable variable=myVariable;]`my script to extract it from gradle.properties`
Then im using template from another repository that needs that variable but I can't use it within task, but when I try use it within - script: echo $variable as a step instead of task it is working.
When i try to use it within task it sees variable as $variable not a value.
Maybe there is a better way to extract variables to azure pipeline instead of using this approach?
Check the error message:
We get the error before the pipeline run the bash task, Since it cannot create the variable parampass, we get the parameters value is $(parampass) instead of the variable value.
Check this doc:
In a pipeline, template expression variables ${{ variables.var }} get processed at compile time, before runtime starts. Macro syntax variables $(var) get processed during runtime before a task runs. Runtime expressions $[variables.var] also get processed during runtime but were designed for use with conditions and expressions.
As a workaround:
pipeline.yml
pool:
vmImage: ubuntu-20.04
resources:
repositories:
- repository: common
type: git
name: organisation/repo-name
variables:
- name: parampass
value: xxx
stages:
- stage: "Build"
jobs:
- job: "Build"
steps:
- template: templatename.yml#common
parameters:
par1: ${{ variables.parampass}}
Result:
Probably you do not provide variable to your template
Example execution of template with provided parameter
- template: my/path/to/myTemplate.yml#MyAnotherRepositoryResourceName
parameters:
projectsToBeTested: $(someVariable)
And example template accepting parameters
steps:
- task: DotNetCoreCLI#2
displayName: 'Just testing'
inputs:
command: test
projects: ${{ parameters.projectsToBeTested}}
Please provide more information if it does not help.
Code looks like this:
pipeline.yml
pool:
vmImage: ubuntu-20.04
resources:
repositories:
- repository: common
type: git
name: organisation/repo-name
stages:
- stage: "Build"
jobs:
- job: "Build"
steps:
- bash: |
echo "##vso[task.setvariable variable=parampass]anything"
- template: templatename.yml#common
parameters:
par1: $(parampass)
templatename.yml
parameters:
- name: par1
steps:
- task: SonarCloudPrepare#1
displayName: SonarCloud analysis prepare
inputs:
SonarCloud: ${{ parameters.par1}}
organization: 'orgname'
scannerMode: 'Other'
extraProperties: |
# Additional properties that will be passed to the scanner,
# Put one key=value per line, example:
# sonar.exclusions=**/*.bin
sonar.projectKey= # same param pass case
sonar.projectName= # same param pass case
Generally, it does not matter if i do have parameters passed or if I'm using the template as if it were part of the pipeline code within. Output is always $(parampass) could not be found or smth
We have a collection of Azure DevOps pipeline templates that we re-use across multiple repositories. Therefore we wanted to have a file that contains variables for all of our templates.
The repo structure looks like this
template repo
├── template-1.yml
├── template-2.yml
└── variables.yml
project repo
├── ...
└── azure-pipelines.yml
The variables.yml looks like this
...
variables:
foo: bar
In template-1.yml we are importing the variables.yml as described in here
variables:
- template: variables.yml
In the azure-pipelines.yml we are using the template like this
resources:
repositories:
- repository: build-scripts
type: git
name: project-name/build-scripts
steps:
...
- template: template-1.yml#build-scripts
When we now try to run the pipeline, we get the following error message:
template-1.yml#build-scripts (Line: 10, Col: 1): Unexpected value 'variables'
The issue is because you used variable template at steps scope. And variables simply doesn't exists at that level. This should work for you:
resources:
repositories:
- repository: build-scripts
type: git
name: project-name/build-scripts
variables:
- template: template-1.yml#build-scripts
steps:
...
this is available to use at any place where variables are possible to use. So for instance you can use this in that way:
jobs:
- job: myJob
timeoutInMinutes: 10
variables:
- template: template-1.yml # Template reference
pool:
vmImage: 'ubuntu-16.04'
steps:
- script: echo My favorite vegetable is ${{ variables.favoriteVeggie }}.
If your template file only has variables, you can refer to Krzysztof Madej's answer.
If your template file has both variables and steps as shown below, it can only be used by extends.
# File: template-1.yml
variables: ...
steps: ...
Or you can write them in a stage, as shown below.
# File: template-1.yml
stages:
- stage: {stage}
variables: ...
jobs:
- job: {job}
steps: ...
Then insert it as a separate stage.
# azure-pipelines.yml
stages:
- stage: ...
- template: template-1.yml
Replying as I think there was something missed in the original explanations:
First:
If you are referencing variables in a template then you must ensure that you are extending that template when you call it. Calling a template that defines variables that is not extended will result in a failed pipeline execution.
template-1.yml#build-scripts (Line: 10, Col: 1): Unexpected value 'variables'
Second: When referencing variables in multiple templates do not use:
${{ variables.foo }} #even though this is seen in templates examples.
but rather use the normal variable syntax
$(foo) #this works
So for the originally reference example the following should work.
Variables.yml:
variables:
foo: bar
template-1.yml:
variables:
- template: variables.yml
....
steps:
...
- task: CmdLine#2
displayName: Display Variable
inputs:
script: |
echo $(foo)
azure-pipelines.yml:
resources:
repositories:
- repository: build-scripts
type: git
name: project-name/build-scripts
extends:
template: template-1.yml#build-scripts
template-2.yml
steps:
...
- task: CmdLine#2
displayName: Display Variable
inputs:
script: |
echo $(foo)
This approach can help someone, so I decided to post it here.
One more kind of "workaround" for that case, maybe a bit "dirty" since you need to specify the parameters explicitly each time you execute template, which is not a good idea if you have a lot of parameters to pass. (actually, there is a way, read below improved version) But, if you really want or you have not that much parameters, this should work:
the logic is: you have all your parameters in variables template, like templates/vars.yml:
variables:
- name: myVar1
value: MyValue1
- name: myVar2
value: MyValue2
and since you have everything you need in the variables, probably there is no need to importing variables into template itself, because template will be executed in pipeline, which will have your variables imported and you can substitute it explicitly, like in the example below:
templates/my-template-setup-env.yml's content (without variables inside it):
steps:
- script: |
echo "$(myVar3)"
my-azure-pipeline.yml's content (with importing variables template):
name: my-cute-CI-name
trigger:
- main
pool:
vmImage: ubuntu-18.04
variables:
- template: templates/vars.yml # importing your variables from templates
stages:
- stage: RunTheStage
displayName: 'Run first stage'
jobs:
- job: RunTheJob
displayName: 'Run your job'
steps:
- template: templates/my-template-setup-env.yml # your template
parameters:
myVar3: $(myVar1) # substitute value from vars.yml, so myVar1 will be used in templated and printed
Improved version
But, if you have unique naming of your params and variables across all pipelines and templates, you are in safe to not specify it explicitly during template usage, that will work as well:
edited and shortened version of my-azure-pipeline.yml (in case you have the same name of your variable and parameter in template):
variables:
- template: templates/vars.yml
...
steps:
- template: templates/my-template-setup-env.yml # env preparation
# parameters:
# myVar2: $(myVar2) # you don't need to pass any parameters explicitly to the template since you have same name of variable
templates/my-template-setup-env.yml then should be like this:
steps:
- script: |
echo "$(myVar2)" # not myVar3, since myVar3 is not in initial variables file templates/vars.yml
or you need to add remaining variables (myVar3 in our first case) into templates/vars.yml file as well.
I have a template that defines an array of branches for each environment, so that I can control which branches cause a deploy to a particular environment. For example, the only branch I want deployed to production is master, but for UAT I want release, hotfix, and master. I've set-up a parent template that calls downstream templates in an 'each' loop. Inside the deploy template, I want to compare the array of allowed branches against the current branch to determine whether to proceed or not. Here's part of the parent template that passes the branches arrays:
- template: pipeline-templates/environment-pipeline.yml
parameters:
envs:
- name: devtest
branches:
- 'refs/heads/master'
- 'refs/heads/develop'
- 'refs/heads/hotfix/'
- 'refs/heads/release/'
- name: nightly
branches:
- 'refs/heads/master'
- 'refs/heads/develop'
- 'refs/heads/hotfix/'
- 'refs/heads/release/'
- name: qa
branches:
- 'refs/heads/master'
- 'refs/heads/hotfix/'
- 'refs/heads/release/'
- name: prod
branches:
- 'refs/heads/master'
The environment-pipeline.yml below then calls the deploy template for each environment.
parameters:
- name: envs # collection of environments to create/deploy
type: object
default:
env:
name: ''
branches: []
stages:
- ${{ each env in parameters.envs }}:
- template: deploy.yml
parameters:
environmentName: ${{ env.name }}
onlyForBranches: ${{ env.branches }}
The above all works fine (note, I've deleted a lot of the YAML to simplify the example). The next stage is to use a condition with the array of branches in the deploy template. This is how I was hoping it would work, but doesn't.
parameters:
environmentName: ''
onlyForBranches: []
stages:
- stage: branch_gate
condition: and(succeeded(), in(variables['Build.SourceBranch'], ${{ parameters.onlyForBranches }}))
# then the deploy stages go here if above condition passes
This results in the following error:
Unable to convert from Array to String.
Is there some way to make this work, or should I use a different approach?
in is not the function to use. It takes n arguments and is true if the first argument equals any of the rest: in('B', 'A', 'B', 'C') => true
Try containsvalue instead:
condition: and(succeeded(), containsvalue(${{ parameters.onlyForBranches }}, variables['Build.SourceBranch']))
I agree with schummar on the use of containsValue, however the suggested approach did not work for me either.
Instead try this:
condition: and(succeeded(), ${{ containsValue(parameters.onlyForBranches, variables['Build.SourceBranch']) }})
I defined a YAML build-pipeline in azure:
variables:
test: '${{ variables.Environment }}'
pool:
vmImage: 'ubuntu-latest'
steps:
- script: |
echo $(test)
displayName: 'Show test'
I queue that pipeline with the Environment variable defined as 'abc':
I expect it to echo abc but instead abc is replaced with nothing - variables.Environment seems to be undefined.
Later on I want to load a different Variable Group depending on the Environment variable, which is why I do not echo $(Environment) directly in the script. It's just a simplified example.
I expect it to echo abc but instead abc is replaced with nothing -
variables.Environment seems to be undefined.
According to this document:
Runtime happens after template expansion. Template variables are processed at compile time, and are replaced before runtime starts. Template variables silently coalesce to empty strings when a replacement value isn't found.
So in your case echo $(test) print out nothing but empty string. Cause the queue variables are used for runtime. For this, you can consider using macro or runtime expression which is for runtime. Both test: $(Environment) and test: $[variables.Environment] work well on my side.
Later on I want to load a different Variable Group depending on the
Environment variable, which is why I do not echo $(Environment)
directly in the script. It's just a simplified example.
As I know, linking different variable groups depending on the dynamic Environment variable is not supported yet, here's one discussion about that topic. And this is one good workaround in that scenario.
Nowadays Azure Devops Service is rolling out the new feature runtime parameters, I think it can meet most of your requirements. It could be a better choice for you, use runtime parameters instead of not supported dynamic Environment variable.
My simple test about this option:
1.Content in yaml:
parameters:
- name: group
displayName: Group Name
type: string
default: TestGroup
values:
- TestGroup
- Group2
- Group3
- Group4
variables:
- group: ${{ parameters.group }}
steps:
- script: |
echo $(Name)
displayName: 'Show group name'
2.My variable group TestGroup:
3.Click run pipeline:
4.The pipeline runs well and the displays the variable defined in variable group:
Hope it helps :)
A variation of this problem:
TL;DR:
Group variables do not seem to be available at template compile time, e.g. when conditionally setting a template to run, based on a variable from that group - only variables explicitly set in the pipeline seem to be available.
Long version:
I have a set of variable groups; 'Group-NonProd', 'Group-Prod', etc. Each contains variable 'identifier', with differing values ("dev", "prod", for the below example).
I have a master pipeline, 'main-pipeline.yml', and in that are several stages, each corresponding to the value of 'identifier', e.g.:
- stage: NonProd
variables:
- group: 'Group-NonProd' ## includes the variable 'identifier' with value "nonprod"
jobs:
- template: nonprod.yml
parameters:
identifier: $(identifier)
- script: echo "environment is $(identifier)"
- stage: Prod
variables:
- group: 'Group-Prod' ## includes the variable 'identifier' with value "prod"
jobs:
- template: prod.yml
parameters:
identifier: $(identifier)
- script: echo "environment is $(identifier)"
However, when I run the pipeline, the parameters 'identifier' do not expand to the value in the group variable 'identifier' - it is blank - I see that when I try using the variable in the below conditional logic (the aim being to use this logic to determine which template to call - see commented out lines - and pass that parameter down to them):
steps:
- ${{ if eq(parameters.identifier, 'nonprod') }}:
# - template: nonprod.yml
- script: echo "Using nonprod template, environment is ${{ parameters.identifier }}"
- ${{ if not(eq(parameters.identifier, 'prod')) }}:
# - template: prod.yml
- script: echo "Using prod template, environment is ${{ parameters.identifier }}"
The above script always resorts to the 2nd condition, because the result is always "Using prod template, environment is " (blank).
Here's the odd thing though - if I explicitly set the variable 'identifier' at each stage, it does work!
e.g. this works:
- stage: NonProd
variables:
- group: 'Group-NonProd'
- name: identifier
value: nonprod
jobs:
- template: nonprod.yml
parameters:
identifier: $(identifier)
- script: echo "environment is ${{ parameters.identifier }}"