We are using a canary deployment job in Azure. The job is defined in a template. We want to set the "increments:" parameter at runtime to a single value (i.e. [ 50 ] rather than [ 25, 50 ]). The value is an output of a prior job. I've tried a few different approaches and receive various errors immediately in the screen where one tries to start the pipeline, before it actually starts.
Simplified yaml:
jobs:
- job: preValidationCanary
steps:
- task: PythonScript#0
name: setCanaryDeploymentVariables
script: |
# Do stuff to determine percentage
print("##vso[task.setvariable variable=%s;isOutput=true]%s" % ("deploy_canaryTrafficSplitPercentage_global", trafficPercentageInt ))
- deployment: deployCanary
dependsOn:
- preValidationCanary
variables:
deploy_canaryTrafficSplitPercentage: $[ dependencies.preValidationCanary.outputs['setCanaryDeploymentVariables.deploy_canaryTrafficSplitPercentage_global'] ]
strategy:
canary:
increments: $[deploy_canaryTrafficSplitPercentage]
I've tried passing just the number from the prior job, and then using this syntax:
increments: [ $[deploy_canaryTrafficSplitPercentage] ]
This results in the error: "While parsing a flow sequence, did not find expected ',' or ']'."
I've tried using the macro syntax:
increments: [ $(deploy_canaryTrafficSplitPercentage) ]
This results in the error: "Mapping values are not allowed in this context."
In case there was an issue expanding a number into the brackets, I tried passing a string from the prior job, using the runtime syntax:
- task: PythonScript#0
name: setCanaryDeploymentVariables
script: |
# Do stuff to determine percentage
print("##vso[task.setvariable variable=%s;isOutput=true]%s" % ("deploy_canaryTrafficSplitPercentage_global", str(trafficPercentageInt) ))
This also provided the error: "While parsing a flow sequence, did not find expected ',' or ']'."
So I tried this with macro syntax, which provided the error: "Mapping values are not allowed in this context."
I'll continue to beat my head against this, but if anyone either knows how this can be done, or knows that it cannot be done, please reply.
Related
I struggle with setting up a condition statement to run a specific task only on main and when there is a CI build.
While trying without success, I came across this statement which is an AND of two exactly the same statements:
condition: and(contains(variables['build.sourceBranch'], 'refs/heads/main'), contains(variables['build.sourceBranch'], 'refs/heads/main'))
When running it on my branch, I see this result:
Evaluating: and(contains(variables['build.sourceBranch'], 'refs/heads/main'), contains(variables['build.sourceBranch'], 'refs/heads/main'))
Expanded: and(contains('refs/heads/task/mybranch', 'refs/heads/main'), contains(variables['build.sourceBranch'], 'refs/heads/main'))
Result: False
Why is the second variable not expanded?
To reach my aim of that condition, I'd like to run something like this, but here the expansion does not work as well:
and(endsWith(variables['build.sourceBranch'], 'refs/heads/main'), endsWith(variables['build.reason'], 'CI'))
What am I doing wrong?
Thanks in advance
For often used tasks in azp I created an own repository with a yml file, I'll show you a subpart of that:
create-and-upload-docu.yml:
parameters:
- name: Documentation
type: string
default: ''
- name: Language
type: string
default: ''
- name: ArchiveBaseDir
type: string
default: ''
steps:
- script: |
ARCHIVERELPATH=${{parameters.Documentation}}-${{parameters.Language}}.zip
ARCHIVEDIR=$(echo -n ${{parameters.ArchiveBaseDir}} | sed -e 's#/$##')/${{parameters.Documentation}}/${{parameters.Language}}
echo "##vso[task.setvariable variable=archiveRelPath;isOutput=true]$ARCHIVERELPATH"
echo "##vso[task.setvariable variable=archiveDir;isOutput=true]$ARCHIVEDIR"
name: ${{parameters.Documentation}}_${{parameters.Language}}_params
- task: DeleteFiles#1
inputs:
Contents: '$(Build.ArtifactStagingDirectory)/$(${{parameters.Documentation}}_${{parameters.Language}}_params.archiveRelPath)'
The relevant part is: the "script" has the name which is unique in a job - so I can use this kind of expansion for setting variables within the template:
$(${{parameters.Documentation}}_${{parameters.Language}}_params.archiveRelPath)
This works fine as long as I had called the template with fixed values, like
- template: create-and-upload-docu.yml#templates
parameters:
Documentation: 'adocuvalue'
Language: 'en_US'
ArchiveBaseDir: '$(Build.ArtifactStagingDirectory)/build/'
But now I want to use a matrix to have a few documentations with a few languages:
jobs:
- job: Documentation_CI
displayName: "Docu CI"
timeoutInMinutes: 30
strategy:
matrix:
main_en_US:
Documentation: main
Language: en_US
main_de_AT:
Documentation: main
Language: de_AT
steps:
- checkout: self
- template: create-and-upload-docu.yml#templates
parameters:
Documentation: ${{variables.Documentation}}
Language: ${{variables.Language}}
ArchiveBaseDir: '$(Ws)/build/'
But at the time where ${{}} expressions are expanded, it seems that the matrix variables are not already set; this means that the template script part is called __params and the pipeline has the following error
Publishing build artifacts failed with an error: Input required: ArtifactName
Is there a somewhat simple way to achive what I want (being able to set some variables within templates with a unique naming schema):
can I somehow use ${{ expressions but need a different naming to get to the hard-coded matrix style variables
can I workaround my problem any simple way?
Additional Info: we run a Azure 2020 on prem.
Is there a somewhat simple way to achive what I want (being able to set some variables within templates with a unique naming schema):
Sorry for any inconvenience.
I am afraid there is no such way to resolve this at this moment.
Just as you test, the syntax ${{}} is parsed at compile time. We could not get the value when we use it as name or display name in the task, since it will be parsed at compile time. But the matrix variables have not been set during compilation. That the reason why we get the value _params.
There is a feature request about this. And you could add your request for this feature on our UserVoice site (https://developercommunity.visualstudio.com/content/idea/post.html?space=21 ), which is our main forum for product suggestions:
I am really struggling with some variables which I have in my variable group named 'android-pipeline'.
Inside this variable group, I have some variables with values.
But when I am running the pipeline it cannot read the values inside my variable group. :(
Example:
Inside the variable group, I have a variable called
$(key.alias)
I am trying to get this value which is behind the variable, see my code below.
I think something is wrong with the syntax (or the way I am using it), but I cannot find the right syntax for using my $(key.alias) variable.
Also, inside the variable group I have made sure that All pipelines have access to this Variable group.
Can someone, please tell me how I can get the value behind the $(key.alias) variable and use this in a task? I tried to follow many guides, but none are clear enough for me or not working
variables:
group: android-pipeline
buildConfiguration: 'Release'
stages:
stage: Publish
dependsOn: Build
displayName: Sign Apps
jobs:
- task: AndroidSigning#3
displayName: Android App signing
inputs:
apkFiles: '**/*.apk'
apksignerKeystoreFile: '$(androidKeyStore)'
apksignerKeystorePassword: '********'
apksignerKeystoreAlias: '$(key.alias)'
apksignerKeyPassword: '*******'
apksignerArguments: --out $(outputDirectory)/app.release.apk
zipalign: true
Since you're mixing groups and inline variables, you may need to change this from a mapping to a sequence, as in:
variables:
- group: android-pipeline
- name: buildConfiguration
value: Release
Normally when you declare variables, you can do them like a mapping, or hashtable, of name/value pairs:
variables
var1: value1 # note there's no dash at the beginning of the line
var2: value2
var3: value3
# etc
When you want to use a group, you have to change your syntax a little, so that the parser doesn't think you want to create a variable named "group" - you turn it into a sequence, or array:
variables
- group: groupname1 # note there's a dash at the beginning of the line
- group: groupname2
# etc
Here's the final wrinkle - once you've gone from the first format to the second (mapping to sequence), you have to declare new variables that are local to your file in the "sequence" style:
variables
- group: groupname1 # note there's a dash at the beginning of the line
- name: varname1
value: value1
- name: varname2
value: value2
# etc
You reference the variable further down in your pipeline the same way, with $(varname1) syntax.
If you're having problems with this, I recommend a couple of things (actually, 3):
Use script or pwsh tasks to echo or Write-Host everything you want to see but aren't, as in "pwsh: Write-Host "My var should be $(varname1)"
Turn on system diagnostics when you run the pipeline and see if the output has any useful details
Edit the pipeline through the portal - Pipelines - select your pipeline -> Edit. Then, from the ellipsis menu in the top right of the page, select "Download full YAML" - this will give download what the compiler would create. Now, it won't give you variable values, but what it can do is give you clues as to possible format or declaration errors.
I need to read settings from a Json file for the execution of a pipeline. I use a Powershell task for this. In the next step I want to use those properties as input. However when I do that the variable does not get rendered. How can I achieve this?
- task: PowerShell#2
displayName: 'Main -> Read App Setting'
inputs:
targetType: 'inline'
script: |
$settings = Get-Content "$(Build.Repository.LocalPath)\settings\appsettings.json" | ConvertFrom-Json
Write-Host "##vso[task.setvariable variable=type]$($settings.pipeline.type)"
- template: ./.lib/create_resources.yml
parameters:
type: $(type)
However when I do that the variable does not get rendered. How can I
achieve this?
You can't pass parameter to template in this way. The variable did not get rendered because we can't use runtime variables in template parameters. It's the direct cause why it didn't get rendered.
This is by design of Azure Devops Service, check Process the pipeline:
To turn a pipeline into a run, Azure Pipelines goes through several steps in this order:
1.First, expand templates and evaluate template expressions.
2.Next, evaluate dependencies at the stage level to pick the first stage(s) to run.
3.For each stage selected to run, two things happen:
All resources used in all jobs are gathered up and validated for authorization to run.
Evaluate dependencies at the job level to pick the first job(s) to run.
4.For each job selected to run, expand multi-configs (strategy: matrix or strategy: parallel in YAML) into multiple runtime jobs.
5.For each runtime job, evaluate conditions to decide whether that job is eligible to run.
6.Request an agent for each eligible runtime job.
So your variable comes after the task is executed(step6) while the parameter is evaluated at the very first of this process(step1 of pipeline run process). See:
This ordering helps answer a common question: why can't I use certain variables in my template parameters? Step 1, template expansion, operates solely on the text of the YAML document. Runtime variables don't exist during that step. After step 1, template parameters have been completely resolved and no longer exist.
In addition: Shayki Abramczyk's comment is right though it's not the main cause of your issue, you should use the Write-Host "##vso[task.setvariable variable=type]$($settings.pipeline.type)" so that the value of $(type) would be $settings.pipeline.type instead of $settings.
To use variable in next step:
We can check this document:
In order to use a variable as a task input, you must make the variable an output variable, and you must give the producing task a reference name.
More details about how to use that in yaml pipeline please check my another issue.
As Lance described the problem is that template parameters are expanded first, so setting the parameters using a variable does not work when used as input for a tasks in the template. However I found out that in a PowerShell task in the template, the exact same parameters are showing the latest value.
Additionally I wanted to run templates sequential and have the output of template 1 be used for template 2 etc, without template1 knowing about template2, no dependencies. This way individual templates can still be run manually from DevOps.
So I used the following setup, which fulfills both requirements. It passes the output from template 1 to template 2, without hardcoding those variables in the templates itself & it shows template 1 passing an incoming parameter as input for a seconds step by using PowerShell to convert the parameter to a variable first. This way one could add a PowerShell script to each template and convert all parameters to variables and use those variables in the template instead. One extra step, but until MS includes dynamic parameters, quite acceptable I think.
Main:
steps:
- task: PowerShell#2
displayName: 'Main -> Read App Setting'
inputs:
targetType: 'inline'
script: |
$settings = Get-Content "$(Build.Repository.LocalPath)\settings\appsettings.json" | ConvertFrom-Json
Write-Host "##vso[task.setvariable variable=type]$($settings.pipeline.type)"
- template: ./template1.yml
parameters:
input: $(type)
outputVariable: template1Output
- template: ./template2.yml
parameters:
input: $(template1Output)
Template1:
parameters:
- name: input
type: string
- name: outputVariable
type: string
steps:
- task: PowerShell#2
displayName: Task in Template 1
inputs:
targetType: inline
script: |
$input = "${{parameters.input}}"
$output = $input + " some extra value"
Write-Host "##vso[task.setvariable variable=input]${{parameters.input}}"
Write-Host "##vso[task.setvariable variable=${{parameters.outputVariable}}]$output"
- template: ./template2.yml
parameters:
input: $(input)
Template2:
parameters:
- name: input
type: string
steps:
- task: PowerShell#2
displayName: Task in Template 2
inputs:
targetType: 'inline'
script: |
echo "${{parameters.input}}"
The consistency of Variable support & the syntax vary wildly in Azure DevOps YAML.
Case in point:
trigger:
- master
# Variable Group has $(testCategory1) with value
# 'TestCategory=bvttestonly | TestCategory=logintest'
variables:
- group: DYNAMIC_VG
jobs:
- job:
pool: 'MyPool' #Has about 10+ self hosted agents
strategy:
parallel: $[ variables['noOfVMsDynamic']]
variables:
indyx: '$(testCategories$(System.JobPositionInPhase))'
indyx2: $[ variables['indyx'] ]
testCategories: $[ variables[ 'indyx2' ] ]
steps:
- script: |
echo "indyx2 - $(indyx2)"
echo "testCategories $(testCategories)"
displayName: 'Display Test Categories'
The step prints:
"indyx2 - $(testCategories1)"
"testCategories $(testCategories1)"
I need to print the value of $(testCategories1) defined in the Variable Group:
'TestCategory=bvttestonly | TestCategory=logintest'
This may work to you:
variables
indyx: $[ variables[format('{0}{1}', 'testCategories', variables['System.JobPositionInPhase'])] ]
It worked for me, in a slightly different situation which also required some dynamic variable names.
Howto: Dynamically resolve a nested variable in Azure DevOps YAML
That because the value of nested variables (like $(testCategories$(System.JobPositionInPhase))) are not yet supported in the build pipelines at this moment.
That the reason why you always get the value $(testCategories1) rather than the real value of variable testCategories1.
I encountered this issue many times in my past posts and we do not have a perfect solution before Azure Devops supports this feature.
For the convenience of testing, I simplified your yaml like following:
jobs:
- job: ExecCRJob
timeoutInMinutes: 800
pool:
name: MyPrivateAgent
displayName: 'Execute CR'
variables:
testCategories1: 123456
testCategoriesSubscripted: $(testCategories$(System.JobPositionInPhase))
strategy:
parallel: $[variables['noOfVMs']]
steps:
- template: execute-cr.yml
parameters:
testCategories: $(testCategoriesSubscripted)
The execute-cr.yml:
steps:
- script: echo ${{ parameters.testCategories }}
We always get the $(testCategories1)NOT the value of it.
If I change the $(testCategories$(System.JobPositionInPhase)) to $(testCategories1), everything work fine.
Since nested variables are not yet supported, As workaround, we need to expand the nested variables for each value of testCategories, like:
- job: B
condition: and(succeeded(), eq(dependencies.A.outputs['printvar.skipsubsequent'], 'Value1'))
dependsOn: A
steps:
- script: echo hello from B
Check the Expressions Dependencies for some more details.
Hope this helps.
If I'm understanding your issue correctly, the problem is that the pipeline evaluates all variables at the runtime of the job. The solution in this scenario is to split your tasks into separate jobs with dependencies.
Have a look at my answer in this post and let me know if it's what you're after : YAML pipeline - Set variable and use in expression for template
I manage to get dynamic name resolution by using get-item to read the corresponding environment variable, allowing construction of the name of the variable and then getting the value.
In our case we save the name of an autogenerated branch into a variable group and each repository will have its own variable.
$branchVarName = "$(Build.Repository.Name).BranchName".replace(".","_")
$branchName = (get-item -Path Env:$branchVarName).value
write-host "/$(System.TeamProject)/_apis/build/builds?&repositoryId=$(Build.Repository.ID)&repositoryType=TFSGit&branchName=refs/heads/$branchName&api-version=6.0"
Notice in the second line that I reference the variable content using .value because get-item returns a name/value key pair.
This extraction has to be done in script but could be exposed as an output variable if needed in another context.