I have template
dockerbuild.yml
steps:
- task: Bash#3
displayName: Build an image to container registry
inputs:
script : echo $(PATH)
then a variable file
var.build.yml
- name: PATH
value: 'docker/path'
- name: PATH1
value: 'docker/oldpath'
- name: PATH2
value: 'docker/newpath'
azurepipeline1.yml
resources:
repositories:
- repository: templates
type: git
name: components/pipeline_templates
trigger:
- none
pool:
name: PoolA
variables:
- template: variabletemplates/var.build.yml#templates
jobs:
steps:
- template: CI-CD/dockerbuild.yml#templates # Template reference
parameters:
PATH: ${{ variables.PATH }}
- template: CI-CD/dockerbuild.yml#templates # Template reference
parameters:
PATH: ${{ variables.PATH1 }}
- template: CI-CD/dockerbuild.yml#templates # Template reference
parameters:
PATH: ${{ variables.PATH2 }}
The above code works
But if I change the parameter to
parameters:
PATH: ${{ variables.PATH1 }}
OR
parameters:
PATH: ${{ variables.PATH2 }}
In the output it shows
Output:
docker/path
$(PATH1)
$(PATH2)
I need to reuse the template with many times within the pipeline with different paths. Please help in resolving the issue
Your template needs a parameters block.
parameters:
- name: PATH
type: string
steps:
- task: Bash#3
displayName: Build an image to container registry
inputs:
script : echo ${{ parameters.PATH }}
Related
I want to get the environments and the associated variableGroup - loop thru the environments array in Stages.yml and create a stage per environment... insert correct variableGroup per stage... use variablesGroup values to perform jobs in jobs.yml - each variable group contains the same vars.. with different values.
``
I have a main.yml file in which I reference my environments and the associated group variable
main.yaml :
name: $(BuildID)
trigger: none
pool:
vmImage: 'leetchi-agentspool-windows'
variables:
- template: vars/variables-api.yaml
extends:
template: deploy-stages.yaml
parameters:
environements:
- environement: 'DV1'
variableGroupName: tf-dv1
- environement: 'IN1'
variableGroupName: tf-in1
A stage template which is supposed to apply with the environment parameters and the group variable.
stage.yaml :
---
parameters:
- name: environements
type: object
default: []
stages:
- ${{ each environement in parameters.environements }}:
- stage: 'Deploy to ${{ environement.environements }}'
variables:
- group: ${{ environement.variableGroup }}
jobs:
- template: template-terraform-deployment.yml
parameters:
environement: ${{environement.environements}}
project: $(project)
definition: $(def)
project1: $(project1)
definition1: $(def1)
stateKey: ${{environement.environement}}$(tfkey)
TerraformDirectory: $(TfDirectory)
azureSubscription: $(service_connection_name)
TfStateResourceGroupName: $(tf_resource_group)
TfStateStorageAccountName: $(tf_storage_account)
TStateContainerName: $(tf_container)
commandOptions: -out=$(System.DefaultWorkingDirectory)/terraform.tfplan -detailed-exitcode
I start in YAML. Could you help me please? Thank you very much in advance
You can modify your main.yaml as below:
trigger: none
pool:
vmImage: 'leetchi-agentspool-windows'
variables:
- template: vars/variables-api.yaml
extends:
template: deploy-stages.yaml
parameters:
- name: environment
values:
- DV1
- IN1
- name: variableGroupName
values:
- tf-dv1
- tf-in1
stage.yaml :
stages:
- stage: 'Deploy to ${{ parameters.environment }}'
variables:
- group: ${{ parameters.variableGroupName}}
jobs:
- template: template-terraform-deployment.yml
parameters:
environment: ${{ parameters.environment }}
project: $(project)
definition: $(def)
project1: $(project1)
definition1: $(def1)
stateKey: ${{ parameters.environment }} + $(tfkey)
TerraformDirectory: $(TfDirectory)
azureSubscription: $(service_connection_name)
TfStateResourceGroupName: $(tf_resource_group)
TfStateStorageAccountName: $(tf_storage_account)
TStateContainerName: $(tf_container)
commandOptions: -out=$(System.DefaultWorkingDirectory)/terraform.tfplan -detailed-exitcode
This is part of my Yml file
I need to re-call this template if it failed.it should be rerun again. After a few seconds, ideally.I am new to yml files
I tried to use retryCountOnTaskFailure but it should be under task but the template calling different hierarchy
https://learn.microsoft.com/en-us/azure/devops/release-notes/2021/sprint-195-update#automatic-retries-for-a-task
- template: ${{variables['System.DefaultWorkingDirectory']}}
parameters:
Test: ${{ parameters.isTestRelease }}
Environment: ${{ parameters.deploymentTarget }}
Component: '${{ parameters.component }}'
The retryCountOnTaskFailure feature is applied to individual tasks within a pipeline. Templates aren't tasks, they act more like an include that expands the contents of the template into your pipeline.
# pipeline.yml
trigger: none
parameters:
- name: isTestRelease
type: boolean
default: false
- name: deploymentTarget
type: string
default: DEV
values:
- DEV
- QA
- UAT
- PROD
- name: component
type: string
default: 'x'
stages:
- template: my-template.yml
parameters:
Test: ${{ parameters.isTestRelease }}
Environment: ${{ parameters.deploymentTarget }}
Component: ${{ parameters.component }}
And within the template, you'd want to add the retry logic to specific tasks:
# my-template.yml
parameters:
- name: isTestRelease
type: boolean
- name: component
type: string
stages:
- stage: Test
jobs:
- job: 1
steps:
- task: ...
- task: ...
- task: ...
retryCountOnFailure: 2
- task: ...
It's also located under "Control Options" when the "Command Line" task is added:
I am facing an issue I could not resolve.
regarding this documentation it seams to be possible to set a variable as output of a job in the first stage, then use it as input to set a job variable in second stage.
The pbm I'm facing, is that my stages are parametrised, so their name are not constant.
here my sample code :
name: mypipeline
parameters:
- name: Application
displayName: Application to deploy the front end using front door
type: string
- name: stage_vars
displayName: stage VARS
type: object
default:
- stage_1
- stage_2
- none
trigger: none
stages:
- stage: Build
displayName: ${{ parameters.Application }} - Build.
pool:
name: Azure Pipelines
vmImage: windows-2019
jobs:
- template: ../templates/job_BuildAndPublish.yml
parameters:
BuildId: $(Build.BuildId)
importTerratest: false
importARM: true
- ${{ each stageregion in parameters.stage_vars }}:
- ${{ if ne(stageregion, 'none') }}:
- template: ../templates/isolated_web/tf_plan_stage.yml
parameters:
BuildId: $(system.BuildId)
Predecessors: 'Build'
- template: ../templates/isolated_web/tf_apply_stage.yml
parameters:
Predecessors: '${{stageregion}}_prepare'
build template is not usefull, but the ones from tf_plan_stage and tf_apply_stage are below :
tf_plan_stage.yml :
parameters:
- name: BuildId
type: string
default: $(system.BuildId)
- name: stage
type: string
default: Deploy_prepare
- name: Predecessors
type: string
default: none
stages:
- stage: ${{ parameters.stage }}
? ${{ if and(ne(parameters.Predecessors, 'previous'), ne(parameters.Predecessors, 'none') ) }}
: dependsOn: ${{parameters.Predecessors}}
displayName: ${{ parameters.TF_VAR_STAGE }} Terraform plan & publish artifact for ${{ parameters.TF_VAR_APPLICATION }}
pool:
name: Azure Pipelines
vmImage: windows-2019
jobs:
- deployment: PreDeployTerraform
displayName: ${{ parameters.TF_VAR_STAGE }} Terraform plan & publish artifact
environment: PLAN_${{ parameters.TF_VAR_ENVIRONMENT }}
timeoutInMinutes: 480
strategy:
runOnce:
deploy:
steps:
- checkout: none
# set output planAttempt output available for later Apply job
- powershell: |
echo "jobAttempt is $(System.JobAttempt)"
echo "##vso[task.setvariable variable=planAttempt;isOutput=true;]$(System.JobAttempt)"
name: setVarJobAttempt
- powershell: |
echo "variable setVarJobAttempt.planAttempt value is : $(setVarJobAttempt.planAttempt)"
name: getplanAttemptVar
tf_apply_stage.yml :
parameters:
- name: stage
type: string
default: Deploy_prepare
- name: Predecessors
type: string
default: none
stages:
- stage: ${{ parameters.stage }}
? ${{ if and(ne(parameters.Predecessors, 'previous'), ne(parameters.Predecessors, 'none') ) }}
: dependsOn: ${{parameters.Predecessors}}
displayName: ${{ parameters.TF_VAR_STAGE }} download artifact & Terraform apply changes to ${{ parameters.TF_VAR_APPLICATION }}
pool:
name: Azure Pipelines
vmImage: windows-2019
jobs:
- deployment: DeployTerraform
displayName: ${{ parameters.TF_VAR_STAGE }} download artifact & Terraform apply changes
variables:
# this one fails in pipeline : "[not recognise"
planJobAttempt: $[ stageDependencies.[parameters.Predecessors].PreDeployTerraform.outputs['setVarJobAttempt.planAttempt'] ]
# this one runs with no result in pipeline : planJobAttempt is the string "stageDependencies['parameters.Predecessors'].PreDeployTerraform.outputs['setVarJobAttempt.planAttempt']"
# planJobAttempt: $[ stageDependencies['parameters.Predecessors'].PreDeployTerraform.outputs['setVarJobAttempt.planAttempt'] ]
environment: ${{ parameters.TF_VAR_ENVIRONMENT }}
timeoutInMinutes: 480
strategy:
runOnce:
deploy:
steps:
- checkout: none
- powershell: |
echo "jobAttempts outputs are : $[ stageDependencies.[parameters.Predecessors].PreDeployTerraform.outputs ]"
- powershell: |
echo "jobAttempt is $[ stageDependencies['parameters.Predecessors'].PreDeployTerraform.outputs['setVarJobAttempt.planAttempt'] ]"
- powershell: |
echo "plan file for terraform is : $(pipeline_artifact_folder_download)/$(planJobAttempt)_${{ parameters.Predecessors }}/$(artefact_terraform_plan)_${{ parameters.TF_VAR_STAGE }}"
I tried differents things to get my system.jobAttempts from tf_plan_stage stage to tf_apply_stage stage, but without any success, as the variable in the last stage (tf_apply) seams unable to find the value from a "parametrised" stage name.
is there a way for that ?
thank-you for any answers.
Okay,
Once again, I will answer my own question :D
I've finallt find the way to use the parameters.Predecessors value in my variable declaration.
And, second, the Microsoft documentation seams not to be updated.
The tf_apply_stage.yml should be like that below ;
parameters:
- name: stage
type: string
default: Deploy_prepare
- name: Predecessors
type: string
default: none
stages:
- stage: ${{ parameters.stage }}
? ${{ if and(ne(parameters.Predecessors, 'previous'), ne(parameters.Predecessors, 'none') ) }}
: dependsOn: ${{parameters.Predecessors}}
displayName: ${{ parameters.TF_VAR_STAGE }} download artifact & Terraform apply changes to ${{ parameters.TF_VAR_APPLICATION }}
pool:
name: Azure Pipelines
vmImage: windows-2019
jobs:
- deployment: DeployTerraform
displayName: ${{ parameters.TF_VAR_STAGE }} download artifact & Terraform apply changes
variables:
*# this one give he good value for Variable*
planJobAttempt: $[ stageDependencies.${{ parameters.Predecessors }}.PreDeployTerraform.outputs['PreDeployTerraform.setVarJobAttempt.planAttempt'] ]
environment: ${{ parameters.TF_VAR_ENVIRONMENT }}
timeoutInMinutes: 480
strategy:
runOnce:
deploy:
steps:
- checkout: none
- powershell: |
echo "plan file for terraform is : $(pipeline_artifact_folder_download)/$(planJobAttempt)_${{ parameters.Predecessors }}/$(artefact_terraform_plan)_${{ parameters.TF_VAR_STAGE }}"
I put in BOLD the main changes above :
planJobAttempt: $[ stageDependencies.${{ parameters.Predecessors
}}.PreDeployTerraform.outputs['PreDeployTerraform.setVarJobAttempt.planAttempt']
]
The Job name have to be set twice in variable declaration !
Thanks for those who read my post ;)
And many thanks to this post from #Kay07949 who gives the good answers ;)
I'm trying to create a multi-stage pipeline which has Variable Groups defined for each stage of the pipeline. The goal is to pass values from the group as parameters to a sub template. It seems the value of the Group, defined at the stage, is not getting passed in to the sub template. It overrides the "DEFAULTVALUE" with an empty string.
pipeline.yml
trigger:
- none
pool:
name: 'Azure Pipelines'
vmImage: windows-latest
stages:
- stage: DEV
variables:
- group: my-group-dev
jobs:
- template: sub-template.yml
parameters:
env: 'dev'
subscriptionName: '$(SubscriptionName)' # This reference from the variable group doesn't get passed in
subscriptionId: '$(SubscriptionId)'
- stage: TEST
variables:
- group: my-group-test
jobs:
- template: sub-template.yml
parameters:
env: 'test'
subscriptionName: '$(SubscriptionName)' # This reference from the variable group doesn't get passed in
subscriptionId: '$(SubscriptionId)'
sub-template.yml
parameters:
env: 'DEFAULTVALUE'
subscriptionName: 'DEFAULTVALUE'
subscriptionId: 'DEFAULTVALUE'
jobs:
- deployment: ResourceDeployment
displayName: Deploy Resources ${{ parameters.env }}
environment: ${{ parameters.env }}
strategy:
runOnce:
deploy:
steps:
- task: AzureFileCopy#4
displayName: 'Upload ARM Templates'
inputs:
sourcePath: '$(Pipeline.Workspace)/templates'
azureSubscription: '${{ parameters.subscriptionName }}'
destination: 'azureBlob'
storage: 'my-storage-account'
containerName: 'arm'
name: AzureFileCopy
- task: AzureResourceManagerTemplateDeployment#3
inputs:
deploymentScope: 'Resource Group'
azureResourceManagerConnection: '${{ parameters.subscriptionName }}'
subscriptionId: '${{ parameters.subscriptionId }}'
action: 'Create Or Update Resource Group'
resourceGroupName: 'my-resource-group'
location: 'eastus2'
templateLocation: 'URL of the file'
csmFileLink: '$(AzureFileCopy.StorageContainerUri)${{ parameters.env }}/templates/main.json$(AzureFileCopy.StorageContainerSasToken)'
I have also tried adding the variable group within the sub-template but that also doesn't parse correctly...
Does anyone know if this is possible?
It is a known issue that the service connection endpoint cannot be referenced in variable groups defined under stage level.
You can fix this issue by below workarounds:
1, Define the variable groups in the global level instead of stage level See below:
trigger:
- none
pool:
name: 'Azure Pipelines'
vmImage: windows-latest
# define the variable group under global level.
variables:
- group: my-group-dev
- group: my-group-test
stages:
- stage: DEV
jobs:
- template: sub-template.yml
parameters:
env: 'dev'
subscriptionName: '$(SubscriptionName)' # This reference from the variable group doesn't get passed in
subscriptionId: '$(SubscriptionId)'
- stage: TEST
jobs:
- template: sub-template.yml
parameters:
env: 'test'
subscriptionName: '$(SubscriptionName)' # This reference from the variable group doesn't get passed in
subscriptionId: '$(SubscriptionId)'
2, Link the variable groups on the UI page.
On the yaml pipeline edit page--> Click the 3dots-->Triggers-->Variables Tab-->Link Variable group
Please see below threads for more information.
Using a variable for the service connection
Azure subscription endpoint ID cannot be provided through a variable in build definition YAML file
You need to pass variable group name to template and on template level apply it on variables scope like this:
parameters:
env: 'DEFAULTVALUE'
subscriptionName: 'DEFAULTVALUE'
subscriptionId: 'DEFAULTVALUE'
variableGroupName: 'DEFAULTVALUE`
jobs:
- deployment: ResourceDeployment
displayName: Deploy Resources ${{ parameters.env }}
environment: ${{ parameters.env }}
variables:
- group: {{ paramaters.variableGroupName }}
strategy:
runOnce:
deploy:
steps:
- task: AzureFileCopy#4
displayName: 'Upload ARM Templates'
inputs:
sourcePath: '$(Pipeline.Workspace)/templates'
azureSubscription: '${{ parameters.subscriptionName }}'
destination: 'azureBlob'
storage: 'my-storage-account'
containerName: 'arm'
name: AzureFileCopy
- task: AzureResourceManagerTemplateDeployment#3
inputs:
deploymentScope: 'Resource Group'
azureResourceManagerConnection: '${{ parameters.subscriptionName }}'
subscriptionId: '${{ parameters.subscriptionId }}'
action: 'Create Or Update Resource Group'
resourceGroupName: 'my-resource-group'
location: 'eastus2'
templateLocation: 'URL of the file'
csmFileLink: '$(AzureFileCopy.StorageContainerUri)${{ parameters.env }}/templates/main.json$(AzureFileCopy.StorageContainerSasToken)'
This is similar to this topic:
For security reasons, we only allow you to pass information into templated code via explicit parameters.
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops
The means the author of the pipeline using your template needs to commit changes where they explicitly pass the needed info into your template code.
There are some exceptions to this, where the variable is statically defined in the same file or at pipeline compile time, but generally speaking, it’s probably better to use parameters for everything that does not involve system-defined read-only dynamic variable and custom-defined dynamic output variables.
I have to variables groups
QA
PROD
both has isProd variable
and then my template is
parameters:
variableGroupName: 'QA'
jobs:
- job: ResourceDeployment
variables:
- group: ${{ parameters.variableGroupName }}
steps:
- script: echo '$(isProd)'
and main file is
trigger: none
pr: none
stages:
- stage: QA
jobs:
- template: sub-template.yml
parameters:
variableGroupName: 'QA'
- stage: PROD
jobs:
- template: sub-template.yml
parameters:
variableGroupName: 'PROD'
and I got:
and
I have a Azure DevOps YAML Pipeline to execute a Terraform deployment using the Terraform by MS DevLabs extension and an Azure Resource Manager service connection.
The last working state was using a pipeline template yaml file however I had to configure a parameter within the template and call the variable using the template expression syntax.
...
...
stages:
- stage: Plan
displayName: Terrafom Plan
jobs:
- job: DEV PLAN
displayName: Plan (DEV)
pool:
vmImage: "ubuntu-latest"
variables:
az_service_connection: "MyServiceConnection"
tf_environment: "DEV"
tf_state_rg: "DEV"
tz_state_location: "canadacentral"
tf_state_stgacct_name: "mystorageaccuontname1231231"
tf_state_container_name: "tfstate"
steps:
- template: templates/terraform-plan.yml
parameters:
az_service_connection: ${{ variables.az_service_connection }}
...
...
steps:
- task: terraformInstaller#0
displayName: "Install Terraform $(tf_version)"
inputs:
terraformVersion: $(tf_version)
- task: TerraformTaskV1#0
displayName: "Run > terraform init"
inputs:
command: "init"
commandOptions: "-input=false"
backendServiceArm: ${{ parameters.az_service_connection }}
...
...
I believe the reason why this works is because the template expression syntax ${{ variables.varname}} evaluates at compile time vs. runtime. If I didn't do it this way, i'd either get $(az_service_connection) passed into the backendServiceArm input or an empty value.
With the introduction of variable groups, i'm now facing similar behavior. I expect that the variable group evaluates after the template expression variable which causes ${{ variables.az_service_connection }} to have an empty value. I am unsure how to get this working.
How can I use variable groups with a pipeline template that uses a service connection?
I used $() syntax to pass arm connection to template:
Template file:
parameters:
- name: 'instances'
type: object
default: {}
- name: 'server'
type: string
default: ''
- name: 'armConnection'
type: string
default: ''
steps:
- task: TerraformTaskV1#0
inputs:
provider: 'azurerm'
command: 'init'
backendServiceArm: '${{ parameters.armConnection }}'
backendAzureRmResourceGroupName: 'TheCodeManual'
backendAzureRmStorageAccountName: 'thecodemanual'
backendAzureRmContainerName: 'infra'
backendAzureRmKey: 'some-terrform'
- ${{ each instance in parameters.instances }}:
- script: echo ${{ parameters.server }}:${{ instance }}
Main file:
trigger:
branches:
include:
- master
paths:
include:
- stackoverflow/09-array-parameter-for-template/*
# no PR triggers
pr: none
pool:
vmImage: 'ubuntu-latest'
variables:
- group: my-variable-group
- name: my-passed-variable
value: $[variables.myhello] # uses runtime expression
steps:
- template: template.yaml
parameters:
instances:
- test1
- test2
server: $(myhello)
armConnection: $(armConnection)
Note: Group my-variable-group contains armConnection variable