I'm currently trying to setup my complete build/release pipeline with yaml files.
First I tried with different stages (dev/staging/prod) and it worked.
Now I wanted to add an approval that the deploy doesn't not happen automatically on each system.
Therefore I added an environment in the TFS with an approval check.
But when I try to setup the yaml file I always get an error.
I don't know how to setup this properly.
This is my main yaml file called release-pipeline.yaml
trigger:
- master
pool:
name: POOL
stages:
- stage: BuildSolution
jobs:
- job: BuildSolutionJob
displayName: Build
workspace:
clean: all
steps:
- template: yaml/BuildSolution.yml
- template: yaml/CopyFiles.yml
- template: yaml/PublishArtifact.yml
- stage: DeployOn_STAGING_System
dependsOn: BuildSolution
jobs:
- job: Deploy_STAGING
- template: yaml/Deploy.yml
parameters:
Environment: 'SITE'
Staging: 1
- stage: Deploy_DEV_System
dependsOn: BuildSolution
jobs:
- deployment: Deploy_DEV
environment: ENVCHECK_DEV
strategy:
runOnce:
deploy:
steps:
- template: yaml/Deploy.yml
parameters:
Environment: 'SITE'
ViewDeploy: 1
This is my Deploy.yml file which i want to execute (only some snips):
parameters:
- name: Environment
type: string
- name: ProdSystem
type: number
default: 0
- name: Staging
type: number
default: 0
- name: ViewDeploy
type: number
default: 0
jobs:
- job:
variables:
artifactName: $[stageDependencies.BuildSolution.BuildSolutionJob.outputs['SetVariables.artifactName']]
version: $[stageDependencies.BuildSolution.BuildSolutionJob.outputs['SetVariables.version']]
steps:
- task: PowerShell#2
displayName: Display given parameters
inputs:
targetType: inline
script: >
Write-Host "ArtifaceName: " $(artifactName)
Write-Host "Environment for Deploy: " ${{ parameters.Environment }}
Write-Host "##vso[task.setvariable variable=isStaging]${{ parameters.Staging }}"
failOnStderr: true
When I try to execute I get the following error:
/release-pipeline.yml: Unexpected value 'parameters'.
How do I need to change this that it will work with the template in both cases, with and without the environment approval check?
I tried https://samlearnsazure.blog/2020/02/05/approvals-in-environments/
and of course different structure for the calling. But nothing helped.
In the tutorials they always have steps below the "deploy" but because I have different sites and environments I want/need the template file to save work.
I found another post which goes in the same direction.
I reworked my complete template so that I can use the approach of this: DevOps template with conditional manual approval job
It's not the same as I wanted, but it works.
My goal was that I don't want to create a environment if I have no checks on this site. Only for sites where I wanted the approval check.
With the above mentioned solution I need to create an environment for each site, independent if I have checks or not.
Related
azure-pipeline.yml
trigger:
- master
parameters:
- name: config
displayName: Execution Environment
type: string
default: QA
values:
- QA
- PreProd
- Prod
pool:
vmImage: 'windows-latest'
The above works perfectly, so in Azure the Execution Environment parameter is shown when I run the pipeline.
If however I attempt to put the parameters in a template as follows:
azure-pipeline.yml
trigger:
- master
extends:
template: parameters.yml
pool:
vmImage: 'windows-latest'
parameters.xml
parameters:
- name: config
displayName: Execution Environment
type: string
default: QA
values:
- QA
- PreProd
- Prod
Then when I run the pipeline the parameter is not shown.
In summary I'm trying to re-use a parameters.yml in different pipelines but extends: template: does not seem to work even though per this link it should:
https://learn.microsoft.com/en-us/azure/devops/pipelines/security/templates?view=azure-devops#set-required-templates
Runtime parameters are something different than templates parameters and having the second in your pipeline will not cause them to show on the UI. There is no way to template runtime parameters. You need to repeat them in each pipeline you expect to have them.
In my azure devops project, I have created a pipeline using template. This is the main yaml files for my build pipeline
name: Test-$(Date:yyyyMMdd)$(Rev:.r)
resources:
repositories:
- repository: api
type: git
name: porject/api
ref: master
- repository: front
type: git
name: project/front
ref: master
- repository: strapi
type: git
name: project/strapi
ref: master
trigger:
branches:
include:
- master
pool:
vmImage: 'ubuntu-latest'
workspace:
clean: all
variables:
- name: workingDir
value: project
- name: tfVersion
value: 0.12.29
- name: backendServiceGCP
value: test
- name: backendGCPBucketName
value: test
- name: tfpath
value: test
- name: env
value: dev
stages:
- stage: Terraform
jobs:
- job: Build
displayName: Build Terraform Infra
steps:
# Set and Export env var for api version to deploy
- template: templates/fetch-tag.yml
parameters:
repo: 'api'
envVar: TERRAFORM_API_TAG
# Set and Export env var for front version to deploy
- template: templates/fetch-tag.yml
parameters:
repo: 'front'
envVar: TERRAFORM_FRONT_TAG
# Set and Export env var for strapi version to deploy
- template: templates/fetch-tag.yml
parameters:
repo: 'strapi'
envVar: TERRAFORM_STRAPI_TAG
# Init Terraform providers
- template: templates/tf-init.yml
parameters:
backendServiceGCP: '$(backendServiceGCP)'
backendGCPBucketName: '$(backendGCPBucketName)'
workingDir: '$(workingDir)'
variableFilePath: $(buildSubscription)-common.tfvars
# Plan Terraform Infra to Deploy
- template: templates/tf-plan.yml
parameters:
backendServiceGCP: '$(backendServiceGCP)'
workingDir: '$(workingDir)'
variableFilePath: $(buildSubscription)-common.tfvars
# Publish Public Artifact with Terraform ressources to deploy
- template: templates/publish-artifact.yml
parameters:
tfpath: '$(tfpath)'
When I am trying to run the pipeline I have the following error:
Encountered error(s) while parsing pipeline YAML:
Job Build: The step name version appears more than once. Step names must be unique within a job.
Job Build: The step name version appears more than once. Step names must be unique within a job.
Job Build: The step name version appears more than once. Step names must be unique within a job.
Job Build: The step name version appears more than once. Step names must be unique within a job.
Job Build: The step name version appears more than once. Step names must be unique within a job.
I don't really understand why.
This is an example of templates I am using in the pipeline named publish-artifact.yml:
parameters:
tfPath: ''
steps:
- task: CopyFiles#2
inputs:
sourceFolder: ${{ parameters.tfpath }}
contents: |
tfplan
**/*.tf
**/*.tfvars
**/*.json
!**/.terraform
**/*.sh
targetFolder: $(Build.ArtifactStagingDirectory)
- task: PublishBuildArtifacts#1
inputs:
pathtoPublish: $(Build.ArtifactStagingDirectory)
artifactName: tf
What am I doing wrong with the job name ?
Azure Devops supports passing output variable from one step as next step's inputs. See this ticket:
We can name a step like this:
steps:
- script: echo test
name: ScriptName
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
Write-Host '##vso[task.setvariable variable=xxx;isOutput=true;]xxx'
name: PSName
The name must be unique so that we can use format $(referencename.variablename) to access output variable from specific step.
The error indicates that some steps in your templates have the same name version! And this is not supported. About why this issue occurs:
1.You called the same template several times, this is the main cause of your issue.
Devops expand templates first when processing the pipeline, so if your fetch-tag template has one step named version, the final expended azure-pipeline.yml would be:
stages:
- stage: Terraform
jobs:
- job: Build
displayName: Build Terraform Infra
steps:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
Write-Host "Hello World"
name: version
...
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
Write-Host "Hello World"
name: version
...
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
Write-Host "Hello World"
name: version
...
2.You may also need to check if there're steps with same name in your different templates.
We can call same template more than once in one pipeline, but we can't call same templates in which there's named step. Cause the pipeline will expand the templates more than once and the final pipeline would contain many steps with same name. And this is not supported since the name should be unique.
Solution:
1.Remove the name element of your steps if you don't need to use output variables mentioned above.
2.Or you can make several copies of fetch-tag.yml and name them fetch-tag-api.yml, fetch-tag-front.yml and fetch-tag-strapi.yml. Rename the referenceName version in these three files to version1, version2 or what. Then you can run the pipeline with:
steps:
# Set and Export env var for api version to deploy
- template: templates/fetch-tag-api.yml
parameters:
repo: 'api'
envVar: TERRAFORM_API_TAG
# Set and Export env var for front version to deploy
- template: templates/fetch-tag-front.yml
parameters:
repo: 'front'
envVar: TERRAFORM_FRONT_TAG
# Set and Export env var for strapi version to deploy
- template: templates/fetch-tag-strapi.yml
parameters:
repo: 'strapi'
envVar: TERRAFORM_STRAPI_TAG
I am trying to use extends as part of my pipeline. I am trying to get the first basic step working from Azure docs ie
# pythonparameter-template.yml
parameters:
- name: usersteps
type: stepList
default: []
steps:
- ${{ each step in parameters.usersteps }}
- ${{ step }}
# azure-pipelines.yml
trigger: none
resources:
repositories:
- repository: CI-CD-Templates
type: git
name: /CI-CD-Templates
ref: refs/heads/master
extends:
template: /pythonparameter-template.yml#CI-CD-Templates
parameters:
usersteps:
- script: echo This is my first step
- script: echo This is my second step
I keep getting the below error:
The directive 'each' is not allowed in this context. Directives are not supported for expressions that are embedded within a string. Directives are only supported when the entire value is an expression
Unexpected value '${{ each step in parameters.usersteps }} - ${{ step }}'
Also after I extend from a template can azure-pipelines.yml also have it's own custom steps ie
# azure-pipelines.yml
resources:
repositories:
- repository: templates
type: git
name: MyProject/MyTemplates
ref: tags/v1
extends:
template: template.yml#templates
parameters:
usersteps:
- script: echo This is my first step
- script: echo This is my second step
steps:
- template: /validation-template.yml#CI-CD-Templates
parameters:
commitMessage: $(commitMessage)
- template: /shared-template.yml#CI-CD-Templates
parameters:
buildArtifactDir: $(buildArtifactDir)
Update
Please refer the response in this DC link-- Yaml extends feature erroring out when looping through steps.
The YAML has a validation task in #pythonparameter-template.yml
Comment out these 2 lines and your YAML will succeed. The template shown here is preventing any tasks to be used. This could be a scenario for someone with specific security requirements.
${{ if eq(pair.key, 'task') }}:
'${{ pair.value }}': error
Are you trying to combine two yml pythonparameter-template.yml azure-pipelines.yml in the same file? It's not supported.
parameters:
- name: usersteps
type: stepList
default: []
steps:
- ${{ each step in parameters.usersteps }}
- ${{ step }}
According to the error Directives are not supported for expressions that are embedded within a string. Directives are only supported when the entire value is an expression Unexpected value '${{ each step in parameters.usersteps }} - ${{ step }}'
You maybe use the wrong format. About the formats you could refer our official doc here--
Template types & usage
Besides, you can make azure-pipelines.yml also have it's own custom steps. But you need to put them under parameters in the pipeline, not like the way you use.
azure-pipelines.yml
trigger:
- master
extends:
template: pythonparameter-template.yml
parameters:
buildSteps:
- bash: echo Test #Passes
displayName: succeed
- bash: echo "Test"
displayName: succeed
- script: echo "Script Test"
displayName: Fail
In a previous SO question, I was trying to get create an Azure DevOps template.
Now that I know how to do that, I'm trying to pass in a list of strings to the template and use them in the template, like so:
azure_pipelines.yml
extends:
template: AzurePipelines/job.yml
parameters:
projects:
- FirstProject.csproj
- SecondProject.csproj
and in this template...
job.yml
steps:
(for each projectFileName in projects)
- bash echo $(projectFileName)
- run code in the build.yml template using $(projectFileName)
- run code in the test.yml template using $(projectFileName)
- run code in the publish.yml template using $(projectFileName)
So for each project name handed into the template, run the template steps.
I'm not sure how to pass in an array of strings and how to test that it works?
It seems you are looking for each function:
# my-template.yml
parameters:
- name: 'projects'
type: projectList
default: []
steps:
- ${{ each project in parameters.projects }}:
- task: PublishBuildArtifacts#1
displayName: 'Publish ${{ project }}'
inputs:
PathtoPublish: '$(Build.ArtifactStagingDirectory)/${{ project }}.zip'
# ci.yml
steps:
- template: my-template.yml
parameters:
projects:
- test1
- test2
The easiest way I found to accomplish the use of templates is by using the templates from within the main yaml file pipeline. This way you will have access to all the variables and variable groups defined in the main template. Using this architecture you will be able to bundle several steps into a template then run it from the main yaml file. Eg:
azure_pipelines.yml
variables:
- name: projectFileName
value: "Variable 1 value"
- name: VAR2
value: "Variable 2 value"
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
steps:
- template: templates/job.yml
- template: templates/test.yml
templates/job.yml
steps:
- task: Bash#3
displayName: 'Run Bash Command'
inputs:
targetType: 'inline'
script: |
print "Variable value: " $(projectFileName)
Hope it helps
I'm trying to set the variable group according to one varible that exists in the pipeline. The yaml looks like this:
But i'm getting the following error when i'm running the pipeline:
If i remove the "- group : QA" or "- group : PROD" the pipeline runs without any problem. What am I doing wrong?
This is slightly different solution but you may achieve your goal - which is if I understood conditional selection of variable group.
You can use runtime parameters:
parameters:
- name: environment
displayName: Environment
type: string
default: QA
values:
- QA
- PROD
stages:
- stage:
displayName: 'Build and Restore'
variables:
- group: ${{ parameters.environment }}
jobs:
- job:
steps:
- script: echo $(name)
than running a build you can select your envrionment:
Note: I have defined two variable groups QA and PROD with variable name in both groups.
Try with below schema:
variables:
isProd: true
stages:
- stage: Test
displayName: Build and restore
variables:
- ${{ if eq(variables['isProd'], 'false') }}:
- group: QA
- ${{ if eq(variables['isProd'], 'true') }}:
- group: PROD
jobs:
- job: A
steps:
- bash: echo $(groupname)
Note: You may receive some red warning when you defining above YAML scipt. Ignore that confused warning and continue to run it.