I am referencing a repository in my azure-pipelines template like this:
resources:
repositories:
- repository: MyRepo
type: git
name: MyRepoName
ref: MyRepoRef
I would like to know if it is possible to read the content of a file that is inside the referenced repository, inside this repository is another yaml that is being executed in the pipeline.
If we use another repo in the pipeline, we could read the content of a file in the referenced repository. We could refer this link for more details
In my test, project name is test, the referenced repository is test and current repository is sample, then I read the content of file pull_request_template.md
YAML build definition:
trigger: none
resources:
repositories:
- repository: test
type: git
name: test/test
ref: master
steps:
#checkout referenced repository
- checkout: test
#List SourcesDirectory files
- task: Bash#3
inputs:
targetType: 'inline'
script: 'ls ''$(Build.SourcesDirectory)'''
#Read the contents of the file pull_request_template.md
- task: PowerShell#2
inputs:
targetType: 'inline'
script: 'Get-Content -Path $(Build.SourcesDirectory)\pull_request_template.md'
Result:
Related
How to authorize variable in a yaml template in another repo to be used in a different repo. IOW, how to declare variables in a template once and use in multiple repos in azure devops
I am trying to migrate from classic pipelines to yaml in azure devops. So i am trying to set up a repo to host all yaml templates so it can be referenced and reused by multiple repos for builds, etc.
I wrote this yaml pipeline to sample prototyping it:
`name: FirstPL
trigger:
- my_test_branch
pool: my-agent
resources:
repositories:
- repository: blah
type: git
name: foo/bar
ref: refs/heads/poc
variables:
- template: pipeline_vars.yml#blah
steps:
- script: echo $(variable_from_pipeline_vars)
`
However when i run this i get the follwoing error:
An error occurred while loading the YAML build pipeline. Variable group was not found or is not authorized for use. For authorization details, refer to https://aka.ms/yamlauthz.
How can i declare my variables and variables groups once in a template in a repo that is dedicated to host those templates and then use them over and again in multiple repos using the resourcs syntax above? Also, I tried to find a way authorize the variables template but couldn't find anything to enable this.
How to authorize variable in a yaml template in another repo to be
used in a different repo. IOW, how to declare variables in a template
once and use in multiple repos in azure devops. However when I run
this i get the follwoing error:
An error occurred while loading the YAML build pipeline. Variable group was not found or is not authorized for use. For authorization
details, refer to
https://aka.ms/yamlauthz.
You can directly add the variable group in your azure DevOps project in the Library tab and save all your variables from pipeline_vars.yml in the variable group like below:-
Now, You can access this variable group in your YAML pipeline of multiple repos like the below:-
# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml
pool:
vmImage: ubuntu-latest
workspace:
clean: all
resources:
repositories:
- repository: repo_a
type: git
name: InternalProjects/repo_a
trigger:
- main
- release
- repository: repo_b
type: git
name: InternalProjects/repo_b
trigger:
- main
variables:
- group: SharedVariables
steps:
- checkout: repo_a
- checkout: repo_b
- script: |
echo $(databaseServerName)
- task: AzureCLI#2
inputs:
azureSubscription: 'xxx subscription(xxxxxxxxx-f598-44d6-b4fd-xxxxxxxxxxxx)'
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: 'az resource list --location uksouth'
Output:-
It asks for approving permission for the Variable group to run in the pipeline like below:-
Console:-
Tried the same with another repo repo_b in the project and it asks to approve access for repositories and variable groups like the below:-
Output:-
If you want this variable to be accessed in multiple stages/repos/pipelines within the project without authorization prompt. You can click on Security on top and allow it:-
I created one variables template and referenced it in the YAML pipeline to use across multiple repos by checking out another repo like below:-
# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml
pool:
vmImage: ubuntu-latest
workspace:
clean: all
resources:
repositories:
- repository: repo_a
type: git
name: InternalProjects/repo_a
trigger:
- main
- release
- repository: repo_b
type: git
name: InternalProjects/repo_b
trigger:
- main
variables:
- template: pipeline_vars.yml
steps:
- checkout: repo_a
- checkout: repo_b
- script: |
echo $(environmentName)
- task: AzureCLI#2
inputs:
azureSubscription: 'xxx subscription(xxxxxxxx-f598-44d6-b4fd-e2b6e97xxxxxx)'
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: 'az resource list --location uksouth'
Output:-
I tried to reference the same template in another repo where it does not exist it could not read the pipeline_vars.yml file as it does not exist in the repo.
You can make use of variable groups like the above to reference the variables in this pipeline.
One of the possible reasons for this is that the project that hosts the repository with the variables does not allow access to it's repositories from yaml pipelines.
To verify, go to your project's settings -> Pipelines -> Settings -> Verify "Protect access to repositories in YAML pipelines" . This setting is enabled by default. You could set it to off or add a checkout step to your pipeline yaml. See here for more information.
I'm trying to edit an ADO YAML file down to the bare minimum in order to isolate another issue.
When I run Validate, it comes back with the following error:
No repository found by name templates
Here's the general gist of my YAML:
#resources:
# repositories:
# - repository: templates
# type: git
# name: TemplateProject/TemplateRepo
name: $(VersionName)
trigger:
branches:
include:
- main
batch: true
paths:
exclude: $(ListOfExclusions)
stages:
- template: core/setVersion.yml#templates
- stage: Build
pool: linux
jobs:
- job: BuildDocker
displayName: Build and Push Docker Image
pool: linux
steps:
- task: Docker#2
displayName: Build and push an image to container registry
inputs:
command: buildAndPush
repository: $(RepoName)
dockerfile: $(Build.SourcesDirectory)/Dockerfile
containerRegistry: $(dockerRegistryServiceConnection)
tags: |
$(Tag)
What could be going wrong? The error message makes me think the YAML isn't clean.
It turns out I caused a simple typo when commenting out the resources section of the YAML. I had a template part of the stage that also needed to be commented out, and I neglected to do this.
Once I updated the code to read:
stages:
# - template: core/setVersion.yml#templates
- stage: Build
pool: linux
jobs:
- job: BuildDocker
# etc...
Now my YAML validates with OK.
In my azure devops project, I have created a pipeline using template. This is the main yaml files for my build pipeline
name: Test-$(Date:yyyyMMdd)$(Rev:.r)
resources:
repositories:
- repository: api
type: git
name: porject/api
ref: master
- repository: front
type: git
name: project/front
ref: master
- repository: strapi
type: git
name: project/strapi
ref: master
trigger:
branches:
include:
- master
pool:
vmImage: 'ubuntu-latest'
workspace:
clean: all
variables:
- name: workingDir
value: project
- name: tfVersion
value: 0.12.29
- name: backendServiceGCP
value: test
- name: backendGCPBucketName
value: test
- name: tfpath
value: test
- name: env
value: dev
stages:
- stage: Terraform
jobs:
- job: Build
displayName: Build Terraform Infra
steps:
# Set and Export env var for api version to deploy
- template: templates/fetch-tag.yml
parameters:
repo: 'api'
envVar: TERRAFORM_API_TAG
# Set and Export env var for front version to deploy
- template: templates/fetch-tag.yml
parameters:
repo: 'front'
envVar: TERRAFORM_FRONT_TAG
# Set and Export env var for strapi version to deploy
- template: templates/fetch-tag.yml
parameters:
repo: 'strapi'
envVar: TERRAFORM_STRAPI_TAG
# Init Terraform providers
- template: templates/tf-init.yml
parameters:
backendServiceGCP: '$(backendServiceGCP)'
backendGCPBucketName: '$(backendGCPBucketName)'
workingDir: '$(workingDir)'
variableFilePath: $(buildSubscription)-common.tfvars
# Plan Terraform Infra to Deploy
- template: templates/tf-plan.yml
parameters:
backendServiceGCP: '$(backendServiceGCP)'
workingDir: '$(workingDir)'
variableFilePath: $(buildSubscription)-common.tfvars
# Publish Public Artifact with Terraform ressources to deploy
- template: templates/publish-artifact.yml
parameters:
tfpath: '$(tfpath)'
When I am trying to run the pipeline I have the following error:
Encountered error(s) while parsing pipeline YAML:
Job Build: The step name version appears more than once. Step names must be unique within a job.
Job Build: The step name version appears more than once. Step names must be unique within a job.
Job Build: The step name version appears more than once. Step names must be unique within a job.
Job Build: The step name version appears more than once. Step names must be unique within a job.
Job Build: The step name version appears more than once. Step names must be unique within a job.
I don't really understand why.
This is an example of templates I am using in the pipeline named publish-artifact.yml:
parameters:
tfPath: ''
steps:
- task: CopyFiles#2
inputs:
sourceFolder: ${{ parameters.tfpath }}
contents: |
tfplan
**/*.tf
**/*.tfvars
**/*.json
!**/.terraform
**/*.sh
targetFolder: $(Build.ArtifactStagingDirectory)
- task: PublishBuildArtifacts#1
inputs:
pathtoPublish: $(Build.ArtifactStagingDirectory)
artifactName: tf
What am I doing wrong with the job name ?
Azure Devops supports passing output variable from one step as next step's inputs. See this ticket:
We can name a step like this:
steps:
- script: echo test
name: ScriptName
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
Write-Host '##vso[task.setvariable variable=xxx;isOutput=true;]xxx'
name: PSName
The name must be unique so that we can use format $(referencename.variablename) to access output variable from specific step.
The error indicates that some steps in your templates have the same name version! And this is not supported. About why this issue occurs:
1.You called the same template several times, this is the main cause of your issue.
Devops expand templates first when processing the pipeline, so if your fetch-tag template has one step named version, the final expended azure-pipeline.yml would be:
stages:
- stage: Terraform
jobs:
- job: Build
displayName: Build Terraform Infra
steps:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
Write-Host "Hello World"
name: version
...
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
Write-Host "Hello World"
name: version
...
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
Write-Host "Hello World"
name: version
...
2.You may also need to check if there're steps with same name in your different templates.
We can call same template more than once in one pipeline, but we can't call same templates in which there's named step. Cause the pipeline will expand the templates more than once and the final pipeline would contain many steps with same name. And this is not supported since the name should be unique.
Solution:
1.Remove the name element of your steps if you don't need to use output variables mentioned above.
2.Or you can make several copies of fetch-tag.yml and name them fetch-tag-api.yml, fetch-tag-front.yml and fetch-tag-strapi.yml. Rename the referenceName version in these three files to version1, version2 or what. Then you can run the pipeline with:
steps:
# Set and Export env var for api version to deploy
- template: templates/fetch-tag-api.yml
parameters:
repo: 'api'
envVar: TERRAFORM_API_TAG
# Set and Export env var for front version to deploy
- template: templates/fetch-tag-front.yml
parameters:
repo: 'front'
envVar: TERRAFORM_FRONT_TAG
# Set and Export env var for strapi version to deploy
- template: templates/fetch-tag-strapi.yml
parameters:
repo: 'strapi'
envVar: TERRAFORM_STRAPI_TAG
I have an azure devops multi-stage pipeline which requires 1 stage that copies files from another repository to $(build.artifactstagingdirectory)
For example my YAML looks like
trigger:
- master
resources:
- repo: self
variables:
tag: '$(Build.BuildId)'
stages:
- stage: Build
displayName: Build image
jobs:
- job: Build
...
- stage: Build
... define other resource/repository ...
- task: CopyFiles#2
inputs:
SourceFolder: 'k8s'
Contents: '**'
TargetFolder: '$(build.artifactstagingdirectory)'
This pipeline is connect to a repository, which is probably defined by repo: self. So the question is, can I change this repository for a specific stage?
You can run git command in powershell script. Add a step in your stage to execute git command like below example. In this way the other repo will be cloned to artifact directory.
- powershell: |
cd $(Build.artifactstagingdirectory)
git clone "https://<<Your PAT>>#dev.azure.com/_organization/_project/_git/_repo"
Note: use your personal access token (PAT) as authentication.
I'm setting up several pipelines in Azure DevOps. To make my teams life easier, I'm using job templates.
These job templates are in a a proper repository, just for them.
For every pipeline I define the repository to get the templates from.
Some tasks in these templates run powershell code, and I want this code to be in a script file, to be reusable and stored in the same repo as the template.
When the pipelines runs, the template is embeded, it tries to locate the powershell script inside project repo actually being built/deployed.
How can i achieve this?
The workaround is to have inline code which I really don't want to have.
Any constructive answer will be very appreciated.
Thanks
After some digging I couldn't find any way to specify a script file as source to powershell task in a template.
Inside pipeline definition:
resources:
repositories:
- repository: templates
type: git
name: deploy-templates
variables:
artifactName: 'Trade Data ETL - $(Build.SourceBranchName)'
stages:
- stage: Build
displayName: Build
variables:
- group: DEV-Credential-Group
- group: COMMON-Settings-Group
jobs:
- template: ssis/pipelines/stage-build.yml#templates # Template reference
parameters:
artifactName: '$(artifactName)'
Inside template file:
- task: PowerShell#2
inputs:
filePath: ssis/pipelines/scripts/build-ssis-project.ps1
arguments: '-ProjectToBuild "tradedata-ldz-ssis/tradedata-ldz-ssis.dtproj'
pwsh: true
Update 2021
According to learn.microsoft.com, you can now also check out multiple repositories without custom scripting.
If you check out more than one repository, a separate folder containing the repository is created below $(Build.SourcesDirectory).
You can define multiple repositories like this:
resources:
repositories:
- repository: devops
type: git
name: DevOps
ref: main
- repository: infrastructure
type: git
name: Infrastructure
ref: main
And in the steps simply check them out as follows:
steps:
- checkout: self
- checkout: devops
- checkout: infrastructure
# List all available repositories
- script: ls
Original Answer
Currently the resources command only supports yml files in other repositories. However, you could simply checkout the repository in a task and then run the desired powershell script.
steps:
- task: PowerShell#2
inputs:
targetType: inline
script: |
git clone -b <your-desired-branch> https://azuredevops:$($env:token)#dev.azure.com/<your-organization>/<your-project>/_git/<your-repository> <target-folder-name>
./<target-folder-name>/foo.ps1
env:
token: $(System.AccessToken)
This script would checkout an arbitrary branch and execute a script foo.ps1 in the root of the target repository.
Call - checkout: templates inside the template file. This might only work when you insert a template but it successfully sees the repository resource and pulls it down.
You can copy the script files from source directory. Currently, you have not mentioned the root folder -
ssis/pipelines/scripts/build-ssis-project.ps1
Assuming, you are building on a repo where the powershell script resides -
Try -
- task: PowerShell#1
inputs:
scriptName: '$(ScriptsDir)/ssis/pipelines/scripts/build-ssis-project.ps1'
Pass the value of ScriptsDir where it could be the build source directory or build working directory