Migrating azure-pipelines.yaml to separate repo, but running on code of other repo - azure

Ultimately, I'm trying to do this:
Move azure-pipelines.yaml and associated templates out of the code repository (code-repo).
Move them into a separate dedicated repository (pipeline-repo).
Have the pipeline look at the config for the pipeline in pipeline-repo, but run the pipeline on the code in the code-repo.
I'm referring the following documentation:
Use other repositories: this one refers to "templates in other repositories," but I'm trying to remove any pipeline configs so the code-repo is just purely application code... and the Dockerfile.
Define a repositories resource
For testing, I have this simple test.yaml:
# Triggers when PR is created due to branch policies
trigger: none
resources:
repositories:
- repository: code-repo
type: git
name: code-repo
pool:
vmImage: 'ubuntu-latest'
stages:
- stage: Testing
displayName: Test stage
jobs:
- job: ParallelA
steps:
- bash: echo Hello from parallel job A
displayName: 'Run a one-line script'
When I create a PR on code-repo, it is triggering the pipeline, which is to say branch policies are configured to refer to that pipeline. I do get the print out the Hello from parallel job A print out.
But I don't see in the run logs it pulling code-repo.
I do see the following, however:
My actual PR pipeline would look something like this:
trigger: none
resources:
repositories:
- repository: code-repo
type: git
name: code-repo
variables:
- template: templates/variables.yaml
pool:
vmIMage: $(vmImageName)
stages:
- template: templates/build/buildStage.yaml
...
Testing that, it confirms that it isn't running on the code-repo PR, but the pipeline-repo so everything fails.
So it is unclear to me what I need to do from here to get the pipeline to run on the PR code from code-repo.
Suggestions?

Ok, I think I have it sorted out, at least some of my stages are now succeeding.
I came across this documentation which informed me of checkout.
So in addition to doing something like:
resources:
repositories:
- repository: code-repo
type: git
name: code-repo
Then you need to add a step called checkout like the following:
# Triggers when PR is created due to branch policies
trigger: none
resources:
repositories:
- repository: code-repo
type: git
name: code-repo
pool:
vmImage: 'ubuntu-latest'
stages:
- stage: Testing
displayName: Test stage
jobs:
- job: ParallelA
steps:
- checkout: code-repo
- task: task1
- task: task2
The checkout should set the context for the subsequent steps.

Related

Azure Devops: how configure my pipeline script to trigger another pipeline with different repo?

I have my own pipeline p-A with repository r-A, and pipeline p-B with another repo r-B.
I want to update the pipeline script for p-A only, to trigger the p-B actively, without any modification in p-B.
Below is the yaml pipeline script for p-B, which is already set up to run with schedule
pool:
name: 'workflow_test_pool'
schedules:
- cron: "0 19 * * *"
displayName: run test every day at 8PM CET
branches:
include:
- main
always: true
trigger: none
jobs:
- job:
timeoutInMinutes: 30
steps:
- script: |
python -m pytest tests/ -s
displayName: 'Run the test'
below is the pipeline script main.yaml for p-A
pool:
name: 'workflow_test_pool'
stages:
#########################
- template: pipeline2/p1.yaml
############################
- template: pipeline2/p2.yaml
parameters:
dependsOn:
- FirstPipeline
so the question is, how to trigger the pipeline p-B in pipeline2/p2.yaml(from p-A)?
You can create PowerShell script task as the last step of the pipeline to trigger pipeline p-B through REST API. You will have to maintain Personal Access Token, ideally as secret variable.
REST API call you will use:
https://learn.microsoft.com/en-us/rest/api/azure/devops/pipelines/runs/run-pipeline?view=azure-devops-rest-7.1
Detailed step-by-step guide:
https://blog.geralexgr.com/cloud/trigger-azure-devops-build-pipelines-using-rest-api
Azure DevOps supports multiple repositories checkout, you can just reference resources function in your YAML script and call another repository to trigger from the pipeline.
YAML code :-
# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml
pool:
vmImage: ubuntu-latest
workspace:
clean: all
resources:
repositories:
- repository: repo_a
type: git
name: InternalProjects/repo_a
trigger:
- main
- release
- repository: repo_b
type: git
name: InternalProjects/repo_b
trigger:
- main
steps:
- checkout: repo_a
- checkout: repo_b
- script: dir $(Build.SourcesDirectory)
I am running this pipeline from repo_a and the repo_a and repo_b both ran successfully like below:-
Output :-
You can directly run any task from pipeline with multiple repositories like below:-
# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml
pool:
vmImage: ubuntu-latest
workspace:
clean: all
resources:
repositories:
- repository: repo_a
type: git
name: InternalProjects/repo_a
trigger:
- main
- release
- repository: repo_b
type: git
name: InternalProjects/repo_b
trigger:
- main
steps:
- checkout: repo_a
- checkout: repo_b
- task: AzureCLI#2
inputs:
azureSubscription: 'Subscription-name(sub-id)'
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: 'az resource list --location uksouth'
Output:-
References :-
Check out multiple repositories in your pipeline - Azure Pipelines | Microsoft Learn
Trigger azure Devops pipeline from another repository - GeralexGR
Multiple Repositories in a Single Azure Pipeline - DEV Community 👩‍💻👨‍💻

ADO YAML failing: No repository found by name templates

I'm trying to edit an ADO YAML file down to the bare minimum in order to isolate another issue.
When I run Validate, it comes back with the following error:
No repository found by name templates
Here's the general gist of my YAML:
#resources:
# repositories:
# - repository: templates
# type: git
# name: TemplateProject/TemplateRepo
name: $(VersionName)
trigger:
branches:
include:
- main
batch: true
paths:
exclude: $(ListOfExclusions)
stages:
- template: core/setVersion.yml#templates
- stage: Build
pool: linux
jobs:
- job: BuildDocker
displayName: Build and Push Docker Image
pool: linux
steps:
- task: Docker#2
displayName: Build and push an image to container registry
inputs:
command: buildAndPush
repository: $(RepoName)
dockerfile: $(Build.SourcesDirectory)/Dockerfile
containerRegistry: $(dockerRegistryServiceConnection)
tags: |
$(Tag)
What could be going wrong? The error message makes me think the YAML isn't clean.
It turns out I caused a simple typo when commenting out the resources section of the YAML. I had a template part of the stage that also needed to be commented out, and I neglected to do this.
Once I updated the code to read:
stages:
# - template: core/setVersion.yml#templates
- stage: Build
pool: linux
jobs:
- job: BuildDocker
# etc...
Now my YAML validates with OK.

Unable to fetch source and target branches of a Pull Request - Azure Devops

I have my source code repository in github and my pipeline in Azure devops. I am trying to execute certain tasks in my pipeline based on the source branch of a pull request. The pipeline gets triggered on PR. However, when I try to get the below attributes of a Pull request from my yml pipeline, I get the message shown in the screenshot below. It basically states command not found for all the values. Is there anything obvious that could cause this. Or is this not how these values are expected to be fetched? Any help is much appreciated.
trigger:
branches:
include:
- feature/azure-pipeline
- develop
- release/*
exclude:
- features/*
- master
pr:
branches:
include:
- develop
- main
stages:
- stage: TestStage
jobs:
- job: unit_test
displayName: 'Unit test Job'
pool:
vmImage: 'macos-latest'
variables:
- name: currentBranch
${{ if eq(variables['Build.Reason'], 'PullRequest') }}:
value: $(System.PullRequest.TargetBranch)
${{ if ne(variables['Build.Reason'], 'PullRequest') }}:
value: $(Build.SourceBranch)
steps:
- task: DownloadSecureFile#1
displayName: 'Download CSSM secrets'
name: secureKeys
inputs:
secureFile: 'cssmkeys.properties'
- script: |
echo Target Branch is $(System.PullRequest.TargetBranch)
echo Source Repository URI is $(System.PullRequest.SourceRepositoryURI)
echo PullRequest Id is $(System.PullRequest.PullRequestId)
echo Source Branch is $(System.PullRequest.SourceBranch)
echo Current Branch is $(value)
Output screenshot
EDIT
Link to the system variables page - https://learn.microsoft.com/en-us/azure/devops/pipelines/build/variables?view=azure-devops&tabs=yaml

How to run a template(.yml) belonging to another repository from my current repository in azure devops?

With the below setup, when my pipeline corresponding to my repo runs, it runs the template (template.yml) file belonging to 'anotherRepo'. But when it checks out, it checks out my repo instead of 'anotherRepo'.
Is there any issue in my setup?
Looks like checkout:self does not have any impact and it does not work
My current Repo:
azurepipeline.yml file:
variables:
acceptanceTestsRepoName: 'anotherRepo'
resources:
repositories:
- repository: 'anotherRepo'
name: ProjectName/anotherRepo
type: git
ref: master
stages:
- stage: acceptance_tests
displayName: 'Run Acceptance Tests in Dev'
jobs:
- template: 'azure-pipelines-templates/template.yml#${{variables.acceptanceTestsRepoName}}'
Repo:anotherRepo
template.yml
jobs:
- job: AcceptanceTest
displayName: Run Acceptance Test in Dev
pool:
vmImage: 'ubuntu-latest'
steps:
- checkout: self
- task: UsePythonVersion#0
inputs:
versionSpec: '$(python.version)'
self always refers to a repo associated with the build pipeline. In your case, you need to checkout anotherRepo manually:
# Azure Repos Git repository in the same organization
- checkout: git://anotherRepo
This assumes that anotherRepo is in the same Azure DevOps organization. If it's not or stored somewhere else (GitHub, Bitbucket, etc...) you also need to add it as a resource to the pipeline defintion. See Check out multiple repositories in your pipeline for details.
According to the document about checkout: self represents the repo where the initial Azure Pipelines YAML file was found.
So your pipeline checkout the repo where your azurepipeline.yml file is located.
If you want to checkout your anotherRepo, the checkout step in your template.yml should be - checkout: anotherRepo:
jobs:
- job: AcceptanceTest
displayName: Run Acceptance Test in Dev
pool:
vmImage: 'ubuntu-latest'
steps:
- checkout: anotherRepo
- task: UsePythonVersion#0
inputs:
versionSpec: '$(python.version)'
You can also use Inline syntax checkout to directly check out another repo in azure-pipelines.yml file:
stages:
- stage: acceptance_tests
displayName: 'Run Acceptance Tests in Dev'
jobs:
- job: checkout
steps:
- checkout: git://ProjectName/anotherRepo
Thanks all for your response expecially #beatcracker
By replacing the checkout step from self , I was able to run successfully
No change to My Current Repo.
Below change done to another repo
jobs:
- job: AcceptanceTest
displayName: Run Acceptance Test in Dev
pool:
vmImage: 'ubuntu-latest'
steps:
- checkout: git://ProjectName/anotherRepo
- task: UsePythonVersion#0
inputs:
versionSpec: '$(python.version)'

Azure DevOps - two dependent YAML pipelines in one repository

I have two .yml files in my repo. One for build, one for deployment. The main reason why I would like to keep build separate from the deployment is that I also would like to store variables for environments in my repo, e.i. in variables-dev.yml and variables-prod.yml files. So there is no need to create a new build every time (which includes running tests, docker image build etc.).
The file build.yml:
trigger:
paths:
exclude:
- build.yml
- deploy.yml
stages:
- stage: build
jobs:
...
And the deploy.yml, which I want to be triggered only on the completion of the build pipeline. That's why I add the first exclusion of all paths, but add one on pipeline resource.
trigger:
paths:
exclude:
- '*'
resources:
pipelines:
- pipeline: build
source: build
trigger:
branches:
include:
- '*'
stages:
- stage: dev
variables:
- template: variables-dev.yml
jobs:
- deployment: deploy_dev
environment: 'dev'
strategy:
runOnce:
deploy:
steps:
...
- stage: prod
dependsOn: dev
variables:
- template: variables-prod.yml
jobs:
- deployment: deploy_prod
environment: 'prod'
strategy:
runOnce:
deploy:
steps:
...
Unfortunately it does not seem to work. The top trigger blocks lower trigger. And if I remove the top trigger than the deploy pipeline is triggered at the same time with the build one.
you have to start your deploy.yml with trigger: none
trigger: none
resources:
pipelines:
- pipeline: ci-pipeline
source: my-build-pipeline
trigger:
enabled: true
branches:
include:
- master
Set your triggers for the second yml to none, then add this setting in the "Triggers" section of the UI. It will stage your builds as you describe

Resources