I am working on having one yaml template file job dependOn a deployment job before continuing. I need my "other-template" job below to depend on the deployment job name "DeployApi"
# api-deploy.yml
jobs:
- deployment: DeployApi
...
release.yml
stages:
- stage: 'deploy_dev'
displayName: Deploy to dev environment
dependsOn: []
jobs:
- template: api/api-deploy.yml
parameters:
environment: ...
varTemplate: ...
- template: other/other-template.yml
parameters:
dependsOn: ['DeployApi']
Is it possible to have the second template "other/other-template.yml" depend on the "DeployApi" deployment job? I know it's possible if it was a standard job name but I couldn't find any information on if it's possible to use dependsOn for a deployment job name.
Little bit more information: I have about 10 release templates that run parallel (I believe). I have one release template that I want to wait to run until every other template is ran.
Yes, you can.
release.yml
trigger:
- none
pool:
vmImage: ubuntu-latest
stages:
- stage: 'deploy_dev'
displayName: Deploy to dev environment
dependsOn: []
jobs:
- template: api/api-deploy.yml
parameters:
environment: ...
varTemplate: ...
- template: other/other-template.yml
parameters:
dependsOn: ['DeployApi']
api/api-deploy.yml
jobs:
- job: DeployApi
steps:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
# Write your PowerShell commands here.
Write-Host "This is api-deploy.yml."
other/other-template.yml
parameters:
dependsOn: []
jobs:
- job: getDependsOn
dependsOn: ${{parameters.dependsOn}}
steps:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
# Write your PowerShell commands here.
Write-Host "This is other-template.yml"
Result successfully:
Related
I have a multi stage Azure DevOps pipeline in yaml. Each stage includes multiple jobs. The pipeline runs properly by using Azure-hosted agents. I need to run it by our company's self-hosted agent. The self-hosted agent is a virtual machine scale set (VMSS) which is defined as an agent pool in DevOps, it has been tested and is working properly. Because the self-hosted agent does not have all the capabilities we need, our company's policy is to run the jobs inside docker containers. The docker images reside in our company's private Azure Container Registy.
My question is how can I run a whole template stage in a docker container? Please note that my question is not about executing a job but all the jobs in a stage template. This link explains how to execute jobs inside a container. It doesn't give any examples of how to run all the jobs in a stage in a container, especially when the stage is defined as a template.
My pipeline in simplified form is as follows:
## Name of self-hosted agent
pool: Platform-VMSS
resources:
containers:
- container: base
image: myazurecontainerregistryname.azurecr.io/platform/myimg:1.0.0
endpoint: 'serviceconnection-acr'
trigger:
branches:
include:
- "feature/ORGUS-4810"
exclude:
- "main"
- "release"
paths:
include:
- "usecase-poc/**"
variables:
pythonVersion: 3.8
whlPackageName: py_core
srcDirectory: usecase-poc/$(whlPackageName)
${{ if eq(variables['Build.SourceBranch'], 'refs/heads/main') }}:
BuildFrom: main
PackageName: $(whlPackageName)
versionOption: custom
deployToDevPipeline: True
deployToQAPipeline: True
deployToProdPipeline: True
${{ else }}:
BuildFrom: branch_${{ lower(variables['Build.SourceBranchName'] ) }}
PackageName: $(whlPackageName)_$(BuildFrom)
versionOption: patch
deployToDevPipeline: True
deployToQAPipeline: False
deployToProdPipeline: False
stageName: 'deployToDev'
stages:
- stage: DownloadArtifact
displayName: "Download python whl from artifactory"
jobs:
- job: DownloadArtifactJob
steps:
- checkout: self
- task: UniversalPackages#0
displayName: 'Download Artifact with Universal Packages'
inputs:
vstsFeed: 'some-value/00000000-0000-0009-0000-d24300000000'
vstsFeedPackage: '00000000-0000-0000-0000-0000000'
vstsPackageVersion: 0.8.4
downloadDirectory: $(Build.ArtifactStagingDirectory)
- task: Bash#3
name: GetWheelName_Task
inputs:
targetType: 'inline'
script: |
echo $(Build.ArtifactStagingDirectory)
find $(Build.ArtifactStagingDirectory) -name '*.whl'
ArtifactName=$(find $(Build.ArtifactStagingDirectory) -name '*.whl')
echo "Artifact name value is " $ArtifactName
echo "##vso[task.setvariable variable=ArtifactName;isOutput=true]$ArtifactName"
displayName: 'Get downloaded artifact in source directory'
- task: PublishBuildArtifacts#1
displayName: "Publish downloaded artifact to pipeline's output"
inputs:
pathToPublish: $(Build.ArtifactStagingDirectory)
artifactName: whl
- stage: ConstructSharedVariablesForAllStages
displayName: Construct Shared Variables For All Stages
dependsOn: DownloadArtifact
variables:
- group: proj-shared-vg
- name: ArtifactName
value: $[stageDependencies.DownloadArtifact.DownloadArtifactJob.outputs['GetWheelName_Task.ArtifactName']]
jobs:
- job: DownloadArtifact
container: base
steps:
- task: Bash#3
displayName: "Print variable value"
inputs:
targetType: 'inline'
script: |
echo $(ArtifactName)
- task: Bash#3
name: extractWheelName
displayName: Extract Wheel Name
inputs:
targetType: inline
script: |
echo $(ArtifactName) | awk -F"/" '{print $NF}'
WhlName="py_core-0.8.4-py3-none-any.whl"
echo "##vso[task.setvariable variable=WhlName;isOutput=true]$WhlName"
- task: DownloadPipelineArtifact#2
displayName: "Download artifact from previous stage"
inputs:
buildType: 'current'
project: 'Project Name'
buildVersionToDownload: 'latest'
artifactName: whl
targetPath: '$(System.ArtifactsDirectory)'
- pwsh: |
$whlFile = Get-ChildItem -Filter *.whl -Path "$(System.ArtifactsDirectory)" | ForEach-Object { $_.fullname } | Select-Object -First 1
Write-Host "##vso[task.setvariable variable=whlFile]$whlFile"
name: SetVars
displayName: Get wheel name
## This is the section where my question is about. How can I make sure each stage runs in the self-hosted agent pool. The stage contains multiple jobs.
- template: ../templates/azure-pipeline-stage-template.yaml
parameters:
deploy: ${{variables.deployToDevPipeline}}
variableGroup: databricks-sp-vg-dev
stageName: DeployToDev
environment: DBRKS_Dev_WHL
conditionParameter: deployToDev
dependsOnStage: ConstructSharedVariablesForAllStages
- template: ../templates/azure-pipeline-stage-template.yaml
parameters:
deploy: ${{variables.deployToQAPipeline}}
variableGroup: databricks-sp-vg-qa
stageName: DeployToQA
environment: DBRKS_QA_WHL
conditionParameter: deployToQA
dependsOnStage: DeployToDev
- template: ../templates/azure-pipeline-stage-template.yaml
parameters:
deploy: ${{variables.deployToProdPipeline}}
variableGroup: databricks-sp-vg-prod
stageName: DeployToProd
environment: DBRKS_Prod_WHL
conditionParameter: deployToProd
dependsOnStage: DeployToQA
In the code above in resources the container resides in our Azure Container Registry (ACR), the endpoint is our DevOps service connection of type container registry to pull and push images to and from ACR. In the code above I have commented where the issue is. In templates where I am refering to stage templates, I would like to run them all inside a container where I have defined as a resource at the beginning of the pipeline. The stage template has multiple jobs. Here is just a sample of stage template when running to emphasize it has multiple jobs:
The highlighted stage is the one created by template:
- template: ../templates/azure-pipeline-stage-template.yaml
parameters:
deploy: ${{variables.deployToDevPipeline}}
variableGroup: databricks-sp-vg-dev
stageName: DeployToDev
environment: DBRKS_Dev_WHL
conditionParameter: deployToDev
dependsOnStage: ConstructSharedVariablesForAllStages
Question is how to enforce all the jobs in the above template run in the docker container defined as resource in our pipeline. Thank you very much for your valuable input.
Add a container field at the job level as shown below. Then all the jobs in the template will be running on specified container.
pool:
vmImage: ubuntu-latest
resources:
containers:
- container: testcontainer
image: ubuntu
stages:
- stage: template01
displayName: tempate test
jobs:
- job: template
container: testcontainer
steps:
- template: templates/template01.yaml
- stage: template02
displayName: template test
jobs:
- job: template
container: testcontainer
steps:
- template: templates/template02.yaml
Else, add a step target field to all the required tasks in a template, as referred to in this link Build and Release Tasks - Azure Pipelines | Microsoft Learn](https://learn.microsoft.com/en-us/azure/devops/pipelines/process/tasks?view=azure-devops&tabs=yaml#step-target) "https://learn.microsoft.com/en-us/azure/devops/pipelines/process/tasks?view=azure-devops&tabs=yaml#step-target)")
resources:
containers:
- container: pycontainer
image: python:3.11
steps:
- task: AnotherTask#1
target: pycontainer
Currently, our inhertited pipelines (not the best designed, but it's what I've got for now), look something like this:
build -> provision + deploy (pipeline per env) -> acceptance test
I only want to be able to do one deployment at a time, so I'm looking at Exclusive Locks.
Unfortunately, these seem to work at the step level, not the pipeline level. So given my provision + deploy pipeline contains multiple steps, how can I prevent step 1 from pipeline run 1 running then step 1 from pipeline 2 running etc?
I can't see a way to apply the exclusive lock at the pipeline level.
Our QA provision + deploy pipeline yml looks like this (before adding the locks):
pool:
vmImage: "ubuntu-latest"
resources:
pipelines:
- pipeline: apiBuild
source: "API/Build"
trigger:
enabled: true
branches:
include:
- main
trigger: none
pr: none
variables:
- template: _pipeline/variables/allVariables.yml
parameters:
deployEnvironment: Qa
stages:
- stage: provisionInfrastructureTemplates
displayName: Provision Templates infrastructure
dependsOn: []
jobs:
- template: _pipeline/stages/jobs/provisionTemplates.yml
- stage: templatesAcceptanceTests
displayName: Templates acceptance tests
dependsOn: provisionInfrastructureTemplates
jobs:
- template: _pipeline/stages/jobs/acceptanceTestsTemplates.yml
- stage: provisionInfrastructureClients
displayName: Provision Clients infrastructure
dependsOn: []
jobs:
- template: _pipeline/stages/jobs/provisionClients.yml
- stage: clientsAcceptanceTests
displayName: Clients acceptance tests
dependsOn: provisionInfrastructureClients
jobs:
- template: _pipeline/stages/jobs/acceptanceTestsClients.yml
- stage: provisionInfrastructureReports
displayName: Provision Reports infrastructure
dependsOn: []
jobs:
- template: _pipeline/stages/jobs/provisionReports.yml
- stage: reportsAcceptanceTests
displayName: Reports acceptance tests
dependsOn: provisionInfrastructureReports
jobs:
- template: _pipeline/stages/jobs/acceptanceTestsReports.yml
- stage: upgradePreviewImage
displayName: Upgrade preview image
dependsOn: []
jobs:
- template: _pipeline/stages/jobs/upgradePreviewImage.yml
- stage: provisionInfrastructureDocuments
displayName: Provision Documents infrastructure
dependsOn: upgradePreviewImage
jobs:
- template: _pipeline/stages/jobs/provisionDocuments.yml
- stage: documentsAcceptanceTests
displayName: Documents acceptance tests
dependsOn: provisionInfrastructureDocuments
jobs:
- template: _pipeline/stages/jobs/acceptanceTestsDocuments.yml
- stage: provisionInfrastructureNotifications
displayName: Provision Notifications infrastructure
dependsOn: []
jobs:
- template: _pipeline/stages/jobs/provisionNotifications.yml
- stage: provisionEventGridSubscriptions
displayName: Provision Event Grid Subscriptions
dependsOn: [clientsAcceptanceTests, templatesAcceptanceTests, reportsAcceptanceTests, documentsAcceptanceTests, provisionInfrastructureNotifications]
jobs:
- template: _pipeline/stages/jobs/provisionEventGridSubscriptions.yml
- stage: workflowTests
displayName: Workflow tests
dependsOn: provisionEventGridSubscriptions
jobs:
- template: _pipeline/stages/jobs/workflowTests.yml
As an aside, I know our services ought to be independently deployable, they're not currently, that's part of the tech debt we're dealing with so, as things stand, they need deploying together.
When you put the jobs in separate stages. The exclusive lock of the environment will be judged separately.
It is not able to apply the exclusive lock at the pipeline level.
To meet your requirement, you can define required the jobs in the same stage. The Exclusive Lock will be judged before running.
When set the environments exclusive lock in the same stage, all environments referenced in a stage will be locked. Only when the current Pipeline Run Stage finished, the next Pipeline run will continue to run.
Here is an example:
stages:
- stage: Test
displayName: Test
lockBehavior: sequential
jobs:
- deployment: Test1
displayName: 'Test'
environment: Test2
strategy:
runOnce:
deploy:
steps:
- script: echo "Test"
- deployment: Test
displayName: 'Test2'
environment: Test2
strategy:
runOnce:
deploy:
steps:
- script: echo "Test"
Context:
I have two pipelines. Let's call them trigger and trigger-two.
Trigger pipeline has multiple stages, called Troubleshooting, Troubleshooting2, and Troubleshooting3.
My aim is to trigger trigger-two pipeline when the stages Troubleshooting and Troubleshooting2 from trigger pipeline are completed.
Issue:
I have followed Microsoft documentation ( https://learn.microsoft.com/en-us/azure/devops/pipelines/process/pipeline-triggers?view=azure-devops ), but no matter how I implement it, the trigger-two pipeline is triggered only when the trigger pipeline is fully completed. The stage filter is not taken into consideration for unknown reasons.
Code for the trigger-two pipeline:
trigger: none
resources:
pipelines:
- pipeline: triggertest
source: triggertest
project: project_name
trigger:
stages:
- Troubleshooting
- Troubleshooting2
branches:
include:
- refs/heads/branchtest
I tried to reproduce the issue, but it works in my pipeline. Please try my my yaml：
pipeline trigger：
trigger:
- none
pool:
vmImage: windows-latest
stages:
- stage: Troubleshooting1
jobs:
- job: Troubleshooting1
steps:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
Write-Host "Hello World"
- stage: Troubleshooting2
jobs:
- job: Troubleshooting2
steps:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
Write-Host "Hello World"
- stage: Troubleshooting3
jobs:
- job: Troubleshooting3
steps:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
# Write your PowerShell commands here.
Write-Host "Hello World"
#in order to add the time for this stage, install a Module
Install-Module Az -Scope CurrentUser -Force
Pipeline trigger-two：
trigger: none
resources:
pipelines:
- pipeline: trigger-resource
source: trigger
project: testproject
trigger:
stages:
- Troubleshooting1
- Troubleshooting2
branches:
include:
- refs/heads/main
pool: Default # Specify a self hosted agent pool by name. Because I only have one MS-hosted agent parallel job. So I use different pool in two yaml.
#pool:
# vmImage: windows-latest
steps:
- task: CmdLine#2
inputs:
script: |
echo Write your commands here
echo Hello world
Edit:
I'm trying to integrate a QA pipeline in Azure DevOps that triggers during a dev pipeline. The dev pipeline has 4 environments that the build is deploying to, where each environment is a stage in the dev pipeline
Currently, I'm able to trigger the QA builds for each environment using 4 separate QA pipelines using syntax similar to this in each pipeline:
resources:
pipelines:
- pipeline: Dev_Env_1
source: Dev
trigger:
stages:
- Env_1
My goal is to only have one QA pipeline that is triggered multiple times by the dev pipeline when it completes each stage. It feels like syntax in the yml file like this should work:
resources:
pipelines:
- pipeline: Dev_Env_1
source: Dev
trigger:
stages:
- Env_1
- pipeline: Dev_Env_2
source: Dev
trigger:
stages:
- Env_2
However, this only triggers after Env_1 is completed, when I would like a build triggered for the completion of the Env_1 and Env_2 stages in the Dev pipeline.
Is there a way to do this without drastically changing the way either pipeline currently works?
The below steps can help you achieve your requirements.
For example,
Dev Pipeline
#Dev Pipeline
trigger:
- none
pool:
vmImage: 'windows-latest'
stages:
- stage: Env_1
displayName: Env_1
jobs:
- job:
steps:
- task: CmdLine#2
inputs:
script: |
echo Stage1
- stage: Env_2
displayName: Env_2
jobs:
- job:
steps:
- task: CmdLine#2
inputs:
script: |
echo Stage2
- stage: Env_3
displayName: Env_3
jobs:
- job:
steps:
- task: CmdLine#2
inputs:
script: |
echo Stage3
- stage: Env_4
displayName: Env_4
jobs:
- job:
steps:
- task: CmdLine#2
inputs:
script: |
echo Stage4
1, Create an incoming webhook service connection, and an incoming service webhook that is triggered by the complete and success of stages of the Dev Pipeline.
incoming service hook URI: https://dev.azure.com/<ADO Organization>/_apis/public/distributedtask/webhooks/<WebHook Name>?api-version=6.0-preview
Official document:
https://learn.microsoft.com/en-us/azure/devops/release-notes/2020/pipelines/sprint-172-update#generic-webhook-based-triggers-for-yaml-pipelines
After that, write QA pipeline as below:
QA Pipeline
# QA pipeline
trigger:
- none
resources:
webhooks:
- webhook: bowmantest
connection: BowmanIncommingWebHook
pool:
vmImage: 'windows-latest'
steps:
- task: CmdLine#2
inputs:
script: |
echo QA pipeline
Everything works fine:
I have the following pipeline
variables:
azureSubscription: ...
stages:
- stage: Deploy
displayName: Deploy stage
dependsOn: Build
condition: succeeded()
jobs:
- deployment: Deploy
displayName: Deploy
environment: 'development'
strategy:
runOnce:
deploy:
steps:
- task: AzureResourceGroupDeployment#2
inputs:
action: 'Create Or Update Resource Group'
resourceGroupName: '...'
location: '...'
templateLocation: 'Linked artifact'
csmFile: '$(Pipeline.Workspace)/azure-deploy.json'
deploymentMode: 'Incremental'
The repo has the following files (at the root directory)
azure-pipelines.yaml
azure-deploy.json
and only a master branch.
I have tried:
azure-deploy.json
**azure-deploy.json
**/*azure-deploy.json
$(Build.SourcesDirectory)/azure-deploy.json
$(Pipeline.Workspace)/azure-deploy.json
$(System.DefaultWorkingDirectory)/azure-deploy.json
Having read:
Azure Pipeline Error: Could not find any file matching the template file pattern
VSTS Pipeline Deployment of ARM Error: Could not find any file matching the template file pattern
https://github.com/microsoft/azure-pipelines-tasks/issues/11520
to no avail. Any ideas?
Update: I have added a publish pipeline as suggested by #ShaykiAbramczyk
Now I get a Template file pattern matches a directory instead of a file: /home/vsts/work/1/azure-deploy.json
- stage: Build
displayName: Build stage
jobs:
- job: Build
displayName: Build
pool:
vmImage: $(vmImageName)
steps:
- task: PublishPipelineArtifact#1
inputs:
targetPath: '$(Pipeline.Workspace)'
artifact: 'azure-deploy.json'
publishLocation: 'pipeline'
"A deployment job doesn't automatically clone the source repo. You can checkout the source repo within your job with checkout: self."
Source: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/deployment-jobs?view=azure-devops
Example from my setup, I put checkout: self as the first step and now my repository is cloned before before executing the Azure PowerShell:
strategy:
runOnce:
deploy:
steps:
- checkout: self
- task: AzurePowerShell#5
displayName: Setup Network
inputs:
It is a good strategy to go for the multi-stage pipelines for what you are doing.
Build is for composing your artifacts.
Deployment jobs is for the publishing part.
So you are on the right track.
If you need sources during the deployment jobs then use the checkout step to fetch the sources. ref. Repo Checkout docs.
Just my two cents
Because you use deployment job the sources from the master branch didn't download into the agent.
You need to publish the files in the build stage and consume them in the deployment - with pipeline artifacts.
Or, just run the AzureResourceGroupDeployment in a regular job, than the .json file will be in the agent.
My objective was to create a Blank and Starter Resource Group in Azure and nothing else. And this was repeatedly giving me the error
Template file pattern matches a directory instead of a file: /home/vsts/work/1/s
And thats what got me here.
I finally sorted that out with Stringfello's
- checkout self
step.
My full pipeline is as follows.
variables:
- name: finalBuildArtifactName
value: 'aspNetCoreDropFromYaml123'
- name: BuildParameters.RestoreBuildProjects
value: '**/*.csproj'
- name: BuildParameters.TestProjects
value: '**/*[Tt]ests/*.csproj'
- name: buildConfiguration
value: 'Release'
trigger:
- master
name: $(date:yyyyMMdd)$(rev:.r)
stages:
- stage: Build ##Basically prints out some vars and copies the template files.
jobs:
- job: buildWebApp
displayName: Build Stage for somepurpose
pool:
vmImage: ubuntu-latest
steps:
- script: |
echo build.artifactstagingdirectory and build.buildnumber are as follows.
echo $(build.artifactstagingdirectory) $(build.buildnumber)
echo $(projects)
echo $(BuildConfiguration)
echo Pipeline.Workspace is $(Pipeline.Workspace)
echo The current branch is - $(Build.SourceBranchName)!!.
echo $(finalBuildArtifactName)
echo "This is the build pipe line. This produces the necessary artifacts for subsequent release pipeline."
displayName: 'Command Line Script to write out some vars'
- powershell: |
# Write your PowerShell commands here.
Write-Host "This is from power shell command task"
Write-Host "This writes out the env vars"
get-childitem -path env:*
displayName: 'PowerShell script to write out env vars'
# The following task is needed. This copies the arm template files.
# Created these templates from Visual studio 2019 as follows.
# Right click your solution and Add -> New Project -> Azure Resource Group and gave the name Vivek_Aks_Rg
- task: CopyFiles#2
inputs:
SourceFolder: 'iac/ArmTemplates/Vivek_Aks_Rg/'
Contents: 'azuredeploy*.json'
TargetFolder: '$(build.artifactStagingDirectory)/ArmTemplates'
- task: PublishBuildArtifacts#1
displayName: Publish Artifact
condition: succeededOrFailed()
inputs:
PathtoPublish: '$(build.artifactstagingdirectory)'
ArtifactName: '$(finalBuildArtifactName)'
PublishLocation: 'Container'
- stage: DeployToDev
displayName: Deploy to Dev Env
jobs:
- deployment:
pool:
vmImage: ubuntu-latest
environment: Dev
strategy:
runOnce:
deploy:
steps:
- checkout: self
- task: AzureResourceManagerTemplateDeployment#3
displayName: 'Create Azure App Service in a Given Resource Group'
inputs:
deploymentScope: 'Subscription'
azureResourceManagerConnection: 'Pay-As-You-Go(123YourSubscriptionId)'
subscriptionId: '123YourSubscriptionId'
action: 'Create Or Update Resource Group'
resourceGroupName: 'YourResourceGroupName'
location: 'Central India'
csmFile: '$(Pipeline.Workspace)/$(finalBuildArtifactName)/ArmTemplates/azuredeploy.json'
csmParametersFile: '$(Pipeline.Workspace)/$(finalBuildArtifactName)/ArmTemplates/azuredeploy.parameters.json'
deploymentMode: 'Incremental'