Currently, our inhertited pipelines (not the best designed, but it's what I've got for now), look something like this:
build -> provision + deploy (pipeline per env) -> acceptance test
I only want to be able to do one deployment at a time, so I'm looking at Exclusive Locks.
Unfortunately, these seem to work at the step level, not the pipeline level. So given my provision + deploy pipeline contains multiple steps, how can I prevent step 1 from pipeline run 1 running then step 1 from pipeline 2 running etc?
I can't see a way to apply the exclusive lock at the pipeline level.
Our QA provision + deploy pipeline yml looks like this (before adding the locks):
pool:
vmImage: "ubuntu-latest"
resources:
pipelines:
- pipeline: apiBuild
source: "API/Build"
trigger:
enabled: true
branches:
include:
- main
trigger: none
pr: none
variables:
- template: _pipeline/variables/allVariables.yml
parameters:
deployEnvironment: Qa
stages:
- stage: provisionInfrastructureTemplates
displayName: Provision Templates infrastructure
dependsOn: []
jobs:
- template: _pipeline/stages/jobs/provisionTemplates.yml
- stage: templatesAcceptanceTests
displayName: Templates acceptance tests
dependsOn: provisionInfrastructureTemplates
jobs:
- template: _pipeline/stages/jobs/acceptanceTestsTemplates.yml
- stage: provisionInfrastructureClients
displayName: Provision Clients infrastructure
dependsOn: []
jobs:
- template: _pipeline/stages/jobs/provisionClients.yml
- stage: clientsAcceptanceTests
displayName: Clients acceptance tests
dependsOn: provisionInfrastructureClients
jobs:
- template: _pipeline/stages/jobs/acceptanceTestsClients.yml
- stage: provisionInfrastructureReports
displayName: Provision Reports infrastructure
dependsOn: []
jobs:
- template: _pipeline/stages/jobs/provisionReports.yml
- stage: reportsAcceptanceTests
displayName: Reports acceptance tests
dependsOn: provisionInfrastructureReports
jobs:
- template: _pipeline/stages/jobs/acceptanceTestsReports.yml
- stage: upgradePreviewImage
displayName: Upgrade preview image
dependsOn: []
jobs:
- template: _pipeline/stages/jobs/upgradePreviewImage.yml
- stage: provisionInfrastructureDocuments
displayName: Provision Documents infrastructure
dependsOn: upgradePreviewImage
jobs:
- template: _pipeline/stages/jobs/provisionDocuments.yml
- stage: documentsAcceptanceTests
displayName: Documents acceptance tests
dependsOn: provisionInfrastructureDocuments
jobs:
- template: _pipeline/stages/jobs/acceptanceTestsDocuments.yml
- stage: provisionInfrastructureNotifications
displayName: Provision Notifications infrastructure
dependsOn: []
jobs:
- template: _pipeline/stages/jobs/provisionNotifications.yml
- stage: provisionEventGridSubscriptions
displayName: Provision Event Grid Subscriptions
dependsOn: [clientsAcceptanceTests, templatesAcceptanceTests, reportsAcceptanceTests, documentsAcceptanceTests, provisionInfrastructureNotifications]
jobs:
- template: _pipeline/stages/jobs/provisionEventGridSubscriptions.yml
- stage: workflowTests
displayName: Workflow tests
dependsOn: provisionEventGridSubscriptions
jobs:
- template: _pipeline/stages/jobs/workflowTests.yml
As an aside, I know our services ought to be independently deployable, they're not currently, that's part of the tech debt we're dealing with so, as things stand, they need deploying together.
When you put the jobs in separate stages. The exclusive lock of the environment will be judged separately.
It is not able to apply the exclusive lock at the pipeline level.
To meet your requirement, you can define required the jobs in the same stage. The Exclusive Lock will be judged before running.
When set the environments exclusive lock in the same stage, all environments referenced in a stage will be locked. Only when the current Pipeline Run Stage finished, the next Pipeline run will continue to run.
Here is an example:
stages:
- stage: Test
displayName: Test
lockBehavior: sequential
jobs:
- deployment: Test1
displayName: 'Test'
environment: Test2
strategy:
runOnce:
deploy:
steps:
- script: echo "Test"
- deployment: Test
displayName: 'Test2'
environment: Test2
strategy:
runOnce:
deploy:
steps:
- script: echo "Test"
Related
I have two stages in my Azure DevOps pipeline. One with Pulumi Preview (let's call it Preview) and one with Pulumi Up (Up) in order to run my infrastructure as code.
Both run from the same container and it takes a while to pull it. I want to manually approve the Preview before the implementation.
Can I pull and run the container for both stages simultaneously but wait for the last job of the UP-Stage until the Preview-Stage is approved?
Currently they depend on eachother as follows:
trigger:
- master
pool:
vmImage: 'ubuntu-latest'
stages:
- stage: Pulumi_Preview
jobs:
- job: Preview
container:
image: REGISTRY.azurecr.io/REPOSITORY:latest
endpoint: ServiceConnection
steps:
- task: Pulumi#1
displayName: pulumi preview
inputs:
azureSubscription: 'Something'
command: 'preview'
args: '--diff --show-config --show-reads --show-replacement-steps'
stack: $(pulumiStackShort)
cwd: "./"
- stage: Pulumi_Up
displayName: "Pulumi (Up)"
dependsOn: Pulumi_Preview
jobs:
- job: Up
container:
image: REGISTRY.azurecr.io/REPOSITORY:latest
endpoint: ServiceConnection
steps:
- task: Pulumi#1
inputs:
azureSubscription: 'Something'
command: 'up'
args: "--yes --skip-preview"
stack: $(pulumiStackShort)
cwd: "./"
You could use the Manual Validation Task.
Use this task in a YAML pipeline to pause a run within a stage, typically to perform some manual actions or validations, and then resume/reject the run.
jobs:
- job: waitForValidation
displayName: Wait for external validation
pool: server
timeoutInMinutes: 4320 # job times out in 3 days
steps:
- task: ManualValidation#0
timeoutInMinutes: 1440 # task times out in 1 day
inputs:
notifyUsers: |
test#test.com
example#example.com
instructions: 'Please validate the build configuration and resume'
onTimeout: 'resume'
I am working on having one yaml template file job dependOn a deployment job before continuing. I need my "other-template" job below to depend on the deployment job name "DeployApi"
# api-deploy.yml
jobs:
- deployment: DeployApi
...
release.yml
stages:
- stage: 'deploy_dev'
displayName: Deploy to dev environment
dependsOn: []
jobs:
- template: api/api-deploy.yml
parameters:
environment: ...
varTemplate: ...
- template: other/other-template.yml
parameters:
dependsOn: ['DeployApi']
Is it possible to have the second template "other/other-template.yml" depend on the "DeployApi" deployment job? I know it's possible if it was a standard job name but I couldn't find any information on if it's possible to use dependsOn for a deployment job name.
Little bit more information: I have about 10 release templates that run parallel (I believe). I have one release template that I want to wait to run until every other template is ran.
Yes, you can.
release.yml
trigger:
- none
pool:
vmImage: ubuntu-latest
stages:
- stage: 'deploy_dev'
displayName: Deploy to dev environment
dependsOn: []
jobs:
- template: api/api-deploy.yml
parameters:
environment: ...
varTemplate: ...
- template: other/other-template.yml
parameters:
dependsOn: ['DeployApi']
api/api-deploy.yml
jobs:
- job: DeployApi
steps:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
# Write your PowerShell commands here.
Write-Host "This is api-deploy.yml."
other/other-template.yml
parameters:
dependsOn: []
jobs:
- job: getDependsOn
dependsOn: ${{parameters.dependsOn}}
steps:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
# Write your PowerShell commands here.
Write-Host "This is other-template.yml"
Result successfully:
I'm trying to integrate a QA pipeline in Azure DevOps that triggers during a dev pipeline. The dev pipeline has 4 environments that the build is deploying to, where each environment is a stage in the dev pipeline
Currently, I'm able to trigger the QA builds for each environment using 4 separate QA pipelines using syntax similar to this in each pipeline:
resources:
pipelines:
- pipeline: Dev_Env_1
source: Dev
trigger:
stages:
- Env_1
My goal is to only have one QA pipeline that is triggered multiple times by the dev pipeline when it completes each stage. It feels like syntax in the yml file like this should work:
resources:
pipelines:
- pipeline: Dev_Env_1
source: Dev
trigger:
stages:
- Env_1
- pipeline: Dev_Env_2
source: Dev
trigger:
stages:
- Env_2
However, this only triggers after Env_1 is completed, when I would like a build triggered for the completion of the Env_1 and Env_2 stages in the Dev pipeline.
Is there a way to do this without drastically changing the way either pipeline currently works?
The below steps can help you achieve your requirements.
For example,
Dev Pipeline
#Dev Pipeline
trigger:
- none
pool:
vmImage: 'windows-latest'
stages:
- stage: Env_1
displayName: Env_1
jobs:
- job:
steps:
- task: CmdLine#2
inputs:
script: |
echo Stage1
- stage: Env_2
displayName: Env_2
jobs:
- job:
steps:
- task: CmdLine#2
inputs:
script: |
echo Stage2
- stage: Env_3
displayName: Env_3
jobs:
- job:
steps:
- task: CmdLine#2
inputs:
script: |
echo Stage3
- stage: Env_4
displayName: Env_4
jobs:
- job:
steps:
- task: CmdLine#2
inputs:
script: |
echo Stage4
1, Create an incoming webhook service connection, and an incoming service webhook that is triggered by the complete and success of stages of the Dev Pipeline.
incoming service hook URI: https://dev.azure.com/<ADO Organization>/_apis/public/distributedtask/webhooks/<WebHook Name>?api-version=6.0-preview
Official document:
https://learn.microsoft.com/en-us/azure/devops/release-notes/2020/pipelines/sprint-172-update#generic-webhook-based-triggers-for-yaml-pipelines
After that, write QA pipeline as below:
QA Pipeline
# QA pipeline
trigger:
- none
resources:
webhooks:
- webhook: bowmantest
connection: BowmanIncommingWebHook
pool:
vmImage: 'windows-latest'
steps:
- task: CmdLine#2
inputs:
script: |
echo QA pipeline
Everything works fine:
Azure Pipelines support multiple stages in YAML. One typical example would be to have something like :
trigger:
- master
pool:
name: Default
demands:
- npm
- msbuild
- visualstudio
stages:
- stage: build
jobs:
- job: Build app
- stage: deploy
jobs:
- job: Deploy to dev
I'm not used to working like this. Usually, I would run the pipeline to build my application and drop artifacts to a drop folder. The pipeline would be the same regardless the environment that would later be targeted by the release.
Then I would choose to run a release, either Integration, UAT, or Production.
However, having multi-stages pipeline we are mixing the build and the release together. So how would I release in a given environment ? Do I have to duplicate this pipeline per environment ?
You can use template structure here. In this you will need to create separate files for different jobs and variables. Then you will need to execute templates with suitable variable template files for each stage.
Directory structure
Directory Structure
Pipeline:
Pipeline stages
Environments
Environments
Here's the sample pipeline:
trigger:
- master
variables:
- name: vmImage
value: 'ubuntu-latest'
stages:
- stage: Build
displayName: Build stage
jobs:
- job: BuildJob
pool:
vmImage: $(vmImage)
steps:
- template: Jobs/build.yml
- stage: NonProd
displayName: Deploy non prod stage
jobs:
- deployment: DeploymentJob1
pool:
vmImage: $(vmImage)
environment: non-prod
variables:
- template: Variables/non-prod.yml
strategy:
runOnce:
deploy:
steps:
- template: Jobs/deploy.yml
- stage: Prod
displayName: Deploy prod stage
jobs:
- deployment: DeploymentJob2
pool:
vmImage: $(vmImage)
environment: prod
variables:
- template: Variables/prod.yml
strategy:
runOnce:
deploy:
steps:
- template: Jobs/deploy.yml
Jobs/build.yml
steps:
- script: echo I am building!
displayName: 'Run Build'
Jobs/deploy.yml
steps:
- script: echo I am deploying to $(Environment)!
displayName: 'Run Deployment'
Variables/non-prod.yml
variables:
- name: Environment
value: non-prod
Variables/prod.yml
variables:
- name: Environment
value: prod
I have two .yml files in my repo. One for build, one for deployment. The main reason why I would like to keep build separate from the deployment is that I also would like to store variables for environments in my repo, e.i. in variables-dev.yml and variables-prod.yml files. So there is no need to create a new build every time (which includes running tests, docker image build etc.).
The file build.yml:
trigger:
paths:
exclude:
- build.yml
- deploy.yml
stages:
- stage: build
jobs:
...
And the deploy.yml, which I want to be triggered only on the completion of the build pipeline. That's why I add the first exclusion of all paths, but add one on pipeline resource.
trigger:
paths:
exclude:
- '*'
resources:
pipelines:
- pipeline: build
source: build
trigger:
branches:
include:
- '*'
stages:
- stage: dev
variables:
- template: variables-dev.yml
jobs:
- deployment: deploy_dev
environment: 'dev'
strategy:
runOnce:
deploy:
steps:
...
- stage: prod
dependsOn: dev
variables:
- template: variables-prod.yml
jobs:
- deployment: deploy_prod
environment: 'prod'
strategy:
runOnce:
deploy:
steps:
...
Unfortunately it does not seem to work. The top trigger blocks lower trigger. And if I remove the top trigger than the deploy pipeline is triggered at the same time with the build one.
you have to start your deploy.yml with trigger: none
trigger: none
resources:
pipelines:
- pipeline: ci-pipeline
source: my-build-pipeline
trigger:
enabled: true
branches:
include:
- master
Set your triggers for the second yml to none, then add this setting in the "Triggers" section of the UI. It will stage your builds as you describe