Parallelisation of Azure Pipelines - azure

I have a pipeline (in YAML) which upgrades an infrastructure(I have 2 stages each containing a series of jobs)
I now want to upgrade multiple infrastructures simultaneously i.e. pass a list of identifiers which represents deployments to the pipeline and then let it upgrade each.
What is the best practice here for organising the pipeline? It feels like I need to generate a set of parallel jobs using a loop?
As I understand it any job failure will result in a total failure which could leave us in a very confused state.

If you purchased parallel jobs for your organization. You can use Template to generate multiple jobs according the identities parameter using expression ${{each id in parameters.identities}}.
So you can move the job which upgrades the infrastructures into a template and define your yaml pipeline as below. See below example:
Template file: upgrade-infrastructure.yml
parameters:
id: 1
jobs:
- job: upgradeinfra${{parameters.id}}
steps:
- powershell: echo "upgrade-infra-${{parameters.id}}"
azure-pipelines.yml:
#define the identities as a object to hold a array object of ids
parameters:
- name: identities
type: object
default:
ids:
- 1
- 2
trigger: none
stages:
- stage: Upgrage
pool:
vmImage: windows-latest
jobs:
- job: A
steps:
- powershell: echo "job A"
#loop through the ids array object and the each id to the template paramter to generate multiple jobs for each id.
#indentation is very important, bad indentation may cause pipeline compile error.
- ${{ each id in parameters.identities.ids }}:
- template: upgrade-infrastructure.yml
parameters:
id: ${{id}}
After you set up your yaml pipeline as above, you can enter the identities in the parameter when executing the pipeline:
Then you will see the multiple jobs are generated and run in parallel:

To make your deployments run in parallel, all you need to do is to set the dependencies. (The dependency on the previous step is automatically set). Here is an example of a stage that only depends on the build before all stage will run in parallel:
stage:
-stages : DeployTo${{ parameters.environment }}
dependsOn: ["Build"] //The stage that build the code is called "Build"
The Result looks like this:
Without the dependsOn property your pipeline stages will run sequentially and looks like this:
stages:
-stage : DeployTo${{ parameters.environment }}

Related

How to get yaml-pipeline demands?

How to get the demands of a pipeline created from a yaml-file? The yaml-file contains the demands:
...
jobs:
- job: my_job
displayName: My Job
pool:
name: my_pool
demands:
- Agent.Name -equals My-Agent-1
step:
...
If the pipeline is created through the user interface, then I can get the demands using the get-request. The response body will contain the demands.
But if I usually use the get-request for a yaml-pipeline, then the response body will not contain the demands! Do I need to parse the yaml-file myself?!
Unfortunately, I did not find an answer to my question.
REST API Definitions - List doesn't support get the definitions inside the YAML. You need to parse the yaml-file yourself.

Azure YAML if-else always fail

I am recently doing an CI/CD setup using Azure. The goal is to have the developer select the type of build to be created i.e Staging / Prod.
Thanks to How to write if else condition in Azure DevOps Pipeline, I have added following code -
parameters:
- name: selectConfiguration
displayName: Select build configuration
type: string
default: Debug
values:
- Debug
- Release
variables:
- name: config
${{ if eq(variables['parameters.selectConfiguration'], 'Debug') }}:
value: Debug
${{ else }}:
value: Release
This gives me the following result -
But no matter what I select in this radio group, it always run the else block. i.e. the if-else always fails. Any help to understand what I am doing wrong here?
Try the below, it should work. I am using the same logic to switch between different agent pools.
variables:
${{ if eq(parameters.selectConfiguration, 'Debug') }}:
config: Debug
${{ else }}:
config: Release
In YAML pipeline, you can not use the if...else expression to assign different values to a variable in different conditions.
You can only use the if expression to determine a variable which has one specified value can be available in the specified condition. See "Conditionally assign a variable".
The if...else expression can be used to:
assign different values to an input of the task in different conditions. See "Conditionally set a task input".
run different steps in a job in different conditions. See "Conditionally run a step".

Run only one build pipeline task on a schedule

Is it possible to run only one build pipeline task on a schedule and not the whole pipeline on a schedule? I have a task in a build pipeline to generate some report about the pipeline. I would want the task to run once every month.
Yes.. it is possible in several ways:
you can set previously to the task the condition:
${{ if eq(variables['isBuild'], true) }}:
You can configure conditions to run the tasks that you want, depending of whatever, for example in this case using the variable isBuild, defined previously:
task: PublishBuildArtifacts#1
displayName: 'Publish artifact: drop'
inputs:
PathtoPublish: 'whatever'
ArtifactName: 'drop'
publishLocation: 'Container'
condition: eq(variables['isBuild'], true)
You can configure conditions to run the stages that you want, so previously you can group your tasks in stages, depending of whatever, for example using in this case in stage the variable isBuild, defined previously:
stage: Build
displayName: 'Build'
condition: eq(variables['isBuild'], true)
In every example in case of IsBuild be different to true will not be run.
You can find more info in https://github.com/MicrosoftDocs/azure-devops-docs/blob/main/docs/pipelines/process/conditions.md
Also If you want to schedule your task to be executed only 1st day of every month at 07:00:
trigger:
master #This is the trigger for other stages. It is not needed for the scheduled stage.
schedules:
cron: '0 7 1 * *'
displayName: 'Deploy every 1st day of every month at 07:00Z'
branches:
include:
main
always: true
Then to ensure that task runs as part of schedule, use the condition:
- stage: 'Test'
displayName: 'Deploy to the test environment'
dependsOn: Dev
condition: eq(variables['Build.Reason'], 'Schedule')
For more detail you can to to:
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/scheduled-triggers?view=azure-devops&tabs=yaml#scheduled-triggers
Put the task in a second pipeline, and run that on a monthly schedule.
If you want to avoid yaml code duplication, you could define a template containing that one task and have that template called from both pipelines.

Run Azure DevOps CI Pipeline with Scheduler and Variable Group

We have one pipeline and one only (we cannot and do not want to create a 2nd pipeline or do it with a separate pipeline, it has to be done in the same pipeline), that pipeline has a task to either stop or start a function while accepting a variable group (its required) from the library (we specify those store A-E variables on the YAML but they also exist in the library), and specify the Azure subscription. Currently, we run this pipeline manually, this is what it looks like before I run it
What I'm looking for is a feature to automate this pipeline to run at 7 PM CST with the start function as the task, accept a variable group, and specify which azure subscription that i want. Then, at 6 AM CST the next day, I need to have that SAME pipeline to run a build with stop function as the task, accept a variable group, and specify which azure subscription i want.
I found a scheduler feature in the CI pipeline but it doesnt allow me to specify which variable group I want from the library, no option to select either start or stop the function, and no option to select the subscription. This is what I'm expecting to see
If it any helps this is the .YAML code that i have (some stuff has been removed for privacy purposes)
trigger:
- none
pool:
vmImage: 'windows-latest'
parameters:
- name: variableGroup
displayName: Variable Group
type: string
values:
- 'variable for store A'
- 'variable for store B'
- 'variable for store C'
- 'variable for store D'
- 'variable for store E'
- name: artifactVersion
displayName: ArtifactVersion (* (latest) or 1.{sprintNumber}.{ReleaseNo})
type: string
default: '*'
- name: Function
displayName: Function
type: string
default: 'deploy'
values:
- deploy
- name: task
displayName: ExecuteTask
type: string
default: ''
values:
- start thefunction
- stop the function
- name: Subscription
displayName: Subscription
type: string
values:
- 'sandbox'
- 'production '
I am afraid that there is no such method can meet your requirements for the time being.
Refer to this doc: Scheduled triggers
schedules:
- cron: string # cron syntax defining a schedule
displayName: string # friendly name given to a specific schedule
branches:
include: [ string ] # which branches the schedule applies to
exclude: [ string ] # which branches to exclude from the schedule
always: boolean # whether to always run the pipeline or only if there have been source code changes since the last successful scheduled run. The default is false.
The Schedule trigger does not support setting the target value for Parameters.
On the other hand, when you set the runtime parameters, the option or drop-down list to select the value can only be displayed when the pipeline is run manually.
I can fully understand your requirements.
You could add your request for this feature on our UserVoice site (https://developercommunity.visualstudio.com/content/idea/post.html?space=21 ), which is our main forum for product suggestions:

How to combine Git branch and tag triggers in Azure Pipelines

Context
Azure Pipelines supports different kinds of CI triggers (docs).
Example A:
trigger:
branches:
include:
- master
The pipeline will run if new commits are detected in the master branch.
Example B:
trigger:
branches:
include:
- refs/tags/v*
The pipeline will run if new tags beginning with v are detected.
I want my pipeline to run when both of the above conditions are true: branch and tag.
Example C (not working as desired):
trigger:
branches:
include:
- master
- refs/tags/v*
Combining the two triggers seems to act like an or-condition, not an and-condition.
Question
How can I trigger a pipeline on the master branch only when there are new v* tags?
In my opinion it is not possible. Following documentation:
If you specify tags in combination with branch filters that include file paths, the trigger will fire if the branch filter is satisfied and either the tag or the path filter is satisfied.
Unfortunatelly adding condition will not help here because we don't have sufficient information. For instance for tag trigger we have this
BUILD_SOURCEBRANCH=refs/tags/release-07
BUILD_SOURCEBRANCHNAME=release-07
and for branch trigger this:
BUILD_SOURCEBRANCH=refs/heads/master
BUILD_SOURCEBRANCHNAME=master
And even if you try to check manually branch name with git branch you will get:
* (HEAD detached at 154ce86)
For me this is a good candidate for a feature request at developer community.
I found a way of doing this by using templates and parameters.
Example
First, create a common.yaml with parameterized tasks and scripts:
parameters:
- name: myMessage
type: string
default: 'Hello from template!'
- script: |
echo ${{ parameters.myMessage }}
This yaml can contain any number of parameters, shared tasks and scripts.
Then, create a master.yaml for the master branch:
trigger:
branches:
include:
- master
extends:
template: common.yml
parameters:
myMessage: 'Hello from master!' # Overrides default parameter value
Create another tags.yaml to run for each new version tag:
trigger:
branches:
include:
- refs/tags/v*
extends:
template: common.yml
parameters:
myMessage: 'Hello from tags!' # Overrides default parameter value
Lastly, create two Pipelines within Azure DevOps web interface. Connect one of the Pipelines to master.yaml and the other to tags.yaml.
With this setup, each trigger is independent with minimal yaml duplication.

Resources