I have 2 pipelines in 2 different repositories:
lib.yml in repo.git produces a re-usable library artifact A (needs multiple stages on different nodes for that) and also has a stage that runs autotests after A has been produced.
app.yml in app.git builds and tests an application that needs A to build.
In app.yml I want to integrate A without duplicating lib.yml. I was told that templates are a solution, but I am not so sure. Multiple stages of lib.yml must run before I can consume the desired artifact. Using job templates would only complicate both pipelines and create a dependency on pipeline internals. app.yml should not know how lib.yml builds A.
After consulting the docs I think that a pipeline resource is closer to what I need. But I do not fully understand how it works. Let us assume we want artifact A from lib.yml on branch B.
Will app.yml use the latest available artifact A of branch B or will it kick lib.yml on B?
Is there a way to tell app.yml: use the latest A of lib.yml for branch B if available, otherwise run lib.yml on B and wait for A to become ready?
You, you are right - resource pipelines should do a job for you.
And to run app.yml you should have sth like this:
resources:
pipelines:
- pipeline: hadar
source: kmadof.hadar
trigger:
branches:
- B
and use just:
- download: hadar
to get latest artifact from pipeline of branch B.
You can also select pipeline running your app.yaml manual and select artifact:
And - download: hadar also gets correct artifact.
So
app.yml will use latest available artifact A of branch B.
If you want to app.yaml trigger lib.yaml and then get artifact from lib.yaml it would be difficult and not possible using out of the box functionality. So there is no out fo the box way to have app.yaml trigger un lib.yaml and waiting for artifacts from lib.yaml. And if your app yaml will be triggered by an other trigger than resoure pipeline trigger it will use latest available artifact A of branch B.
Related
I have a release pipeline that deploys multiple components using multiple stages. With the current setup i have if a release pipeline triggers then deployment happens in all stages even if new version of artifact is not generated.
My requirement is, deployment into stage should happen only if new version of artifact is generated i.e., out of 10 components if only one component got new version of artifact then the stage respective to that should only triggered and other components should be skipped.
FYI:
i don't want to create separate release pipelines for the components
I think the condition of the tasks should be able to achieve your requirements.
Just give an idea, for example, if you publish new artifact to DevOps feed, you can use this REST API to get the packages versions:
https://learn.microsoft.com/en-us/rest/api/azure/devops/artifacts/artifact-details/get-package-versions?view=azure-devops-rest-6.0
Then, you can compare the latest version that the REST API returned with the artifact version in your release pipeline.
condition: and(succeeded(), eq(<version in pipeline>, <latest version in feed>))
You can refer to this official document:
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/conditions?view=azure-devops&tabs=yaml
If you have continuous deployment triggers enabled for an artifact, the pipeline creates a new release (with all stages) every time. Since the build artifact trigger is abstracted away from the stages, you can't select a stage per artifact.
You said you didn't want to create a separate pipeline per artifact but that really sounds like the correct design. That way only the associated stage(s) will be deployed when the artifact is updated.
I have a .Net project that uses the Azure DevOps pipelines. The setup is that I have a build pipeline that creates an artifact. The artifact then automatically gets published through the release pipeline. This is working perfectly.
The problem is after I turned on the policy Build Validation, pull requests now triggers the build pipeline which then triggers the release pipeline. So every pull requests gets published. The build step is correct, but the release should not happen. The pre-deployment trigger "Pull request deployment" is disabled.
What I did to try to solve this is that I added a condition to the build step where the artifact gets created. So pull requests does not create artifacts, while merges does. This also works as intended. However, the release pipe still gets triggered, but this time without an artifact (which fails the pipe).
TLDR:
Release pipe triggers on pull requests, settings for this behavior is off. WTD?
My CI/CD settings:
Your release triggers on any of your builds and branches (PR also has a branch). You have to add the branch filter: Continuous deployment triggers. Restrict your filter with the master branch or any other. Also, you can define 2 build definitions:
A pipeline to validate your pull requests without linked releases.
CI pipeline that triggers a release.
Additionally, I think, this is a bug. Because the PR trigger is not enabled. Let's check dev community comments: https://developercommunity.visualstudio.com/content/problem/1292039/release-pipelines-ignore-pull-request-settings.html
What we have here is that we've set up build pipelines tied to YAML files stored in the repository, together with source code. And release pipelines have their Source set up to each of the build pipelines.
This is part of the Master Build:
trigger:
batch: false
branches:
include:
- master
And this is part of the Pull Request Build:
trigger: none
pr:
- master
We have Release pipelines for each of the Source builds, having Pull Request triggers enabled in one of them only, but you can have only one for your master artifacts, so PRs won't be published.
We have all our pipelines checked into code, but if we deliver a PR with changes in those pipelines, the PR Build Policies will run with the YAML-files in MASTER, not the ones that we want to check into master. It's basically a deadlock.
Say you want to remove a validation that makes all your PRs fail, so you make a PR but you can't merge it cause the build policies fail :P
PS: I know I can remove the policies, complete the merge, and add the policies back in as a manual job, but that is not really a good solution.
Create a separate yaml pipeline with the pre merge build steps that you then set in the PR policies for. It will always run the code from the current branch that the PR is created from.
We do it like this:
(All in same repo)
build_steps.yml - Yaml template with build steps
azure-pipelines-yml - Main pipeline that has a reference to build_steps.yml to build the project
pre_merge.yml - Secondary pipeline that is run only by PR request which has a reference to build_steps.yml so there are no differences in the build and two places to update if something changes.
Whole yaml definition:
#pre_merge.yml
trigger: none #Pipeline should never trigger on any branches. Only invoked by the policy.
variables:
- name: system.buildConfiguration
value: 'Release'
- name: system.buildPlatform
value: 'win10-x86'
- name: system.debug
value: 'false'
pool:
vmImage: 'windows-latest'
name: $(SourceBranchName)_$(date:yyyyMMdd)$(rev:.r)
steps:
- template: build_steps.yml
And then in the policies you setup it like this:
All this applies also to classic pipelines. You need to create a seperate pre-merge build pipeline that could reference a TaskGroup with the steps used in the main build pipeline. In both cases you dont have to use a template or Taskgroup and and create the steps manually. But if the build would change in the future you have 2 places to update.
Just throwing this out there as an alternative. However; if desired can maintain one yaml pipeline that can do both the pre merge and the deployment.
Would want to create a variable to detect if the pipeline is running against the "master/main" branch similar to:
IsMainBranch: $[eq(variables['Build.SourceBranch'], 'refs/heads/master')]
From there the variable will need to be a condition on subsequent jobs/stages. Personally I do not like this approach as it limits the portability of the pipelines. In addition it interjects another variable to account for; however, felt it fair to provide an alternative option.
Say you have a build and its corresponding release definition on Azure Devops.
Generally, the way it's used is like:
Repository "A" changes -> Build repo "A" -> Release to a given environment
Within this scenario (using only one build artifact), it's easy to trigger deployments (after successful build) for different branches.
But, say you also have a release definition with multiple build artifacts (from different repos), and you want to, changes coming from branch "dev" you want to deploy to environment "Development", but if changes come from branch "hotfix" you want to deploy to environment "Staging".
Something like this
You can play with branch filters on the specific environment and artifact filters on the artifact triggers.
The above scenario works very well for "dev" changes when you define your build artifacts to use "dev" as default branch.
But if you make changes on Repo A (branch hotfix), but not in Repo B, the build artifacts for that release will come from different branches, hence the "artifacts will not meet requirements", cause all changes must come from hotfix branch in order to trigger the deployment.
In my opinion, the "Default Version" of the Build Artifact itself, makes this confusion for the pipeline.
See this image example
That being said, how can we trigger a deployment into "Staging" environment if only one of the triggered changes come from the branch "hotfix" ?
The way I can see, when that happens, "Staging" environment never "meets the artifacts requirements", cause other artifacts will use version from "dev" as it's the default source branch.
Has anyone worked with a similar scenario ?
Thanks for reading my question.
Update:
According to your scenario, the best solution at present is to create two release pipelines to deploy separately, because Artifact A and Artifact B have no dependencies.
In the Continuous deployment trigger of the build artifact, you can set Build branch filters to include the dev and hotfix branches.
Then enable Artifact filters in the Pre-deployment conditions of the stage to set the build branch artifact conditions that trigger the deployment.
Artifact filters: Select artifact condition(s) to trigger a new deployment. A release will be deployed to this stage only if all artifact conditions match.
A build pipeline can be tied to only 1 source branch.
In release pipeline, we configure an artifact by selecting the source build pipeline. So a release artifact can be tied to only 1 build pipeline.
What is the purpose of Build branch filters in Continuous deployment trigger?
Let say that your code base was updated. You want deploy only if build was made over specific branch (example develop).
In that scenario Build Branch Filter looks fairly redundant... but what if:
You want to trigger new deployment for every latest artifact, built from each feature branch to get them tested. You have to filter for feature/*.
You want to trigger new deployments if your release branches are updated, but some of them are deprecated and you need to filter them out. I that scenario specify one include filter release/* and second exclude filter release/old*.
Anyway.. in most of the cases the filter and the branch name (in artifact) will match each other. Still sometimes it can be heady to trigger deployment from multiple branches or filter something out.
In regards to your comment, I uploaded part of the yaml build. In fact one build can create artifacts from all branches in repository if you want.
trigger:
branches:
include:
- feature/*
- bugfix/*
- release/*
- develop
- master
exclude:
- experimental/*