I am using different YAML files (with different tasks) in different branches.
While triggering it executes multiple times. I don't know what is the reason.
I am using different triggers for different pipelines. eg:
In branch SFCC:
trigger:
- SFCC
In branch SFCC-QA:
trigger:
- SFCC-QA
In branch SFCC-DEV:
trigger:
- SFCC-DEV
When I run the pipeline for SFCC triggered, the same pipeline would start running in another pipeline also with same trigger, because of that app is deploying 2-3 times instead of single time.
Could you please help me with this.
SFCC-QA pipeline:
SFCC pipeline:
You can see in first screenshot(for the date 9th sep and 14th sep), the pipeline is for QA and here pipeline is executing for other branch, which is not mentioned in this particular pipeline. While in the second screenshot pipeline is executing fine for the same date. So you can see 2 times same pipeline is execting on 9th and 14th sep.
Azure build pipeline definition is associated with a repo and YAML file in that repo. The branch is associated at the time only when the pipeline is run. So, it is not something you associate at the time of pipeline definition.
I think in all 3 of your pipelines, the YAML file path is same. You have different trigger (and different other content) in the YAML file in different branches, but your file name/path is same. Hence the same file will trigger the build for each branch where relevant within pipeline.
To get desired results you can use different file name for your YAML file for each pipeline definition. So in SFCC branch have SFCC.yaml and in SFCC-QA branch, have SFCC-QA.yaml.
Related
Context
I’m using CI for developing a solution using a monorepo. It has two custom python libraries, and high level orchestration scripts using these packages.
The CI is split into 3 parts: build, test and deploy.
During build, I create an image (tag being a fixed name pattern followed by pipeline id) in ECR (staging repository) with kaniko.
This "base" image is then used in test stage to run unit tests and integration tests.
These two stages run always for any merge request into default branch (main). If both of these pass, I’ll merge the development branch into main branch, and a new pipeline gets triggered post merge.
This will repeat the build and test stage, and then deploy stage starts.
Some very high level wrapper scripts are added to the "base" image created in build stage and publish it with a versioned tag (identified by a fixed name pattern followed by commit id) and also as latest tag in ECR (deploy repository).
Question
Everything works in this approach, but I don't like the fact that the build and test stages repeat between pre-merge (triggered as merge request pipeline) and post-merge (triggered as branch pipeline) pipelines. These are identical time consuming steps, and shall always be identical as we allow only fast forward merges and each merge creates a merge commit. Is there a way to handle this scenario?
Issues
The main challenge I’m facing is how to identify the "base" image created during build stage (identified by pipeline id). If I can do that, it becomes simple. But I do not know how to get that pipeline id. I have to use an identifier in image tag as a lot of people are working and there can be simultaneous pipelines by other people, but as soon as I use pipeline id or commit id, it is becoming impossible to track those after merge.
I have a Release pipeline with three environments. I have a single build pipeline (_TestCM) that builds different branches (develop, Master, release). All branch builds are CI builds.
I applied an artifact filter to each environment.
I want to deploy
Develop branch --> Environment 1 at 11:30 AM, 3 PM daily
Master branch --> Environment 2 at 5 AM, 7 PM daily
Release branch --> Environment 3 at 6 AM, 2 PM daily.
How can I configure the above scheduling requirement in a single release pipeline? Creating multiple release pipelines with single environment in each pipeline is not an option as we have hundreds of release pipelines. Release pipeline need to trigger at the scheduled time only if there are any changes from that branch.
If it is not possible in a single Release pipeline, what are the options we have for accomplishing this task considering the hundreds of Release pipelines?
Based on your requirement, I am afraid that it is not possible in a single release pipeline.
Release pipeline need to trigger at scheduled time only if there any changes from that branch.
For this requirement, there is an Option in Release Pipeline(
Only schedule releases if the source or pipeline has changed).
According to your deploy process, you need to determine the environment via the branches. So you need to add multiple artifacts for multiple branches.
In this case, this option will not meet your needs. It treats all atifacts as a whole. When one of the branches changes, the release will be triggered by the schedule and all artifacts of different branches will be deployed at the same time.
You can achieve your needs by creating multiple pipelines.
The following are the settings you can refer to：
When you add artifacts, you can select the branch and enable the option:Only schedule releases if the source or pipeline has changed.
For example:
In environemnt, you can directly set schedule trigger without other settings.
You need to create the corresponding number of release pipelines according to your branches.
I have created a pipeline in my repository which is used to validate code by executing unit tests for code that is being pushed to features/* branches. The same pipeline is used as Build validation pipeline set as Branch Policy on the develop branch to validate incoming PRs. This is the trigger of the pipeline.
# pipeline.yml
trigger:
batch: false
branches:
include:
- features/*
However we have come across the following condition: Given an open PR from refs/heads/features/azure-pipelines -> refs/heads/develop we push a commit on the features/azure-pipelines branch.
This causes the pipeline to trigger twice. To my understanding one of the runs is due to the trigger of the pipeline (The one marked as Individual CI on the screenshot) and the second run is due to branch policy trying to validate code being pushed onto the open PR to develop. (The PR Automated)
Is there any way to disable one of the executions since it's essentially a duplicate? I was maybe looking for a way to retrieve open PRs and abort execution of a pipeline for Individual CI if there is an open PR for a branch but I am not sure that's the best way around that and I am looking for options.
You can set
trigger: none
This way only the branch policy will trigger the pipeline.
Is there any way to disable one of the executions since it's essentially a duplicate?
As we know, we could not disable the Build validation pipeline set as Branch Policy on the develop branch to validate incoming PRs unless we cancel the Build validation.
For your situation, you could try to include [skip ci] in the commit message or description of the HEAD commit to make the Azure Pipelines skip running CI when you plan to merge the features branch to the develop branch.
You could check the document Skipping CI for individual commits for some more details.
Here it depends if they does the same. You can have conditional checks in the pipeline which does a different things for PR and CI runs. However, I'm pretty sure that this is not possible, because one is defined on the YAML and the second on the Azure DevOps portal. So even if you disnle PR trigger here in YAML, a branch policy still runs a PR. And you can specify antyhing in YAML to block branch policy.
I have a Question about the Pipline trigger in Devops.
My team and I are using Azure Devops to develop a software.
We use a Branch Specific trigger, to start the pipeline only in out Master Branch. The other branches are ignored in the YAML File.
First part of my question is, that we don't know a way how to trigger the Pipeline over a commit message in our Git tool.
For example: "We work in a different branch than the Master branch -->No Pipeline is running. But we want to trigger the pipeline in this Branch for a Specific test just one time. Our way would be to insert a specific command in the commit text to trigger the pipeline."
My second question is, if it's possible to run different stages in different Branches in one YAML file.
Here again an example: " In our different Branch, we just want to run our unit tests every push. In our Master Branch we want to run our unit tests and after that, we want to build our application.
So far, we started the pipeline at every push and build a new image everytime. But we dont want that, because some pushs arent working and we just push it. We want to decide when the pipeline is running and whitch stage is running.
I hope you can understand my problem. For further questions, please comment here.
Thanks
Question 1:
You can consider using tag triggers to do this. Here is an example:
trigger:
branches:
include:
- master
tags:
include:
- test.*
Then the pipeline will be triggered when working on the master branch or the commit tag is test.*.
Question 2:
You can use conditions. Here is an example:
condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/master'))
Add the condition to your stage, and then the stage will be only triggered by the master branch.
Question 3:
So far, we started the pipeline at every push and build a new image everytime. But we dont want that, because some pushes aren't working and we just push it.
You can skip CI for individual commits.
Just include [skip ci] in the commit message or description of the
HEAD commit and Azure Pipelines will skip running CI.
Update 1:
For Question 1, the test.* means a tag which started with test., such as test.1, test.0.1, and so on. You can change the test.* to anything you want.
As for the question your encountered, you can't create a tag called test.* directly because a tag name cannot contain *.
To avoid confusion, you need to create a tag for the commit to trigger the tag CI, rather than writing it directly in the commit text.
The idea
insert a specific command in the commit text to trigger the pipeline.
I don't think is currently supported and tag trigger is an alternative.
Click this document for detailed information about git tag.
Update 2:
Trigger the stage by master branch or 1620-to-PipelineTrigger branch:
condition: or(eq(variables['Build.SourceBranch'], 'refs/heads/main'), eq(variables['Build.SourceBranch'], 'refs/heads/1620-to-PipelineTrigger'))
You can set only one condition per stage, but you can create more complex conditions using and, or and ().
Click this document to get more detailed information and examples about conditions.
I have 2 pipelines in 2 different repositories:
lib.yml in repo.git produces a re-usable library artifact A (needs multiple stages on different nodes for that) and also has a stage that runs autotests after A has been produced.
app.yml in app.git builds and tests an application that needs A to build.
In app.yml I want to integrate A without duplicating lib.yml. I was told that templates are a solution, but I am not so sure. Multiple stages of lib.yml must run before I can consume the desired artifact. Using job templates would only complicate both pipelines and create a dependency on pipeline internals. app.yml should not know how lib.yml builds A.
After consulting the docs I think that a pipeline resource is closer to what I need. But I do not fully understand how it works. Let us assume we want artifact A from lib.yml on branch B.
Will app.yml use the latest available artifact A of branch B or will it kick lib.yml on B?
Is there a way to tell app.yml: use the latest A of lib.yml for branch B if available, otherwise run lib.yml on B and wait for A to become ready?
You, you are right - resource pipelines should do a job for you.
And to run app.yml you should have sth like this:
resources:
pipelines:
- pipeline: hadar
source: kmadof.hadar
trigger:
branches:
- B
and use just:
- download: hadar
to get latest artifact from pipeline of branch B.
You can also select pipeline running your app.yaml manual and select artifact:
And - download: hadar also gets correct artifact.
So
app.yml will use latest available artifact A of branch B.
If you want to app.yaml trigger lib.yaml and then get artifact from lib.yaml it would be difficult and not possible using out of the box functionality. So there is no out fo the box way to have app.yaml trigger un lib.yaml and waiting for artifacts from lib.yaml. And if your app yaml will be triggered by an other trigger than resoure pipeline trigger it will use latest available artifact A of branch B.