Context
I’m using CI for developing a solution using a monorepo. It has two custom python libraries, and high level orchestration scripts using these packages.
The CI is split into 3 parts: build, test and deploy.
During build, I create an image (tag being a fixed name pattern followed by pipeline id) in ECR (staging repository) with kaniko.
This "base" image is then used in test stage to run unit tests and integration tests.
These two stages run always for any merge request into default branch (main). If both of these pass, I’ll merge the development branch into main branch, and a new pipeline gets triggered post merge.
This will repeat the build and test stage, and then deploy stage starts.
Some very high level wrapper scripts are added to the "base" image created in build stage and publish it with a versioned tag (identified by a fixed name pattern followed by commit id) and also as latest tag in ECR (deploy repository).
Question
Everything works in this approach, but I don't like the fact that the build and test stages repeat between pre-merge (triggered as merge request pipeline) and post-merge (triggered as branch pipeline) pipelines. These are identical time consuming steps, and shall always be identical as we allow only fast forward merges and each merge creates a merge commit. Is there a way to handle this scenario?
Issues
The main challenge I’m facing is how to identify the "base" image created during build stage (identified by pipeline id). If I can do that, it becomes simple. But I do not know how to get that pipeline id. I have to use an identifier in image tag as a lot of people are working and there can be simultaneous pipelines by other people, but as soon as I use pipeline id or commit id, it is becoming impossible to track those after merge.
Related
In their documentation regarding pull request pipelines, bitbucket says:
Pull requests:
a special pipeline that only runs on pull requests initiated from within your repository. It merges the destination branch into your working branch before it runs. If the merge fails, the pipeline stops.
So I'm wondering, why merging before running the pipeline? Why not just running against the coming branch without merging?
Could the reason be detecting merge conflicts early on in the pipeline before the real merge?
If you want to run a pipeline against the coming branch, this is very doable by using Branch workflows. PR merge trigger is just a slightly different idea, as the result of a PR merge is not necessarily the same as the coming branch. For example, merge conflicts can be introduced, which will make your pipeline fail.
There's one thing that documentation is not quite clear about, so I'll clarify it: all this pre-pipeline merging only occurs inside your build environment. Git history of your repository is absolutely safe, and Bitbucket Pipelines won't introduce any changes to it on your behalf.
Finally, you can run a PR merge pipeline manually from the Pipelines UI, without actually merging a PR (see the same link). This way, you can make sure that the merge result build is passing without actually doing a merge.
How do we create multi branch pipeline in azure.
Say there are 4 developers committing to the same branch, for each commit there should be a seperate build pipeline with unit tests.
for each commit there should be a seperate build pipeline with unit tests.
If you want to have a seperate build pipeline with unit tests for each commit, you need to create multi pipeline for each branch:
And if just want to one build pipeline for those multi branch, you may need to have create multi branch triggers:
I think what you need is a CI pipeline. So lets say, there is a main/master branch, 4 developers have pulled this main branch in their PC/laptop. And all 4 of them are about to work on 4 different defects/bugs say bug1,Bug2, Bug3 and Bug4
Each one of them are supposed to create a branch from main. Say bug/Bug1, bug/Bug2, bug/Bug3 and bug/Bug4. So now we got 4 branches.
What you can do now is switch on pull request branch policies in main branch. Please see below screenshot
Once you switch on this pull request policy, no code can be merged in your main branch without creating a pull request.
Now second part, you have to create a pipeline that will for example, build the code and run some tests.
Now part three: In the same branch policies page, if you scroll down a bit, you will see a setting for "Build Validation" as shown below
What you have to do here is
Press that + sign and configure the pipeline that you have created earlier here.
What this means is
When say Dev 1 whose branch is bug/Bug1 will raise a pull request to merge in main branch, this Build validation will trigger the pipeline that you have configured here.
So when all 4 Devs will raise 4 pull requests, based on their code changes, this pipeline will get triggered and validate those code changes for build + tests.
Also, based on the code review, if there are any further code changes pushed into these branches (bug/Bug1 etc.), you can configure this Build validation in such a way that this pipeline will get automatically triggered whenever new code changes are pushed into a branch which is connected to a pull request.
I have a great experience of pipeline creating automation (in case of huge amount of repos).
For example, a project has 20 similar repos with Java app (like a microservice) and a pipeline for each of them is differing only by repo url (and a few more minor attributes). The CI/CD process for each of them is the same.
So, we can create a separated devops-repo with declaration configuration for our services. Also we can create a single pipeline which will pull the devops repo and create all needed pipelines for each repo in the configuration (this operation is going to be executed only once in the beginning and in case if we want to change the devops-configuration)
I have implemented that using Jenkins. Now, I am going to do so using GitLab CI. But I can't get how is it possible.
Is it possible to create a pipeline from another one (dynamically)?
Any suggestions?
You can use include and put the generic pipeline in your devops repo.
In your java repos you can include the devops pipeline and set the variables which are specific for the respective java repo.
So the pipeline for your java repos can be as short as this:
include:
- project: 'your-group/devops-repository'
file: '.generic-ci.yml'
variables:
FOO: bar
For my team where we have partner teams providing us SW pieces that need to be integrated on HW systems and tested together, our code footprint is small and hence churn is small, while number of changes from partner teams is frequent. In such a scenario, I see the need to trigger the release part of the yaml many more times than the build part. Is multi-stage pipelines the way to go? I want to trigger new release instances using RestAPI invoke only the Release stages on the YAML file, using AzureDevOps Rest API.
Regards,
You don't have to use multi-stage pipelines to be able to trigger repeated releases, it just makes the management of the pipeline cleaner.
It's possible to create a pipeline that include a build stage and release stages for each of your environments, trigger the build stage (manually or based on a CI trigger), and then from that Pipeline "Run", deploy as many times as you see fit to whatever environments you like. That can be done from API or portal.
It's also possible to create a pipeline that is "release-only" - that is, it gets created manually, or as the result of seeing a specified build having been run.
Personally, I like the multi-stage build because it's a little easier to see what build created the release that you're deploying around. It's not a requirement, though.
I have a pipeline build for a .NET application called Master Site and then I have three other build definitions using different repos. Every time I have to do a build of the Master site and then the subsequent build. I need to know if we can do multiple builds by triggering a single build request. (all the builds are built off of different repos)
I need to know if we can do multiple builds by triggering a single
build request.
For this issue,the answer is yes. You can do this by adding multiple Trigger Build Tasks to the agent job in a build pipeline .
This tasks allows to trigger a new Build (add it to the queue) as part of a Build Definition.