I am seeing an error in our release build process. Need your help/suggestions on it.
In our organization, we follow the release strategy like this:
our codebase has microservices/applications located on individual folders like this format
MS-A
MS-B
MS-C
MS-D
Our release processes follow this workflow:
branch-1 has made changes on MS-A and merged it to master
branch-2 has made changes on MS-B and merged it to master
from master we create Release branch R1 and did the release process, where only MS-A and MS-B gets deployed
afterwards similarly,
branch-3 has made changes on MS-C and merged it to master
branch-4 has made changes on MS-D and merged it to master
then from from master we create Release branch R2 and did the release process, where only MS-C and MS-D gets deployed
During release, we basically do a git diff R2 R1 and calculate which folders got updated, and then start deployment of only those folders(i.e. microservices).
Now when there is any error during deployment - we create a bugfix branch from master(which naturally gets ahead of the release branch HEAD now), push the fixed codes there, merge it to master and then we just cherry pick that commit id on the release branch and redeploy it.
So as a result, we had been seeing an issue occuring - when git diff is calculating the changes on two release branches, it isn't taking into account the cherry-picked ids as a part of the previous release but considering it as a part of the next release branch itself.
a short example using the previously described relese workflow example will easily visualize this error:
Lets say, on R1 release MS-A and MS-B got deployed and there was some issue with MS-A for which we raise a bugfix branch B-1. we push the commits on B1 and merge it to master and cherry-pick those commits onto R1 branch. Then after the release R1 is completed, we made changes on MS-C and MS-D and merged it to master and create new release branch R2.
So if i do git diff R2 R1, i want the output to be MS-C and MS-D only, but here we get MS-A, MS-C and MS-D. Git is not considering cherry picked B-1 commits as a part of R1.
Considering the above scenario and keeping in mind I can't make any changes in the entire release workflow process whatsoever, is there any way to fix this issue?
Related
In our development using GitLab we usually squash commits from feature branches when they are merged in main branch (we use rebase strategy instead of merge if it matters).
But if the commits from feature branch are squashed to a single commit (which is later merged in main branch) and the branch itself is deleted, how can GitLab still show the original commits? Are they kept in the repository forever or it is a matter of time when they finally get dropped?
I believe GitLab does some garbage collection on the repository, but I wonder why it does not affect commits from MR.
The squash and merge commit page does mention:
Each time a branch merges into your base branch, up to two commits are added:
The single commit created by squashing the commits from the branch.
A merge commit, unless you have enabled fast-forward merges in your project.
Fast-forward merges disable merge commits
That merge commit might very well keep a reference to the branch before squashing, which can explain why the original commits are still visible.
If that is not the case, meaning nothing actually references those commits, then housekeeping (described here for self-managed GitLab, but also active on Cloud offering) will remove them the repository eventually.
However:
Gitaly cannot ever delete unreachable objects from object pools because they might be used by any of the forks that are connected to it.
Gitaly must keep all objects reachable due to the same reason. Object pools thus maintain references to unreachable “dangling” objects so that they don’t ever get deleted.
In case of squashing, the feature branch commits are squashed into a single commit to be pushed to the target branch. On the target branch, you will see only a single commit.
See this image to see the commit on the target branch i.e. vighnesh_develop
But you will still be able to see the individual commits on the merge request from the deleted feature branch. The feature branch is deleted but all the merge requests are maintained for lifetime unless we manually set an expiration period.
See this merge request for squashed commit
Hence, the individual commits are linked to the merge requests and not to the deleted branch or the target branch. Git will maintain a list of ALL the commits even though the branch from which the commit came is deleted. That is how it is able to do effective version control.
As in the title: there's always the statement The source branch is 1 commit behind the target branch. How is it even possible: both master and develop branches are up to date, no one made any modifications in the meantime, and the merge request gets created via GUI operating on the most up to date version of the repository, isn't it?
I read GitLab documentation saying that it could be the merge commit counted as this 1 commit behind, so I changed my project settings to:
It didn't change anything however: there's still the same message:
So far, I make a rebase in the following way:
git checkout master
git pull
git checkout develop
git merge master
git push
to line up the branches, but I'd like to save myself these few additional steps. Is it possible to achieve?
Updated 26-Feb-2020
In our Project, I have a pipeline "MyPipeline" which restores the NuGet packages, builds the solutions and runs the tests.
On the master branch, I have a policy which does things like add code reviewers, and it has a "build validation" step which executes "MyPipeline". All well and good.
However, I created another branch from Master called NewBranch and synced (pushed) this up to Azure. After doing some minor changes in Visual Studio, I did a merge from master, committed and sync'd to Azure.
I was a bit surprised then to see "MyPipeline" executing. It seems to have been triggered when I pushed my changes to NewBranch to Azure. I don't have a branch policy on "NewBranch". The trigger in the YAML file is:
trigger:
- master
What kicked this off? I'll soon burn through my free agent minutes if this continues...
Update 26-Feb-2020
As per the comments below:
The history of commits on the master branch are:
Tue 9:10 PM
Tue 7:47 PM
Mon 2:46 PM
The history of the pipeline execution show:
Wed 9:14 AM "PR Automated"
So, nothing new has been committed to the Master branch. However, I think I know what's causing this. Just that I'm not convinced on the timing....
So, we have two branches, master and NewBranch.
Master has a policy that requires two code reviewers to Approve, and it requires the build to succeed. Because of this policy, a developer can't therefore merge directly into master - they have to generate a Pull Request.
So, Developer A creates a Pull Request to merge NewBranch to Master. There is then the rather lengthy code review process which may take multiple additional commits to the "NewBranch"'s Pull Request before it's deemed acceptable.
What seems to be happening is that everytime one of these new Commits is sync'd to the Pull Request, a build is triggered. Is this a good thing? Maybe, Maybe not. If the build is going to trigger just once, then the compilation should occur when all the code in the Pull Request has been approved, not before hand. Why trigger a build at such an early stage; the master branch may be updated by multiple other Pull Requests before this is ready to be merged in. However, with unlimited resources, then I guess there's no harm in building as often as possible, but a) this can delay other builds (representing an impediment to other developers) and b) this uses up the free limit on agent time.
The detailed info you share in question is much help for me to understand the whole workflow of yours.
Though you did not express too clear, but, yeah, what you are guessing is correct. The action you faced is expected and by designed. The root cause is you are using branch policy and the Build validation also included into this policy.
The nutshell is you are using Pull request trigger.
Explanation:
Let's pay attention to its definition:
Pull request (PR) triggers cause a build to run whenever a pull request is opened with one of the specified target branches, or when
changes are pushed to such a pull request.
Based on your added contents, your developers are pushing changes (new commit) into the opening Pull request (Note: The key point is Pull request is still be opening.) This belongs to the work scope of the PR trigger because of above definition. That is why the build triggered every y new changes pushed into NewBranch branch.
Work around:
I agree with the logic of #devpro's answer. But its script does not available for your scenario. Because the pr in YAML only work for GitHub and Bitbucket Cloud repos.
Here you are using VSTS repo, and configured the policy for it. So, you can only via branch policy configuration to avoid such burning trouble.
In your Build validation panel, you are setting build policy with Automatic, right?
Please change the Automatic to Manual, also keep the Policy requirement value as Required.
Now, the corresponding build pipeline will not be ran automatically once new commit pushed, it can only built when someone run it manually.
For the delay timing you noticed which make you unsure, I guess it would because of conflict.
For sample, the Pull request P1 which merge from NewBranch to Master is opening. I commit a new change C1 into NewBranch. BUT, it causes the conflict. At this time, the build will not be ran because the changes does not actually pushed into PR.
Note: The commit is true to NewBranch. BUT changes does not accepted by PR yet, because PR detect out there has conflict here and you must tell PR which changes you want to keep. Only the changes pushed into PR can work with PR trigger.
Only the conflict solved, the changes, maybe I can say commit, can truly be accepted and pushed into PR. Then the triggered build ran.
This would be the time period delay you noticed.
I've seen the same behaviour today.
Can you try rewriting the trigger part in your pipeline like this:
trigger:
branches:
include:
- master
Does that help ? I can't verify it myself yet, since I've done that change but the PullRequest isn't approved yet :)
I've also another repository where I do not see that behaviour, but there, my pipeline trigger looks like this:
trigger:
- master
(Notice the 2 spaces before - master)
I think you need to add pr: none to your pipeline definition to disable the automated run.
If pr is not set I think the default is to run on every PR.
That would give something like this:
trigger:
batch: true
branches:
include:
- master
paths:
exclude:
- README.md
pr: none
I am trying to implement a CI pipeline where development is merged to a release branch which, in turn, is merged to master.
The release branches are named x.y.z and reflect the version number of the release.
The release job modifies the pom to create a SNAPSHOT version (using the branch name). The master job modifies the pom to create a release version by removing '-SNAPSHOT'. Both jobs commit and push their changes.
The downside is that this causes a conflict on the next iteration. The previous commit on master puts it out-of-sync with development and the MergeRequest flags a conflict. The only difference is the version number in the pom and is straightforward to fix.
In the diagram above, it is not possible to merge 1.0.1 with master because of the commit 'Create release version (1.0.0)'.
I am wondering if it is possible (in Gitlab) to force an override when merging. Can I force an override of "theirs" with "mine" within the MergeRequest?
Background
This CI process seems to give a solution to the problem of who sets the pom version? This way the release branch defines the version and SNAPSHOTs and Releases are defined by the target branches.
This is for a web application and I want WAR files to be stored in Nexus as releases or SNAPSHOTs
GitLab has an interactive editor for solving merge conflicts:
https://docs.gitlab.com/ee/user/project/merge_requests/resolve_conflicts.html
It is possible to click use this on either their changes or our changes. There is a quirk with this though: The conflict resolution will be in the source branch and not in the target branch. This means that if you merge 1.0.1 into master and there's a conflict, it will show up as a merge of master into 1.0.1! It is then possible to accept the merge request, and this will result in a merge of 1.0.1 into master. This is a bit unexpected, but it makes it possible for a branch author to fix conflicts, even if they are not allowed to push fixes to the target branch.
There is an issue for this on GitLab:
https://gitlab.com/gitlab-org/gitlab/issues/25014
But they write that they don't have any plans to move forward with that at the moment.
I am using the Android Studio functionality to manage my branches/commits etc.
I have the master branch which is reserved for final versions, then I have my development branch and then I open new branches for any features I want to work on.
I've just noticed that my changes in my new branches are automatically merged into my previous branch.
For example, if I'm on the development branch then create a new branch "Image gallery" and add something, when I go back to the development branch, that thing I've added is already there!
This beats the whole purpose of using branches, what is going on? Did I click anything I shouldn't have clicked?
If we are not commit the changes which we are done in the feature branch(image branch), it will added to the development branch when we change the branch local. First we need to commit the changes which is done feature branch(image branch). Then we need to change the branch to the development.