I need to set up a pipeline for a branch that will do the following,
On push, it will trigger a pipeline (which happens as per GitLab)
1st Job to trigger a build (.net5) for the branch (new code pushed to the branch). If the build succeeds, it will begin the next job. However, I am to trigger build (MSBuild) on the runner location check out when the pipeline runs. But is there any way to directly run a build on the branch?
If the above build fails, the push should be reverted on a branch. I applied git revert and reset commands, but it only gets used to the build location of the runner. I need to revert commit at the branch level.
When GitLab starts a pipeline, it is a "detached HEAD" which basically means it creates a temporary branch that points to your specific commit. It does this because there might be a commit after it which the current build doesn't know about.
To get out of the detached head, you need to switch to the branch directly.
git checkout branch-name
To get the branch name, you find an appropriate predefined variable which works for your.
Once you have that, you need to get the latest version so you can revert. I usually needed the reset here to make things work.
git reset --hard
git pull
From there, you can revert. Now, the CI process doesn't have write permissions, so you need to have a GitLab access token with write permissions, which means you have to set up a new origin or parent.
git remote add changes https://oauth2:$PUSH_GITLAB_TOKEN#gitlab.com/group-name/project-1
git push -f changes branch-name
Now... that said, this is probably not what you want to do. Since you said you were new to CI, I'm going to say I'm 99.99999% sure you don't want to go down this path because it only leaves to heartache and frustration.
The reason is the branch may have a second or additional commit. It could be you missed something at the last minute and threw up a new one, an automated process starts up, or just your typical race conditions. Reverting a commit means you either will blow up with those changes or you end up erasing every other commit as you have to force push to get it up.
This also means that this is a very fragile process and will break many times, usually at the worst times (when you have a rapid series of commits), and then you have to tell everyone to stop working while you fix it.
(A good sign is how much work you have to do to fight the CI process.)
Instead, I recommend you create a merge request (MR) against your branch and set up the rules to make sure that is valid before you merge it into your branch. That way, the process doesn't have to worry about future or past commits, it just says what you have in your merge request (which is another branch) can't be applied after merging it with branch-name.
In our team, we have many branches (main, next, release/2.3.4, release/2.4.0, etc.) and doing MRs against each one has worked out well. We do the merge requests against each of the release branches, and the CI process says it can merge and pass tests before it tries to commit, but then we can commit knowing that 98+% of the time, it will be fine.
And it doesn't require jumping through hoops.
Related
I use GitLab on-premise and need to change the contents of a file automatically when I push into my feature branch so when tests run and code is executed, one of the files called from within the repo itself has modified content. When I merge that branch into master, I need to undo that change.
It's not enough to override the file contents just when the test runs, because of how the application works. It will end up pulling the repo from GitLab on it's own and execute one of the files contained within.
I have been looking into hooks a little bit, but I can't find any references or examples on how to accomplish something like this.
Currently I am changing the file manually so my CI tests run accurately. If the tests then pass, I can manually change the file back and skip the CI tests on the final push, and then merge into master.
There's not really a default way to automatically change, commit and push back into Gitlab from a pipeline, as the pipeline does not have authorization to write into the repo.
However, you can provide a "Personal Access Token" (PAT) for one of Gitlab's users (or even a special service-account created for that purpose) - either commit that to your repo (which is quite unsafe) or provide it through the "CI/CD Variables" setting from within Gitlab.
Your pipeline will then need to do something like:
# change file.txt
# add the remote with credentials authorized to commit; do not fail if the remote already exists
git remote remove pushorigin || true
git remote add pushorigin https://commituser:${PAT}#gitlab.local/path/to/project.git
# add and commit the file
git add file.txt
git commit -m "Commit Message"
# move the remote's branch tip:
git push pushorigin "HEAD:${CI_COMMIT_REF_NAME}"
I don't have a clue how to revert that automatically when/before finally merging the branch. Don't you mind those testing commits being merged back into main, possibly creating conflicts on those files?
I guess overall, the application should be modified to better support testing on those branches.
For example, if the dev branch is behind the build branch when the two branches merge, the merge request can be created. But it is clear that it cannot merge because it is not possible to merge a backward branch into the current branch. In this case, I want to use the.gitlab-ci.yml configuration, To determine if the dev branch is behind the build branch, I wonder, can this be done? How to configure the.gitlab-ci.yml file if possible?
Basically, gitlab attempts the merge using git to determine if there's a conflict. It actually calls to gitaly to do this.
Basically the process goes like this:
The rails app in repository.rb will call to the gitaly conflict service to check for conclicts
The conflict service list_conflict_file in gitaly will call to git2go
The git2go conflicts subcommand will basically perform the git merge operation and return any conflicts that were encountered. This eventually makes its way back as a response to the call from rails in step (1)
So, if you wanted to do something similar in your CI/CD pipeline, you could use git (or a programatic API to git in your favorite language) to attempt a local merge of the two branches in order to detect conflicts.
Updated 26-Feb-2020
In our Project, I have a pipeline "MyPipeline" which restores the NuGet packages, builds the solutions and runs the tests.
On the master branch, I have a policy which does things like add code reviewers, and it has a "build validation" step which executes "MyPipeline". All well and good.
However, I created another branch from Master called NewBranch and synced (pushed) this up to Azure. After doing some minor changes in Visual Studio, I did a merge from master, committed and sync'd to Azure.
I was a bit surprised then to see "MyPipeline" executing. It seems to have been triggered when I pushed my changes to NewBranch to Azure. I don't have a branch policy on "NewBranch". The trigger in the YAML file is:
trigger:
- master
What kicked this off? I'll soon burn through my free agent minutes if this continues...
Update 26-Feb-2020
As per the comments below:
The history of commits on the master branch are:
Tue 9:10 PM
Tue 7:47 PM
Mon 2:46 PM
The history of the pipeline execution show:
Wed 9:14 AM "PR Automated"
So, nothing new has been committed to the Master branch. However, I think I know what's causing this. Just that I'm not convinced on the timing....
So, we have two branches, master and NewBranch.
Master has a policy that requires two code reviewers to Approve, and it requires the build to succeed. Because of this policy, a developer can't therefore merge directly into master - they have to generate a Pull Request.
So, Developer A creates a Pull Request to merge NewBranch to Master. There is then the rather lengthy code review process which may take multiple additional commits to the "NewBranch"'s Pull Request before it's deemed acceptable.
What seems to be happening is that everytime one of these new Commits is sync'd to the Pull Request, a build is triggered. Is this a good thing? Maybe, Maybe not. If the build is going to trigger just once, then the compilation should occur when all the code in the Pull Request has been approved, not before hand. Why trigger a build at such an early stage; the master branch may be updated by multiple other Pull Requests before this is ready to be merged in. However, with unlimited resources, then I guess there's no harm in building as often as possible, but a) this can delay other builds (representing an impediment to other developers) and b) this uses up the free limit on agent time.
The detailed info you share in question is much help for me to understand the whole workflow of yours.
Though you did not express too clear, but, yeah, what you are guessing is correct. The action you faced is expected and by designed. The root cause is you are using branch policy and the Build validation also included into this policy.
The nutshell is you are using Pull request trigger.
Explanation:
Let's pay attention to its definition:
Pull request (PR) triggers cause a build to run whenever a pull request is opened with one of the specified target branches, or when
changes are pushed to such a pull request.
Based on your added contents, your developers are pushing changes (new commit) into the opening Pull request (Note: The key point is Pull request is still be opening.) This belongs to the work scope of the PR trigger because of above definition. That is why the build triggered every y new changes pushed into NewBranch branch.
Work around:
I agree with the logic of #devpro's answer. But its script does not available for your scenario. Because the pr in YAML only work for GitHub and Bitbucket Cloud repos.
Here you are using VSTS repo, and configured the policy for it. So, you can only via branch policy configuration to avoid such burning trouble.
In your Build validation panel, you are setting build policy with Automatic, right?
Please change the Automatic to Manual, also keep the Policy requirement value as Required.
Now, the corresponding build pipeline will not be ran automatically once new commit pushed, it can only built when someone run it manually.
For the delay timing you noticed which make you unsure, I guess it would because of conflict.
For sample, the Pull request P1 which merge from NewBranch to Master is opening. I commit a new change C1 into NewBranch. BUT, it causes the conflict. At this time, the build will not be ran because the changes does not actually pushed into PR.
Note: The commit is true to NewBranch. BUT changes does not accepted by PR yet, because PR detect out there has conflict here and you must tell PR which changes you want to keep. Only the changes pushed into PR can work with PR trigger.
Only the conflict solved, the changes, maybe I can say commit, can truly be accepted and pushed into PR. Then the triggered build ran.
This would be the time period delay you noticed.
I've seen the same behaviour today.
Can you try rewriting the trigger part in your pipeline like this:
trigger:
branches:
include:
- master
Does that help ? I can't verify it myself yet, since I've done that change but the PullRequest isn't approved yet :)
I've also another repository where I do not see that behaviour, but there, my pipeline trigger looks like this:
trigger:
- master
(Notice the 2 spaces before - master)
I think you need to add pr: none to your pipeline definition to disable the automated run.
If pr is not set I think the default is to run on every PR.
That would give something like this:
trigger:
batch: true
branches:
include:
- master
paths:
exclude:
- README.md
pr: none
I have setup a Git project + CI (using Gitlab-runner) on Gitlab v12.3.5. I have a question about issues and pipelines. Let's say I create an issue and assign it to myself. So this create a branch/merge request. Then, I open up the WebIDE to modify some files in an attempt to fix the issue. Now I want to see what if my changes will fix the issue. In order to run the pipeline, is it necessary to commit the changes into the branch or is there some other way?
The scenario I have is that it may take me 20 times to fix the files to make the pipeline 'clean'. In that case, I would have to keep committing on each change to see the results. What is the preferred way to accomplish this? Is it possible to run the pipeline by just staging the changes to see if they work?
I am setting up the gitlab-ci.yaml file. Hence it is taking a lot of trials to get it working properly.
You should create a branch and push to that. Only pushed changes will trigger pipeline runs. After you're done, you can squash and merge the branch so that the repo's history will be clean.
Usually though, you won't have to do this because you'll have automated tests set up to check whether your code works. You should also try testing the Linux commands (or whichever commands you're running in your GitLab CI scripts) locally first. If you're worried about whether your .gitlab-ci.yml syntax is correct, you can navigate to the file in your repository and check there (there's a button at the top which lints it).
I am working on my master branch and have some uncommitted changes inside the master branch. I have already created a feature branch and I am able to switch to my feature branch using the git checkout feature_branch command. I have couple of doubts here which I am listing below:
Why am I seeing the uncommitted changes in my branch when I have made my changes inside my master branch. If I commit my changes in the master branch and switch to my feature branch, I do not see the committed changes specific to master. I am expecting the same behavior for uncommitted changes also.
I came across the stash command in Git and there it was mentioned that if you have any uncommitted changes inside the master branch, and would like to switch to some other branch, git will not allow you and hence you need to stash the changes inside the master branch and then only switch to the development branch. I did not face this issue. Am I missing something here and if so, what is the significance of using the stash command?
If I commit my changes in the master branch and switch to my feature branch, I do not see the committed changes specific to master. I am expecting the same behavior for uncommitted changes also.
That's not a sensible behavior. Uncommitted changes are by definition not made to a branch. Branches are pointers to commits, and there is no commit associated with uncommitted changes. They are made to your working directory and have nothing tying them to any branch. They aren't even associated with Git. Until you commit (or stash) them, Git does absolutely nothing to track them, and it will never try to manage them for you.
I came across the stash command in Git and there it was mentioned that if you have any uncommitted changes inside the master branch, and would like to switch to some other branch, git will not allow you and hence you need to stash the changes inside the master branch ...
That's wrong. Stashes don't exist inside any branch. There is one global pool of stashes.
... I did not face this issue.
You will only be prevented from switching branches if doing so would cause your uncommitted changes to be overwritten.
Am I missing something here and if so, what is the significance of using the stash command?
You only need to worry about stashing your changes if they affect any files that would be changed by switching branches. If you have modifications to a file that is identical between two branches, switch from one branch to the other will not require a stash. If the file is different on your other branch, Git will not let you switch branches, as this would destroy your uncomitted changes.
It is safe to try switching branches, and if Git warns you that "uncommitted changes would be overwritten" and refuses to switch branches, you can stash your changes and try again.