We have two developers using the same ADF. Each developer creates a git branch and starts working on it. Each developer can save the changes to their own git branch but there can only be one collaboration branch and this branch decides the publishing branch. This is causing a blockade (for one of the developer. How can we solve this ?
ADF publish branch can be set using a publish_config.json but now there is an option to set this in the adf itself. which one takes precedence? What is the best practice here?
You need to manage the work of each developer with standard git branch/merge processes. When one dev is done with work in their feature branch, then they will create a pull request to merge changes into your collaboration branch.
If the second dev has not created a feature branch yet, they can just do so after the pull request from the first dev is complete and then continue work from there. If the second dev has already created a feature branch, then they will need to merge the new changes from the collaboration branch into their feature branch to continue work before later committing to git and creating a pull request to merge changes from their feature branch back into the collaboration branch. From there, you can publish as needed.
This git work can be done through the ADF editor as well as through any other git interface you have. It's up to you.
This article discusses the process in specific detail using the ADF editor.
EDIT:
I believe you now have answers for this from 3 of the other 5 questions you posted about this same topic in the past day.
ADF publish confusion in git mode
Azure data factory working-branch confusion
When ADF publish branch is git protected how to publish?
Here is another article which describes the fundamental git process for ADF to help bring you up to speed with the fundamentals of how the different branches work, and how you can switch publish branches on the fly if needed.
Related
When working with the regular source code, (Java, C++, etc..) there are things like
git pull ..
git fetch ..
git push ..
to synch your remote git repo branch with your local branch.
What is the equivalent of such in the Azure Data Factory world ?
So, I am using azure data factory with the Azure git repo.
I am working in the particular feature branch - "fefature branch"
And my pipeline has a copy activity that hits a data set in its "Sink" stage.
Here is a screen shot but .. it's pretty simple and seems right
I see that my code for Data set definition (Json) in the remote Git repository is different from what I see in the Azure portal gui (being pointed to that same remote branch). ADF Gui in the Azure Portal is correct, the one in the git repo contains some stuff that I already deleted, but it does not gets deleted there (Why??)
So, when I 'Debug' pipeline I get errors which indicate this discrepancy as a problem. I want ty sync the environments and .. given that I do not understand how the discrepancies came about, I don't know how to fix an issue?. Any help is appreciated.
In the ADF world, we use publish and create a new pull request to merge the new changes from a feature branch to the main branch.
it seems like your git repository version is not up to date with the live ADF.
If there are any pending changes in your main branch, then you can click on Publish button to merge the changes
And if you are working on the feature branches, you can merge the changes using the new pull request.
If you have multiple feature branches, then you will need to manually compare the different versions to resolve these conflicts.
I am using Azure Devops in an IT environment with many different development teams and git repositories. Each development team owns one or more repositories. It's my job to work on various application components contained in said repositories. Because I do not own those repositories, I should not make any changes in build/release pipelines, build policies etc, all by myself because that can impact other people's work.
Now let's say I have a feature branch named UpgradedFeature in the repository FeatureRepository, containing my changes. Said changes also introduce a breaking change in the build pipeline used for that repository for the master branch. Let's say that pipeline is named MasterBuildPipeline.
So in order for my build based on the branch UpgradedFeature to succeed and not impact other people's work, I make a clone of the MasterBuildPipeline, name it UpgradedFeatureBuildPipeline and configure the breaking changes. This new build pipeline is used exclusively by me for the UpgradedFeature branch only.
The build, now using the new UpgradedFeatureBuildPipeline pipeline succeeds and now I want to merge into master , so I make a pull request to merge the changes contained in UpgradedFeature into master. The master branch has a branch policy in place named MasterBranchPolicy like described on https://learn.microsoft.com/en-us/azure/devops/repos/git/branch-policies?view=azure-devops&tabs=browser. This branch policy contains the MasterBuildPipeline and prevents completion of that pull request when the build using that pipeline does not succeed.
So my problem is that my pull request triggers the MasterBranchPolicy containing the MasterBuildPipeline and not the UpgradedFeatureBuildPipeline containing the necessary breaking changes for the build to succeed. So the build fails and I cannot complete the pull request.
Of course I could edit the MasterBuildPipeline for a short time, introduce my breaking changes, run the build, then discard the breaking changes again. But there's a chance I may impact other people's work with that and somehow I have a feeling that's not the right approach. I could also edit or disable the MasterBranchPolicy for a short time but again, I may impact other people's work and I feel it's not the right approach.
How should I do this?
So the build fails and I cannot complete the pull request.
To complete the pull request even through the build fails, you could grant yourself Bypass permissions. Bypass permissions let you complete pull requests that don't satisfy branch policies. You can grant bypass permissions to yourself then complete the pull request. Here is Bypass branch policies for reference.
Please navigate to Project setting >> Repositories >> The repo >> Security >> user (yourself) >> Bypass policies when completing pull requests.
Then, you can Override branch policies and enable merge even the MasterBuildPipeline faild.
Please also note that use caution when granting the ability to bypass policies, especially at the repo and project levels. Policies are a cornerstone of secure and compliant source code management. In your scenario, it's suggested to edit the MasterBuildPipeline and the MasterBranchPolicy or disable the MasterBranchPolicy as you mentioned.
Generally, workflow of DevOps Branching Strategy as follows
Developer will create a feature or bugfix branch out of develop. One feature or bugfix branch usually stands for one JIRA bug or feature item. These branches are personal
The changes will be pushed into the developer's feature or bugfix branch.
When the new feature or bugfix is complete.A developer will create a pull request. Pull requests open a code review phase.
Once a pull request has been approved, the team lead or development team will move it into development.
When the development branch has all the epics and bug fixes, i.e., the content planned for the next release, the development team or team lead will create a release branch. This initiates the release regression testing phase.At this stage, only bug fixes are accepted for release, and the workflow is similar to that of the development branch.
Having a separate release branch will enable future development towards the next release in the development branch. Features for the next release are not included in this release. However, bug fixes for this release will be incorporated into the development of the next release as well.
When release content meets the criteria, the release branch will be frozen, which means that it ends. Content from releases will be merged to master and tagged there. For the next release, a new release branch is created when needed.
As per my experience, I would suggest creating branching policies like
A pull request is requested to merge the develop, release, and master branches.
Pull request approvers should be leads.
All developers can create feature branches.
All developers can push to hotfix and feature branches. Commit messages must include the JIRA issue id.
In git mode, when we want to test a pipeline, ADF forces us to publish first.
Publish action does two things is my understanding
Saves to the local ADF (DEV) as given here
Creates arm templates in a branch (adf_publish/the branch we
specify)
But to get the 'Publish' button enabled, we need to be in collaboration branch. This means no two people can work at the same time on a DEV ADF. As both people will be asked to publish by ADF before they could test the pipeline they are building.
If this is the case then why is there an option for us to connect another branch other than collaboration branch? (by changing it from the drop down)
Also what is a 'working branch'?
As we know, we only can 'Publish' in collaboration branch and changes are being pushed to to "adf_publish" branch by default. By default, the collaboration branch is named main.
If you want team work, you need to create several branches.
Working on the own branch, we can validate and debug the pipeline to make sure everything is ok.
Then click save all, it will commit on the own branch. If we want publish, we need to creat a pull request to the main branch.
4.After merged to the main branch, we can publish to the adf_publish branch.
My requirement is whenever developer try to do check-in existing GitLab repository then before doing check-in in repository,build should trigger (Jenkins build) and Junit test case should run on new check-in and if passes then it should go forward and will allow developer to do check-in in main repository.
I am not sure but is pre-hook commit can achieve this requirement?
While you could achieve this with pre-commit hooks, it's more common to do so with post-commit hooks on the server-side.
You can achieve this by operating a branch based workflow, there are multiple to choose from - I would recommend reading through this guidance by Atlassian.
Developers will create branches from a 'main' branch (often master, but can be a 'dev' branch working towards a release for instance), then develop code on that branch. They will then push their branch and commits to the remote repository (GitLab). When ready to merge into the main branch, your developers can open a merge request onto the main branch.
On GitLab you can setup a webhook to trigger Jenkins builds when a push event occurs. I would recommend this guide to guide you through it.
In the GitLab project settings you can require a passing build before merge requests are allowed to merge.
Furthermore, your understanding of Git seems incorrect - check in is not a term used in Git. Please take a look at the Git documentation. In Git a developer creates commits against a local copy of the repository, then pushes these to a remote repository (GitLab/GitHub etc.). There is no direct equivalent of the 'check in' used in various centralised version control systems e.g. SVN.
We're using GitLab for our project and we think it's great.
We're also using git flow to manage the changes in feature, develop, master branches.
Can you use the Merge Request build in GitLab to manage the branches in git flow style?
Meaning when accepting a Merge Request for a release branch, it will merge the release branch into master AND into develop..
Or should we always use git flow on our local machines to accept the Merge Requests.
Like this issue 1628 illustrates, it is best to use GitLab to publish (push) branches, then to rely on a specific GitLab feature related to a particular workflow like git-flow.
So the natural approach is to limit those git-flow features to the local clones, and push any branch you want to GitLab once the merge request is resolved.
By now, there is something called GitLab Flow, "a clearly defined set of best practices".
Cited from the linked source:
It combines feature-driven development and feature branches with issue
tracking.