When using a Git-backed ADFv2, I'm trying to detect when a user publishes directly to the Factory vs when a user publishes to the Factory via the Git collaboration branch.
I've tried looking at the Activity Log, but I can't distinguish between events from a Git Publish vs events from a Direct Publish.
I see that this information is at least visible in the UI. Is this persisted anywhere? Is there any way to obtain this message?
Is there another way to do this?
According to my observation, the messages don't persisted anywhere.It is just temporarily stored in the browser's cache, and disappears once you refreshes it.
In the ADF active log,you only could see the epitomize of operations.
I supposed that you only can distinguish between events from a Git Publish vs events from a Direct Publish in the Dev ops page.
You could see the direct publish will leave the comments in the commits list.
And if you release any updates in the Git publish, you could remark some prefix,like --from git---.
Related
When working with the regular source code, (Java, C++, etc..) there are things like
git pull ..
git fetch ..
git push ..
to synch your remote git repo branch with your local branch.
What is the equivalent of such in the Azure Data Factory world ?
So, I am using azure data factory with the Azure git repo.
I am working in the particular feature branch - "fefature branch"
And my pipeline has a copy activity that hits a data set in its "Sink" stage.
Here is a screen shot but .. it's pretty simple and seems right
I see that my code for Data set definition (Json) in the remote Git repository is different from what I see in the Azure portal gui (being pointed to that same remote branch). ADF Gui in the Azure Portal is correct, the one in the git repo contains some stuff that I already deleted, but it does not gets deleted there (Why??)
So, when I 'Debug' pipeline I get errors which indicate this discrepancy as a problem. I want ty sync the environments and .. given that I do not understand how the discrepancies came about, I don't know how to fix an issue?. Any help is appreciated.
In the ADF world, we use publish and create a new pull request to merge the new changes from a feature branch to the main branch.
it seems like your git repository version is not up to date with the live ADF.
If there are any pending changes in your main branch, then you can click on Publish button to merge the changes
And if you are working on the feature branches, you can merge the changes using the new pull request.
If you have multiple feature branches, then you will need to manually compare the different versions to resolve these conflicts.
We have two developers using the same ADF. Each developer creates a git branch and starts working on it. Each developer can save the changes to their own git branch but there can only be one collaboration branch and this branch decides the publishing branch. This is causing a blockade (for one of the developer. How can we solve this ?
ADF publish branch can be set using a publish_config.json but now there is an option to set this in the adf itself. which one takes precedence? What is the best practice here?
You need to manage the work of each developer with standard git branch/merge processes. When one dev is done with work in their feature branch, then they will create a pull request to merge changes into your collaboration branch.
If the second dev has not created a feature branch yet, they can just do so after the pull request from the first dev is complete and then continue work from there. If the second dev has already created a feature branch, then they will need to merge the new changes from the collaboration branch into their feature branch to continue work before later committing to git and creating a pull request to merge changes from their feature branch back into the collaboration branch. From there, you can publish as needed.
This git work can be done through the ADF editor as well as through any other git interface you have. It's up to you.
This article discusses the process in specific detail using the ADF editor.
EDIT:
I believe you now have answers for this from 3 of the other 5 questions you posted about this same topic in the past day.
ADF publish confusion in git mode
Azure data factory working-branch confusion
When ADF publish branch is git protected how to publish?
Here is another article which describes the fundamental git process for ADF to help bring you up to speed with the fundamentals of how the different branches work, and how you can switch publish branches on the fly if needed.
Is there a way to change the commit message when saving an Azure Data Factory to git?
Whenever you press save it seems you get a commit and that commit will change say Updating pipeline name. While sort of useful it would be nice to actually put in some descriptions of what's changed. Is there a way to do this?
Unfortunately, Azure Data Factory does not currently support to customize the commit message. We can upvote the feedback here to let Microsoft improve this feature.
I’m investigating alternatives in my projects for my team (20+ developers). i want something that can be run on server and has auditing (check) that logs
Push
Pull
Clone of projects
I can’t find anything in the docs about this for GitLab community
Does anyone here know if these features are supported? or any plugin ?
Currently, the level of logging you're requesting has not been implemented into GitLabs, but there are plans in place to implement them, though there is currently no estimated time of arrival.
Audit events are richer now, with GitLab 14.3 (September 2021)
Audit events for merge request approval setting changes
Audit events are now created if changes are made to the merge request approval settings
in a project. You can now see if a change is made to the following policies:
Requiring user password for approvals.
Allowing modifying merge request approvals in a merge request.
Needing to get new approvals when a new commit is added to a merge request.
You can now be confident that once you configure approval settings, you can quickly see
if they are changed. This is a great way to show auditors that controls were put in place
and have not been removed or modified.
Thanks to Adrien Gooris from Michelin for this contribution!
See Documentation and Issue.
And GitLab 15.2 (July 2022) adds (for non-CE only)
Audit events for group-level merge request settings
GitLab now records additional audit events when changes are made to group-level merge request settings. These are in addition to project
audit events that record changes to the same settings on projects. Specifically, audit events are now created when changes are made to groups to:
Prevent approval by author
Prevent approvals by users who add commits
Prevent editing approval rules in projects and merge requests.
Require user password to approve
Remove all approvals when commits are added to the source branch
These audit events can help you know that the settings and default configurations for your group-level merge request settings have been put in place correctly and that they have not been changed.
This is especially important because these group-level settings
will cascade down to child projects.
Governance and visibility over these changes will help you strengthen separation of duties and further simplify audits.
See Documentation and Issue.
GitLab 15.2 (July 2022) also propose to audit a special kind of clone: forks.
But only for GitLab Ultimate, so again, not CE.
Streaming audit events for project forks
You can now monitor the project forking inside your groups with new audit events that are recorded whenever
a project is forked. This includes information such as:
The user name of the user that forked the project.
The timestamp of when the project was forked.
Details of the forked project.
This gives you visibility on where your projects and source code are being copied to, and by
whom, so that you can take action if needed.
These events potentially generate a high volume of data, so they are only available as
streaming audit events.
Thank you Linjie Zhang for this contribution!
See Documentation and Issue.
Where I work, our marketing team is looking for a "quick and easy" method up periodically updating some files on a website of ours. I opened my mouth and said "We can use Azure Websites with DropBox!". It all works fine, except that with DropBox, files only deploy if I log into the Azure Portal and click Sync. Needless to say, this is a deal breaker, because the users want to save a file and have everything appear magically.
Is there a way to setup continuous deployments via DropBox on Azure? I don't mind setting up a job to run every 15 minutes to perform a file upload if needed.. but would prefer to avoid that if possible
Thanks In Advance
Currently we don't support the continuous sync with Dropbox. The challenge is the noise and the reliability of the site given those changes. Imagine users naturally modify file by file and Dropbox sync them one at time. You can get into the situation where your site is in transient bad state.
This is not currently possible using the Dropbox integration in Azure Websites. Best option for this is the local git integration, where Azure will provide you a remote git location that you can push to that causes an update.
So that gives you behavior, but not the dropbox behavior you want, as someone would still need to commit and push.
To get that you could look into implement a Git hook to mimick the behavior, where you would auto commit and push when a file changes.
Something like this would give you that behavior, but you'd need to translate to a server-based model.
Git Repo Auto-commit and Push
Alternatively, you can host the site in GitHub or Visual Studio Online and I beleive you get that hook automatically.