I am getting following error while syncing a azure databricks notebook with an Azure DevOps repo I've linked it to:
Error while syncing Git history: Numeric value (3204746134) out of range of int? at [Source: {"id":
The repository branch specified in azure databricks is successfully created over on Azure DevOps.
I tried removing the revision history of the notebook that is no longer required to reduce the number of changes to be checked-in. This did not help to fix the error.
There was error while syncing git repo with Azure databricks when repo size is above 3 GB. This was confirmed by the products team and a fix has been deployed.
Related
I am using CI/CD for Azure data bricks to import note books to Workspace. Previously I used master/main branch to deploy in the Dev environment. But recently created a branch named "DEV" and using this for deploying to the dev environment. I am getting below error.
Additional information:
"DEV" branch have branch policies
I am using "Deploy Databricks Notebooks" tasks to import notebooks
I'm facing a problem while deploying an ADF project in the stating env using my CI/CD pipeline.
I'm using Azure DevOps.
I'm using a release pipeline, triggered when a new build is available (each validated PR on the main branch triggers a new build).
Pipeline has one stages : staging.
I am using a managed private endpoint connecting an ADLS account. This was causing me some problems.
I am getting following error while doing deployment to staging env,
InvalidResourceRequest: Invalid resource request. Resource type:
'ManagedPrivateEndpoint', Resource name: 'XXX-YYY'
'Error: A managed private endpoint to the data source already exist.'
I know that managed endpoint is already exist but cant we overwrite or skip it while doing the deployment to staging env.
Going through the best practices CI CD ADF
If a private endpoint already exists in a factory and you try to deploy an ARM template that contains a private endpoint with the same name but with modified properties, the deployment will fail.
We have tried to integrate the databricks notebook with a repository in Azure Repo. But, we are getting this below error
Error while syncing Git history: Response from Azure DevOps Services: status code: 500, reason phrase: ?{"$id":"1","innerException":null,"message":"Unable to complete authentication for user due to looping logins","typeName":"Microsoft.VisualStudio.Services.Identity.IdentityLoopingLoginException, Microsoft.VisualStudio.Services.WebApi","typeKey":"IdentityLoopingLoginException","errorCode":0,"eventId":4207}
Both Azure DevOps Services organization and Databricks located in the same tenant.
Repos Link format: https://dev.azure.com/myorg/project/_git/repo
When we tried valid token with GitHub getting an this below error.
I'm trying to deploy an ADF pipeline using CICD (Azure DevOps Deployment) using a release pipeline.
Here I'm trying to merge stuff from my Collaboration branch to master (usig GIT).
I'm also using:
Azure Deployment:
Create Or Update Resource Group action on SiteOpsConsolidatedProd (Agent Job) as a part of the Continuous Deployment Process
I'm getting the following error:
[error]At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.
[error]Details:
##[error]BadRequest: The document creation or update failed because of invalid reference 'irslinked'.
##[error]Check out the troubleshooting guide to see if your issue is addressed: https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/deploy/azure-resource-group-deployment?view=azure-devops#troubleshooting
##[error]Task failed while creating or updating the template deployment.
irslinked is the name of a self-hosted IR (Linked)
this IR: irslinked is also included in the overwrite template parameters list.
As suggested by Microsoft in their ADF CICD best practices, the self-hosted IR should be hosted on a dedicated ADF instance and should be of type shared and linked to the dev/test/prod instances.
Link to the documentation: https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment#best-practices-for-cicd
Please respond if it works.
This may be helpful for someone. I had a similar invalid reference error. Actually, my ADF was imported from git. I have reconfigured the git with "import existing resource to repository" option.
Azure documentation https://learn.microsoft.com/en-us/azure/data-factory/ci-cd-github-troubleshoot-guide#recover-from-a-deleted-data-factory
If customer had a Self-hosted Integration Runtime in deleted ADF, they
will have to create a new instance in new ADF, also uninstall and
reinstall the instance on their On-prem machine/VM with the new key
obtained. After setup of IR is completed, customer will have to change
the Linked Service to point to new IR and test the connection or it
will fail with error invalid reference.
I had this error message but it referenced a resource that didn't even exist.
I was able to solve it using steps from here:
https://learn.microsoft.com/en-us/azure/data-factory/source-control#troubleshooting-git-integration
Notably:
Disconnect from existing git repo.
Reconnect back to the same repo, but select "import existing resources" into a "Create new git branch".
Then use git to create more commits on top of that branch, which remove all the extra crud that got generated, until the codebase is back to the desired state.
Note that I had to have sequential commits deleting stuff in dependency order, before it would work. i.e. Pipelines, then dataflows, then datasets, then linked services, then Int.Runtimes.
Using azure data factory v2 with GIT / Azure DevOps integration:
If you for example create a trigger using Set-AzDataFactoryV2Trigger via powershell according to the documentation, the trigger is created directly in the adf_publish branch. This is an issue, as this will result in a mismatch between the master branch and adf_publish, meaning you'll not be able to publish going forward as this of course raises an error. How do I get the cmdlet to create the trigger in a new or specific branch, which I then can merge into master and publish the correct way?