I'm trying to deploy an ADF pipeline using CICD (Azure DevOps Deployment) using a release pipeline.
Here I'm trying to merge stuff from my Collaboration branch to master (usig GIT).
I'm also using:
Azure Deployment:
Create Or Update Resource Group action on SiteOpsConsolidatedProd (Agent Job) as a part of the Continuous Deployment Process
I'm getting the following error:
[error]At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.
[error]Details:
##[error]BadRequest: The document creation or update failed because of invalid reference 'irslinked'.
##[error]Check out the troubleshooting guide to see if your issue is addressed: https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/deploy/azure-resource-group-deployment?view=azure-devops#troubleshooting
##[error]Task failed while creating or updating the template deployment.
irslinked is the name of a self-hosted IR (Linked)
this IR: irslinked is also included in the overwrite template parameters list.
As suggested by Microsoft in their ADF CICD best practices, the self-hosted IR should be hosted on a dedicated ADF instance and should be of type shared and linked to the dev/test/prod instances.
Link to the documentation: https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment#best-practices-for-cicd
Please respond if it works.
This may be helpful for someone. I had a similar invalid reference error. Actually, my ADF was imported from git. I have reconfigured the git with "import existing resource to repository" option.
Azure documentation https://learn.microsoft.com/en-us/azure/data-factory/ci-cd-github-troubleshoot-guide#recover-from-a-deleted-data-factory
If customer had a Self-hosted Integration Runtime in deleted ADF, they
will have to create a new instance in new ADF, also uninstall and
reinstall the instance on their On-prem machine/VM with the new key
obtained. After setup of IR is completed, customer will have to change
the Linked Service to point to new IR and test the connection or it
will fail with error invalid reference.
I had this error message but it referenced a resource that didn't even exist.
I was able to solve it using steps from here:
https://learn.microsoft.com/en-us/azure/data-factory/source-control#troubleshooting-git-integration
Notably:
Disconnect from existing git repo.
Reconnect back to the same repo, but select "import existing resources" into a "Create new git branch".
Then use git to create more commits on top of that branch, which remove all the extra crud that got generated, until the codebase is back to the desired state.
Note that I had to have sequential commits deleting stuff in dependency order, before it would work. i.e. Pipelines, then dataflows, then datasets, then linked services, then Int.Runtimes.
Related
I need to install a bunch of artifacts in Azure portal. I accidentally tried to install an artifact that will definitely fail due to some permission etc. Now it fails and blocks any further artifacts to install. Please see the screenshot.
I googled, but only found people talking about deleting already installed artifacts, or how to resolve installing problem so they can go ahead installing that failed artifact again.
But here, this artifact will fail to install anyway, so I have to remove it from the to-be-install list.
How can I remove this artifact from the install list?
I tried to reproduce your scenario on my end and got below results:-
AFAIK, Applying artifacts to the VM is a one time manual process. And once the base Artifacts are applied and they fail, there’s no option for now to delete the failed artifacts as they are already deployed inside the VM OS. You can only inspect the reason behind the failure and create a new VM inside the lab with correct artifacts again. While creating a new VM you can choose to delete/add correct artifacts. Apart from azure Portal even Rest API’s do not yet support deleting applied artifacts from the VM refer below :-
https://learn.microsoft.com/en-us/rest/api/dtl/virtual-machines
Created one VM in the Dev-Test lab and applied Artifacts which failed during creation : -
In Portal there’s no option to delete the failed artifacts for now. But, You can inspect the reason behind the failure like below :-
Or Even find the logs in the Dev-test Lab’s storage account like below :-
You can create a new VM by adding correct artifacts like below :-
Reference :-
Azure DevTest lab VM artifacts - Stack Overflow by Erikson Bahr
I want to create an Azure "Managed App" definition, in preparation for making an Azure Marketplace offering. I am following these MS instructions, and I had specifically been using this MS example managed app. There were was an error(s) in the documentation, which I posted to the MS team (along with my proposed fixes). Nevertheless, I did get the MS example working!
My next step was to replace the original MS sample deployment bundle...
https://raw.githubusercontent.com/Azure/azure-managedapp-samples/master/Managed%20Application%20Sample%20Packages/201-managed-storage-account/managedstorage.zip
...with my own deployment bundle...
https://github.com/brentarias/azureStaticEmpty/raw/master/baselinepocapp.zip
This didn't work. When issuing the az managedapp definition create command, I received the following error:
(DownloadItemFromBlobFailed) Download of the item from blob at 'https://github.com/brentarias/azureStaticEmpty/raw/master/baselinepocapp.zip' failed due to a failed connection.
Code: DownloadItemFromBlobFailed
Message: Download of the item from blob at 'https://github.com/brentarias/azureStaticEmpty/raw/master/baselinepocapp.zip' failed due to a failed connection.
It makes no sense to have a "connection" error, so I assumed that the REAL error was something inside of my deployment bundle. To test that theory, I copied the original MS sample bundle to a variety of places that I control, including Azure BLOB storage. One example location I placed the copied MS deployment file was here:
https://github.com/brentarias/azureStaticEmpty/raw/master/managedstorage.zip
When using this latter URL, I still received the same "connection" error.
In short, the only way for me to bypass the "connection" error is if I use the original sample MS deployment, from the original path that MS supplied. Incidentally, I also tried a variant URL of the original MS sample:
https://github.com/Azure/azure-managedapp-samples/raw/master/Managed%20Application%20Sample%20Packages/201-managed-storage-account/managedstorage.zip
Suddenly the deployment works! However, that location is still the original MS-owned repo "azure-managedapp-samples". This simply seems to confirm that if the deployment does not come from a MS-owned repo, I am then unable to make the deployment.
What am I doing wrong?
Update 2/3/2023
I finally found a way to make this work! When using an Azure storage account, simply having a publicly visible URL for the deployment bundle is insufficient. I need to have a "shared access signature" URL for that deployment bundle...and then the az managedapp definition create command works!
However, my overall question still is unanswered:
What are the valid file-share platforms that the az managedapp definition create supports? Besides github and Azure BLOB, what else?
What exact configuration do I need to make with a github raw link, before it is considered "kosher" by the managedapp definition create?
I get the following error in a pipeline that's first activity is to do a lookup on a storage container to get the contents of a file. When I test the connectionns, linked server, datasets or debug the pipeline I do not receive any errors. However when the pipeline is triggered by the storage event, it throws this error:
ErrorCode=AzureBlobCredentialMissing,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Please provide either connectionString or sasUri or serviceEndpoint to connect to Blob.,Source=Microsoft.DataTransfer.ClientLibrary,'
As per your scenario, where the debug is successful but the trigger runs failing. This make me assume that your dev changes have not been published which is why the trigger run fails. In simple terms the most recent published version of your linked service is different than that of your development version which haven't been published.
In case if you are using Source control then I would recommed following this tutorial for best practices - Automated publishing for continuous integration and delivery
If you are using CI-CD, then the issue might indeed cause by the DevOps pipeline not overriding the linked service parameters. Try redeploying the resource bye following below step and it should work as expected. (Linked service parameters had to be overwritten on the Azure resource template)
For example, if you have a linked service such as below:
Then you will still have to add below values into the overrideParameters section of the AzureResourceManagerTemplateDeployment task.
I want to clone an Azure pipeline that builds from a Git repo. Following the instructions here -- https://learn.microsoft.com/en-us/azure/devops/pipelines/get-started/clone-import-pipeline?view=azure-devops&tabs=classic#clone-a-pipeline , I clicked "Clone" when clicking "Edit" on the pipeline ...
and then on the following screen changed the branch from where the repository builds and clicked "Save". After entering my comment and saving, I got this error ...
The pipeline is not valid. Job Job_2: Step input connectedServiceNameARM references service connection 2b45...959e8d3d which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz. Job Job_1: Step input connectedServiceNameARM references service connection 2b45...959e8d3d which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz.
I "think" the error message is referring to the "Agent Job" part of the pipeline, which looks like the below
I'm clueless about what the error message is referring to and am curious about if there is a way to clone my pipeline.
Edit: In response to the answer given, turns out I don't have any service connections defined in my project at all. When I click on the "service connections" link in the project settings, all I see is this ...
I can reproduce your problem on my side. You need to check whether the service connections used in your pipeline exist in the Service connections of Project Settings. If the service connections have been deleted, you will get this error when you clone the pipeline.
Not enough reputation to comment here, but your error message makes it look like you are trying to deploy to Azure through your pipeline. This is probably not necessary for a simple build job. From the menu where you clone the pipeline try export YAML.
Export to YML menu pic
That will show the details of Job_2 and might give you the hint that you need to find the step in the UI.
If there isn't a corresponding step in the UI where you can clear out the service connection then you may be stuck creating a new pipeline and manually copying the steps you do have access to in the UI. You could also create a new pipeline based on the YAML file you just exported minus the problematic job.
I am building a report to summarize release definitions in Azure DevOps. This report will perform various checks to determine the health of the release definition. One of the checks I would like to implement is to determine if the Azure Service Connection the deployment phase is set to is still valid.
Currently I'm able to retrieve the YAML for the release definition, and the release instances however neither seem to indicate the specific service connection that's being used. The PowerShell commands I'm using are:
az pipelines release definition show
az pipelines release show
The YAML returned has several nodes such as deployPhases, deploySteps, releaseDeployPhases, deploymentJobs however for a release definition configured for a service connection I don't see any information in the release definition or the instance that indicates which service connection is being used.
The closest the YAML has to information about a service connection are references to variable such as $(ConnectedServiceName) and $(Parameters.ConnectedServiceName).
I'm also unable to find any information in either the Azure CLI or Rest API documentation on how to find the service connection for a release definition. The documentation I have checked is:
YAML Schema: https://learn.microsoft.com/en-us/azure/devops/pipelines/yaml-schema?view=azure-devops&tabs=schema
Azure-DevOps (Azure CLI): https://learn.microsoft.com/en-us/cli/azure/ext/azure-devops/?view=azure-cli-latest
Is it possible to query the CLI and retrieve Service Connection information based on the YAML returned for a release definition, or a release instance?
Service Connections are stored at Azure DevOps project level, so you can retrieve them from DevOps Rest API:
GET https://dev.azure.com/{organization}/{project}/_apis/serviceendpoint/endpoints?api-version=5.1-preview.2
More information at https://learn.microsoft.com/en-us/rest/api/azure/devops/serviceendpoint/endpoints/get%20service%20endpoints?view=azure-devops-rest-5.1