I get the following error in a pipeline that's first activity is to do a lookup on a storage container to get the contents of a file. When I test the connectionns, linked server, datasets or debug the pipeline I do not receive any errors. However when the pipeline is triggered by the storage event, it throws this error:
ErrorCode=AzureBlobCredentialMissing,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Please provide either connectionString or sasUri or serviceEndpoint to connect to Blob.,Source=Microsoft.DataTransfer.ClientLibrary,'
As per your scenario, where the debug is successful but the trigger runs failing. This make me assume that your dev changes have not been published which is why the trigger run fails. In simple terms the most recent published version of your linked service is different than that of your development version which haven't been published.
In case if you are using Source control then I would recommed following this tutorial for best practices - Automated publishing for continuous integration and delivery
If you are using CI-CD, then the issue might indeed cause by the DevOps pipeline not overriding the linked service parameters. Try redeploying the resource bye following below step and it should work as expected. (Linked service parameters had to be overwritten on the Azure resource template)
For example, if you have a linked service such as below:
Then you will still have to add below values into the overrideParameters section of the AzureResourceManagerTemplateDeployment task.
Related
I want to create an Azure "Managed App" definition, in preparation for making an Azure Marketplace offering. I am following these MS instructions, and I had specifically been using this MS example managed app. There were was an error(s) in the documentation, which I posted to the MS team (along with my proposed fixes). Nevertheless, I did get the MS example working!
My next step was to replace the original MS sample deployment bundle...
https://raw.githubusercontent.com/Azure/azure-managedapp-samples/master/Managed%20Application%20Sample%20Packages/201-managed-storage-account/managedstorage.zip
...with my own deployment bundle...
https://github.com/brentarias/azureStaticEmpty/raw/master/baselinepocapp.zip
This didn't work. When issuing the az managedapp definition create command, I received the following error:
(DownloadItemFromBlobFailed) Download of the item from blob at 'https://github.com/brentarias/azureStaticEmpty/raw/master/baselinepocapp.zip' failed due to a failed connection.
Code: DownloadItemFromBlobFailed
Message: Download of the item from blob at 'https://github.com/brentarias/azureStaticEmpty/raw/master/baselinepocapp.zip' failed due to a failed connection.
It makes no sense to have a "connection" error, so I assumed that the REAL error was something inside of my deployment bundle. To test that theory, I copied the original MS sample bundle to a variety of places that I control, including Azure BLOB storage. One example location I placed the copied MS deployment file was here:
https://github.com/brentarias/azureStaticEmpty/raw/master/managedstorage.zip
When using this latter URL, I still received the same "connection" error.
In short, the only way for me to bypass the "connection" error is if I use the original sample MS deployment, from the original path that MS supplied. Incidentally, I also tried a variant URL of the original MS sample:
https://github.com/Azure/azure-managedapp-samples/raw/master/Managed%20Application%20Sample%20Packages/201-managed-storage-account/managedstorage.zip
Suddenly the deployment works! However, that location is still the original MS-owned repo "azure-managedapp-samples". This simply seems to confirm that if the deployment does not come from a MS-owned repo, I am then unable to make the deployment.
What am I doing wrong?
Update 2/3/2023
I finally found a way to make this work! When using an Azure storage account, simply having a publicly visible URL for the deployment bundle is insufficient. I need to have a "shared access signature" URL for that deployment bundle...and then the az managedapp definition create command works!
However, my overall question still is unanswered:
What are the valid file-share platforms that the az managedapp definition create supports? Besides github and Azure BLOB, what else?
What exact configuration do I need to make with a github raw link, before it is considered "kosher" by the managedapp definition create?
I am trying to get the current pipeline runs using the web activity. I followed the below article, however, after having all the permissions required, it give the below error.
Invoking Web Activity failed with HttpStatusCode - '404 : NotFound',
message - 'The requested resource does not exist on the server. Please
verify the request server and retry
link followed
https://www.moderndata.ai/2021/12/how-to-prevent-concurrent-pipeline-execution-in-azure-data-factory-or-azure-synapse-analytics-design-1/
Any idea why I am facing the above error.
enter image description here
The issue occurs when you run the pipeline using Debug option from the top tool bar. It would work fine if you will run it using Trigger Now option.
When you try to query using the Web API, it only query the non-debug pipeline runs, means only triggered runs. Therefore, run it using trigger.
I want to clone an Azure pipeline that builds from a Git repo. Following the instructions here -- https://learn.microsoft.com/en-us/azure/devops/pipelines/get-started/clone-import-pipeline?view=azure-devops&tabs=classic#clone-a-pipeline , I clicked "Clone" when clicking "Edit" on the pipeline ...
and then on the following screen changed the branch from where the repository builds and clicked "Save". After entering my comment and saving, I got this error ...
The pipeline is not valid. Job Job_2: Step input connectedServiceNameARM references service connection 2b45...959e8d3d which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz. Job Job_1: Step input connectedServiceNameARM references service connection 2b45...959e8d3d which could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz.
I "think" the error message is referring to the "Agent Job" part of the pipeline, which looks like the below
I'm clueless about what the error message is referring to and am curious about if there is a way to clone my pipeline.
Edit: In response to the answer given, turns out I don't have any service connections defined in my project at all. When I click on the "service connections" link in the project settings, all I see is this ...
I can reproduce your problem on my side. You need to check whether the service connections used in your pipeline exist in the Service connections of Project Settings. If the service connections have been deleted, you will get this error when you clone the pipeline.
Not enough reputation to comment here, but your error message makes it look like you are trying to deploy to Azure through your pipeline. This is probably not necessary for a simple build job. From the menu where you clone the pipeline try export YAML.
Export to YML menu pic
That will show the details of Job_2 and might give you the hint that you need to find the step in the UI.
If there isn't a corresponding step in the UI where you can clear out the service connection then you may be stuck creating a new pipeline and manually copying the steps you do have access to in the UI. You could also create a new pipeline based on the YAML file you just exported minus the problematic job.
I'm trying to deploy an ADF pipeline using CICD (Azure DevOps Deployment) using a release pipeline.
Here I'm trying to merge stuff from my Collaboration branch to master (usig GIT).
I'm also using:
Azure Deployment:
Create Or Update Resource Group action on SiteOpsConsolidatedProd (Agent Job) as a part of the Continuous Deployment Process
I'm getting the following error:
[error]At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.
[error]Details:
##[error]BadRequest: The document creation or update failed because of invalid reference 'irslinked'.
##[error]Check out the troubleshooting guide to see if your issue is addressed: https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/deploy/azure-resource-group-deployment?view=azure-devops#troubleshooting
##[error]Task failed while creating or updating the template deployment.
irslinked is the name of a self-hosted IR (Linked)
this IR: irslinked is also included in the overwrite template parameters list.
As suggested by Microsoft in their ADF CICD best practices, the self-hosted IR should be hosted on a dedicated ADF instance and should be of type shared and linked to the dev/test/prod instances.
Link to the documentation: https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment#best-practices-for-cicd
Please respond if it works.
This may be helpful for someone. I had a similar invalid reference error. Actually, my ADF was imported from git. I have reconfigured the git with "import existing resource to repository" option.
Azure documentation https://learn.microsoft.com/en-us/azure/data-factory/ci-cd-github-troubleshoot-guide#recover-from-a-deleted-data-factory
If customer had a Self-hosted Integration Runtime in deleted ADF, they
will have to create a new instance in new ADF, also uninstall and
reinstall the instance on their On-prem machine/VM with the new key
obtained. After setup of IR is completed, customer will have to change
the Linked Service to point to new IR and test the connection or it
will fail with error invalid reference.
I had this error message but it referenced a resource that didn't even exist.
I was able to solve it using steps from here:
https://learn.microsoft.com/en-us/azure/data-factory/source-control#troubleshooting-git-integration
Notably:
Disconnect from existing git repo.
Reconnect back to the same repo, but select "import existing resources" into a "Create new git branch".
Then use git to create more commits on top of that branch, which remove all the extra crud that got generated, until the codebase is back to the desired state.
Note that I had to have sequential commits deleting stuff in dependency order, before it would work. i.e. Pipelines, then dataflows, then datasets, then linked services, then Int.Runtimes.
I've been trying to create an azure function which will move blobs from one container to another. For this purpose, I created a function following this tutorial.
Now when I add blob in container, I can debug the rest in local. Problem starts when I try to deploy the function. I'm using CICD for function deployment. After that, when I'm trying to configure the event subscription, it's failing.
I cannot edit the ENDPOINT DETAILS when creating event subscription
If I try to "Add event grid subscription", it gives a strange error message - Deployment has failed with the following error:
{"code":"Url validation","message":"The attempt to validate the
provided endpoint
https://.azurewebsites.net/admin/extensions/EventGridExtensionConfig
failed. For more details, visit https://aka.ms/esvalidation."}
All the tutorials I found are either irrelevant or outdated. Does anyone have any suggestion what I should do or what am I doing wrong?
Thanks in adance