I am building a report to summarize release definitions in Azure DevOps. This report will perform various checks to determine the health of the release definition. One of the checks I would like to implement is to determine if the Azure Service Connection the deployment phase is set to is still valid.
Currently I'm able to retrieve the YAML for the release definition, and the release instances however neither seem to indicate the specific service connection that's being used. The PowerShell commands I'm using are:
az pipelines release definition show
az pipelines release show
The YAML returned has several nodes such as deployPhases, deploySteps, releaseDeployPhases, deploymentJobs however for a release definition configured for a service connection I don't see any information in the release definition or the instance that indicates which service connection is being used.
The closest the YAML has to information about a service connection are references to variable such as $(ConnectedServiceName) and $(Parameters.ConnectedServiceName).
I'm also unable to find any information in either the Azure CLI or Rest API documentation on how to find the service connection for a release definition. The documentation I have checked is:
YAML Schema: https://learn.microsoft.com/en-us/azure/devops/pipelines/yaml-schema?view=azure-devops&tabs=schema
Azure-DevOps (Azure CLI): https://learn.microsoft.com/en-us/cli/azure/ext/azure-devops/?view=azure-cli-latest
Is it possible to query the CLI and retrieve Service Connection information based on the YAML returned for a release definition, or a release instance?
Service Connections are stored at Azure DevOps project level, so you can retrieve them from DevOps Rest API:
GET https://dev.azure.com/{organization}/{project}/_apis/serviceendpoint/endpoints?api-version=5.1-preview.2
More information at https://learn.microsoft.com/en-us/rest/api/azure/devops/serviceendpoint/endpoints/get%20service%20endpoints?view=azure-devops-rest-5.1
Related
I get the following error in a pipeline that's first activity is to do a lookup on a storage container to get the contents of a file. When I test the connectionns, linked server, datasets or debug the pipeline I do not receive any errors. However when the pipeline is triggered by the storage event, it throws this error:
ErrorCode=AzureBlobCredentialMissing,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Please provide either connectionString or sasUri or serviceEndpoint to connect to Blob.,Source=Microsoft.DataTransfer.ClientLibrary,'
As per your scenario, where the debug is successful but the trigger runs failing. This make me assume that your dev changes have not been published which is why the trigger run fails. In simple terms the most recent published version of your linked service is different than that of your development version which haven't been published.
In case if you are using Source control then I would recommed following this tutorial for best practices - Automated publishing for continuous integration and delivery
If you are using CI-CD, then the issue might indeed cause by the DevOps pipeline not overriding the linked service parameters. Try redeploying the resource bye following below step and it should work as expected. (Linked service parameters had to be overwritten on the Azure resource template)
For example, if you have a linked service such as below:
Then you will still have to add below values into the overrideParameters section of the AzureResourceManagerTemplateDeployment task.
I can't seem to authorize access to my Azure subscription in Azure DevOps to run a build whenever a commit is pushed to master. I keep getting the below error:
Also, when I click Authorize resources, it says the authorization was successful, but the next time I run the pipeline, I get the same exact error. I verified in Project settings -> Service connections that I have an active connection to the subscription.
How can I get around this issue? When I go to Deployment Center in Azure Functions and wire up the connection there, it creates a task-based pipeline, but I want to use yaml.
The above indicates the azureSubscription you specified in your azure function deployment task doesnot exist, or you didnot have the permission.
If the service connection is already correctly setup, but you still encounter above error. You can follow below to troubleshoot the issue.
1, Check your yaml pipeline.
The azure subscription is validated at compile time. If you use variables to reference the azure subscription yaml pipeline. You need to make sure the variable can be retrieved at compile time.
You can check out this thread.
2, Check the service connection security setting.
Go to project settings-->Service Connections under Pipelines--> Select your azure service connection --> More settings(3 dots)-->Security-->Try adding your pipeline to the Pipeline permissions list.
If the azure subscription service connection is not set up. You need to create an service connection of azure Resource Manager type to connect to your azure subscription. See below steps:
1, Go to project settings-->Service Connections under Pipelines--> New Service connection-->Select Azure Resource Manager--> Next
2, Then select the Authentication method. If your azure devops is connected to AAD. You can select Service principal (automatic) as Authentication method. This will automatically create a service principal in your Azure AD.
3, If you want to create new service principal. You can select Service principal (manual). See below document to create service principal in Azure
Use the portal to create an Azure Active Directory application and a service principal that can access resources
Use Azure PowerShell to create an Azure service principal with a certificate
Then enter the related information in the service connection configuration page.
After the your azure subscription service connection is created. You can use it in your yaml pipeline task by specify the service connection name. See below example:
- task: AzureFunctionApp#1
displayName: Azure Function App Deploy
inputs:
azureSubscription: myAzureSubscription
Note: You need to add the correct role assignment for above service principal to enable the service principal to deploy to your azure resources.
You must create a new connection from the task itself (you may need to use the advanced options to add an existing service principal).
under "Azure subscription" click the name of the subscription you wish to use
Click the drop down next to "Authorize" and open advanced options
Click " use the full version of the service connection dialog."
Enter all your credentials and hit save
I spent a while trying to figure out why I got the same problem. Compared my yaml to another yaml I had worked on previously and couldn't spot any problems, also verified the service connections.
But as #Levi Lu-MSFT mentions, verifying the yaml lead me to finding what caused my issue so I thought I'd share it here even though it's not 100% related:
My variables weren't indented correctly. I was a bit tired and thought DevOps was just goofing with me. So verify that your yaml is properly setup. Sometimes it can be really small things that causes these issues.
I am currently planning on a type of multi-tenant system, were different resource groups with a set of AppServices are deployed for customers via ARM Templates. Hence, each customer has its own Resource Group and set of AppServices. Currently we use Azure DevOps to deploy to a set of AppServices used for Development and Quality Assurance before it gets to Production. I am now trying to incorporate DevOps into the mix, automating a pipeline creation of some sort... (it would be a copy of an existing pipeline but only changing the Target AppServices). Which is were my question comes from, Is there a way to dynamically create or edit a Release pipeline to add the deployment of those new AppServices, without the need of manually edit or create a pipeline an adding those newly created AppServices, I was thinking something around the lines of being able to copy a yaml file template then replacing the necessary info to point to those AppServices after they have been created, but I am not totally sure where could I store the new yaml file so that it is picked up by Azure DevOps, or how could I would accomplish these, with the main idea being that all of this continues to be part of an automated process (if possible).
Thanks a lot for any help, any suggestion is appreciated.
EDIT:
The question is not about how to Deploy an ARM Template through the DevOps release pipeline (I plan on using a PowerShell Script/REST API to accomplish that), instead, is about when the AppServices Resources are created, I need to deploy code to those newly created AppServices and also update that code when necessary (Hopefully through a Release Pipeline), somehow generate a new release pipeline each time I deploy a new set of Resources. So that, when there is a new update, I could easily have that pipeline triggered and that set if AppServices can be updated (created as part of the automation process "dynamically"). (I Already have a similar pipeline that deploys to a "static" set of AppServices).
This is possible as you eluded to with YAML Pipelines. Based upon the scenario you have subscribed each repository would have it's own pipeline.yml file that will define the trigger, pool etc. It would also reference a repository that will house your yaml template.
The template would accept whichever parameters you may required (resource group, app service name, etc...) The triggering pipeline associated with each repository would pass this information leveraging the teamplate.
By doing this CI/CD can be set up to trigger on the individual pipelines and deploy the appropriate code all while leveraging the same YAML template.
The repository reference would be similar to:
resources:
repositories:
- repository: YAMLTemplates
type: git
name: OrginazationName/YAML Project Name
With the call to the template being similar to:
- template: azure-ARM-template.yml#YAMLTemplate
parameters:
appServiceName: 'AppServiceName'
resourceGroupName: 'ResourceGroupName'
UPDATE
At a high level the YAML pipeline would consist of the following. If all App Services are similar as stated and ARM Templates are similar this how it could be constructed and triggered based on a folder path:
Build necessary artifacts
Publish Pipeline
Deploy Azure Resource Group Task
Deploy App Settings Task (if applicable)
Deploy App Service
Release the deployment pieces for each environment in appropriate stages to help alleviate the amount of copying and pasting each of the above tasks can be part of a template either individually at a task, combination of tasks, or all in one. This would allow for defining the YAML once and referencing it and including app specific components as needed as parameters to the templates.
I'm trying to deploy an ADF pipeline using CICD (Azure DevOps Deployment) using a release pipeline.
Here I'm trying to merge stuff from my Collaboration branch to master (usig GIT).
I'm also using:
Azure Deployment:
Create Or Update Resource Group action on SiteOpsConsolidatedProd (Agent Job) as a part of the Continuous Deployment Process
I'm getting the following error:
[error]At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.
[error]Details:
##[error]BadRequest: The document creation or update failed because of invalid reference 'irslinked'.
##[error]Check out the troubleshooting guide to see if your issue is addressed: https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/deploy/azure-resource-group-deployment?view=azure-devops#troubleshooting
##[error]Task failed while creating or updating the template deployment.
irslinked is the name of a self-hosted IR (Linked)
this IR: irslinked is also included in the overwrite template parameters list.
As suggested by Microsoft in their ADF CICD best practices, the self-hosted IR should be hosted on a dedicated ADF instance and should be of type shared and linked to the dev/test/prod instances.
Link to the documentation: https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment#best-practices-for-cicd
Please respond if it works.
This may be helpful for someone. I had a similar invalid reference error. Actually, my ADF was imported from git. I have reconfigured the git with "import existing resource to repository" option.
Azure documentation https://learn.microsoft.com/en-us/azure/data-factory/ci-cd-github-troubleshoot-guide#recover-from-a-deleted-data-factory
If customer had a Self-hosted Integration Runtime in deleted ADF, they
will have to create a new instance in new ADF, also uninstall and
reinstall the instance on their On-prem machine/VM with the new key
obtained. After setup of IR is completed, customer will have to change
the Linked Service to point to new IR and test the connection or it
will fail with error invalid reference.
I had this error message but it referenced a resource that didn't even exist.
I was able to solve it using steps from here:
https://learn.microsoft.com/en-us/azure/data-factory/source-control#troubleshooting-git-integration
Notably:
Disconnect from existing git repo.
Reconnect back to the same repo, but select "import existing resources" into a "Create new git branch".
Then use git to create more commits on top of that branch, which remove all the extra crud that got generated, until the codebase is back to the desired state.
Note that I had to have sequential commits deleting stuff in dependency order, before it would work. i.e. Pipelines, then dataflows, then datasets, then linked services, then Int.Runtimes.
Currently, trying to configure a release pipeline in Azure Devops. I've successfully created a CI pipeline that is working and running tests successfully. My release pipeline uses the artifacts from latest the CI build and should be deploying to azure. When I look at the logs I can see the the following were completed successfully:
Initialize Agent
Initialize job
Download arifact
and then the "Deploy to Azure App Service" fails with the logs showing the following details:
2019-02-07T19:06:57.5433805Z Got service connection details for Azure App Service:'MyApp'
2019-02-07T19:06:57.5487495Z ##[error]Error: Failed to get resource ID for resource type 'Microsoft.Web/Sites' and resource name 'MyAPP'. Error: Could not fetch access token for Managed Service Principal. Please configure Managed Service Identity (MSI) for virtual machine 'https://aka.ms/azure-msi-docs'. Status code: 400, status message: Bad Request
2019-02-07T19:06:57.5496340Z Failed to add release annotation. TypeError: Cannot read property 'getApplicationSettings' of undefined
I haven't been able to find much to help with this so far. I created the release pipeline by selecting Azure App Service deployment from the templates for stage 1 of the pipline (I only have one stage) and I added an artifact who's source is from my build pipeline. I then selected the Tasks tab and selected my Pay-as-you-go subscription under "Available Azure Service Connections". After selecting my subscription, my app service name became available in the "App service name" dropdown and I selected it. I then hit save and tried to create a release but every release fails with the above error message.
What can I do to overcome this?
Finally found a solution to my issue. The problem was in the way I set up my service connection. I went to:
project settings -> service connections -> new service connection -> selected azure resource manager -> selected service principal authentication INSTEAD OF managed identity authentication -> a popup came up that allowed me to log in and authorize.
Then I used that service connection when creating the release pipeline instead of the pre-existing Pay-As-You-Go subscription that I used last time.
Previously when I tried this, I had an adblocker running and it blocked the popup which allowed me to authorize when I selected service principal authentication. That resulted in an "Unable to authorize account" error message which is why I went with managed identity authentication in the first place.
In my case, it was related to a chrome extension I have added 'enable CORS' plugin. After disabling, release pipeline worked nicely