When running an Azure Powershell task in an Azure DevOps Release Pipeline with system.debug=true, you will get an output similar to this:
# anonymized
...
2019-09-05T12:19:41.8983585Z ##[debug]INPUT_CONNECTEDSERVICENAMEARM: '7dd40b2a-1c37-4c0a-803e-9b0044a8b54e'
2019-09-05T12:19:41.9156487Z ##[debug]ENDPOINT_URL_7dd40b2a-1c37-4c0a-803e-9b0044a8b54e: 'https://management.azure.com/'
2019-09-05T12:19:41.9188051Z ##[debug]ENDPOINT_AUTH_7dd40b2a-1c37-4c0a-803e-9b0044a8b54e: '********'
2019-09-05T12:19:41.9221892Z ##[debug]ENDPOINT_DATA_7dd40b2a-1c37-4c0a-803e-9b0044a8b54e: '{"subscriptionId":"b855f753-d5b3-48f4-b7cd-5beb58fb5508","subscriptionName":"Entenhausen","environment":"AzureCloud","creationMode":"Automatic","azureSpnRoleAssignmentId":"5ddcc3fe-f93c-4771-8041-50b49f76b828","azureSpnPermissions":"[{\"roleAssignmentId\":\"5ddcc3fe-f93c-4771-8041-50b49f76b828\",\"resourceProvider\":\"Microsoft.RoleAssignment\",\"provisioned\":true}]","spnObjectId":"76055cb6-3b75-4191-9309-306b32dad443","appObjectId":"e4b90b9d-7a73-42a3-ae6e-4daec910def4","environmentUrl":"https://management.azure.com/","galleryUrl":"https://gallery.azure.com/","serviceManagementUrl":"https://management.core.windows.net/","resourceManagerUrl":"https://management.azure.com/","activeDirectoryAuthority":"https://login.microsoftonline.com/","environmentAuthorityUrl":"https://login.windows.net/","graphUrl":"https://graph.windows.net/","managementPortalUrl":"https://manage.windowsazure.com/","armManagementPortalUrl":"https://portal.azure.com/","activeDirectoryServiceEndpointResourceId":"https://management.core.windows.net/","sqlDatabaseDnsSuffix":".database.windows.net","AzureKeyVaultDnsSuffix":"vault.azure.net","AzureKeyVaultServiceEndpointResourceId":"https://vault.azure.net","StorageEndpointSuffix":"core.windows.net","EnableAdfsAuthentication":"false"}'
2019-09-05T12:19:41.9284444Z ##[debug]AuthScheme ServicePrincipal
...
I need to add the SPN of the Azure DevOps connection to a resource. When changing subscriptions or pipelines, the SPN also changes and I do not want to hardcode the value.
As the value is printed in the system.debug=true output, I am wondering how to access my own SPN within a pipeline task. Is it possible to read out spnObjectId":"76055cb6-3b75-4191-9309-306b32dad443" somehow using Powershell?
Information about the Service Principal can be accessed using Get-AzureRmContext but the information is limited and some is obfuscated in the logs so you need to make a second call to Get-AzureRmServicePrincipal to access the ObjectId
$Context = Get-AzureRmContext
$AzureDevOpsServicePrincipal = Get-AzureRmADServicePrincipal -ApplicationId $Context.Account.Id
$ObjectId = $AzureDevOpsServicePrincipal.Id
The Id exposed in $Context.Account.Id is the Service Principals ApplicationId
SPN within a pipeline task is nothing but the Azure subscription you have passed on to the task. You can click on manage connections and copy the details of the SPN under connections and use them as you need. But, I am not sure why do you want to use the SPN directly as you can always use an Azure Powershell Task and just select the subscription. Once you store the Connection, you can always reuse it in different pipelines.
Related
There are many questions similar to mine, but I hit a wall finding what I want, please help.
I am running Azure Dev ops pipeline within the organization "MyOrganization" and in the project "MyProject". The service connection or subscription I am connected to is "Subscription-Id" (something like this: abc123-def456..xyz3265)
Pipeline has several tasks.
The first task is a powershell taks that creates Azure Resource Group and then an Azure Key vault. (NewKeyVault)
Second task, will scan another existing key vault (SourceKeyVault) and copy its secrets into the NewKeyVault. I know how to do this and it works just fine when I run the powershell tasks from within my PCwhen I explicitly log in with my log in to azure.
Howerer, here when ran under Azure Dev Ops pipleline, I get error that the "logged in" user has no permissions to Create, get list, etc.. to the secrets.
I want to automatically assign access policy to newly created key vault.
If using the web portal, I can see the Devops as a registered application and can do it. I don't know how to access it from within power shell task within the devops running pipeline.
Based on your expectation, you may consider using the Azure CLI pipeline task, where you can run az keyvault set-policy command to configure keyvault access policies for specific user.
az keyvault set-policy -n $(TheAzureKeyVaultName) --secret-permissions get list --object-id $(UserPrincipalGUID)
See more information on az keyvault set-policy.
steps:
- task: AzureCLI#2
displayName: 'Azure CLI '
inputs:
azureSubscription: 'ARM_Svc_Cnn_Auto_Sub1'
scriptType: ps
scriptLocation: inlineScript
inlineScript: 'az keyvault set-policy -n $(TheAzureKeyVaultName) --secret-permissions get list --object-id $(UserPrincipalGUID)'
This task requires to use the Azure Resource Manager service connection to authenticate and login Azure.
For the automatically created ARM service connection, you can use this API to find out which service principal that the ARM service connection is referenced to az login.
In my case, the ARM service connection is referencing the service principal MyDevOpsOrg-TheProjectName-MySubID which is the contributor of my target KeyVault (inherited from subscription) and has sufficient permission to set key vault access policy.
I was able to resolve an issue in a simple manner - I am using Azure Power shell
script that creates the key vault itself
New-AzKeyVault -VaultName $newKvName -ResourceGroupName $resourceGroupName -Location $location
(all the variables referenced with $ are arguments passed to the script)
And then I get the context for a logged in principal
(AzDevOps pipeline is a principal in itself, registered in Azure AD)
#This gives a context object, which has principal
#id as a property of.
$Context = Get-AzContext
Then I set that principal as access policy
Set-AzKeyVaultAccessPolicy -VaultName $newKvName -ServicePrincipalName $Context.Account.Id -PermissionsToSecrets Get,List,Set
Problem
I have an azure pipeline YAML file. It is able to run through a service connection which accesses a service principal with all the proper authority, etc.
But I am now trying to clean up the code; we have multiple service principals running on multiple subscriptions and resource groups. They need to create storage accounts, which need to be unique.
So I am trying to create a storage account built partially from the associated subscription and resource group of the service principal creating the storage account.
Example Solution
For the subscription, it is fairly easy. I can do something like this, from within a PowerShell script called inside the pipeline:
$subscriptionId = $(az account show --query 'id' -o tsv)
Write-Output "##vso[task.setvariable variable=AZURE_SUBSCRIPTION_ID;isoutput=true;issecret=true]$subscriptionId"
Now I have the variables $subscription ID and AZURE_SUBSCRIPTION_ID set, and can access subscription information within the pipeline itself.
Question
But how can I do something similar with resource groups?
There is no equivalent to az account show with resource groups, without knowing the resource group name itself. (Eg, I have to type az group show -name <RG-name>, but it is precisely the name that I am trying to get.)
Again, to be clear, I am running inside of a particular resource group and subscription, it is those that are associated with the service connection. Now I simply want that information available to the pipeline.
I'm not sure if I completely understand what you are trying to accomplish. But I suspect that the options below might help.
Get role assignments
If you created separate service connections for each individual resource group you can simply check the role assignments for the SPN and determine the scope of the service connection.
If you, for example, use the Azure PowerShell task, you have configured it with a Service Connection. So when the task starts, it has the context of the service principal. You can then do Get-AzRoleAssignment which should output the Resource Groups to which its authorised. Again, this is only useful if you use a service connection per RG, as you otherwise get results for multiple RGs. (Or for subscriptions and Management groups, if you also assigned a role to those scropes)
Use the Azure DevOps API
You can use the Get Service Endpoint request of the Azure DevOps API to get the service connections. The JSON output will contain information regarding the scope of the service connection.
If you find working with the API directly a bit hard, you can try the PSDevOps PowerShell module to interact with the Azure DevOps API. It has the Get-ADOServiceEndpoint command that allows you to get the available service endpoints.
In my Azure Pipeline (YAML), I am deploying an ARM template to create a key vault (among other resources), then running a PowerShell script to generate a certificate and store it in the key vault. This gives me a Forbidden error. In Use secrets from Azure Key Vault in Azure Pipelines, Microsoft suggests that I need to create an access policy on the key vault for the pipeline service principal. When I do this, the script succeeds.
Image from cache404
I would now like to create this access policy programmatically as part of the ARM template, but I don't know how to retrieve the object id for the pipeline service principal programmatically within the pipeline. Could someone please help?
Whilst Hugh Lin's answer is valid, I found it simpler to adapt Nick Graham's answer for granting key vault access to the pipeline service principal as part of the PowerShell script itself:
$Context = Get-AzContext
Set-AzKeyVaultAccessPolicy -VaultName $vaultName -ServicePrincipalName $Context.Account.Id -PermissionsToCertificates Get,List,Create
I would now like to create this access policy programmatically as part
of the ARM template, but I don't know how to retrieve the object id
for the pipeline service principal programmatically within the
pipeline.
If you want to automatically obtain the service principal object ID in the ARM template, I am afraid this is impossible.
You can try to create a script(Get-AzADServicePrincipal) to get the service principal and pass it to the arm template.
Here is the ticket with similar issue you can refer to.
I have created SPN for Azure devops pipeline and I need to access multiple subscription resources in a powershell task/deploymentScript ARM.I am using below command to switch between subscriptions
#Linking appinsight with storage account in Secondary Region
Get-AzSubscription -SubscriptionId $secSubscriptionId | Set-AzContext
This command works from my local powershell (as I have access to both subscriptions). But with CICD another subscription is not visible.I get below error even though both subs are under same tenant.
2020-05-12T17:17:14.9377680Z ##[error]Subscription XXXXXXX was not found in tenant ***. Please verify that the subscription exists in this tenant.
when you created the service principal, did you give it access to resources in both subscriptions?
I was trying to invoke data factory pipeline from azure function programmatically. Its throwing following error.
link:
http://eatcodelive.com/2016/02/24/starting-an-azure-data-factory-pipeline-from-c-net/
AuthorizationFailed: The client 'XXXX-XXXXX-XXXX' with object id 'XXX829e05'XXXX-XXXXX' does not have authorization to perform action
'Microsoft.DataFactory/datafactories/datapipelines/read' over scope
'/subscriptions/XXXXXX-4bf5-84c6-3a352XXXXXX/resourcegroups/fffsrg/providers/Microsoft.DataFactory/datafactories/ADFTestFFFS/datapipelines/ADFTutorialPipelineCustom'.
tried to search similar issues, but none of the search result gave me solution to my problem, Can you please guide us what could be the issue?
Objective is to, run data factory pipeline whenever file being added to blob. so to achieve the result we are trying to invoke data factory pipeline from azure function using blob trigger.
Step 1: login to your azure portal
Step 2: find Subscriptions in left side menu bar and click.
step 3: Click on Access Control IAM and then click on Add.
Step 4: In Add Permission window, select contributor for role. In select input box, type the app name you created in Azure AD (Created in Azure Active Directory)and select it. In my case I created Azure Resource Management.
Step 5:After you have given successful permission, click on Refresh in your subscription window and you will see your app showing in the list. See below example.
SEE Common problem when using Azure resource groups & RBAC
https://blogs.msdn.microsoft.com/azure4fun/2016/10/20/common-problem-when-using-azure-resource-groups-rbac/
This issue is more likely to happen in newer subscriptions and usually happens if a certain resource type has never been created before in that subscription.
Subscription admins often fix this issue by granting resource group owners contributor rights on the subscription level which contradicts with their strategy of isolating access down to the level of resource group level not the subscription level.
Root cause
Some admins say, that some resources require access to the subscription level to be able to create these resources and that ‘owner’ rights on a resource group level is not sufficient. That is not true.
Let’s take a step back to understand how this all works first.
To provision any resources in azure (using the resource manager model) you need to have a resource provider that supports the creation of that resource. For example, if you will provision a virtual machine, you need to have a ‘Microsoft.Compute’ resource provider available in the subscription first before you can do that.
Resource providers are registered on the level of the subscription only.
Luckily, the Azure Resource Manager (ARM) is intelligent enough to figure that out for you. When a new Azure resource gets provisioned, if the resource provider required for that resource type is not registered in the subscription yet, ARM will attempt to register it for you. That action (resource provider registration) requires access to the subscription level.
By default, any new azure subscription will be pre-registered with a list of commonly used resource providers. The resource provider for IoTHub for instance, is not one of them.
When a user is granted owner rights only on a specific resource group, if that user tries to provision a resource that requires registering a resource provider for the first time, that operation will fail. That is what happened in our case above when trying to provision IoThub.
So the bottom line is, we DO NOT need to grant access permissions to the subscription level for users to be able to create resources like HDInsight, IotHub and SQLDW …etc within their resource groups that they have owner rights on, as long as the resource providers for these resources is already registered.
You get the error that you are not authorized to perform action 'Microsoft.DataFactory/datafactories/datapipelines/read' over scope of pipeline because you don't have the relevant permissions on the datafactory.
You either need to have "Contributor" /"DataFactoryContributor" permissions to create & manage data factory resources or child resources. More details of the azure RBAC roles in the following link:
https://learn.microsoft.com/en-us/azure/active-directory/role-based-access-built-in-roles
Since the customer is trying to use the ADF client from inside Azure Function, the recommendation is to use AAD application and service principal for authentication of ADF client. You can find the instructions for creating AAD application and service principal here:
https://learn.microsoft.com/en-us/azure/azure-resource-manager/resource-group-authenticate-service-principal
Please follow the instructions on how to create the Active Directory application, service principal, and then assign it to the Data Factory Contributor role in the following link and the code sample for using service principal with ADF client.
We recently had this issue with the same message and found that it was caused by the user being logged in with a different subscription (we have 2). Using az login --subscription resolved the problem for us.
For anyone else running into a similar issue with the same error message - After "az login" I was recieving the same error when attempting to create a resource group as Owner, I solved this with:
az account set --subscription "Azure Subscription 1"
Basically it stems from the subscription not being set, you can find the details here:
https://learn.microsoft.com/en-us/cli/azure/manage-azure-subscriptions-azure-cli#get-the-active-subscription
Solution:
Step 1: Register an app in Azure Active directory.
Step 2: Assign 'Data Factory Contributor' role to the same app. we can achieve this by using power shell.
The below code works for me. Please try out in power shell after logged in with Azure credential.
Implementation:
Step 1: $azureAdApplication = New-AzureRmADApplication -DisplayName <AppName> -HomePage <URL> -IdentifierUris <URL with domain> -Password <Password>
Step 2: New-AzureRmRoleAssignment -RoleDefinitionName "Data Factory Contributor" -ServicePrincipalName $azureAdApplication.ApplicationId
Follow this post : https://learn.microsoft.com/en-us/azure/azure-resource-manager/resource-group-create-service-principal-portal
In this post , Role is given as "Reader" which should be "Owner" instead otherwise it would give permission error on deployment.
I solved by following this post:
https://www.nwcadence.com/blog/resolving-authorizationfailed-2016
with the command in PowerShell:
Get-AzureRmResourceProvider -ListAvailable | Select-Object ProviderNamespace | Foreach-Object { Register-AzureRmResourceProvider -ProviderName $_.ProviderNamespace}
I solved by finding the Enterprise Application > Object ID.
(it is weird that it does not use App Reg > Application Id)
https://jeanpaul.cloud/2020/02/03/azure-data-factory-pipeline-execution-error/