You can create scopes in Databricks backed by Azure Keyvault instead of using the Databricks CLI. However, when you try to create a Scope, an obscure error message (with a spelling mistake!) is shown. It appears as not many people encounter this error:
"Internal error happened while granting read/list permission to Databricks ervice principal to KeyVault: XYZ"
Setting the Manage Principal to All Users does NOT help in this case.
I figured that this was a Service Principal issue in Azure AD. This particular user I was logged on to Databricks with was not an AD contributer and only had Contributer role on the Databricks and Keyvault service. I could not find any default Object ID in AD for Databricks so I assumed it was creating a service principal on the fly and connecting Databricks with Keyvault (I might be wrong here - it might already exist in AD when you enable the Databricks resource provider).
Logging in as an Admin with the rights to create service principals solved the problem. After that you can see in the Key Vault the DB service principal used in for the key retrieval:
As mentioned by #rcabr in his above comment there is already an SP by name 'AzureDatabricks' inside Enterprize Application, you need to get the object id details and add it in the access policy of the key vault. With this, the Databricks will be able to access the KeyVault
Related
I have a python script running on an Ubuntu Linux virtual machine that needs to access a KeyVault in Azure. If have configured the following environment variables based on an "App Registration".
AZURE_CLIENT_ID=<Client ID from App Registration>
AZURE_TENANT_ID=<Tenent ID from App Registration>
AZURE_CLIENT_SECRET=<SECRET Created in App Registration>
When I try accessing the KeyVault I get an access denied error.
Code: Forbidden
Message: The user, group or application 'appid=<APP_ID>;oid=
<GUID_VALUE>;iss=https://sts.windows.net/<GUID_VALUE>/' does not have secrets get permission
on key vault '<KEY_VAULT_NAME>;location=westus'. For help resolving this issue, please see
https://go.microsoft.com/fwlink/?linkid=2125287
Inner error: {
"code": "AccessDenied"
}
I assume I am getting this, because there is no Role Assignment in the KeyVault, however I may be wrong.
When I try adding a new Role Assignment to the KeyVault, there is no option under "Assign access to" for "App Registration". I tried both "User, Group or Service Principal" and "User Assigned Managed Identity", but it I always get the error "No users, groups, or service principals found.".
What am I missing here?
What am I missing here?
There are a few issues here:
You're trying to assign an RBAC role to an App Registration. What you need to do is assign an RBAC role to the Service Principal. To get the Service Principal Id, you would need to go to Enterprise Applications section in Azure AD and find the Id of your Service Principal (Object ID).
Assigning Reader RBAC role will not do the trick as this role is for managing the Key Vault itself and not the data inside it. There are two ways to solve this:
Use Access Policies: You can define appropriate access policies in your Azure Key Vault to give access to keys, secrets and certificates to your Service Principal.
Use Data RBAC Roles: Instead of using Management RBAC roles (like Reader, Contributor etc.), you will need to use RBAC roles for managing data inside the Key Vault.
Please see this link for more details: https://learn.microsoft.com/en-us/azure/key-vault/general/assign-access-policy-portal.
I need to create a service connection in Azure DevOps for service type "Azure Resource Manager".
When I am trying "Service principal (automatic)", I am able to do with proper permission. But when I am trying "Service principal (manual)", it needs service principle key. May I know how to create it from Azure Portal?
Yes, as mentioned by #juunas, the service principle key is also named client secret, you can create it in your App Registration -> Certificates & secrets in the portal, follow this link.
For the Forbidden error, it means your service principal does not have the correct RBAC role in your subscription, just navigate to your subscription in the portal -> add an RBAC role e.g. Contributor for your service principal like below, details here.
I'm trying to set up my Azure CDN endpoint to use HTTPS for the custom domain I already set up.
When I tried to point at the SSL cert in my Azure KeyVault, I got an error stating that I need to grant Azure CDN access to KeyVault. Any idea how I can do this -- hopefully through the Portal and NOT Powershell, though I have a feeling it'll end up requiring Powershell commands.
Basically, I'm trying to get my Azure CDN endpoint to use the SSL cert in my Azure KeyVault.
Anyway, I'd appreciate someone pointing to me an article or a set of instructions please. Thanks!
Instructions for enabling SSL with your own certificate stored in Azure KeyVault are described here:
https://learn.microsoft.com/en-us/azure/cdn/cdn-custom-ssl?tabs=option-2-enable-https-with-your-own-certificate#ssl-certificates.
There's something broken with this lately.
Azure tells you to do the thing that doesn't work:
New-AzADServicePrincipal -ApplicationId "205478c0-bd83-4e1b-a9d6-db63a3e1e1c8" -Role Contributor
If you're curious as to what it actually does, it just gives the hidden Microsoft.AzureFrontDoor-Cdn the role Contributor in your subscription's IAM. You can view it in the portal:
It seems that the "Contributor" role no longer contains the necessary permissions to read keyvaults - namely the Microsoft.KeyVault/vaults/secrets/readMetadata/action permission.
I've noticed that:
this permission is given to the built in Key Vault Secrets User role
all the roles that seem "superior" to the secrets user like Key Vault Administrator don't work, it HAS to be Key Vault Secrets User
So what fixed it for me is going into my keyvault and giving MYSELF (not azure cdn) the permission to read and list secrets. Seems that even as owner and god-emperor of your azure instance you can't access secrets by default ("AT" is me in the screenshot):
I am trying to link Azure Key Vault secrets to a variable group in Azure Pipelines (part in Azure DevOps). Microsoft documentation here.
However, the "Authorize" button does not seem to work. It spins endlessly. Screenshot.
My target Azure Key Vault already has the service principal included in its access policy with Get and List permissions. Screenshot.
Anyone seen this issue before?
This workaround also seems like a bug for Azure Key Vault deployments using ARM templates.
If the service principal in question is added to the Azure Key Vault (AKV) access policies through an ARM template by referencing the service principal's Object ID (as Microsoft documentation calls for), permission errors with Azure Pipelines follow.
However, if I manually add the service principal to the AKV's access policies by referencing the service principal's application (client) ID, the permissions errors go away entirely.
Again, feels like a bug. And now my automated deployment pipeline doesn't quite work because of this manual step.
Also, in the AKV ARM template, if I were to combine the mandatory field objectId with the optional field applicationId, the service principal shows up as a "compound identity". That does not fix the permissions issues in Azure Pipelines. I do not see a way of adding a service principal properly without doing it manually.
Firstly, please make sure the service connection is working correctly. Then refresh the page and try it again. Alternately you can also try in browser inprivate session.
Just as the message said "The specified Azure service connection needs to have "Get, List" secret management permissions on the selected key vault."
Basically, we need to click the "Authorize" button to enable Azure Pipelines to set these permissions for the specific service connection.
If that doesn't work, we can also manually set the permissions for the specific service connection.
Go to Project settings - > Service connections -> Select the
specific ARM service connection
Click Edit to popup the Update Authentication for xxx dialog
Click the "use the full version of the service connection dialog."
link, to get the Service principal client ID
Go to your key vault in Azure portal -> Access Policies -> Add a new
Access Policy -> Select a template (e.g Key&Secret Management) - >
Select Get, List for Secret permissions.
Click Select Principal -> Copy and paste the Service principal client ID
to search the user/application -> Select the searched
user/application
After that you can see the new APPLICATION access policy.
Try it again after successfully adding the application access policy.
UPDATE:
Generally in Azure DevOps we need to create a ARM service connection (the client which can access the azure sources) first before deploying an Azure Key Vault through an ARM template.
Actually when you select the Azure subscription then click Authorize in Azure resource group deployment task
, the ARM service connection is created automatically. You just need to check the AppID and get the ObjectID to use in the ARM template.
We can get the Service principal client ID (AppID) by following above steps. After that we can get ObjectId by the AppID with running the following command: (See Find service principal object ID using PowerShell for details.)
$(Get-AzureADServicePrincipal -Filter "AppId eq 'a89c3dee-f5bf-4ea1-a805-d4c729a4add3'").ObjectId
Then you can specific the ObjectId when deploying the Azure Key Vault through an ARM template.
When using Azure Key Vault management REST API or cmdlet Add-AzureRmKeyVaultNetworkRule to allow a virtual network to access a key vault, I get the following error:
The client '{guid}' with object id '{guid}' does not have authorization to perform
action 'microsoft.network/virtualnetworks/taggedTrafficConsumers/validate/action'
over scope '/subscriptions/{guid}/resourcegroups/{resource-group}/providers/microsoft.network/virtualnetworks/{vnet-name}/taggedTrafficConsumers/Microsoft.KeyVault'
What is wrong?
Your subscription is not giving Microsoft.KeyVault resource provider permission to access Microsoft.Network resources. The fix is to register your subscription to Microsoft.KeyVault again:
Register-AzureRmResourceProvider -ProviderNamespace Microsoft.KeyVault
This will add required permissions for Microsoft.KeyVault and Microsoft.Network integrations, including the ability to limit access to a given Virtual Network.
For more information: https://learn.microsoft.com/en-us/azure/azure-resource-manager/resource-manager-supported-services
This are the steps required to solve it:
https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/resource-providers-and-types#azure-portal
You just need to register the resource provider in the subscription, this doesn't only happens with Key Vault, my issue was with Sql Server as well :)
So I leave this answer here in case someone else needs it
This feels like a bug/limitation in both the Azure Portal and Azure CLI. We ran into this when trying to allow a subnet of a VNET in subscription X to access a storage account in subscription Y.
For us, the workaround was to look-up the name of the service principal that was mentioned in the error in our Azure AD directory using the "Search your tenant" box on the "Overview" tab of the directory (NOT the subscription but the Azure AD directory for the tenant). The name of the SP turned out to be "Storage Resource Provider" (yours may be different, so that's why you need to look it up in Azure AD), so we granted that SP "Owner" role (temporarily) in the other subscription. Then provisioning worked!
There should be a finer-grained set of permissions you need to grant than just "Owner" but when we granted just the "validate" permission, we got a new error:
Failed to save firewall and virtual network settings for storage account 'XXX'. Error: An operation is currently performing on this storage account that requires exclusive access.
Also experienced this error when adding a vnet to a storage-account in another subscription.
Fixed by adding a storage-account to the subscription using the portal. Then the vnet could be added to the storage-account.
Note: the result is the same as #fernacolo does with a powershell command.