I am trying to create a keyvault backed scope in databricks. I am able to successfully create the scope but when I try to add a key to the scope I see the following error:
Error: b'{"error_code":"BAD_REQUEST","message":"Cannot write secrets to Azure KeyVault-backed scope abc"}'
These are steps I have followed and all commands were run on windows cmd:
Create key vault in Azure
Generate AAD token for databricks - az account get-access-token --resource 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d
Add AAD token to environment variables on windows
Add AAD token to databricks cfg file on windows - databricks configure --aad-token
Create scope - databricks secrets create-scope --scope abc --scope-backend-type AZURE_KEYVAULT --resource-id <keyvault-id> --dns-name <keyvault-dns> --initial-manage-principal users
Add key to scope - databricks secrets put --scope abc --key abc-key << this where I see the error
According to the documentation this is not possible:
To reference secrets stored in an Azure Key Vault, you can create a secret scope backed by Azure Key Vault. You can then leverage all of the secrets in the corresponding Key Vault instance from that secret scope. Because the Azure Key Vault-backed secret scope is a read-only interface to the Key Vault, the PutSecret and DeleteSecret Secrets API 2.0 operations are not allowed. To manage secrets in Azure Key Vault, you must use the Azure SetSecret REST API or Azure portal UI.
Using Az CLI, you could use the az keyvault secret set command.
Related
I am using Terraform cloud and I don't want to use permanent keys in it. So, is there any to create a temporary keys in Azure Cloud(like we can create in AWS).
When you are authenticating to Azure Cloud via Azure service principal, by default, the Az CLI command will get a password for this service principal with a one-year expiration date.
az ad sp create-for-rbac --role="Contributor" --scopes="/subscriptions/<subscription_id>"
from your comments, in fact you want to get this password to expire in a short time. You can use az ad app credential reset to append or overwrite an application's password or certificate credentials.
For example, reset the application password with the following Az CLI commands.
az ad app credential reset --id <appId> --password <sp_password> --end-date 2020-08-13T11:59:59+00:00
For more information, you could read the Relationship between application objects and service principals
By creating a ServicePrincipal in AzureAD you're also able to assign a LifetimePolicy (tokenLifetimePolicies). This way you're able to have an "end of life" for the token.
Here's also a short how to on creating a new ServicePrincipal.
Alternatevily you could use this new preview feature: Configurable token lifetimes in Microsoft identity platform (Preview).
As it is a preview feature you're not supposed to use it in production environments.
I created a keyvault and secrets from the MS docs and confirmed that I can list the keyvault and secrets from an Azure CLI session. When somebody else granted my access to another keyvault in our Azure tenant, I am not able to list that vault from the Azure CLI.
The CLI command I am using is : 'az keyvault list'
I have compared the individual policies applied in two vaults, and have 'owner' role access to both. I can see the vaults and secrets when I use a browser to navigate to the azure portal.
Thanks in advance for any suggestions on what I should check.
I suppose the keyvault is in another subscription in your Azure AD tenant, in Azure CLI, you could just use az keyvault list to list the keyvaults in the default subscription, if you want to do operations in another subscription, you need to set the subscription with it.
Navigate to the keyvault which you want to list in the portal, copy the Subscription ID like below.
Then run the command below before you list the keyvaults.
az account set --subscription <Subscription ID>
az keyvault list
In my case, my login had expired, but unlike with other commands, I did not get a warning to that effect. It simply returned the [] empty list.
Renewing my login with az login allowed az keyvault list to work.
In my case I had to go to the subscription / RG where the key-vault was and give the user / service principal the Reader role. You can do that by clicking on the subscription/RG and then selecting "Access Control (IAM)" on the left side. And then add the role assignment. Instructions - https://learn.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal?tabs=current
I'm looking for a code sample which demonstrates how to connect to an azure key vault and grab a secret from the vault without having to store creds in plain text but instead using certificates and an SPN for authentication to the vault. Has anyone done anything like this before who is willing to shed some light on this?
I'm trying to securely retrieve credentials from azure without ever having to save creds locally, and this has proven to be harder than I originally thought.
yes you can use certificate based authentication while accessing Azure key vault.
Here is a very good article on the same using dot net.
Certificate base authentication
Alternatively Azure Key Vault provides a way to securely store credentials and other keys and secrets, but your code needs to authenticate to Key Vault to retrieve them. Managed Service Identity (MSI) makes solving this problem simpler by giving Azure services an automatically managed identity in Azure Active Directory (Azure AD). You can use this identity to authenticate to any service that supports Azure AD authentication, including Key Vault, without having any credentials in your code.
Run the assign-identity command to create the identity for this application:
az webapp identity assign --name <app_name> --resource-group "<YourResourceGroupName>"
This command is the equivalent of going to the portal and switching Managed service identity to On in the web application properties.
Assign permissions to your application to read secrets from Key Vault
{
"principalId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"tenantId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"type": "SystemAssigned"
}
Then, run this command using the name of your Key Vault and the value of PrincipalId copied from above:
az keyvault set-policy --name '<YourKeyVaultName>' --object-id <PrincipalId> --secret-permissions get
Deploy the Node App to Azure and retrieve the secret value
Deploy your node js app ,After this when you browse https://.azurewebsites.net you can see the secret value. Make sure that you replaced the name with your vault name
I am trying to connect MS Azure databricks with data lake storage v2, and not able to match the client, secret scope and key.
I have data in a Azure data lake v2. I am trying to follow these instructions:
https://docs.azuredatabricks.net/spark/latest/data-sources/azure/azure-datalake-gen2.html#requirements-azure-data-lake
I have created a 'service principle' with the role "Storage Blob Data Contributor", obtained
I have created secret scopes in both Azure Keyvault and Databricks with keys and values
when I try the code below, the authentication fails to recognize the secret scope & key. It is not clear to me from the documentation if it is necessary to use the Azure Keyvault or Databricks secret scope.
val configs = Map(
"fs.azure.account.auth.type" -> "OAuth",
"fs.azure.account.oauth.provider.type" -> "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",
"fs.azure.account.oauth2.client.id" -> "<CLIENT-ID>",
"fs.azure.account.oauth2.client.secret" -> dbutils.secrets.get(scope = "<SCOPE-NAME>", key = "<KEY-VALUE>"),
"fs.azure.account.oauth2.client.endpoint" -> "https://login.microsoftonline.com/XXXXXXXXXX/oauth2/token")
If anybody could help on this, please advise / confirm:
what should be CLIENT-ID : I understand this to be from the storage account;
where should the SCOPE-NAME and KEY-VALUE be created, in Azure Keyvault or Databricks?
The XXXX in https://login.microsoftonline.com/XXXXXXXXXX/oauth2/token should be your TenantID (get this from the Azure Active Directory tab in the Portal > Properties > DirectoryID).
The Client ID is the ApplicationID/Service Principal ID (sadly these names are used interchangeably in the Azure world - but they are all the same thing).
If you have not created a service principal yet follow these instructions: https://learn.microsoft.com/en-us/azure/storage/common/storage-auth-aad-app#register-your-application-with-an-azure-ad-tenant - make sure you grant the service principal access to your lake once it is created.
You should create a scope and secret for the Principal ID Key - as this is something you want to hide from free text. You cannot create this in the Databricks UI (yet). Use one of these:
CLI - https://docs.databricks.com/user-guide/secrets/secrets.html#create-a-secret
PowerShell - https://github.com/DataThirstLtd/azure.databricks.cicd.tools/wiki/Set-DatabricksSecret
REST API - https://docs.databricks.com/api/latest/secrets.html#put-secret
Right now I do not think can create secrets in Azure KeyVault - though I expect to see that in the future. Technically you could manually integrate with Key Vault using their API's but it would give you another headache in needing a secret credential to connect to key vault.
I was facing the same issue , the only thing i did extra was to assign the default permission of the application to datalake gen2's blob container in azure storage explorer . It required the object id of the application , which is not the one available on the UI , it can be taken by using the command "az ad sp show --id " on azure-cli .
After assign the permission on blob container, create a new file, and then try to access it,
I am trying to use azure keyvault in my MVC application to get storage account keys. This MVC application is hosted in different azure subscriptions with storage accounts belonging to those subscriptions. Is it possible to have a one Azure Keyvault resource in any subscription and serve keys for storage accounts residing in different subscriptions?
Yes it is possible.
You will need to make identities for the apps using the Key Vault in the Azure AD tenant where the Key Vault's subscription is. And then assign permissions to access the vault to those service principals.
Then you can put the client id, client secret and tenant id to the apps that need to access the Key Vault. They should then be able to call it, since they have an identity that is allowed access. Key Vault uses an HTTP API so whether the apps and the vault are in the same subscription/data center/cloud provider is irrelevant.
Hi you can use below to do via azure CLI
az webapp config ssl import -n 'webappname' -g 'webappresourcegroup' --key-vault "/subscriptions/[provide subscriptionID]/resourceGroups/[Provide resource group Name]/providers/Microsoft.KeyVault/vaults/[Provide Vault Name] --key-vault-certificate 'Provide certificate Name'