Is there a way to determine if an already existing Azure Databricks Secret Scope is backed by Key Vault or Databricks via a python notebook? dbutils.secrets.listScopes() does not output this. Assume that I have Manage permissions on the scope.
(Unfortunately, Google didn't help)
You can do it either via Secrets REST API - if you use List Secret Scopes API, then backend_type field shows the backend - Datbricks or KeyVault. From notebook you can do it with following code:
import requests
ctx = dbutils.notebook.entry_point.getDbutils().notebook().getContext()
host_name = ctx.tags().get("browserHostName").get()
host_token = ctx.apiToken().get()
cluster_id = ctx.tags().get("clusterId").get()
response = requests.get(
f'https://{host_name}/api/2.0/secrets/scopes/list',
headers={'Authorization': f'Bearer {host_token}'}
).json()
scopes = dict([(s['name'], s.get('backend_type', 'DATABRICKS'))
for s in response['scopes']])
backend = scopes['scope-name']
Or you can do the same via databricks-cli, using the databricks secrets list-scopes command (see docs)
Related
How can we find existing secret scopes in databricks workspace. And which keyvault is referred by specific SecretScope in Azure Databricks?
This command lists available scopes on databricks:
dbutils.secrets.listScopes()
You can do this with either:
Databricks Secrets REST API - the list secret scopes API will give that information
Databricks CLI - the databricks secrets list-scopes command will show your KeyVault URL
You can try this snippet here in Python:
import pandas
import json
import requests
# COMMAND ----------
# MAGIC %md ### define variables
# COMMAND ----------
pat = 'EnterPATHere' # paste PAT. Get it from settings > user settings
workspaceURL = 'EnterWorkspaceURLHere' # paste the workspace url in the format of 'https://adb-1234567.89.azuredatabricks.net' Note, the URL must not end with '/'
# COMMAND ----------
# MAGIC %md ### list secret scopes
# COMMAND ----------
response = requests.get(workspaceURL + '/api/2.0/secrets/scopes/list',\
headers = {'Authorization' : 'Bearer '+ pat,\
'Content-Type': 'application/json'})
pandas.json_normalize(json.loads(response.content), record_path = 'scopes')
I have happened to have written a blog post about this where a full Python script is provided to manage secret scopes in Azure Databricks.
I got a blob storage which I use as website.
This blob has a system assigned managed identity.
This identity is added to a key vault as access policy.
So actually it should be able to access the secrets.
But when I try it the way microsoft documented it I got an error.
const getSecret = async () => {
var credential = new DefaultAzureCredential({
ManagedIdentityClientId: "<blob-id>",
} as DefaultAzureCredentialOptions)
const keyVaultName = "<key-vault-name>"
const url = "https://" + keyVaultName + ".vault.azure.net"
const client = new SecretClient(url, credential)
const secret = await client.getSecret("function-key")
}
I got the error
Error: DefaultAzureCredential is not supported in the browser. Use InteractiveBrowserCredential instead.
at Module.60308 (defaultAzureCredential.browser.js:5:34)
Is this even possible?
Thanks!
Please check this azure-sdk-for-js issue according to which ,
interactive credentials is recommended instead of default
credentials.And for client side applications that run in the browser,
the InteractiveBrowserCredential is the only credential type that is
supported.Please check this github reference
So for interactive credentials for Node.js, if a clientId is
provided, the Azure AD app need to be configured to have a "Mobile and desktop applications" as redirect endpoint instead of web. See
set up redirect uri
See DefaultAzureCredential and examples
Also according to DefaultAzureCredential Class | Microsoft Docs
The following credential types if enabled will be tried, in order:
EnvironmentCredential >ManagedIdentityCredential >
SharedTokenCacheCredential > VisualStudioCredential >
VisualStudioCodeCredential > AzureCliCredential>
AzurePowerShellCredential >InteractiveBrowserCredential: uses browser to auth users - not enabled by default. Pass true to the DefaultAzureCredential to enable it.
I am using Azure function to do an action, in this action I need to get a secret from a keyvault.
I am using this code in order to get the secret
var azureServiceTokenProvider = new AzureServiceTokenProvider();
var keyVaultClient = new KeyVaultClient((authority, resource, scope) => azureServiceTokenProvider.GetAccessTokenAsync(resource));
var secret= await keyVaultClient.GetSecretAsync($"https://{KeyVaultName}.vault.azure.net/", "SecretName");
When I run it locally it's work but when I run the function in azure I am getting an error "Forbidden"
How can I get the secret from a keyVault inside my azure function?
Thanks!
Forbidden might indicate that the identity assumed by the Azure Function does not have access rights over the specific Azure Key Vault.
From the Azure Portal or via CLI/API, head into the relevant Azure Key Vault resource -> Access Policies -> Add Access Policy -> Assign the Azure Function identity with the following permissions:
Secret List
Secret Get
Instead of using the KeyVault client inside your Function, you can have it much more simple if you can use KeyVault-referenced AppSettings (depends a bit of course on your scenario if thats an option). https://learn.microsoft.com/en-us/azure/app-service/app-service-key-vault-references
Then you can simple use the secret like any setting and read it as an Env variable.
I want to get list of all instances in a project using python google client api google-api-python-client==1.7.11
Am trying to connect using method googleapiclient.discovery.build this method required credentials as argument
I read documentation but did not get crdential format and which credential it requires
Can anyone explain what credentials and how to pass to make gcp connection
The credentials that you need are called "Service Account JSON Key File". These are created in the Google Cloud Console under IAM & Admin / Service Accounts. Create a service account and download the key file. In the example below this is service-account.json.
Example code that uses a service account:
from googleapiclient import discovery
from google.oauth2 import service_account
scopes = ['https://www.googleapis.com/auth/cloud-platform']
sa_file = 'service-account.json'
zone = 'us-central1-a'
project_id = 'my_project_id' # Project ID, not Project Name
credentials = service_account.Credentials.from_service_account_file(sa_file, scopes=scopes)
# Create the Cloud Compute Engine service object
service = discovery.build('compute', 'v1', credentials=credentials)
request = service.instances().list(project=project_id, zone=zone)
while request is not None:
response = request.execute()
for instance in response['items']:
# TODO: Change code below to process each `instance` resource:
print(instance)
request = service.instances().list_next(previous_request=request, previous_response=response)
Application default credentials are provided in Google API client libraries automatically. There you can find example using python, also check this documentation Setting Up Authentication for Server to Server Production Applications.
According to GCP most recent documentation:
we recommend you use Google Cloud Client Libraries for your
application. Google Cloud Client Libraries use a library called
Application Default Credentials (ADC) to automatically find your
service account credentials
In case you still want to set it manaully, you could, first create a service account and give all necessary permissions:
# A name for the service account you are about to create:
export SERVICE_ACCOUNT_NAME=your-service-account-name
# Create service account:
gcloud iam service-accounts create ${SERVICE_ACCOUNT_NAME} --display-name="Service Account for ai-platform-samples repo"
# Grant the required roles:
gcloud projects add-iam-policy-binding ${PROJECT_ID} --member serviceAccount:${SERVICE_ACCOUNT_NAME}#${PROJECT_ID}.iam.gserviceaccount.com --role roles/ml.developer
gcloud projects add-iam-policy-binding ${PROJECT_ID} --member serviceAccount:${SERVICE_ACCOUNT_NAME}#${PROJECT_ID}.iam.gserviceaccount.com --role roles/storage.objectAdmin
# Download the service account key and store it in a file specified by GOOGLE_APPLICATION_CREDENTIALS:
gcloud iam service-accounts keys create ${GOOGLE_APPLICATION_CREDENTIALS} --iam-account ${SERVICE_ACCOUNT_NAME}#${PROJECT_ID}.iam.gserviceaccount.com
Once it's done check whether the ADC path has been set properly by checking:
echo $GOOGLE_APPLICATION_CREDENTIALS
Having set the ADC path, you don't need to import from code the service access key, which undesirable, so the code looks as follows:
service = googleapiclient.discovery.build(<API>, <version>,cache_discovery=False)
I'm trying to add a key in my Azure AD application using Azure CLI.
But looking throught the Azure CLI API it seems that there is no such command.
For exmaple:
I'm trying to automate the task from the link below via Azure CLI:
http://blog.davidebbo.com/2014/12/azure-service-principal.html
I can create AD application, service principal, but I can't find a way to add key for newly create AD application.
I'll appreciate any ideas and directions :)
Thanks in advance !
For a new AD application, you can specify a key with -p while creating. For example,
azure ad app create -n <your application name> --home-page <the homepage of you application> -i <the identifier URI of you application> -p <your key>
For an existing AD application, surely the Graph API is able to update the AD Application Credential. Read this API reference, and you can see that the password credential is able to use "POST, GET, PATCH". However, it's too complicated to use the Graph API. I have check the Azure CLI. That functionality is not yet implemented, and the source is unreadable for me. Then, I took a look at Azure SDK for Python, because I am familiar with python, and I found out that they have already implemented it in 2.0.0rc2. See the GitHub Repo
I have written a python script. But, in order to use my script you need to install not only azure2.0.0rc2, but also msrest and msrestazure.
from azure.common.credentials import UserPassCredentials
from azure.graphrbac import GraphRbacManagementClient, GraphRbacManagementClientConfiguration
from azure.graphrbac.models import ApplicationCreateParameters, PasswordCredential
credentials = UserPassCredentials("<your Azure Account>", "<your password>")
subscription_id = "<your subscription id>"
tenant_id = "<your tenant id>"
graphrbac_client = GraphRbacManagementClient(
GraphRbacManagementClientConfiguration(
credentials,
subscription_id,
tenant_id
)
)
application = graphrbac_client.application.get('<your application object id>')
passwordCredential = PasswordCredential(start_date="2016-04-13T06:08:04.0863895Z",
end_date="2018-04-13T06:08:04.0863895Z",
value="<your new key>")
parameters = ApplicationCreateParameters(application.available_to_other_tenants,
application.display_name,
"<the homepage of your AD application>",
application.identifier_uris,
reply_urls=application.reply_urls,
password_credentials = [passwordCredential])
application = graphrbac_client.application.update('<your application object id>', parameters)
The only problem with this script is that you are only able to override all the existing keys of you AD application. You are not able to append a new key. This is a problem of the Graph API. The Graph API does not allow users to read an existing key. One possible solution would be storing your existing keys somewhere else. But, this will bring extra security risk.
I don't have any experience of automating adding the key, I'm not sure it's even possible to be honest. However have a look at the ApplicationEntity documentation in the Graph API, it might be possible using a POST request to the web service.