Hope you are doing well.
Is there a command in the Spark Utilities to see all secrets in an Azure Key Vault run in an Azure Synapse Spark Notebook?
Page https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/microsoft-spark-utilities?pivots=programming-language-python#credentials-utilities
I can reference a Secret Like so mssparkutils.credentials.getSecret('azure key vault name','secret name') but can I list all the secrets?
Thanks
Unfortunately, there is no command available to list all secrets in Key Vault.
You may checkout my answer on MS Q&A platform on how to use Access Secret from vault using Synapse pyspark notebook.
Appreciate if you could share the feedback on our Azure Synapse feedback channel. Which would be open for the user community to upvote & comment on. This allows our product teams to effectively prioritize your request against our existing feature backlog and gives insight into the potential impact of implementing the suggested feature.
You can call Azure Key Vault (AKV) via its REST API and the GetSecret methods, which returns a list of secrets in their full URL form. You could use a Web activity in Synapse pipelines to call this. Example settings:
Setting
Value
Notes
URL
{vaultBaseUrl}/secrets?api-version=7.2
See below for sample URL
Method
GET
Authentication
Managed Identity
Resource
https://vault.azure.net
Sample Key Vault URL
https://yourKeyVault-akv.vault.azure.net/secrets?api-version=7.2
Sample results:
{
"value": [
{
"id": " https://yourKeyVault-akv.vault.azure.net/secrets/somepassword ",
"attributes": {
"enabled": true,
"created": 1635948403,
"updated": 1635948403,
"recoveryLevel": "Recoverable+Purgeable",
"recoverableDays": 90
},
"tags": {}
},
{
"id": " https://yourKeyVault-akv.vault.azure.net/secrets/someusername ",
"attributes": {
"enabled": true,
"created": 1635949171,
"updated": 1635949171,
"recoveryLevel": "Recoverable+Purgeable",
"recoverableDays": 90
},
"tags": {}
}
],
You are able to loop through the values with a For Each activity, eg the Items value would be:
#activity('Web Get AKV Secrets').output.value
Refer to the individual secret inside the For Each activity like this:
#item.id
Get the actual secret name by using split and last functions, eg
#last(split(item().id, '/'))
You could then pass the individual secret name or the collection into a Synapse notebook as a parameter.
Related
I have a requirement, where I needed to fetch the tenantDirectoryId for a given subscription.
I could find a rest get api https://management.azure.com/subscriptions/[subscription]/versions...
The error response to this gave the tenant directory.
Is there any better way to fetch directory for a subscriptionId.
As of now (06/07/2018), an easy approach would be running az account show in the Azure Cloud Shell (requires a Storage Account) in the Azure Portal.
--- Command ---
az account show
--- Command Output ---
{
"environmentName": "AzureCloud",
"id": "{Subscription Id (GUID)}",
"isDefault": true,
"name": "{Subscription Name}",
"state": "Enabled",
"tenantId": "{Tenant Id (GUID)}",
"user": {
"cloudShellID": true,
"name": "{User email}",
"type": "user"
}
}
See this doc for more details on Azure Cloud Shell: https://learn.microsoft.com/en-us/azure/cloud-shell/overview
We have recently had the same problem we wanted to fix. After speaking to Microsoft there is no supported way to do this through the Graph API when accessing as an application using the client_credentialflow.
An approach we have taken to get the tenant id is to call the OAuth2 metadata document endpoint for your tenant
https://login.microsoftonline.com/{tenant}/v2.0/.well-known/openid-configuration
By calling this you’ll see your token endpoint as well as other useful information listed which you can utilise to log in. Or you can grab the tenant id using string stripping.
I want to set up CI/CD (ARM template) with StreamAnalytics Job with output set to DataLake Store.
https://learn.microsoft.com/en-us/azure/templates/microsoft.streamanalytics/streamingjobs/outputs#microsoftdatalakeaccounts
The issue comes with refreshToken:
"It is recommended to put a dummy string value here when creating the data source
and then going to the Azure Portal to authenticate the data source
which will update this property with a valid refresh token"
Furthermore after 90-days refresh token is outdated and you need to do "Renvew Authorization"
https://learn.microsoft.com/en-us/azure/stream-analytics/stream-analytics-data-lake-output#renew-data-lake-store-authorization
I tried to authorize ServicePrincipal.
How to do automatic deployment for ASA with DataLake?
How to handle issue with this 90-days token validitiy?
Maybe you wiped the trail :)
at this time it is not yet possible. Sorry for the inconvenience. However we know this is very important and we will add Service Principal auth in the near future (we are looking at the exact ETA).
In the meantime you need to renew manually the token. This can be done without losing any data by (1) stopping the job, (2) changing the token, then (3) restarting the job last time it was stopped.
Let me know if you have any further question.
As far as I know quite soon MSI-based authentication will be in preview.
But if you need an immediate solution (to e.g. be able to have a VSTS pipeline running through without error) you can do the following:
Create template (e.g. with the CICD NuGet Package [1])
Manipulate the ARM Template <jobName>.JobTemplate.json
Add the output datasource object for the ADLS output object
If you work with Visual Studio you can get the values quite easily from the ADLS output JSON
It is important to set refreshToken to some fake value
Like the following:
"outputs": [
{
"name": "xxx",
"properties": {
"serialization": {
"type": "Json",
"properties": {
"encoding": "UTF8",
"format": "LineSeparated"
}
},
"datasource": {
"type": "Microsoft.DataLake/Accounts",
"properties": {
"accountName": "xxx",
"tenantId": "xxx-xxx-xxx-xxx-xxx",
"tokenUserPrincipalName": "xxx#xxx.com",
"tokenUserDisplayName": "xxx, xxx",
"filePathPrefix": "xxx/{date}/{time}",
"dateFormat": "yyyy/MM/dd",
"timeFormat": "HH",
"refreshToken": "faketoken"
}
}
}
},
...
Deploy the ARM Template
The job will start successfully but it is necessary to renew the token, therefore
Stop the job
Renew the authentication of the ADLS output
Start the job
Resources
[1] CICD NuGet Package
I am using a Classic Web Service with a non-default endpoint for a Update Resource activity on the Azure Data Factory. This is the error I get:
Screenshot of Error
I didn't find any info on the web and couldn't figure it out myself. This website shows an example that I used by just filling in my values for mlEndpoint, apiKey and updateRessourceEndpoint:
{
"name": "updatableScoringEndpoint2",
"properties": {
"type": "AzureML",
"typeProperties": {
"mlEndpoint": "https://ussouthcentral.services.azureml.net/workspaces/xxx/services/--scoring experiment--/jobs",
"apiKey": "endpoint2Key",
"updateResourceEndpoint": "https://management.azureml.net/workspaces/xxx/webservices/--scoring experiment--/endpoints/endpoint2"
}
}
}
There is no mention of a token that needs to be passed...
this error is basically saying the apiKey you provided is invalid to perform the update resource operation. Here is some posts for your reference: https://social.msdn.microsoft.com/Forums/azure/en-US/3bb77e37-8860-43c6-bcaa-d6ebd70617b8/retrain-predictive-web-service-programmatically-when-do-not-have-access-to-managementazuremlnet?forum=MachineLearning
Please also be noted that if you modified your linked service in ADF, remember to re-deploy the pipeline as well to reflect your change in time.
I am trying to automate creating an API Connection for a storage account in Azure using Resource Manager templates.
I am using the listKeys method in ARM to retrieve the access key of the storage account. I went through this question and it is not working for me.
When I use the method in the outputs section of the template, it is working fine and successfully retrieving and displaying the access key.
"outputs": {
"listKeysOutput": {
"type": "string",
"value": "[listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('storagename')), providers('Microsoft.Storage', 'storageAccounts').apiVersions[0]).keys[0].value]"
}
}
However, when I try to use the same function inside a connection resource (as shown below), the template executes without any error. But on accessing the API Connection from the Azure portal, it says 'parameter is missing'.
"parameterValues": {
"accesskey": "[listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('storagename')), providers('Microsoft.Storage', 'storageAccounts').apiVersions[0]).keys[0].value]",
"accountName": "[parameters('storagename')]"
}
Am I missing something here? Or the output of listKeys is not being accepted by the 'accesskey' property?
I had a similar experience a few months ago, and resolved it by using a connection string directly in my code and then passing the connection string into the connections. The value looked like this:
[concat('DefaultEndpointsProtocol=https;AccountName=', variables('storageConfigs')[0].name,';AccountKey=', listKeys(resourceId('Microsoft.Storage/storageAccounts/', variables('storageConfigs')[0].name), variables('defaultStorageApiVersion')).key1)]
I used a storage config object as an input, so that's why it looks like above you could replace variables('storageConfigs')[0].name with whatever name or variable function you use in your code. Looks like above it may be storagename
Two things that might be causing the issue:
Ensure the API Connection has a dependency on the storage account
Capitalise the key in "accessKey" (some things in templates are case sensitive)
#Naren, I recommend you can use this API function to get your Storage Key
POST
https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Storage/storageAccounts/{accountName}/listKeys?api-version={api-version}
You could get the same result as the template.
{
“keys”: [
{
“keyName”: “key1”,
“value”: "key1Value”,
“permissions”: “FULL”
},
{
“keyName”: “key2”,
“value”: "key2Value”,
“permissions”: “FULL”
},
]
}
Just for your reference:
https://msdn.microsoft.com/en-us/library/mt163589.aspx
Dependency is indeed a requirement so that the storage account is already created before the deployment of the api connection is initiated.
The problem with the OP template code is the use of accesskey while the correct parameter name is accessKey (Note the capital K) for an Azure Blob api connection resource.
For anyone who struggles with the lack of documentation for the required parameters of API Connection resources - initiate this API Call:
https://management.azure.com/subscriptions/<YOUR SUBSCRIPTION ID>/providers/Microsoft.Web/locations/<YOUR LOCATION>/managedApis/<API TYPE>?api-version=2016-06-01
The <API TYPE> should be the api type of the connection to check for example azureblob, azurequeues or documentdb.
A description of all the expected parameters is returned along side other descriptive information for that resource.
I'm trying to derive the Registration Key and Url of my Azure Automation DSC account inside the ARM template at runtime. I've tried using the same syntax as you would for a storage account, ie.
listKeys(resourceId('Microsoft.Storage/storageAccounts', 'StorageAccountName'), '2015-05-01-preview').key1)
by doing this:
listKeys(resourceId('Microsoft.Automation/automationAccounts', 'AutomationAccountName'), '2015-05-01-preview').key1)
but no luck (it appears the function simply returns null). This would naturally make provisioning an automation account and and a VM and wiring up the VM to the automation account in the same template easy as pie. Has anyone successfully got something similar to work?
As per this GitHub Ticket, this is still under development.
https://github.com/azureautomation/automation-packs/issues/7
With version 2015-10-31 of the Azure Automation API, the following seems to work.
Getting the registration URL:
reference(resourceId('Microsoft.Automation/automationAccounts/', 'AutomationAccountName'), '2015-10-31').RegistrationUrl
Getting the Primary key:
listKeys(resourceId('Microsoft.Automation/automationAccounts/', 'AutomationAccountName'), '2015-10-31').keys[0].value
Getting the Secondary key:
listKeys(resourceId('Microsoft.Automation/automationAccounts/', 'AutomationAccountName'), '2015-10-31').keys[1].value
For reference, the object returned from the listKeys() template function for an Automation account resource looks like this (can easily be found by adding an output value using listKeys() to the outputs section of an ARM template):
{
"keys": [
{
"KeyName": "Primary",
"Permissions": "Full",
"Value": "VALUE OF PRIMARY KEY"
},
{
"KeyName": "Secondary",
"Permissions": "Full",
"Value": "VALUE OF SECONDARY KEY"
}
]
}