I need to generate Azure Databricks token using Powershell script.
I am done with creation of Azure Databricks using ARM template , now i am looking to generate Databricks token using powershell script .
Kindly let me know how to create Databricks token using Powershell script
The only way to generate a new token is via the api which requires you to have a token in the first place.
Or use the Web ui manually.
There is no official powershell commands for databricks, there are some unofficial ones but they still require you to generate a token manually first.
https://github.com/DataThirstLtd/azure.databricks.cicd.tools
Disclaimer I'm the author of these.
UPDATE: these powershell commands can now authenticate using a service principal instead of a bearer token (or can generate a bearer token for you).
so right now there is no way to use the API directly after deploying an Azure Databricks Workspace. I assume that you want to use it as part of an CI/CD pipeline - right? Reason is that you first need to manually create an API token which you can then use for all subsequent API requests.
But I will investigate and keep you updated here!
another option is to create it via terraform.
https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs/resources/token
mind you, it creates the token as whomever you az login'd as. so if you az login as yourself (when it spawns a browser asking who to log in as), that's who the token will be created as (assuming that user has permissions in the databricks workspace) and contributor (or custom read role, reader role doesn't grant the right permissions) permissions into the resource group that houses the workspace.
you can always use az login -u username#email.com -p to log in as someone else, assuming that user doesn't have MFA then run the terraform init/plan/apply. mind you, if you have a backend storage, that user also has to have permissions to that backend storage as well so it can create/update any tfstate files stored there.
Related
We actually have multiple azure accounts (for some valid reason) and I want to be able to run azure-cli commands for different accounts at the same time from the same machine.
The problem with that is, once I login to one azure account with azure login, token will be stored in ~/.azure directory so I am not sure if I can login into another account exactly at the same time on that machine.
Is there any way to tell azure-cli not to store token in local profile so that I can use azure-cli to connect to multiple accounts at the same time from same machine?
If you are using a windows or mac machine then the tokens are stored in Windows token manager or OSx key chain respectively. Only on Linux systems the tokens are stored in ~/.azure/azureProfile.json
However, you should still be able to login with multiple accounts on Win/Mac or Linux machines.
azure account set "subscription-name" will set the subscription as your default subscription and all the commands that you execute will run against that subscription.
Every command has a -s or --subscription switch where you can explicitly specify the subscription id. Even if the subscription belongs to a different account, it should still work if you have authenticated with that account.
For Linux system, I would suggest to create multiple user accounts and then run the CLI from those accounts. I think there could be a race condition when two commands from different accounts try to access ~/.azure/azureProfile.json.
The latest update is that the environment variable AZURE_CONFIG_DIR has been introduced and that can be set differently for each environment before az login is called.
export AZURE_CONFIG_DIR=/tmp1
az login
and on other window
export AZURE_CONFIG_DIR=/tmp2
az login
Reference: configure the AZURE_CONFIG_DIR for fixing concurrency issue
For Windows, here are steps
Go to env variables and add AZURE_CONFIG_DIR with the value of new config folder (e.x. C:\Users\YourUser\.azure-personal)
restart your cli, then run this az login --use-device-code
use the code given on step 2 and use it with whatever browser to login to new azure account
Now, one of your accounts config is in default azure folder config (C:\Users\YourUser\.azure) and new one lives in the place you specified on step 1.
if you wanna switch between them, you need to flip that env variable to point to whatever config you want
My deploy task using PowerShell script, which use Service Principal for connection to Azure KeyVault for pull secret. Secret (password) store in PowerShell script's code as plain text. Maybe there is another solution how to minimize token viewing.
And also i use powershell inline mode (not separate script) with Azure DevOps Secret Variable in deploy task, but this solution difficult to support (script has several different operations, so you have to keep many versions of the script).
Script is store in Git repository, anyone who has access to it will be able to see the secret and gain access to other keys. Perhaps I don't understand this concept correctly, but if keys cannot be stored in the code, then what should I do?
I devops you can use variable groups and define that the variables is pulled directly from a selected keyvault (if the service principal you have selected have read/list access to the KV) LINK.
This means that you can define all secrets in keyvault, and they would be pulled before any tasks happens in your yaml. To be able to use them in the script you can define them as a env variable or parameter to your script and just reference $env:variable or just $variable, instead of having the secret hardcoded in your script.
I copied a container to another storage account based on the document linked below.
(DataLake Storage Gen2).
When trying, I got the following error:
this request not authorized to perform this operations using this permission
https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-v10
If you are using AAD Token, this error is telling you that you need to add a role assignment to the user. Please go to Storage account -> Access Control -> Add -> Add role assignment, then add Storage Blob Data Owner to your login account.
If this problem persists, please provide more details.
I also faced the same problem. For me to work I just log out and log in again on the azcopy cli after doing the #BowmanZhu solution
azcopy logout
azcopy login --tenant-id xxxx-xxxx-xxxx
If you don't want to login that way there is always the option to add a SAS token at the end of the URL. If you don't want to attach the token always at the end you can try for permanent access by going through any one of these steps you find in the official documentation page.
After granting myself with role Storage Blob Data Owner on the container, then AzCopy will now behave itself and succeed in copying a file to the blob storage container.
go to storageaccount -> container -> Access control rules -> add role assignement -> Storage Blob Data Owner
In my case, my azure storage account vnet address was blocking the azcopy from copying the data over the storage account.
I added my client IP to allow a firewall address.
The SAS token has probably expired.
When I had this, I discovered it was because I'd used Azure Storage Explorer to generate a SAS that didn't have read permission, and I think it was trying to read the size/existence of a blob before writing it.
I got a clue from https://github.com/Azure/azure-storage-azcopy/issues/790 but ultimately I just regenerated a new SAS with read permission and it worked out..
I probably could ahve looked to modify the C# code using Azure Data Movement lib, to not perform a length check, but the spec was later changed to "don't overwrite" so the read permissions are probably needed anyway
Give appropriate permissions(read, write, create) while generating SAS tokens
as here
Had a similar issue. That's how was resolved
Command used was .\azcopy.exe copy "C:\Users\kriof\Pictures" "https://test645676535storageaccount.blob.core.windows.net/images?sp=rw&st=2022-02-23T11:03:50Z&se=2022-02-23T19:03:50Z&spr=https&sv=2020-08-04&sr=c&sig=QRN%2SMFtU3zaUdd4adRddNFjM2K4ik7tNPSi2WRL0%3D"
SAS token had default(Read) permission only. Adding Write permission
in Azure Portal, resolved the issue.
I am trying to understand how can I run a python application inside azure container that can spin up various resources like VM, route-table inside resource group of Azure.
http://azure-sdk-for-python.readthedocs.io/en/latest/
Looks like the examples provided in azure python sdk sets authentication stuff like credentials, subscription ID, etc.
Is there a way to avoid setting up of this information by using Role Based Access Control ? What if I provide container an Contributor access to the resource group, in that case, would i able to avoid setting up authentication variables and still achieve creating resources or atleast read/access resources in resource group ?
No, how do you think that would work? In order to be able to create\read\modify resources you need permissions (else anyone would be able to do that). There is no way to do that without some sort of Auth.
We actually have multiple azure accounts (for some valid reason) and I want to be able to run azure-cli commands for different accounts at the same time from the same machine.
The problem with that is, once I login to one azure account with azure login, token will be stored in ~/.azure directory so I am not sure if I can login into another account exactly at the same time on that machine.
Is there any way to tell azure-cli not to store token in local profile so that I can use azure-cli to connect to multiple accounts at the same time from same machine?
If you are using a windows or mac machine then the tokens are stored in Windows token manager or OSx key chain respectively. Only on Linux systems the tokens are stored in ~/.azure/azureProfile.json
However, you should still be able to login with multiple accounts on Win/Mac or Linux machines.
azure account set "subscription-name" will set the subscription as your default subscription and all the commands that you execute will run against that subscription.
Every command has a -s or --subscription switch where you can explicitly specify the subscription id. Even if the subscription belongs to a different account, it should still work if you have authenticated with that account.
For Linux system, I would suggest to create multiple user accounts and then run the CLI from those accounts. I think there could be a race condition when two commands from different accounts try to access ~/.azure/azureProfile.json.
The latest update is that the environment variable AZURE_CONFIG_DIR has been introduced and that can be set differently for each environment before az login is called.
export AZURE_CONFIG_DIR=/tmp1
az login
and on other window
export AZURE_CONFIG_DIR=/tmp2
az login
Reference: configure the AZURE_CONFIG_DIR for fixing concurrency issue
For Windows, here are steps
Go to env variables and add AZURE_CONFIG_DIR with the value of new config folder (e.x. C:\Users\YourUser\.azure-personal)
restart your cli, then run this az login --use-device-code
use the code given on step 2 and use it with whatever browser to login to new azure account
Now, one of your accounts config is in default azure folder config (C:\Users\YourUser\.azure) and new one lives in the place you specified on step 1.
if you wanna switch between them, you need to flip that env variable to point to whatever config you want