I'm trying to obtain the owner of a list of jobs on databricks using CLI. The issue is, the command databricks jobs list doesn't have any information related with that, any suggestions?
Thanks in advance!
You need to use Permissions API for that. Specifically the Get Job Permissions Levels API. Just do a GET request against endpoint https://<databricks-instance>/api/2.0/preview/permissions/jobs/{job_id}/permissionLevels - replace {job_id} with actual Job ID
Related
I'm trying to figure out if I'm a member of a specific group on Databricks using either the website or Databricks CLI.
I found a reference to managing groups via an account console, but this requires an admin account. I also found the Databricks groups CLI, but the documentation says you need an admin account to use the API.
(Answering my own question.) It turns out that the Databricks groups CLI documentation is incorrect and you can use it to view your own groups as a user without admin access.
The command is:
databricks groups list-parents --user-name <your-user-name>
However, I can't view parents of other users, and I can't access the other databricks groups commands without getting an error message about lacking valid scope.
I need to generate Azure Databricks token using Powershell script.
I am done with creation of Azure Databricks using ARM template , now i am looking to generate Databricks token using powershell script .
Kindly let me know how to create Databricks token using Powershell script
The only way to generate a new token is via the api which requires you to have a token in the first place.
Or use the Web ui manually.
There is no official powershell commands for databricks, there are some unofficial ones but they still require you to generate a token manually first.
https://github.com/DataThirstLtd/azure.databricks.cicd.tools
Disclaimer I'm the author of these.
UPDATE: these powershell commands can now authenticate using a service principal instead of a bearer token (or can generate a bearer token for you).
so right now there is no way to use the API directly after deploying an Azure Databricks Workspace. I assume that you want to use it as part of an CI/CD pipeline - right? Reason is that you first need to manually create an API token which you can then use for all subsequent API requests.
But I will investigate and keep you updated here!
another option is to create it via terraform.
https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs/resources/token
mind you, it creates the token as whomever you az login'd as. so if you az login as yourself (when it spawns a browser asking who to log in as), that's who the token will be created as (assuming that user has permissions in the databricks workspace) and contributor (or custom read role, reader role doesn't grant the right permissions) permissions into the resource group that houses the workspace.
you can always use az login -u username#email.com -p to log in as someone else, assuming that user doesn't have MFA then run the terraform init/plan/apply. mind you, if you have a backend storage, that user also has to have permissions to that backend storage as well so it can create/update any tfstate files stored there.
I was working with azure runbooks in automation accounts for quite a while but recently I was tasked to identify who was executing the runbook.
I noticed that there is a fied called "Executed by" when you get information from a job, but seems that field is being removed by MSFT.
Checking the logs I can see the calls to the runbook but the job id stated in the log doesn't match the one in the jobid in the runbook inside the automation account.
I was wondering how can I match an execution on a runbook with a entry in the log.
any idea with powershell or by calling the REST API directly?
Thanks!
You can get the user who started the Automation job using the startedBy field returned in the Get-AzureRmAutomationJob and REST API.
This will require passing in the job id, which you can get using:
$PsPrivateMetadata.JobId.Guid
I am unable to do anything with my Azure account such as create servers or databases or anything. It seems like it all revolves around resource groups which I cannot create> I get this error:
It's especially tough because I can't even use a cloud shell since I get this:
Requesting a Cloud Shell.
PowerShell may take up to a minute.Failed to provision a Cloud Shell: {"code":"AuthenticationFailed","message":"Authentication failed. The 'Authoriza
tion' header is missing."}
Any idea what this could mean?
Do you have a valid subscription?
If you do make sure you have the right permissions to it.
Cloud shell need to create a storage to work. If you can't create any resource on subscription cloud shell will surely fail.
I am trying to copy a file from a accessible data lake to blob storage while spinning up the cluster.
I am using this command from Azure documentation
hadoop distcp adl://data_lake_store_account.azuredatalakestore.net:443/myfolder wasb://container_name#storage_account_name.blob.core.windows.net/example/data/gutenberg
Now, If I am trying to automate this instead of hardcoding, how do I use this in script action. To be specific how can I dynamically get the the container name and storage_account_name associated while spinning up the cluster.
First as below,
A Script Action is simply a Bash script that you provide a URI to, and parameters for. The script runs on nodes in the HDInsight cluster.
So you just need to refer to the offical tutorial Script action development with HDInsight to write your script action and know how to run it. Or you can call the REST API Run Script Actions on a running cluster (Linux cluster only) to run it automatically.
For how to dynamically get the container name & storage account, a way for any language is to call the REST API Get configurations and extract the property of you want from the core-site in the JSON response, or just to call Get configuration REST API with parameter core-site as {configuration Type} in the url and extract the property of you want from the JSON response.
Hope it helps.