i am facing below error while executing databricks notebook.
INVALID_ARGUMENT(CLIENT_ERROR): databricks_error_message:Cannot fetch secrets referred in the Spark configuration. Please check that the secrets exists and the cluster's owner has the read permission.
i know below cmd will give permission details for the user but i don't know how to get cluster's owner details. and the cluster creator has already access to the cluster.
>>databricks secrets get-acl --scope scope_name --principal username
how to check the databricks cluster's owner details and resolve this issue?
i have granted access to cluster created but still facing same error.
Related
Creating a metastore for Unity Catalog through terraform fails with this error message:
Error: cannot create metastore: Only account admin can create metastores. \
Using databricks-cli auth: host=https://{wsname}.azuredatabricks.net, \
token=***REDACTED***, profile=DEFAULT
My config is setup like this:
resource "databricks_metastore" "this" {
name = "primary"
storage_root = format("abfss://%s#%s.dfs.core.windows.net/",
azurerm_storage_account.storage.name,
azurerm_storage_container.container.name)
force_destroy = true
}
resource "databricks_metastore_assignment" "this" {
metastore_id = databricks_metastore.this.id
workspace_id = data.azurerm_databricks_workspace.oat.workspace_id
}
The documentation regarding Azure Databricks administrators lists up four different kinds of administrators, but I fail to see how I can see check who is Azure Databricks account admins.
In short, I have the same error as Not finding Unity Catalog "Create Metastore" in Azure Databricks
Your account is a workspace admin. Account administrator is one level above.
This Unity Catalog documentation describes how you become account administrator:
The first Azure Databricks account admin must be an Azure Active
Directory Global Administrator at the time that they first log in to
the Azure Databricks account console. Upon first login, that user
becomes an Azure Databricks account admin and no longer needs the
Azure Active Directory Global Administrator role to access the Azure
Databricks account.
Thus, you need Global Administrator role in Azure for the user that is running Terraform.
When deploying new jobs and services to Azure Kubernetes Service cluster, the pods fail to request valid AAD access tokens with all permissions available. If new permissions were added on the same day, before or after a deployment, the tokens still do not pick them up. This issue has been observed with permissions granted to Active Directory Groups over Key Vaults, Storage Accounts, and SQL databases scopes so far.
Example: I have a .NET 5.0 C# API running on 3 pods with antiaffinity rules located each on a separate node. The application reads information from a SQL database. I made a release and added the database permissions afterwards. Things I have tried so far to make the application reset the access tokens:
kubectl delete pods --all -n <namespace> which essentially created 3 new pods again failing due to insufficient permissions.
kubectl apply -f deployment.yaml to deploy a new version of the image running in the containers, again all 3 pods kept failing.
kubectl delete -f deployment.yaml followed by kubectl apply -f deployment.yaml to erase the old kubernetes object and create a new one. This resolved the issue on 2/3 pods, however, the third one kept failing due to insufficient permissions.
kubectl delete namespace <namespace> to erase the entire namespace with all configuration available and recreated it again. Surprisingly, again 2/3 pods were running with the correct permissions and the last one did not.
The commands were ran more than one hour after the permissions were added to the group. The database tokens are active for 24 hours and when I have seen this issue occur with cronjobs, I had to wait 1 day for the task to execute correctly (none of the above steps worked in a cronjob scenario). The validity of the tokens kept changing which implied that the pods are requesting new access tokens, again excluding the most recently added permissions. The only solution I have found that works 100% of the time is destroy the cluster and recreate it which is not viable in any production scenario.
The failing pod from my example was the one always running on node 00 which made me think there may be an extra caching layer on the first initial node of the cluster. However, I still do not understand why the other 2 pods were running with no issue and also what is the way to restart my pods or refresh the access token to minimise the wait time until resolution.
Kubernetes version: 1.21.7.
The cluster has no AKS-managed AAD or pod-identity enabled. All RBAC is granted to the cluster MSI via AD groups.
Please check if below can be worked around in your case.
To access the Kubernetes resources, you must have access to the AKS cluster, the Kubernetes API, and the Kubernetes objects. Ensure that you're either a cluster administrator or a user with the appropriate permissions to access the AKS cluster
Things you need to do, if you haven't already:
Enable Azure RBAC on your existing AKS cluster, using:
az aks update -g myResourceGroup -n myAKSCluster --enable-azure-rbac
Create Role that allows read access to all other Pods and Services:
Add the necessary roles (Azure Kubernetes Service Cluster User Role , Azure Kubernetes Service RBAC Reader/Writer/Admin/Cluster Admin) to the user. See ( Microsoft Docs).
Also check Troubleshooting
Also check if you need to have "Virtual Machine Contributor" and storage account contributer for your resource group containing pods and see if namespace is mentioned in that pod , if you have missed . Stack Overflow refernce.Also do check if firewall is restricting the access to the network in that pod.
Resetting the kubeconfig context using the az aks get-credentials command may clear the previously cached authentication token for some xyz user:
az aks get-credentials --resource-group myResourceGroup --name myAKSCluster --overwrite-existing >Reference
Please do check Other References below:
kubernetes - Permissions error - Stack Overflow
create-role-assignments-for-users-to-access-cluster | microsoft docs
user can't access to AKS cluster with RBAC enabled (github.com)
kubernetes - Stack Overflow
I have a user named rbacWriterAndConsoleNsReader in my azure portal and assigned him Azure Kubernetes Service Cluster User role and Azure Kubernetes Service RBAC Writer role (scope= full cluster)
But when I try to list namespaces or even pods, it shows the below error :
Error from server (Forbidden): namespaces is forbidden: User "rbacWriterAndConsoleNsReader#mayankprac2outlook.onmicrosoft.com" cannot list resource "namespaces" in API group "" at the cluster scope
PS: NO YAML files should be used (as role here is in-built role provided by Azure), All must be done through Azure CLI
Where am I going wrong ? Can anyone pls explain ?
Thanks in advance !
Azure Admins created a cluster for us.
On VM I installed "az cli" and "kubectl".
With my account from Azure Portal I can see that Kubernetes Service and Resource Group to which it belongs.
From the level of that cluster in Azure Portal I can see that I have a role:
"AKS Cluster Admin Operator"
I am logged on VM with kubectl with my account. I need to config my kubectl to work with our cluster.
When I try to execute:
az aks get-credentials --resource-group FRONT-AKS-NA2 --name front-aks
I am getting error:
ForbiddenError: The client 'my_name#my_comp.COM' with object id
'4ea46ad637c6' does not have authorization to perform action
'Microsoft.ContainerService/managedClusters/listClusterUserCredential/action'
over scope
'/subscriptions/89e05d73-8862-4007-a700-0f895fc0f7ea/resourceGroups/FRONT-AKS-NA2/providers/Microsoft.ContainerService/managedClusters/front-aks'
or the scope is invalid. If access was recently granted, please
refresh your credentials.
In my case to refresh recently granted credentials helped this one:
az account set --subscription "your current subscription name"
It led to re-login and fix the issue.
Well, I see the comment, and you already get the solution. So I just can explain the difference to you. Hope it will help you!
When you use the command az aks get-credentials without parameter --admin, it means the CLI command uses the default value: Cluster user. And when you use the cluster user, it just works if you integrate AKS with the AAD. But you said you just have the AKS Cluster Admin Operator role, so the appropriate parameter is --admin. You can get more details here.
And on my side, it's a little dangerous. If the AKS cluster is just for the test, there is no problem. But if it's for production, I recommend you integrate with the AAD, and then give the appropriate permissions to the user. Because the admin user means you have all the permissions, you know, it's not safe.
According to the documentation, Azure Kubernetes Service Cluster User Role allows access to Microsoft.ContainerService/managedClusters/listClusterUserCredential/action API call only.
My user is part of an AD group that has Azure Kubernetes Service Cluster User Role permissions on the AKS cluster and all the cluster role and cluster role bindings have been applied via kubectl.
I can double check and verify that access to dashboard and permissions work with these steps:
1. az login
2. az aks get-credentials --resource-group rg --name aks
3. kubectl proxy
4. Open web connection
5. Get prompt on terminal to login via device code flow
6. Return to web connection on dashboard
7. I can correctly verify that my permissions apply,
i.e. deleting a job does not work and this falls in line with my
kubectl clusterrole bindings to the Azure AD group.
However when I try to use the az aks browse command to open the browser automatically like this, i.e. without kubectl proxy:
1. az login
2. az aks get-credentials --resource-group rg --name aks
3. az aks browse --resource-grouprg --name aks
I keep getting the following error:
The client 'xxx' with object id 'yyyy' does not have authorization to perform action
'Microsoft.ContainerService/managedClusters/read' over scope
'/subscriptions/qqq/resourceGroups/rg/providers/Microsoft.ContainerService/managedClusters/aks'
or the scope is invalid. If access was recently granted, please refresh your credentials.
A dirty solution was to apply Reader role on the AKS cluster for that AD group - then this issue goes away but why does az aks browse require Microsoft.ContainerService/managedClusters/read permission and why is that not included in Azure Kubernetes Service Cluster User Role?
What is happening here?
Currently, the command
az aks browse --resource-grouprg --name aks isn't working with the more recent version of AKS, you can find the full details here.
https://github.com/MicrosoftDocs/azure-docs/issues/23789
Also, your current problem might also be that your user XXX doesn't have the right IAM access level at the Subscription/ResourceGroup level.