I get this issue on my aks cluster
Your service principal has expired or is invalid. Please check if you
are using the correct secret or if the key has expired.
I think the problem is the current service principal needs to be reset because I think it is already expired.
I have followed the instruction in this link, but am still unable to do it even using the account with owner access
Using Global Administrator user
root#root~ % SP_ID=$(az aks show --resource-group myclusterk8s --name myclusterk8s \
--query servicePrincipalProfile.clientId -o tsv)
I already get the id from the first command, but when we do the second command the id we get from the first command is cannot be used
root#root ~ % SP_SECRET=$(az ad sp credential reset --name "$SP_ID" --query
password -o tsv) ERROR: Resource
'6ec95333-4a5f-4sd1-8478-4b367a4b3711' does not exist or one of its
queried reference-property objects are not present.
Does anyone have any suggestions on how to solve this?
Resource 'appID' does not exist or one of its queried
reference-property objects are not present.
The error suggests that the service principal does not exist in your Azure AD
Firstly, you need to check if the service principal exists in your Azure Active Directory
If the service principal does not exist, you can create a new service principal and update the AKS cluster with the new service principal credentials
If the service principal still exists and you are facing the issue, you can create a new client secret in the Azure Portal. Then you can update the AKS cluster with the new credentials
Related
I am testing an environment where I have the credentials of a service principal of an application. My next step is to identify the objects owned by the application or the resources that the app can access.
I could get similar details for an AD user through the cli command
ad signed-in-user list-owned-objects
Running the same command when signed-in with the credentials of the sp results in the following error
Resource not found for the segment 'me'.
My use-case is to enumerate the SP account to understand its access rights in the subscription. Can someone help me out with the right set of azure cli commands.
I tried to reproduce the same in my environment and got below results:
I created one service principal with Storage Blob Data Contributor role at storage account scope like below:
az ad sp create-for-rbac --role "Storage Blob Data Contributor" --name <spname> --scopes /subscriptions/<subID>/resourceGroups/<rgname>/providers/Microsoft.Storage/storageAccounts/<storaccname>
Now I logged in to Azure account successfully using above service principal credentials:
az login --service-principal -u appID -p password --tenant tenantID
When I ran the same command to get the resources that the app can access, I got same error as below:
az ad signed-in-user list-owned-objects
To list RBAC roles assigned to a service principal, you can make use of below command:
az role assignment list --assignee <service_principal_ID> --all
If your use case is to list all the resources/objects a service principal can access/own, currently there is no command available particularly for that.
To know more in detail, you can check below reference:
For a given Azure AD Service Principal, Get a list of the Azure Objects and Rights by AlfredoRevilla-MSFT
I have a k8s cluster running in Azure and I have always reset the service principal credentials by using Azure CLI: az ad sp credential reset --name <xyz> --years 2. Afterwards I have updated the AKS cluster with the new service principal credentials, see update AKS Cluster credentials. After this the cluster will be restarted.
For a production environment I want to avoid restarting the cluster after resetting the credentials, so I was thinking to use the same password as before. So old password=new password. This is achieved using az ad sp credential reset --name <xyz> --years 2 --password <1234>
Now my question is: should I update the AKS cluster with new service principal credentials even if I use the same password as before? Has anyone tried this before?
AFAIK it's not require to update or restart the AKS cluster with new service principal credentials even if you are using the same password as before.
Since you are using the same password at AKS level and only reseting with same password at service principal level for sake of not to expire password.
Note : If you were resting with different password as you are using currently in that case you have to update and reset the AKS cluster.
For the benifits You can use of managed identity rather than service principle as rickvdbosch and Philip Welz Suggested in the comment itself.
You can now update an AKS cluster currently working with service principals to work with managed identities by using the following CLI commands.
az aks update -g <RGName> -n <AKSName> --enable-managed-identity
Azure Admins created a cluster for us.
On VM I installed "az cli" and "kubectl".
With my account from Azure Portal I can see that Kubernetes Service and Resource Group to which it belongs.
From the level of that cluster in Azure Portal I can see that I have a role:
"AKS Cluster Admin Operator"
I am logged on VM with kubectl with my account. I need to config my kubectl to work with our cluster.
When I try to execute:
az aks get-credentials --resource-group FRONT-AKS-NA2 --name front-aks
I am getting error:
ForbiddenError: The client 'my_name#my_comp.COM' with object id
'4ea46ad637c6' does not have authorization to perform action
'Microsoft.ContainerService/managedClusters/listClusterUserCredential/action'
over scope
'/subscriptions/89e05d73-8862-4007-a700-0f895fc0f7ea/resourceGroups/FRONT-AKS-NA2/providers/Microsoft.ContainerService/managedClusters/front-aks'
or the scope is invalid. If access was recently granted, please
refresh your credentials.
In my case to refresh recently granted credentials helped this one:
az account set --subscription "your current subscription name"
It led to re-login and fix the issue.
Well, I see the comment, and you already get the solution. So I just can explain the difference to you. Hope it will help you!
When you use the command az aks get-credentials without parameter --admin, it means the CLI command uses the default value: Cluster user. And when you use the cluster user, it just works if you integrate AKS with the AAD. But you said you just have the AKS Cluster Admin Operator role, so the appropriate parameter is --admin. You can get more details here.
And on my side, it's a little dangerous. If the AKS cluster is just for the test, there is no problem. But if it's for production, I recommend you integrate with the AAD, and then give the appropriate permissions to the user. Because the admin user means you have all the permissions, you know, it's not safe.
According to the documentation, Azure Kubernetes Service Cluster User Role allows access to Microsoft.ContainerService/managedClusters/listClusterUserCredential/action API call only.
My user is part of an AD group that has Azure Kubernetes Service Cluster User Role permissions on the AKS cluster and all the cluster role and cluster role bindings have been applied via kubectl.
I can double check and verify that access to dashboard and permissions work with these steps:
1. az login
2. az aks get-credentials --resource-group rg --name aks
3. kubectl proxy
4. Open web connection
5. Get prompt on terminal to login via device code flow
6. Return to web connection on dashboard
7. I can correctly verify that my permissions apply,
i.e. deleting a job does not work and this falls in line with my
kubectl clusterrole bindings to the Azure AD group.
However when I try to use the az aks browse command to open the browser automatically like this, i.e. without kubectl proxy:
1. az login
2. az aks get-credentials --resource-group rg --name aks
3. az aks browse --resource-grouprg --name aks
I keep getting the following error:
The client 'xxx' with object id 'yyyy' does not have authorization to perform action
'Microsoft.ContainerService/managedClusters/read' over scope
'/subscriptions/qqq/resourceGroups/rg/providers/Microsoft.ContainerService/managedClusters/aks'
or the scope is invalid. If access was recently granted, please refresh your credentials.
A dirty solution was to apply Reader role on the AKS cluster for that AD group - then this issue goes away but why does az aks browse require Microsoft.ContainerService/managedClusters/read permission and why is that not included in Azure Kubernetes Service Cluster User Role?
What is happening here?
Currently, the command
az aks browse --resource-grouprg --name aks isn't working with the more recent version of AKS, you can find the full details here.
https://github.com/MicrosoftDocs/azure-docs/issues/23789
Also, your current problem might also be that your user XXX doesn't have the right IAM access level at the Subscription/ResourceGroup level.
Okay, so I messed up, I accidentally ran az ad sp reset-credentials against the Service Principal that our AKS cluster runs under. And now we are getting errors like:
Error creating load balancer (will retry): error getting LB for service test/admin-api: azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/subscriptions/****/resourceGroups/MC_****/providers/Microsoft.Network/loadBalancers?api-version=2017-09-01: StatusCode=0 -- Original Error: adal: Refresh request failed. Status Code = '401'. Response body: {"error":"invalid_client","error_description":"AADSTS70002: Error validating credentials. AADSTS50012: Invalid client secret is provided.\r\nTrace ID:****\r\nCorrelation ID:**** \r\nTimestamp: 2018-08-23 12:01:33Z","error_codes":[70002,50012],"timestamp":"2018-08-23 12:01:33Z","trace_id":"****","correlation_id":"****"}
and
Failed to pull image "****.azurecr.io/****:****": rpc error: code = Unknown desc = Error response from daemon: Get https://****.azurecr.io/v2/****/manifests/****: unauthorized: authentication required
So now I want to find the original client secret that the Service Principal uses, so that I can re-add that as a key to the Service Principal. That's the only solution I can think of other than recreating the entire cluster.
Any ideas?
In the end the solution was quite simple.
In the Azure portal, navigate to the resource group named MC_<resourcegroup>_<aksName>_<region>.
Click one of the resources of the type "Virtual machine".
Scroll down to "Run command"
Choose "RunShellScript"
Enter cat /etc/kubernetes/azure.json and click "Run"
The command will return the contents of the JSON file. The property you need is aadClientSecret
Whoever comes over this issue there's an updated solution from Microsoft
https://learn.microsoft.com/en-us/azure/aks/update-credentials#update-aks-cluster-with-new-credentials
They also mention (something that's not obvious) that:
By default, AKS clusters are created with a service principal that has a one-year expiration time.
Also,
As of Azure CLI 2.0.68, the --password parameter to create a service principal with a user-defined password is no longer supported to prevent the accidental use of weak passwords.
so the initial solution to change the service principal password doesn't work anymore.
It's an annoying thing that you want to do. For your issue, you cannot pull the image without authentication.
First, you have to find out the service principal of your container registry. You can do this in the Azure portal and navigate to the registry panel, then you can find the service principal like this:
Or you can use the Azure CLI command to find the registry ID like this:
az acr show --resource-group groupName --name registryName --query id --output tsv
Then use the command to find the service principal ID like this:
az role assignment list --scope registryID
You can select the service principal which you want.
Then you would get all the secrets with the command kubectl get secrets and kubectl get secrets secretName -o yaml to get the token of the secret. Then analyze one-by-one to check if the username the same as the service principal ID. You can use tools such as JWT to analyze the secret token. The result will like this:
If the username the same as the service principal ID which you find, that is the secret you want. This step is a trouble. You should check the secret one-by-one or you will have a more great way to check them.
By the way, it seems that you can just see the password of the service principal once when you create it. The Azure will not show you again. But if you create the Kubernetes secret, the password is stored in it.
There is new functionality available in the Azure portal to look at the cluster configuration without using the CLI.
Go to the Azure Portal -> Your Cluster Resource -> Overview
On the right hand side there is a button that says "JSON View". Click it and you will see a json file with the cluster details. The service principal ID is visible under: "servicePrincipalProfile".