Okay, so I messed up, I accidentally ran az ad sp reset-credentials against the Service Principal that our AKS cluster runs under. And now we are getting errors like:
Error creating load balancer (will retry): error getting LB for service test/admin-api: azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/subscriptions/****/resourceGroups/MC_****/providers/Microsoft.Network/loadBalancers?api-version=2017-09-01: StatusCode=0 -- Original Error: adal: Refresh request failed. Status Code = '401'. Response body: {"error":"invalid_client","error_description":"AADSTS70002: Error validating credentials. AADSTS50012: Invalid client secret is provided.\r\nTrace ID:****\r\nCorrelation ID:**** \r\nTimestamp: 2018-08-23 12:01:33Z","error_codes":[70002,50012],"timestamp":"2018-08-23 12:01:33Z","trace_id":"****","correlation_id":"****"}
and
Failed to pull image "****.azurecr.io/****:****": rpc error: code = Unknown desc = Error response from daemon: Get https://****.azurecr.io/v2/****/manifests/****: unauthorized: authentication required
So now I want to find the original client secret that the Service Principal uses, so that I can re-add that as a key to the Service Principal. That's the only solution I can think of other than recreating the entire cluster.
Any ideas?
In the end the solution was quite simple.
In the Azure portal, navigate to the resource group named MC_<resourcegroup>_<aksName>_<region>.
Click one of the resources of the type "Virtual machine".
Scroll down to "Run command"
Choose "RunShellScript"
Enter cat /etc/kubernetes/azure.json and click "Run"
The command will return the contents of the JSON file. The property you need is aadClientSecret
Whoever comes over this issue there's an updated solution from Microsoft
https://learn.microsoft.com/en-us/azure/aks/update-credentials#update-aks-cluster-with-new-credentials
They also mention (something that's not obvious) that:
By default, AKS clusters are created with a service principal that has a one-year expiration time.
Also,
As of Azure CLI 2.0.68, the --password parameter to create a service principal with a user-defined password is no longer supported to prevent the accidental use of weak passwords.
so the initial solution to change the service principal password doesn't work anymore.
It's an annoying thing that you want to do. For your issue, you cannot pull the image without authentication.
First, you have to find out the service principal of your container registry. You can do this in the Azure portal and navigate to the registry panel, then you can find the service principal like this:
Or you can use the Azure CLI command to find the registry ID like this:
az acr show --resource-group groupName --name registryName --query id --output tsv
Then use the command to find the service principal ID like this:
az role assignment list --scope registryID
You can select the service principal which you want.
Then you would get all the secrets with the command kubectl get secrets and kubectl get secrets secretName -o yaml to get the token of the secret. Then analyze one-by-one to check if the username the same as the service principal ID. You can use tools such as JWT to analyze the secret token. The result will like this:
If the username the same as the service principal ID which you find, that is the secret you want. This step is a trouble. You should check the secret one-by-one or you will have a more great way to check them.
By the way, it seems that you can just see the password of the service principal once when you create it. The Azure will not show you again. But if you create the Kubernetes secret, the password is stored in it.
There is new functionality available in the Azure portal to look at the cluster configuration without using the CLI.
Go to the Azure Portal -> Your Cluster Resource -> Overview
On the right hand side there is a button that says "JSON View". Click it and you will see a json file with the cluster details. The service principal ID is visible under: "servicePrincipalProfile".
Related
I get this issue on my aks cluster
Your service principal has expired or is invalid. Please check if you
are using the correct secret or if the key has expired.
I think the problem is the current service principal needs to be reset because I think it is already expired.
I have followed the instruction in this link, but am still unable to do it even using the account with owner access
Using Global Administrator user
root#root~ % SP_ID=$(az aks show --resource-group myclusterk8s --name myclusterk8s \
--query servicePrincipalProfile.clientId -o tsv)
I already get the id from the first command, but when we do the second command the id we get from the first command is cannot be used
root#root ~ % SP_SECRET=$(az ad sp credential reset --name "$SP_ID" --query
password -o tsv) ERROR: Resource
'6ec95333-4a5f-4sd1-8478-4b367a4b3711' does not exist or one of its
queried reference-property objects are not present.
Does anyone have any suggestions on how to solve this?
Resource 'appID' does not exist or one of its queried
reference-property objects are not present.
The error suggests that the service principal does not exist in your Azure AD
Firstly, you need to check if the service principal exists in your Azure Active Directory
If the service principal does not exist, you can create a new service principal and update the AKS cluster with the new service principal credentials
If the service principal still exists and you are facing the issue, you can create a new client secret in the Azure Portal. Then you can update the AKS cluster with the new credentials
I have an Azure CLI script which runs perfectly well in the Cloud Shell but I run into trouble when I try to include it in a devops release pipeline. For debugging purposes I've reduced the script to simply signing-in as a Service Principal and then retrieving the version...
az login --service-principal -u http://[Service Principal name] -p [Service Principal password] --tenant [Service Principal tenant GUID]
az --version
...but it still incurs the same error.
Here are the properties of my Azure CLI task:
Script Type: Shell
Script Location: Inline script
Inline Script: [as above]
Script Arguments:
Access service principal details in script: false
Use global Azure CLI configuration: false
Working Directory:
Fail on Standard Error: false
Enabled: true
Continue on error: false*
Environment Variables:
And the error is:
2020-06-15T12:46:39.8710944Z ##[error]Error Code: [1]
2020-06-15T12:46:39.8724737Z ##[error]Error: Azure login failed
2020-06-15T12:46:39.8728448Z ##[error]Script failed with error: Get Token request returned http error: 401 and server response: {"error":"invalid_client","error_description":"AADSTS7000222: The provided client secret keys are expired. Visit the Azure Portal to create new keys for your app, or consider using certificate credentials for added security: https://learn.microsoft.com/azure/active-directory/develop/active-directory-certificate-credentials
Trace ID: d0f42793-739e-4ce9-9118-5049086aa800
Correlation ID: 2ad50471-9c2a-4c02-a4d0-189efad2f0c8
Timestamp: 2020-06-15 12:46:39Z","error_codes":[7000222],"timestamp":"2020-06-15 12:46:39Z","trace_id":"d0f42793-739e-4ce9-9118-5049086aa800","correlation_id":"2ad50471-9c2a-4c02-a4d0-189efad2f0c8","error_uri":"https://login.microsoftonline.com/error?code=7000222"}
I was able to authenticate as this Service Principal using the Azure Cloud Shell so I'm puzzled as to why I apparently can't do the same from within the release pipeline.
* I have tried with Continue on error: true and although the task completes the same error appears in the output.
To use the service principal in Azure CLI, no need to login manually.
For the correct way, please follow the steps below.
1.From the error message, the client secret(i.e. client key) of your service principal is expired. So you need to check it first, navigate to the azure portal -> Azure Active Directory -> App registrations -> All applications -> find the AD App related to your service principal -> Certificates & secrets -> check if the secret is expired. If it is expired, just click New client secret to create a new one and save the value.
2.In devops, navigate to the Project Settings -> Service connections -> New service connection -> Azure Resource Manager -> Service principal (manual).
Then fix the options with your service principal, you can get the values from your AD App in App registration page. The Service Principal Id is the Application (client) ID, the Service principal key is the client secret.
The Subscription Id and Subscription Name is which you want to use Azure CLI to access, you could find them in azure portal.
After input the values -> Verify -> Verify and save.
3.In your Azure CLI task -> Azure Resource Manager connection -> select the service connection in step 2. In my sample, I test to get a web app in my subscription with az webapp show --name xxxx --resource-group xxxx(Note: no need to use az login manually, it will login with the service principal which was configured in the service connection automatically).
Then run it and check the result, it works fine.
I think this is very similar to the problem i faced earlier, it happens when there is a special character in the client secret.
One workaround is to go to the Azure portal and keep generating new secrets until you get one that does not have special characters in it.
https://github.com/ansible/ansible/issues/54914
I am trying to link Azure Key Vault secrets to a variable group in Azure Pipelines (part in Azure DevOps). Microsoft documentation here.
However, the "Authorize" button does not seem to work. It spins endlessly. Screenshot.
My target Azure Key Vault already has the service principal included in its access policy with Get and List permissions. Screenshot.
Anyone seen this issue before?
This workaround also seems like a bug for Azure Key Vault deployments using ARM templates.
If the service principal in question is added to the Azure Key Vault (AKV) access policies through an ARM template by referencing the service principal's Object ID (as Microsoft documentation calls for), permission errors with Azure Pipelines follow.
However, if I manually add the service principal to the AKV's access policies by referencing the service principal's application (client) ID, the permissions errors go away entirely.
Again, feels like a bug. And now my automated deployment pipeline doesn't quite work because of this manual step.
Also, in the AKV ARM template, if I were to combine the mandatory field objectId with the optional field applicationId, the service principal shows up as a "compound identity". That does not fix the permissions issues in Azure Pipelines. I do not see a way of adding a service principal properly without doing it manually.
Firstly, please make sure the service connection is working correctly. Then refresh the page and try it again. Alternately you can also try in browser inprivate session.
Just as the message said "The specified Azure service connection needs to have "Get, List" secret management permissions on the selected key vault."
Basically, we need to click the "Authorize" button to enable Azure Pipelines to set these permissions for the specific service connection.
If that doesn't work, we can also manually set the permissions for the specific service connection.
Go to Project settings - > Service connections -> Select the
specific ARM service connection
Click Edit to popup the Update Authentication for xxx dialog
Click the "use the full version of the service connection dialog."
link, to get the Service principal client ID
Go to your key vault in Azure portal -> Access Policies -> Add a new
Access Policy -> Select a template (e.g Key&Secret Management) - >
Select Get, List for Secret permissions.
Click Select Principal -> Copy and paste the Service principal client ID
to search the user/application -> Select the searched
user/application
After that you can see the new APPLICATION access policy.
Try it again after successfully adding the application access policy.
UPDATE:
Generally in Azure DevOps we need to create a ARM service connection (the client which can access the azure sources) first before deploying an Azure Key Vault through an ARM template.
Actually when you select the Azure subscription then click Authorize in Azure resource group deployment task
, the ARM service connection is created automatically. You just need to check the AppID and get the ObjectID to use in the ARM template.
We can get the Service principal client ID (AppID) by following above steps. After that we can get ObjectId by the AppID with running the following command: (See Find service principal object ID using PowerShell for details.)
$(Get-AzureADServicePrincipal -Filter "AppId eq 'a89c3dee-f5bf-4ea1-a805-d4c729a4add3'").ObjectId
Then you can specific the ObjectId when deploying the Azure Key Vault through an ARM template.
I am using Azure app service api to view server details like worker process and region etc. for management purpose. After generating token from AuthenticationContext.AcquireTokenAsync method, I am requesting following URL for server details https://management.azure.com/subscriptions/<sub ID>/resourceGroups/<resource group name>/providers/Microsoft.Web/serverfarms/?api-version=2018-02-01
In the response I am getting AuthorizationFailed error code with the detail given bellow:
The client does not have authorization to perform action 'Microsoft.Web/serverfarms/read' over scope '/subscriptions/xxxxxxxx-xxxxxxx-xxxx/resourceGroups/xxxxxxxxxxx/providers/Microsoft.Web/serverfarms/xxxx' or the scope is invalid. If access was recently granted, please refresh your credentials.
But when I try the same verification using https://learn.microsoft.com/en-us/rest/api/appservice/appserviceenvironments/get portal where I can try the APIs for testing, the request is returning expected results.
So, is there any other way to authenticate or should I have to define some permissions to achieve the functionality?
The service principal you are using doesn't have rights within that tenant.
Tenants have subscriptions and service principals belong to tenants. Azure resource manager also exposes role based authorization for a given principal, which would give it rights on Azure resources. It appears the service principal doesn't have rights to read from that subscription.
Go to portal and find your subscription, click on Access Control (IAM) and then click on Add role assignment with correspond service principal which you use to acquire token.
After you have given successful permission, refresh and try again.
I had the same error while running,
az aks get-credentials --resource-group myResourceGroup --name myAKSCluster
I did this shortly after az login.
Problem was I haven't set my subscription properly so what I had to do was run,
az account set --subscription your-subscription-id
After that az aks get-credentials worked fine, the error was gone and you will get an output like,
Merged "myAKSCluster" as current context in C:\Users\UserName\.kube\config
I had the same problem. Initially, I went ahead and added to my user the "Web Plan Contributor" role, as it is the one that should handle those things. Nothing changed - I still had this error.
What turned actual problem turned out to be is a wrong resource group... Turns out I copied some old script where WebAppPlans were in separate RG, and I was searching the app plan there. Completely missleading error. I guess it will bring up the same error message even if the App Plan simply doesn't exist.
You can create scopes in Databricks backed by Azure Keyvault instead of using the Databricks CLI. However, when you try to create a Scope, an obscure error message (with a spelling mistake!) is shown. It appears as not many people encounter this error:
"Internal error happened while granting read/list permission to Databricks ervice principal to KeyVault: XYZ"
Setting the Manage Principal to All Users does NOT help in this case.
I figured that this was a Service Principal issue in Azure AD. This particular user I was logged on to Databricks with was not an AD contributer and only had Contributer role on the Databricks and Keyvault service. I could not find any default Object ID in AD for Databricks so I assumed it was creating a service principal on the fly and connecting Databricks with Keyvault (I might be wrong here - it might already exist in AD when you enable the Databricks resource provider).
Logging in as an Admin with the rights to create service principals solved the problem. After that you can see in the Key Vault the DB service principal used in for the key retrieval:
As mentioned by #rcabr in his above comment there is already an SP by name 'AzureDatabricks' inside Enterprize Application, you need to get the object id details and add it in the access policy of the key vault. With this, the Databricks will be able to access the KeyVault