I have an Azure CLI script which runs perfectly well in the Cloud Shell but I run into trouble when I try to include it in a devops release pipeline. For debugging purposes I've reduced the script to simply signing-in as a Service Principal and then retrieving the version...
az login --service-principal -u http://[Service Principal name] -p [Service Principal password] --tenant [Service Principal tenant GUID]
az --version
...but it still incurs the same error.
Here are the properties of my Azure CLI task:
Script Type: Shell
Script Location: Inline script
Inline Script: [as above]
Script Arguments:
Access service principal details in script: false
Use global Azure CLI configuration: false
Working Directory:
Fail on Standard Error: false
Enabled: true
Continue on error: false*
Environment Variables:
And the error is:
2020-06-15T12:46:39.8710944Z ##[error]Error Code: [1]
2020-06-15T12:46:39.8724737Z ##[error]Error: Azure login failed
2020-06-15T12:46:39.8728448Z ##[error]Script failed with error: Get Token request returned http error: 401 and server response: {"error":"invalid_client","error_description":"AADSTS7000222: The provided client secret keys are expired. Visit the Azure Portal to create new keys for your app, or consider using certificate credentials for added security: https://learn.microsoft.com/azure/active-directory/develop/active-directory-certificate-credentials
Trace ID: d0f42793-739e-4ce9-9118-5049086aa800
Correlation ID: 2ad50471-9c2a-4c02-a4d0-189efad2f0c8
Timestamp: 2020-06-15 12:46:39Z","error_codes":[7000222],"timestamp":"2020-06-15 12:46:39Z","trace_id":"d0f42793-739e-4ce9-9118-5049086aa800","correlation_id":"2ad50471-9c2a-4c02-a4d0-189efad2f0c8","error_uri":"https://login.microsoftonline.com/error?code=7000222"}
I was able to authenticate as this Service Principal using the Azure Cloud Shell so I'm puzzled as to why I apparently can't do the same from within the release pipeline.
* I have tried with Continue on error: true and although the task completes the same error appears in the output.
To use the service principal in Azure CLI, no need to login manually.
For the correct way, please follow the steps below.
1.From the error message, the client secret(i.e. client key) of your service principal is expired. So you need to check it first, navigate to the azure portal -> Azure Active Directory -> App registrations -> All applications -> find the AD App related to your service principal -> Certificates & secrets -> check if the secret is expired. If it is expired, just click New client secret to create a new one and save the value.
2.In devops, navigate to the Project Settings -> Service connections -> New service connection -> Azure Resource Manager -> Service principal (manual).
Then fix the options with your service principal, you can get the values from your AD App in App registration page. The Service Principal Id is the Application (client) ID, the Service principal key is the client secret.
The Subscription Id and Subscription Name is which you want to use Azure CLI to access, you could find them in azure portal.
After input the values -> Verify -> Verify and save.
3.In your Azure CLI task -> Azure Resource Manager connection -> select the service connection in step 2. In my sample, I test to get a web app in my subscription with az webapp show --name xxxx --resource-group xxxx(Note: no need to use az login manually, it will login with the service principal which was configured in the service connection automatically).
Then run it and check the result, it works fine.
I think this is very similar to the problem i faced earlier, it happens when there is a special character in the client secret.
One workaround is to go to the Azure portal and keep generating new secrets until you get one that does not have special characters in it.
https://github.com/ansible/ansible/issues/54914
Related
I have researched the way to push docker images from gitlab container registry to an azure resource: Pushing Docker image from gitlab-ci to Azure Container Registry
I have also found the documentation to create managed identities (both system-assigned and user-assigned) in the Azure docs
I am missing to connect the dots on how I can use
az login —-identity in a gitlab-ci.ymlfile to access an azure app service. The purpose is to push a docker image from gitlab container registry.
How can I do this?
How do I need to configure the azure app service (identity / access control)?
Are there any security concerns? If yes, is az login —-service-principal a more secure way to do this? Or any other authentication procedure? ssh?
Thank you for your help in advance! 🙏
You can use a GitLab CI Job JWT token to login to Azure from within a CI/CD pipeline without needing to store secrets in a GitLab project. In order to do this, you will also need to configure OpenID Connect (OIDC) for ID federation between GitLab and an Azure service principal. This is recommended by Microsoft for authenticating to Azure from CI/CD services, among other use cases.
Note: Using OIDC as described below will only work if you are using gitlab.com or a publicly reachable GitLab instance. This is because Azure needs to connect to the token issuer for the keys to validate the token. If you are self-hosting GitLab and your instance is not publicly accessible, you can choose a different credential type for step 2.
1. Create a registered app
First, you will need to register an Application in Azure. You can do this by following these instructions to register an application and create a service principal.
After doing this, make note of the values for Application (client) ID and Directory (tenant) ID (found in the application Overview pane). These values will be needed for step 3.
2. Add the federated credentials
Once your app is registered, you can add federated credentials to the application's service principal. In the Azure portal, go to registered apps -> your application. In the sidebar, select Certificates & secrets. Under the Federated credentials tab, click the "Add credential" button
Use the following parameters for the credential configuration:
Federated credential sceanrio: Other issuer
Issuer: your gitlab URL e.g. https://gitlab.example.com
Subject Identifier: The value of the sub claim to match. For example, to allow jobs on the main branch of the contoso/myproject project to use this service principal, use project_path:contoso/myproject:ref_type:branch:ref:main
Name: Any descriptive name for the federated credental (e.g. contoso-myproject-main)
Description: Optional, a description for the federated credential.
Audience: your GitLab URL e.g. https://gitlab.example.com
3. Authenticate to Azure in your job
After the federated credentials are created, you can leverage the CI_JOB_JWT_V2 token in your job to authenticate with Azure. In this example, we'll use the Azure CLI (az login).
azure-cli:
image: mcr.microsoft.com/azure-cli
variables:
AZURE_CLIENT_ID: "YOUR Application Client ID"
AZURE_TENANT_ID: "YOUR TENANT ID"
script:
- az login --tenant $AZURE_TENANT_ID --service-principal -u $AZURE_CLIENT_ID --federated-token $CI_JOB_JWT_V2
# now you are logged into Azure and can take other actions using the CLI
# - az resource list # example
CI_JOB_JWT_V2: Predefined variable
AZURE_CLIENT_ID: The Application (Client) ID of the registered application.
AZURE_TENANT_ID: The ID of the Azure Tenant to login to (can be found in the application overview)
Also, don't forget to grant your registered app appropriate permissions for Azure container registry
When I run the below on my local pc via powershell it completes successfully.
Set-AzSqlServerActiveDirectoryAdministrator -ResourceGroupName $resourceGroupName -ServerName $serverName -DisplayName 'AAD_GAOUAT_SQLADMIN' -ObjectId 'd9c6b5a7-079e-44b4-8911-bd0451bfb59a'
Now when I run the same command (via Azure Powershell) in DevOps, I get the below error
Any ideas why this fails from Azure DevOps?
When you run the command in devops azure powershell task, it will let you use a service connection to auth, I suppose you didn't use a correct service principal located in the correct tenant(i.e. the subscription of the sql server located).
Please follow the steps below to fix the issue.
1.In devops, navigate to the Project Settings -> Service connections -> New service connection -> Azure Resource Manager -> Service principal (manual).
If you don't have an AD App, you can also select Service principal (automatic) above, then fix the values with correct information .e.g subscription. If you select Service principal (manual), please make sure the service principal has an RBAC role in the subscription to set the sql server AAD admin e.g. Owner, Contributor.
2.After creating the service connection, click it and select Manage Service Principal.
It will open a page for the related AD App -> API permissions -> Add a permission -> add Directory.Read.All Application permission of Azure Active Directory Graph like below(Not Microsoft Graph), At last, don't forget to click Grant admin consent for xxx button.
This step is because Set-AzSqlServerActiveDirectoryAdministrator will call the AAD Graph to check the object you input, so you need to give the permission.
3.Then in the azure powershell task, select the service connection we created.
Run it, it works fine.
When the release is being triggered on AzureDevops, it fails on 'App Service Deployment' stage with this Error: ExpiredServicePrincipal
The error may be caused by your service principal secret was expired, the service principal is used in your service connection in DevOps org.
Navigate to the Azure Active Directory in the Azure portal, make sure you select the correct AAD tenant which the service principal belongs to, in the App registrations -> search for the client id of the service principal(you can find it in the service connection) -> find the AD App related to the service principal -> Certificates & secrets -> New client secret, then copy it and update the Service Principal Key in your service connection -> verify connection.
I am trying to add the Azure credentials (Microsoft Azure Service Principal) on jenkins server under
Manage Jenkins -> configure System -> Add cloud -> Microsoft Azure VM Agent -> Add Azure Credentials
I have created SPN, provided contributor access at the subscription level, copied subscription ID, Application id, application secrete to create credential but When I click Verify Service Principal I am getting error invalid credentials.
Okay, so I messed up, I accidentally ran az ad sp reset-credentials against the Service Principal that our AKS cluster runs under. And now we are getting errors like:
Error creating load balancer (will retry): error getting LB for service test/admin-api: azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/subscriptions/****/resourceGroups/MC_****/providers/Microsoft.Network/loadBalancers?api-version=2017-09-01: StatusCode=0 -- Original Error: adal: Refresh request failed. Status Code = '401'. Response body: {"error":"invalid_client","error_description":"AADSTS70002: Error validating credentials. AADSTS50012: Invalid client secret is provided.\r\nTrace ID:****\r\nCorrelation ID:**** \r\nTimestamp: 2018-08-23 12:01:33Z","error_codes":[70002,50012],"timestamp":"2018-08-23 12:01:33Z","trace_id":"****","correlation_id":"****"}
and
Failed to pull image "****.azurecr.io/****:****": rpc error: code = Unknown desc = Error response from daemon: Get https://****.azurecr.io/v2/****/manifests/****: unauthorized: authentication required
So now I want to find the original client secret that the Service Principal uses, so that I can re-add that as a key to the Service Principal. That's the only solution I can think of other than recreating the entire cluster.
Any ideas?
In the end the solution was quite simple.
In the Azure portal, navigate to the resource group named MC_<resourcegroup>_<aksName>_<region>.
Click one of the resources of the type "Virtual machine".
Scroll down to "Run command"
Choose "RunShellScript"
Enter cat /etc/kubernetes/azure.json and click "Run"
The command will return the contents of the JSON file. The property you need is aadClientSecret
Whoever comes over this issue there's an updated solution from Microsoft
https://learn.microsoft.com/en-us/azure/aks/update-credentials#update-aks-cluster-with-new-credentials
They also mention (something that's not obvious) that:
By default, AKS clusters are created with a service principal that has a one-year expiration time.
Also,
As of Azure CLI 2.0.68, the --password parameter to create a service principal with a user-defined password is no longer supported to prevent the accidental use of weak passwords.
so the initial solution to change the service principal password doesn't work anymore.
It's an annoying thing that you want to do. For your issue, you cannot pull the image without authentication.
First, you have to find out the service principal of your container registry. You can do this in the Azure portal and navigate to the registry panel, then you can find the service principal like this:
Or you can use the Azure CLI command to find the registry ID like this:
az acr show --resource-group groupName --name registryName --query id --output tsv
Then use the command to find the service principal ID like this:
az role assignment list --scope registryID
You can select the service principal which you want.
Then you would get all the secrets with the command kubectl get secrets and kubectl get secrets secretName -o yaml to get the token of the secret. Then analyze one-by-one to check if the username the same as the service principal ID. You can use tools such as JWT to analyze the secret token. The result will like this:
If the username the same as the service principal ID which you find, that is the secret you want. This step is a trouble. You should check the secret one-by-one or you will have a more great way to check them.
By the way, it seems that you can just see the password of the service principal once when you create it. The Azure will not show you again. But if you create the Kubernetes secret, the password is stored in it.
There is new functionality available in the Azure portal to look at the cluster configuration without using the CLI.
Go to the Azure Portal -> Your Cluster Resource -> Overview
On the right hand side there is a button that says "JSON View". Click it and you will see a json file with the cluster details. The service principal ID is visible under: "servicePrincipalProfile".