I'm trying to use DefaultAzureCredentials to authenticate my Azure function against Azure Service Bus. In my azure function azure-func-service-bus, I call to Azure Service Bus
servicebus_client = ServiceBusClient(
fully_qualified_namespace=MY_SERVICE_BUS_NAMESPACE_NAME+".servicebus.windows.net",
credential=DefaultAzureCredential(additionally_allowed_tenants=['*'])
)
I created and pushed Docker container to ACR. When I run the container locally for testing outside of Azure, it does not know what permissions to use.
az acr login --name acr01
docker push acr01.azurecr.io/azure-func-service-bus:v1
docker pull acr01.azurecr.io/azure-func-service-bus:v1
docker run -it --rm -p 8080:80 acr01.azurecr.io/azure-func-service-bus:v1
but got the following error.
DefaultAzureCredential failed to retrieve a token from the included credentials.
Attempted credentials:
EnvironmentCredential: EnvironmentCredential authentication unavailable. Environment variables are not fully configured.
Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot to troubleshoot.this issue.
ManagedIdentityCredential: ManagedIdentityCredential authentication unavailable, no response from the IMDS endpoint.
SharedTokenCacheCredential: SharedTokenCacheCredential authentication unavailable. No accounts were found in the cache.
VisualStudioCodeCredential: Failed to get Azure user details from Visual Studio Code.
AzureCliCredential: Azure CLI not found on path
AzurePowerShellCredential: PowerShell is not installed
To mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.
Unexpected error occurred (ClientAuthenticationError('DefaultAzureCredential failed to retrieve a token from the included credentials.\nAttempted credentials:\n\tEnvironmentCredential: EnvironmentCredential authentication unavailable. Environment variables are not fully configured.\nVisit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot to troubleshoot.this issue.\n\tManagedIdentityCredential: ManagedIdentityCredential authentication unavailable, no response from the IMDS endpoint.\n\tSharedTokenCacheCredential: SharedTokenCacheCredential authentication unavailable. No accounts were found in the cache.\n\tVisualStudioCodeCredential: Failed to get Azure user details from Visual Studio Code.\n\tAzureCliCredential: Azure CLI not found on path\n\tAzurePowerShellCredential: PowerShell is not installed\nTo mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.')). Handler shutting down.
I'm missing a key piece of the puzzle. How can I handle this?
When the Azure Function runs in Azure, it's configured to support ManagedIdentityCredential. For your case I'd recommend trying to configure EnvironmentCredential to test locally.
You can find the details in the link, but the short version is:
Create a service principle (Docs) and give it the needed access
Run the container with extra Environment Variables:
AZURE_TENANT_ID: service principal's Tenant ID
AZURE_CLIENT_ID: service principal's AppId
AZURE_CLIENT_SECRET: service principle's password
I'd recommend using a .env file to make this easier, but be sure it doesn't get checked in anywhere.
FYI If your account doesn't use MFA, you can instead use the variables AZURE_USERNAME and AZURE_PASSWORD. But then you've put your username and password in a file or your terminal history which is concerning. Admittedly the service principal has the same problem, but you can more easily mitigate that with minimizing it's access and regularly rolling the secret.
P.S. If you're using Visual Studio for making your Azure Function you should be able to use something like: EnvironmentCredentialExample to automate setting up and using the needed .env file.
I created the aks cluster with azure service principal id and i provided the contributer role according to the subscription and resource group.
For each and every time when i executed the pipeline the sign-in is asking and after i authenticated it is getting the data.
Also the "kubectl get" task is taking more than 30 min and is getting "Kubectl Server Version: Could not find kubectl server version"
To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code CRA2XssWEXUUA to authenticate
Thanks in advance
What is the version of the created cluster?
I'm assuming from your snapshot that you are using az in order to get credentials for it.
Old azure auth plugin is deprecated in V1.22+. If you are using V1.22 or above you should use kubelogin in order authenticate.
You will also need to update your kube config accordingly:
kubelogin convert-kubeconfig
and specifically if you're logging via az:
kubelogin convert-kubeconfig -l azurecli
Note that the flag -l azurecli is important here: the default value is "devicecode" which will not consider your az as a logging method - and you will still be requested a browser authentication.
Alternatively, you can set environment variable:
AAD_LOGIN_METHOD=azurecli
Because you are getting sign in request and not the deprecation warning for the auth plugin, I suspect that you already have kubelogin installed on your agent, and you just need to update the kube config file
What task are you using? There is official kubectl task: https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/deploy/kubernetes?view=azure-devops
It requires the service connection.
If you still want to execute kubectl directly, you should run the following before the kubectl inside the AzureCLI task:
az aks get-credentials --resource-group "$(resourceGroup)" --name "$(k8sName)" --overwrite-existing
Please use Selfhosted agents for executing your commands. looks like you have private endpoints for your AKS and requests are only allowed from trusted devices.
I ran into the same issue and for me the fix was to change the Connection Type in the stage definition from Azure Resource Manager to Kubernetes Service Connection - check on the screenshot below.
Then you should be able to also specify the connection type in each of the tasks where you are running kubectl or helm commands. For example, in a kubectl task, under Kubernetes Cluster --> Service connection type use the Kubernetes Service Connection:
As mentioned by #DevOpsEngg, the problem could be related to private endpoints but I wouldn't say that it is regarding selfhosted agents, because I'm using these. As an extra comment - this started happening when I added more than one user to the cluster, so you might want to check user permissions and authentication. Unfortunately, I'm still getting used to K8s so I don't have more info about that.
I am facing issues from azure metadata service when I am trying to run my terraform plan.
Things I tried.
removing ~/.azure folder , re-tried az login from browser.
removed ~/.Identity folder as well , but still the same.
Also added client_id , client_secret , etc from provider block but no luck.
Error: retrieving environments from Azure MetaData service: Get "https:///metadata/endpoints?api-version=2019-05-01": http: no Host in request URL
on main.tf line 1, in provider "azurerm":
1: provider "azurerm" {
Any help would be greatly appreciated. I tried already to check if this was raised before but no luck.
I was able to fix this issue by doing 2 things mentioned below.
remove the ~/.azure and ~/.Identitiy folder from local.
export ARM_MSI_ENDPOINT=false
Once you do that you should now be able to access the metadata API again for authentication to azure.
Explanation of MSI flag is here terraform provider MSI use
I wanted to use service principal to authenticate and somehow I got this enabled in provider.
The Container Setting on the App Service it self look solid:
But the log pane shows errors:
2020-02-11 06:31:40.621 ERROR - Image pull failed: Verify docker image configuration and credentials (if using private repository)
2020-02-11 06:31:41.240 INFO - Stoping site app505-dfpg-qa2-web-eastus2-gateway-apsvc because it failed during startup.
2020-02-11 06:36:05.546 INFO - Starting container for site
2020-02-11 06:36:05.551 INFO - docker run -d -p 9621:8081 --name app505-dfpg-qa2-web-eastus2-gateway-apsvc_0_a9c8277e_msiProxy -e WEBSITE_SITE_NAME=app505-dfpg-qa2-web-eastus2-gateway-apsvc -e WEBSITE_AUTH_ENABLED=False -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=app505-dfpg-qa2-web-eastus2-gateway-apsvc.azurewebsites.net -e WEBSITE_INSTANCE_ID=7d18d5957d129d3dc3a25d7a2c85147ef57f1a6b93910c50eb850417ab59dc56 appsvc/msitokenservice:1904260237
2020-02-11 06:36:05.552 INFO - Logging is not enabled for this container.
Please use https://aka.ms/linux-diagnostics to enable logging to see container logs here.
2020-02-11 06:36:17.766 INFO - Pulling image: a...cr/gateway:1.0.20042.2
2020-02-11 06:36:17.922 ERROR - DockerApiException: Docker API responded with status code=NotFound, response={"message":"pull access denied for a...cr/gateway, repository does not exist or may require 'docker login': denied: requested access to the resource is denied"}
2020-02-11 06:36:17.923 ERROR - Pulling docker image a...cr/gateway:1.0.20042.2 failed:
2020-02-11 06:36:17.923 INFO - Pulling image from Docker hub: a...cr/gateway:1.0.20042.2
2020-02-11 06:36:18.092 ERROR - DockerApiException: Docker API responded with status code=NotFound, response={"message":"pull access denied for a...cr/gateway, repository does not exist or may require 'docker login': denied: requested access to the resource is denied"}
2020-02-11 06:36:18.094 ERROR - Image pull failed: Verify docker image configuration and credentials (if using private repository)
2020-02-11 06:36:19.062 INFO - Stoping site app505-dfpg-qa2-web-eastus2-gateway-apsvc because it failed during startup.
The Service Principal used to deploy the App Service has AcrPush access to the parent resource group of the container registry:
The setting are present:
I did az login with that service principal and then tried az acr login to the registry. It works fine. So what am I missing here?
EDIT 1
I know the credentials are correct, because I tested them like this:
Where I just copied the values from the app service configuration and pasted on the console. docker has no problem logging in.
It must be something else.
EDIT 2
However, I also get this:
C:\Dayforce\fintech [shelve/terraform ≡]> docker pull a...r/gateway
Using default tag: latest
Error response from daemon: pull access denied for a...r/gateway, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
So, I can login, but not pull. Very strange, because the account is configured to have AcrPush access to the container, which includes AcrPull:
EDIT 3
I was able to pull successfully when using the FQDN for the registry:
I updated the pipeline, but I still get the same errors:
2020-02-11 16:03:50.227 ERROR - Pulling docker image a...r.azurecr.io/gateway:1.0.20042.2 failed:
2020-02-11 16:03:50.228 INFO - Pulling image from Docker hub: a...r.azurecr.io/gateway:1.0.20042.2
2020-02-11 16:03:50.266 ERROR - DockerApiException: Docker API responded with status code=InternalServerError, response={"message":"Get https://a...r.azurecr.io/v2/gateway/manifests/1.0.20042.2: unauthorized: authentication required"}
2020-02-11 16:03:50.269 ERROR - Image pull failed: Verify docker image configuration and credentials (if using private repository)
2020-02-11 16:03:50.853 INFO - Stoping site app505-dfpg-qa2-web-eastus2-gateway-apsvc because it failed during startup.
EDIT 4
The only way that I found working was to enable the Admin User on the ACR and pass its credentials in the DOCKER_... variables instead of credentials of the Service Principal.
This is frustrating, I know the Service Principal can login and pull when ran locally, it is a mystery why it does not work for docker running on an App Service Host. We have another team here which faced the same issue and they have not found any solution, but enable the Admin User.
EDIT 5
The entire process runs as part of the Azure DevOps on-prem release pipeline using a dedicated Service Principal. Let me call it Pod Deploy Service Principal or just SP for short.
Let DOCKER_xyz denote the three app settings controlling the docker running on the App Service host:
DOCKER_REGISTRY_SERVER_URL
DOCKER_REGISTRY_SERVER_USERNAME
DOCKER_REGISTRY_SERVER_PASSWORD
I think we need to distinguish two parts here:
App Service needs to talk to the ACR in order to pull from it the details about the image and present them in this GUI - For that to work, the SP must have the AcrPull role in the ACR. Failure to do so results in the GUI presenting a spinning icon for the Image and Tag rows. I stumbled on it before - How to configure an Azure app service to pull images from an ACR with terraform? Now the answer to that question suggests that I have to assign the AcrPull role and set the DOCKER_xyz app settings. I think that the DOCKER_xyz app settings are not for that, but for the second part.
It seems to me that when an App Service is started, the host uses docker to actually pull the right image from the ACR. This part seems to be detached from (1). For it to work, the app settings must have the DOCKER_xyz app settings.
My problem is that part (1) works great, but part (2) does not even if DOCKER_xyz app settings specify the credentials of the SP from part (1). The only way I could make it work if I point DOCKER_xyz at the Admin User of the ACR.
But that why on Earth the DOCKER_xyz app settings cannot point to the pipeline SP, which was good enough for the part (1)?
EDIT 6
The current state of affairs is this. Azure App Service is unable to communicate with an ACR except using ACR admin user and password. So, even if the docker runtime running on the App Service host machine may know how to login using any service principal, the App Service would not use any identity or Service Principal to read metadata from the ACR - only admin user and password. The relevant references are:
https://feedback.azure.com/forums/169385-web-apps/suggestions/36145444-web-app-for-containers-acr-access-requires-admin#%7btoggle_previous_statuses%7d
https://github.com/MicrosoftDocs/azure-docs/issues/49186
On a personal note I find it amazing that Microsoft recommends not to use ACR admin user, yet a very core piece of their offering, namely Azure App Service, depends on it being enable. Makes me wonder whether different teams in Microsoft are aware of what others are doing or not doing...
App service started pulling after doing these steps for me. :D
Enable Admin Access in Azure Container Registry
In the App service configuration, provide container registry admin credentials
DOCKER_REGISTRY_SERVER_PASSWORD(admin enabled password),
DOCKER_REGISTRY_SERVER_USERNAME(crxxxxxx),
DOCKER_REGISTRY_SERVER_URL (https://crxxxxxx.azurecr.io)
Go to your app service and select identity section on the left, and click on system assigned - change status to On.
Now go to IAM Control container registry, add ACR pull role to App Service system assigned identity enabled on step 3.
Restart your App Service and wait .Changes will take few minutes to reflect so refresh your logs. (10 minutes or more)
Good luck :)
After a lot of research I figured out a way to resolve this without enabling Admin user
Create an app registration using Azure Active Directory and store the secret somewhere.
Go to the Azure container registry and add role assignment to this newly created app with permissions of AcrPush (which also contains AcrPull).
In the App service configuration, replace the variables .
DOCKER_REGISTRY_SERVER_PASSWORD with Client Secret of app registration which was saved in the first step
DOCKER_REGISTRY_SERVER_USERNAME with client Id of App registration
This should solve the Docker Api exception.
It's baffling that this is not mentioned in any Azure Container Registry documentation. Although I think it is mentioned somewhere in AAD documentation indirectly 😐.
From the message I got of the talk, let me solve your puzzle about the error.
I guess you deploy the image in ACR to the Web App through the Azure portal. When you use the Azure portal to deploy the Web App from the ACR, it only lets you select the ACR and image and tag, but do not let you set the credential. In this way, Azure will set it itself with the admin user and password if you enable the admin user. If you do not enable it, the error you got happens.
And if you want to use the service principal, I recommend you use the other tools, such as Azure CLI. Then you can set the docker registry credential yourself with the command az webapp config container set.
Here is the example and it works fine on my side:
With the Azure CLI, you can follow the steps here.
Update:
Here are the screenshots of the test on my side:
Found the answer by setting "acrUseManagedIdentityCreds" to True. The second command in this comment: https://stackoverflow.com/a/69120462/17430834
Edit 1: Adding the command
Here is the command that you will need to run to make this change.
az resource update --ids /subscriptions/<subscription-id>/resourceGroups/myResourceGroup/providers/Microsoft.Web/sites/<app-name>/config/web --set properties.acrUseManagedIdentityCreds=True
I was trying to do the same from Azure DevOps pipelines and got the same problem.
I didn't find out how to make it work using the ACR name, but it works if you use your_acr_name.azurecr.io instead.
If you go to the Access Keys page of your ACR you will find two values
Registry name: MyCoolRegistry (doesn't work if you use this one)
Login server: mycoolregistry.azurecr.io
The login server is working - just put it as the containerRegistry in your Pipeline without creating a service connection.
Just in case someone is struggling with that one.
Just to add to mark's amazing job of working it all through and for the fast readers: for everything to work, one of course also has to enable the admin user (who by default is disabled). For example by issuing:
az acr update -n <your-azureregistry-name> --admin-enabled true
on the console.
I experienced this same issue when trying to deploy an Docker application to Azure Web Apps for containers.
When I deployed the application I will get the error:
DockerApiException: Docker API responded with status code=NotFound, response={"message":"pull access denied for a..my-repo/image, repository does not exist or may require 'docker login': denied: requested access to the resource is denied"}.
Here's how I solved it:
The issue was that I was not specifying the full path to the image. I was supposed to include my-registry-url in the docker image-name. That is instead of just image-name I was supposed to use my-registry-url/image-name, since I am trying to pull from a private repository.
So say these are variables:
docker image name is promiseapp
docker-registry_url is promisecicdregistry.azurecr.io
resource-group is dockerprojects
app-service-plan is dockerlinuxprojects
azure-web-app name is promiseapptest
docker-registry-user is test-user
docker-registry-password is 12345678
Then my command will be:
az webapp create --resource-group dockerprojects --plan dockerlinuxprojects --name promiseapptest --deployment-container-image-name promisecicdregistry.azurecr.io/promiseapp
az webapp config container set --resource-group dockerprojects --name promiseapptest --docker-custom-image-name promisecicdregistry.azurecr.io/promiseapp --docker-registry-server-url https://promisecicdregistry.azurecr.io --docker-registry-server-user test-user --docker-registry-server-password 12345678
In my case, I fixed the error by using the fully qualified Azure Container Registery name like this:
xwezi.azurecr.io
The previous value was
xwezi
When I deploy manually to App Services, I wouldn't get that error.
But, when I used Azure App Service deploy task to deploy the container to the App Service, the service won't work correctly.
And, the log stream will show the above errors.
Unfortunately, the error messages weren't helpful for me to find this out. But I hope this will save your time :)
I've been given a personal access token (full access) which allows me to connect to a private Azure git repo within an Azure devops account from another subscription. Connecting to that repo locally using git is working fine.
I would like to set this up as a CI/CD deployment source for my app service but have been unable to find out how to do this. I tried Azure CLI:
az webapp deployment source config ... --repo-url https://anything:{pat}#dev.azure.com/Company/Project/_git/Reponame
This fails with a 500 error.
So I tried calling the Rest API directly but that also fails with the 500 error so not an Azure CLI issue.
Hoping someone can point me in the right direction,
Thanks for the help, much appreciated