Kubernetes pod throwing CredentialUnavailableException: EnvironmentCredential authentication unavailable - azure

I am running a Spring Boot application utilizing Azure Kubernetes Service. I found this strange error in my pod logs recently.
com.azure.identity.CredentialUnavailableException: EnvironmentCredential authentication unavailable. Environment variables are not fully configured.To mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azsdk/java/identity/environmentcredential/troubleshoot
ManagedIdentityCredential authentication unavailable. Connection to IMDS endpoint cannot be established.
SharedTokenCacheCredential authentication unavailable. No accounts were found in the cache.
IntelliJ Authentication not available. Please log in with Azure Tools for IntelliJ plugin in the IDE.
Failed to read Vs Code credentials from Linux Key Ring.
AzureCliCredential authentication unavailable. Azure CLI not installed.To mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azsdk/java/identity/azclicredential/troubleshoot
Unable to execute PowerShell. Please make sure that it is installed in your systemTo mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azure-identity-java-default-azure-credential-troubleshoot
Any hints are much appreciated !
My trails so far:
Upgrade/Downgrade Kubernetes versions
Checking Environment Variable Assignments

Could you please validate that you are setting the following environment variables?
ENVIRONMENT_VARIABLES
ensure that the variables azure_client, azure_tenant and azure_client_secret are properly set.
Below steps will work when authenticate using environment variables:
Please add the following variables in env_path,
export AZURE_CLIENT_ID=XXXXXXXXXXXXXX
export AZURE_TENANT_ID=XXXXXXXXXXXXX
export AZURE_CLIENT_SECRET=XXXXXXXX
Check your environment variables with
System.getenv("AZURE_CLIENT_ID")
MANAGEDIDENTITY_CREDENTIALS
Managed Identity is currently unsupported by the Java, we can use an secret or a certificate authentication
for Sample:
export AZURE_CLIENT_ID=XXXXXXXXXXXXXX
export AZURE_TENANT_ID=XXXXXXXXXXXXX
export AZURE_CLIENT_CERTIFICATE_PATH=XXXXXXXXXXXX
In VS, go to Tools > Options>Azure Service Authentication > Account Selection> Sign_in with your credentials
If you see the "Re-enter your credentials link, click it and sign in again.
if not sign_out and sign_in again.
PROFILE_ENV_APPLICATION
Please check the profile environment for the application
windir\System32\inetsrv\config\applicationHost.config
In application.config file if it setProfileEnvironment is false, change it to True.
If not, add it under <applicationPoolDefaults> tag like below.
<applicationPoolDefaults managedRuntimeVersion="vXX">
<processModel identityType="ApplicationPoolIdentity" loadUserProfile="true" setProfileEnvironment="true">
SHARED_TOKEN_CASHE_CREDENTIAL
for shared token cache credentials we have to add the below command
DefaultAzureCredential(connection_verify=False, exclude_shared_token_cache_credential=True
AZURE_CLI_CREDENTIAL AND AZURE_CLI
in environment variable add your PATH
run the terminal
echo $PATH
POWERSHELL
open the PowerShell and run as a administrator,
run the command to fix the disk and display a status report
Chkdsk c: /F
after this command You will have to restart the computer to work the PowerShell.

Related

Use DefaultAzureCredentials to authenticate Service bus in Docker Container

I'm trying to use DefaultAzureCredentials to authenticate my Azure function against Azure Service Bus. In my azure function azure-func-service-bus, I call to Azure Service Bus
servicebus_client = ServiceBusClient(
fully_qualified_namespace=MY_SERVICE_BUS_NAMESPACE_NAME+".servicebus.windows.net",
credential=DefaultAzureCredential(additionally_allowed_tenants=['*'])
)
I created and pushed Docker container to ACR. When I run the container locally for testing outside of Azure, it does not know what permissions to use.
az acr login --name acr01
docker push acr01.azurecr.io/azure-func-service-bus:v1
docker pull acr01.azurecr.io/azure-func-service-bus:v1
docker run -it --rm -p 8080:80 acr01.azurecr.io/azure-func-service-bus:v1
but got the following error.
DefaultAzureCredential failed to retrieve a token from the included credentials.
Attempted credentials:
EnvironmentCredential: EnvironmentCredential authentication unavailable. Environment variables are not fully configured.
Visit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot to troubleshoot.this issue.
ManagedIdentityCredential: ManagedIdentityCredential authentication unavailable, no response from the IMDS endpoint.
SharedTokenCacheCredential: SharedTokenCacheCredential authentication unavailable. No accounts were found in the cache.
VisualStudioCodeCredential: Failed to get Azure user details from Visual Studio Code.
AzureCliCredential: Azure CLI not found on path
AzurePowerShellCredential: PowerShell is not installed
To mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.
Unexpected error occurred (ClientAuthenticationError('DefaultAzureCredential failed to retrieve a token from the included credentials.\nAttempted credentials:\n\tEnvironmentCredential: EnvironmentCredential authentication unavailable. Environment variables are not fully configured.\nVisit https://aka.ms/azsdk/python/identity/environmentcredential/troubleshoot to troubleshoot.this issue.\n\tManagedIdentityCredential: ManagedIdentityCredential authentication unavailable, no response from the IMDS endpoint.\n\tSharedTokenCacheCredential: SharedTokenCacheCredential authentication unavailable. No accounts were found in the cache.\n\tVisualStudioCodeCredential: Failed to get Azure user details from Visual Studio Code.\n\tAzureCliCredential: Azure CLI not found on path\n\tAzurePowerShellCredential: PowerShell is not installed\nTo mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.')). Handler shutting down.
I'm missing a key piece of the puzzle. How can I handle this?
When the Azure Function runs in Azure, it's configured to support ManagedIdentityCredential. For your case I'd recommend trying to configure EnvironmentCredential to test locally.
You can find the details in the link, but the short version is:
Create a service principle (Docs) and give it the needed access
Run the container with extra Environment Variables:
AZURE_TENANT_ID: service principal's Tenant ID
AZURE_CLIENT_ID: service principal's AppId
AZURE_CLIENT_SECRET: service principle's password
I'd recommend using a .env file to make this easier, but be sure it doesn't get checked in anywhere.
FYI If your account doesn't use MFA, you can instead use the variables AZURE_USERNAME and AZURE_PASSWORD. But then you've put your username and password in a file or your terminal history which is concerning. Admittedly the service principal has the same problem, but you can more easily mitigate that with minimizing it's access and regularly rolling the secret.
P.S. If you're using Visual Studio for making your Azure Function you should be able to use something like: EnvironmentCredentialExample to automate setting up and using the needed .env file.

Not able to restore application from powershell

I accidently deleted an Azure Ad application and want to restore it from Powershell.
To get the Azure Ad deleted application, I found this command Get-AzureADDeletedApplication.
To restore the application, I am using the below command:
Restore-AzureADDeletedApplication
But I am getting Resource not found error.
Please help me with the error and where I am going wrong.
Thank You #NiMux for your suggestion- We have tested in our environment; it is working fine using the objectId there is no option to pass clientId as a parameter of Restore-AzureADDeletedApplication.
If you are passing the ClientId forcefully it will give you an error.
Note:
The AppRegistration can be restored but it won't restore the Enterprise Application once deleted.
Output: After restoring the AppRegistration.

Azure DevOps Release Pipeline || To sign in, use a web browser to open

I created the aks cluster with azure service principal id and i provided the contributer role according to the subscription and resource group.
For each and every time when i executed the pipeline the sign-in is asking and after i authenticated it is getting the data.
Also the "kubectl get" task is taking more than 30 min and is getting "Kubectl Server Version: Could not find kubectl server version"
To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code CRA2XssWEXUUA to authenticate
Thanks in advance
What is the version of the created cluster?
I'm assuming from your snapshot that you are using az in order to get credentials for it.
Old azure auth plugin is deprecated in V1.22+. If you are using V1.22 or above you should use kubelogin in order authenticate.
You will also need to update your kube config accordingly:
kubelogin convert-kubeconfig
and specifically if you're logging via az:
kubelogin convert-kubeconfig -l azurecli
Note that the flag -l azurecli is important here: the default value is "devicecode" which will not consider your az as a logging method - and you will still be requested a browser authentication.
Alternatively, you can set environment variable:
AAD_LOGIN_METHOD=azurecli
Because you are getting sign in request and not the deprecation warning for the auth plugin, I suspect that you already have kubelogin installed on your agent, and you just need to update the kube config file
What task are you using? There is official kubectl task: https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/deploy/kubernetes?view=azure-devops
It requires the service connection.
If you still want to execute kubectl directly, you should run the following before the kubectl inside the AzureCLI task:
az aks get-credentials --resource-group "$(resourceGroup)" --name "$(k8sName)" --overwrite-existing
Please use Selfhosted agents for executing your commands. looks like you have private endpoints for your AKS and requests are only allowed from trusted devices.
I ran into the same issue and for me the fix was to change the Connection Type in the stage definition from Azure Resource Manager to Kubernetes Service Connection - check on the screenshot below.
Then you should be able to also specify the connection type in each of the tasks where you are running kubectl or helm commands. For example, in a kubectl task, under Kubernetes Cluster --> Service connection type use the Kubernetes Service Connection:
As mentioned by #DevOpsEngg, the problem could be related to private endpoints but I wouldn't say that it is regarding selfhosted agents, because I'm using these. As an extra comment - this started happening when I added more than one user to the cluster, so you might want to check user permissions and authentication. Unfortunately, I'm still getting used to K8s so I don't have more info about that.

Unable to add ssh key in azure vm

I am the admin of this particular azure subscription. I had to add my ssh key to a ubuntu server. But when I try to add the ssh key through "Reset Password", After sometime I'll get the following error message.
VM agent on VM 'Server' has not reported latest status for extension 'enablevmaccess'. Please verify the VM has a running VM agent and can establish outbound connections to Azure storage.
What might be the issue? How to resolve this?
Failed to reset ssh key
vmaccess is enabled
Two simple thing you might try;
Uninstall the VMAccess extension and try reset again.
Use the 'Run Command' to set/reset password.
Hope this helps.
your first error tells you exactly why this happens. vm extension needs to talk to azure storage to report extension status. if it cant - portal operations might fail (this doesnt mean the extension failed; its just unable to report actual extension status).

Unable to get access to Key Vault using Azure MSI on App Service

I have enabled Managed Service Identities on an App Service. However, my WebJobs seem unable to access the keys.
They report:
Tried the following 3 methods to get an access token, but none of them worked.
Parameters: Connectionstring: [No connection string specified], Resource: https://vault.azure.net, Authority: . Exception Message: Tried to get token using Managed Service Identity. Unable to connect to the Managed Service Identity (MSI) endpoint. Please check that you are running on an Azure resource that has MSI setup.
Parameters: Connectionstring: [No connection string specified], Resource: https://vault.azure.net, Authority: https://login.microsoftonline.com/common. Exception Message: Tried to get token using Active Directory Integrated Authentication. Access token could not be acquired. password_required_for_managed_user: Password is required for managed user
Parameters: Connectionstring: [No connection string specified], Resource: https://vault.azure.net, Authority: . Exception Message: Tried to get token using Azure CLI. Access token could not be acquired. 'az' is not recognized as an internal or external command,
Kudo does not show any MSI_ environmental variables.
How is this supposed to work? This is an existing App Service Plan.
The AppAuthentication library leverages an internal endpoint in App Service that receives the tokens on your site's behalf. This endpoint is non-static and therefore is set to an environment variable. After activating MSI for your site through ARM, your site will need to be restarted to get two new Environment Variables set in it:
MSI_ENDPOINT and MSI_SECRET
The presence of these variables are essential to the MSI feature working properly during runtime as the AppAuthentication library uses them to get the authorization token. The error message reflects this:
Exception Message: Tried to get token using Managed Service Identity. Unable to connect to the Managed Service Identity (MSI) endpoint. Please check that you are running on an Azure resource that has MSI setup.
If these variables are absent, you might need to restart the site.
https://learn.microsoft.com/en-us/azure/app-service/app-service-managed-service-identity
If the environment variables are set and you still see the same error, the article above has a code sample showing how to send requests to that endpoint manually.
public static async Task<HttpResponseMessage> GetToken(string resource, string apiversion) {
HttpClient client = new HttpClient();
client.DefaultRequestHeaders.Add("Secret", Environment.GetEnvironmentVariable("MSI_SECRET"));
return await client.GetAsync(String.Format("{0}/?resource={1}&api-version={2}", Environment.GetEnvironmentVariable("MSI_ENDPOINT"), resource, apiversion));
}
I would try that and see what kind of response I get back.
I just solved this issue when trying to use MSI with a Function app, though I already had the environment variables set. I tried restarting multiple times to no success. What I ended up doing was manually turning off MSI for the Function, then re-enabling it. This wasn't ideal, but it worked.
Hope it helps!
I've found out that if you enable MSI and then swap out the slot, the functionality leaves with the slot change. You can re-enable it by switching it off and on again but that will create a new identity in AD and will require you to reset permissions on the key vault for it to work.
Enable the identity and give access to your azure function app in keyvault via access policy.
You can find identity in platform feature tab
These two steps works for me
In my case I had forgotten to add an Access Policy for the application in the Key Vault
Just switched ON the Status like #Sebastian Inones showed.
Than add access policy for KeyVault like
This is resolved the issue!!
For the ones, like my self, wondering how to enable MSI.
My scenario:
I have an App Service already deployed and running for a long time.
In addition, on Azure DevOps I have my Pipeline configured to Auto-Swap my Deployment Slots (Staging/Production). Suddenly, after a normal push, Production starts failing because of the described issue.
So, in order to enable MSI again (I don't know why it has to be re-enabled but I believe this is only a workaround, not a solution, as it should be still enabled in the first place)
Go to your App Service. Then Under Settings --> Identity.
Check the status: In my case, it was off
I have attached an image below to make it easier to follow.
For the folks that will come across these answers, I would like to share my experience.
I got this problem with Azure Synapse pipeline run. Essentially I added access policies properly to the KeyVault, and also I added a LinkedService to the Azure Synapse pointing to my KeyVault.
If I trigger the notebook manually it works, but in the pipeline, it fails.
Initially, I used the following statement:
url = TokenLibrary.getSecret("mykeyvault", "ConnectionString")
Then I added the name of the linked service as a third parameter, and the pipeline was able to leverage that linked service to obtain the MSI token for a Vault.
url = TokenLibrary.getSecret("mykeyvault", "ConnectionString", "AzureKeyVaultLinkedServiceName")
Might be unrelated to your issue but I was getting the same error message.
For me, the issue was using pip3's azure-cli. I was able to fix this issue by using brew packages for both azure-cli and azure-functions-core-tools.
Uninstall pip3 azure-cli
pip3 uninstall azure-cli
Install brew azure-cli
brew update
brew install azure-cli
Double check if the error message ends with:
Please go to Tools->Options->Azure Services Authentication, and re-authenticate the account you want to use.

Resources