need some help renewing api-server certs running on 1.18 - azure

My prod AKS cluster is running on 1.18 and i'm not able to run any kubectl command because the api-cert is expired a day ago.
cluster is throwing the error below:
x509: certificate has expired or is not yet valid: current time 2022-12-05T12:30:28+05:30 is
I do see there are few blogs referring to rotate certs using the below command.
az aks rotate-certs -g $RESOURCE_GROUP_NAME -n $CLUSTER_NAME
As this is a cluster running on older k8 version, i'm wondering if the above step works.
can this be run during working hours?
running this command will have any impact on the workloads?
any help is greatly appreciated. thanks.

I tried to renew the API server certificate in my environment and got the below results
Kubernetes uses certificates for authentication with many of components
If we have RABC enabled for the cluster then its enabled with the auto rotation for certificate
For checking the expiry date for the certificate
kubectl config view --raw -o jsonpath="{.users[?(#.name == 'clusterUser_rg_cluster_name')].user.client-certificate-data}" | base64 -d | openssl x509 -text | grep -A2 Validity
To check the expiry date of certificate use the below command
az vm run-command invoke -g Alldemorg -n linuxvmaks --command-id RunShellScript --query 'value[0].message' -otsv --scripts "openssl x509 -in /etc/kubernetes/certs/apiserver.crt -noout -enddate"
To upgrade the cluster use the below command
az aks upgrade -g rg_name -n cluster_name
To sign into the cluster
az aks get-credentials -g rg-name -n cluster-name
To rotate all the certificates
az aks rotate-certs -g rg-name -n cluster-name
To verify the old certificates are valid or not by using below command
kubectl get pods
By using above command I am not able to access the kubectl commands
To enable the kubectl commands use the below command
When I check the kubectl command I am able to access
kubectl get nodes

Related

Workload identity with application on Kubernetes service:kubernetes

I am trying to deploy and manage the Kubernetes cluster using OpenID Connect issuer, I have followed this Microsoft Document to deploy the application on AKS for that I have created the resource group and install the AKS preview extension
`az group create --name myResourceGroup --location eastus
az extension add --name aks-preview
az extension update --name aks-preview
#register the enabled workload identity
az feature register --namespace "Microsoft.ContainerService" --name "EnableWorkloadIdentityPreview"az feature show --namespace "Microsoft.ContainerService" --name "EnableWorkloadIdentityPreview"
az provider register --namespace Microsoft.ContainerService`
After that when I am trying to create the Kubernetes cluster with --enable-oidc-isser I am getting below error, it is taking more than 10 minutes and showing some error
(OIDCIssuerUnsupportedk8sVersion) OIDC issuer feature requires at least Kubernetes version 1.20.0. Code: OIDCIssuerUnsupportedK8sVersion Message: OIDC issuer feature requires at least Kubernetes version 1.20.0
I have upgraded to the latest version but still getting same error.
How can I export OIDC Issuer to set the Environment variables on the cluster.
Thanks in Advance :)
I tried to reproduce the same issue in my environment and got the below results
My current version is 1.23.12
I have created the resource group and install the extension preview and registered the EnabledworkloadidentityPreview using below command
az feature register --namespace "Microsoft.ContainerService" --name "EnableWorkloadIdentityPreview"
To verify the status used the below command
az feature show --namespace "Microsoft.ContainerService" --name "EnableWorkloadIdentityPreview"
I have created the AKS cluster with --enable-oidc-issuer parameter to use the OIDC issuer using below command
az aks create -g rg -n cluster --node-count 1 --enable-oidc-issuer --enable-workload-identity --generate-ssh-keys
When I check in my environment and got the same error
To resolve this issue I have upgraded my AKS version using below commands
My current version is 1.23.12
I have upgraded the to newest version using this SO answer
My current version is 1.24.3 and when I the below OIDC command I am able access
az aks create -g <rg-name> -n cluster --node-count 1 --enable-oidc-issuer --enable-workload-identity --generate-ssh-keys
NOTE: This error will occur if the version is more than 1.20.0 we have to upgrade the version to latest version not to current version then only it will work

Azure VM command fails with no credential error

I am running a command from the azure CLI to execute a script on a linux VM running in Azure.
The command looks like this:
az vm run-command invoke --command-id RunShellScript --name <\VMName> --resource-group <\RGName> --scripts "curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash"
The script installs azure CLI on the VM.
The command results into error:
Could not retrieve credential from local cache for service principal 0cf4e5cf-de8f-4524-8be6-5e2exxxxxx. Please run 'az login' for this service principal.
Can someone help me understand the problem here.
When I run the command without the piped sudo, the command works fine.
I tried to reproduce the same in my environment I got the same error as below
az vm run-command invoke --command-id RunShellScript --name <\VMName> --resource-group <\RGName> --scripts "curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash"
This error may occur if you are not using latest version, please check your version like below and try to update your version:
az version
Also, I have tried logging in with my service principal in Azure CLI and tried running the command to install azure-cli inside the VM and it run successfully.
az login --service principal -u <AppID> -p <secret> --tenant <TenantID>
You can also install latest version locally by using this MsDocs
Reference:
Azure CLI GitHub Action fails with Azure CLI 2.30.0: Could not retrieve credential from local cache for service principal · Issue #20154 · Azure/azure-cli · GitHub

Issue while enabling prometheus monitoring in azure k8s

Getting error while configuring prometheus in azure kubernetes
I tried to reproduce the same issue in my environment and got the below results
I have the cluster and I am trying to configure the Prometheus in azure Kubernetes and I got the successful deployment
To verify the agent is deployed or not use the below commands
kubectl get ds <dep_name> --namespace=kube-system
kubectl get rrs --namespace=kube-system
This error getting because of you are using the service principal instead of managed identity
For enabling the managed identity please follow the below commands
AKS cluster with service principal first disable monitoring and then upgrade to managed identity, the azure public cloud is supporting for this migration
To get the log analytics workspace id
az aks show -g <rg_name> -n <cluster_name> | grep -i "logAnalyticsWorkspaceResourceID"
For disable the monitoring use the below command
az aks disable-addons -a monitoring -g <rg_name> -n <cluster_name>
Or I can get it on portal in the azure monitor logs
I have upgrade the cluster to system managed identity, use the below command to upgrade
az aks update -g <rg_name> -n <cluster_name> --enable-managed-identity
I have enable the monitoring addon with the managed identity authentication
az aks enable-addons -a monitoring --enable-msi-auth-for-monitoring -g <rg_name> -n <cluster_name> --workspace-resource-id <workspace_resource_id>
For more information use this document for Reference

Where are kubelet logs in AKS stored?

I would like to view kubelet logs going back in time in Azure AKS. All I could find from Azure docs was how to ssh into the nodes and list the logs (https://learn.microsoft.com/en-us/azure/aks/kubelet-logs) but I feel like this has to be aggregated in Log Analytics somewhere right ?
However I wasn't able to find anything in Log Analytics for Kubernetes. Am I missing something ?
We have omsagent daemonset installed and Microsoft.OperationalInsights/workspaces is enabled
Thanks :)
I tried to reproduce this issue in my environment and got below results:
I created resource group and VM by setting up the Subscription
az account set --subscription "subscription_name"
az group create --location westus resourcegroup_name
created the AKS Cluster with the parameter to enable the AKS Container Insights
The following Example will creates the Cluster with name AKSRG.
az aks create -g myResourceGroup -n resourceGroup_name --enable-managed-identity --node-count 1 --enable-addons monitoring --enable-msi-auth-for-monitoring --generate-ssh-keys
Here I have configured the kubectl to connect the kubernetes cluster with the get-credentials command
I have created the interactive shell connection to the node
using kubectl debug
kubectl debug node/pod_name -it --image=mcr.microsoft.com/dotnet/runtime-deps:6.0
I have used the below Command in after hash(#) tag
journalctl -u kubelet -o cat
To get the logs check the nodes and pods
We can use the below command to check the KUBE LOGS
kubectl logs pod_name
Reference:
View kubelet logs in Azure Kubernetes Service (AKS) - Azure Kubernetes Service | Microsoft Docs

How to configure kubectl from another pc?Namespaces are empty

I have successfully setup the kubectl cli on my laptop to connect to my azure cluster.
If I make, for example:
kubectl config get-contexts
I get my namespaces and I can see my resources by navigating to the current namespace.
Now I need to replicate this setup on another laptop of mine and I made the following:
az login <--login to azure
az aks install-cli <--install of kubectl
az aks get-credentials --resource-group myResourceGroup --name myCluster <--linking kubectl to my cluster
Problem is that, if I make get-contexts again I only get the default namespace. And, of course, that namespace is empty as I put my deployment in another one.
What am I missing?
so I'm not sure what the actual question is. if your resources are in different namespace, you can query those namespaces like you normally would:
kubectl get pods -n othernamespace
kubectl edit deployment xxx -n othernamespace
you can set the default namespace for the context like so:
kubectl set-context xxx --namespace othernamespace

Resources