To preface, I am new to leveraging Kubernetes in Azure, so I might get some terms wrong. Sorry in advance.
I have been attempting to find a way to create Kubernetes secret objects that are populated with values from Azure key vault (AKV) in Azure Kubernetes services (AKS). The goal is to be able to create an imagePullSecret using values populated from AKV to allow my cluster to pull *images* from an Azure container registry (ACR). I have tested this out with manual creation and got it working, but am open to other solutions to integrating AKS and ACR.
I have found several articles pointing out that I can manage secrets from AKV by using the API secrets-store.csi.x-k8s.io/v1. I attempted to find a way to leverage this, but these appear to be secrets that can be mounted by a pod and not leveraged by the AKS management layer.
Whenever I attempt to search for a way to generate Kubernetes imagePullSecrets, I always find either the API above or manual secret creation. At this point, I am starting to think it is not possible create the imagePullSecrets with a Kubernetes manifest file. If it is possible, a link to documentation would be appreciated. If the mentioned API can achieve the desired goal, I might need help understanding how to leverage it for this task.
TLDR: Can I create a Kubernetes secret object without the need for pods in AKS using AKV?
Alternatively, Is there another integration solution for AKS and ACR that would avoid the need to manual linkage creation?
Edit: Changed secret to image
No, the secret CSI always need a pod to create Kubernetes secrets.
Yes, you can easily attach your ACR to your AKS leverage the Kubelet identity to pull images from your ACR. You just need to export the Identity and add a Role Assignment on your ACR (for an existing ACR and AKS):
export KUBE_ID=$(az aks show -g <resource group> -n <aks cluster name> --query identityProfile.kubeletidentity.objectId -o tsv)
export ACR_ID=$(az acr show -g <resource group> -n <acr name> --query id -o tsv)
az role assignment create --assignee $KUBE_ID --role "AcrPull" --scope $ACR_ID
For a new AKS simply run :
MYACR=myContainerRegistry
az acr create -n $MYACR -g myContainerRegistryResourceGroup --sku basic
az aks create -n myAKSCluster -g myResourceGroup --generate-ssh-keys --attach-acr $MYACR
This can be also done with IaC like Bicep or Terraform (just create the Role Assignment on the Kubelet Identity)
Here you can also find the documentation.
Let's clarify your objective...
You want to pull images from ACR and in order to do so you need to set up imagePullSecrets in your pods/deployments. I am assuming you want to limit access to the ACR to specific namespaces in your cluster. If you want it cluster wide - then you may just go with direct integration setup with AKS --> ACS
One would not pull secrets from ACR as you indicated in your question, they would pull the secret from Azure Key Vault using the Azure version of the secrets store connector you listed above (https://learn.microsoft.com/en-us/azure/aks/csi-secrets-store-driver) - this will set up secrets in your cluster namespace the pods can use for imagePullSecrets to pull container images from ACR.
In short:
set up Azure CSI
use Azure CSI to create secret holding your pull secrets for your ACS. The creation happens on pod deployment
when defining pod spec - set the imagePullSecrets secret name to the one you create in step 2
I would do it this way.
Reconfigure your cluster with the Azure Key Vault provider - you can do this when you create your cluster so you don't have to go through the effort of adding it manually. If you are doing it through the portal it's under the Advanced Tab in the cluster setup pictured below...
or if you are setting up your cluster in terraform like I do then set up the key_vault_secrets_provider section in the resource.
Read the instructions in the link above on how to set up secret pulling from AKV. you will need to mount a volume to your pods to create them. If you haven't done that before it's pretty easy to get the hang of. Once you do, every time you deploy a pod, a kubernetes secret will be created with the value of the secret pulled from AKV.
When you need to assign the image pull secret - you use the secret you created via the mounted volume
your deployment yaml may look something like this..
apiVersion: apps/v1
kind: Deployment
metadata:
name: <your deployment name>
namespace: <namespace name>
<more meta data stuff here>
spec:
<some deployment spec stuff here>
template:
metadata:
<some common metadata for your pods in this deployment>
spec:
imagePullSecrets:
- name: <your secret name that holds the value from AKV here>
containers:
- name: <your container name>
image: <ACR_registry_name_here.repo_name:version>
volumeMounts:
- mountPath: /mnt/<image-pull-secret-mount-for-csi here>
name: <whatever name you want>
readOnly: true
<some other container stuff here>
volumes:
- name: <whatever name you want - must be same as above volume mount>
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: <the name of your secretProviderClass resource - see below>
---
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: <your secretProviderClass name>
namespace: <namespace name>
spec:
parameters:
keyvaultName: <the name of your Azure Key Vault>
objects: <read instructions in the link on how to create these>
resourceGroup: <the azure resource group name of your key vault>
subscriptionId: <your Azure Subscription GUID>
tenantId: <your Azure tenant GUID>
usePodIdentity: "false"
useVMManagedIdentity: "true"
userAssignedIdentityID: <the managed identity id for your core cluster virtual machines - look in the resource group that is automatically created for your cluster VMs when you create your cluster>
provider: azure
secretObjects:
- <this depends upon the secret type you want - again - read instructions in the link I provided>
there are other resources out there i found that this guy's videos are helpful
https://www.youtube.com/watch?v=S2wzCD5L9Mo&ab_channel=HoussemDellai
good luck - I hope this helps
If your objective is to securely pull an image from a ACR on AKS Cluster
imo, the cleanest would be to grant AcrPull (https://learn.microsoft.com/en-us/azure/container-registry/container-registry-roles?tabs=azure-cli) to your AKS NodePool identity ( this can be done at the point of deployment of the cluster, or grant it later using CLI or the portal)
but as an alternative if you want to read some service principal secret from the Azure key vault and directly want it to be mapped to a key-vault secret --> kubernetes secret , you can use external-secrets.io (https://external-secrets.io/v0.7.0/provider/azure-key-vault/)
Related
I working on AKS shared cluster, where have multiple teams are working on the same cluster and have their own ACR for each team.
I want to find ways to allow ACR to pull from specified namespace only.
Currently that I have though is an expensive way by
Using ACR premium tier to enable the scope-map feature, and create the token for authentication on pull secret.
Or someone did know how to pull an image from the service principal with the AcrPull role.
please tell me.
thank you.
I have found the solution without changing the ACR pricing tier, by using only the service principal to access the target ACR.
Solution
Create the service principal and assign AcrPull role.
After that, Create kubernetes secret into your namespace to pull image by ImagePullSecrets
kubectl create secret docker-registry <secret-name> \
--namespace <namespace> \
--docker-server=<container-registry-name>.azurecr.io \
--docker-username=<service-principal-ID> \
--docker-password=<service-principal-password>
reference
A few weeks ago, I was able to use the Azure CLI to create my Container Registry (ACR) and Kubernetes (AKS) cluster. I could push images to my ACR and have AKS pull images successfully - everything worked great. Every now and then, I would have to refresh my login with az acr login --name <acrName>, but not a big deal.
Today, I found that when I go to deploy an updated image to my AKS cluster, I got a status of ImagePullBackOff:
Failed to pull image "MY_ACR.azurecr.io/MY_IMAGE:v1": rpc error: code = Unknown desc = Error response from daemon: Get https://MY_ACR.azurecr.io/v2/MY_IMAGE/manifests/v1: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information.
I couldn't remember what I needed to do to make this work, so I went through my original steps and created an entirely new resource group, ACR, AKS cluster, and service principal connecting them. I pushed images to my ACR and was able to apply my Kubernetes manifest, and everything worked again.
A couple hours later, when I applied an updated manifest, I again got the same error message. As part of my setup, I created a service principal:
az ad sp create-for-rbac --skip-assignment
az role assignment create --assignee <principal's appId> --scope <my ACR's id> --role Reader
I also used --role acrpull. It seems like the authentication has timed out, and the documentation for Authenticate with an Azure container registry says that individual AD identities will time out after 3 hours, but even after running az acr login --name <acrName>, I'm not able to fix the issue.
What are the required steps to get my AKS cluster to be able to authenticate again to my ACR?
I'll note that I also attached the ACR according to the documentation at Authenticate with Azure Container Registry from Azure Kubernetes Service by running:
az aks update -n cluster_name -g resource_group --attach-acr acr_name
I also tried using the ACR id instead of the name. After a minute or so, the command completed, and even a half hour+ later, I get the same permissions issue.
The easiest way to integrate AKS with ACR is to leverage the --attach-acr option during cluster creation. This will have AKS manage the service principal for your and handle the token refresh's
https://learn.microsoft.com/en-us/azure/aks/cluster-container-registry-integration#create-a-new-aks-cluster-with-acr-integration
We are using Azure env injector webhook configuration to inject the env variables from azure keyvault into kubernetes pods. we have a bunch of env variables. each variable is created as a secret in azure keyvault. In POD definition env is loaded from specific secret from azure keyvault.
We would like to create all key:values in one azure keyvault and load all of them using a single secret in POD definition.
please advise
You can leverage the Flex Volume to enable integration between your Kubernetes cluster and your Azure Key Vault instance. The way you can achieve this is explained here. In this article you have all the steps to perform this integration.
Assuming having access to an Azure subscription with a fully configured Azure Kubernetes Service, via
az login
kubectl create clusterrolebinding kubernetes-dashboard --clusterrole=cluster-admin --serviceaccount=kube-system:kubernetes-dashboard
az aks browse --resource-group somegroup --name somecluster
i can get access to Kubernetes Dashboard.
Is there a way to give temporary access to Kubernetes Dashboard to some person who does not have access to the Azure Subscription the AKS is associated with?
yes, just create appropriate kubernetes config (so the user can port-forward the dashboard pod) to the cluster and then the user will be able to connect to the dashboard.
I'm trying to follow this guide to setting up a K8s cluster with external-dns' Azure DNS provider.
The guide states that:
When your Kubernetes cluster is created by ACS, a file named /etc/kubernetes/azure.json is created to store the Azure credentials for API access. Kubernetes uses this file for the Azure cloud provider.
When I create a cluster using aks (e.g. az aks create --resource-group myResourceGroup --name myK8sCluster --node-count 1 --generate-ssh-keys) this file doesn't exist.
Where do the API credentials get stored when using AKS?
Essentially I'm trying to work out where to point this command:
kubectl create secret generic azure-config-file --from-
file=/etc/kubernetes/azure.json
From what I can see when using AKS the /etc/kubernetes/azure.json doesn't get created. As an alternative I followed the instructions for use with non Azure hosted sites and created a service principal (https://github.com/kubernetes-incubator/external-dns/blob/master/docs/tutorials/azure.md#optional-create-service-principal)
Creating the service principal produces some json that contains most of the detail. This can be used to manually create the azure.json file and the secret can be created from it.
Use this command to get credentials:
az aks get-credentials --resource-group myResourceGroup --name myK8sCluster
Source:
https://learn.microsoft.com/en-us/azure/aks/kubernetes-walkthrough
Did you try this command ?
cat ~/.kube/config
It provided all i needed for my CI to connect to the Kubernetes Cluster and use API