Aks SecretProviderClass secret not found - azure

EDIT: Found the issue.I didnt installed the addon for secret driver. Once installed that i was able to make it work
I am facing an issue here and i have no idea what else i can try to figure out the issue.
I have an aks running with a single pod that runs a basic web app todo list. Nothing too fancy or complicated. what i am trying to do here, is to give permission to the aks cluster to access a keyvault and GET a secret to pass to the pod. the secret is just an ASPNETCORE_ENVIRONMENT: Development.
Following the documentations, i used helm to install the repo:
helm repo add csi-secrets-store-provider-azure https://azure.github.io/secrets-store-csi-driver-provider-azure/charts
helm install csi csi-secrets-store-provider-azure/csi-secrets-store-provider-azure
I created a Service Principle in azure:
SERVICE_PRINCIPLE_CLIENT_SECRET = az ad sp create-for-rbac --skip-assignment --name sp-aks-keyvault
i queried the clientId and Secret and passed them to my cluster as follow:
kubectl create secret generic secrets-store-creds --from-literal clientid="ClientID" --from-literal clientsecret="Password"
Once everything was set. I set those deployments.
Deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: webapp-deployment
namespace: default
labels:
app: webapp
spec:
replicas: 1
selector:
matchLabels:
app: webapp
template:
metadata:
labels:
app: webapp
spec:
containers:
- name: webapp
image: dockerimage-acr
ports:
- containerPort: 80
env:
- name: ASPNETCORE_ENVIRONMENT
valueFrom:
secretKeyRef:
name: aspenet-environment
key: environment
securityContext:
allowPrivilegeEscalation: false
volumeMounts:
- name: secrets-mount
mountPath: "/mnt/secrets-store"
readOnly: true
restartPolicy: Always
volumes:
- name: secrets-mount
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: "kv-name"
nodePublishSecretRef: # Only required when using service principal mode
name: secrets-store-creds
And my secretProvider.yaml
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: keyvault-secret-class
namespace: default
spec:
provider: azure
secretObjects:
- secretName: aspenet-environment
type: Opaque
data:
- objectName: aspnetcoreenvironment
key: environment
parameters:
usePodIdentity: "false"
useVMManagedIdentity: "false"
userAssignedIdentityID: ""
keyvaultName: "mykeyvault-name"
objects: |
array:
- |
objectName: aspnetcoreenvironment
objectType: secret
objectVersion: ""
tenantId: "<Tenant-Id>"
In my keyvault i gave access policy to the Service principle created and assigned Secret Permissions: GET and created a secret called
Name: aspnetcoreenvironment
value: Development
So far everything went ok, but when i run the deployment. and use the command kubectl describe pod <podname> i see the error, that prevents the container to start
Warning Failed 8s (x3 over 21s) kubelet Error: secret "aspenet-environment" not found
I tried different solutions but nothing works.
if i run the command kubectl get secretproviderclass i get back my provider i created.
As far as i understand, if no service is requiring a specific secret, i wont be able to find the secret i want to create if i run the command: kubectl get secret
And this is correct, i guess, because my pod is not starting.
Any help or enlightenment here about what i am doing wrong or how to fix it?
Thank you so much guys
EDIT:
Some extra debugging i came across the fact that the volume mount is still required. So i did add the volume to the deployment. But this is still giving an error.
The issue is, as i realized. Is when i run the command kubectl apply -f secretProviderClass.yml, no secret get created at all, reason why is failing.
So i think something is wrong here. Applying the SecretProviderClass shouldnt create automatically a secret service?

Related

Secret is not creating in AKS after fetching it with CSI Driver

By using the reference of https://learn.microsoft.com/en-us/azure/aks/csi-secrets-store-nginx-tls this document, I'm trying to fetch the TLS secrets from AKV to AKS pods.
Initially I created and configured CSI driver configuration with using User Assigned Managed Identity.
I have performed the following steps:
Create AKS Cluster with 1 nodepool.
Create AKV.
Created user assigned managed identity and assign it to the nodepool i.e. to the VMSS created for AKS.
Installed CSI Driver helm chart in AKS's "kube-system" namespace. and completed all the requirement to perform this operations.
Created the TLS certificate and key.
By using TLS certificate and key, created .pfx file.
Uploaded that .pfx file in the AKV certificates named as "ingresscert".
Created new namespace in AKS named as "ingress-test".
Deployed secretProviderClass in that namespace are as follows.:
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: azure-tls
spec:
provider: azure
secretObjects: # secretObjects defines the desired state of synced K8s secret objects
- secretName: ingress-tls-csi
type: kubernetes.io/tls
data:
- objectName: ingresscert
key: tls.key
- objectName: ingresscert
key: tls.crt
parameters:
usePodIdentity: "false"
useVMManagedIdentity: "true"
userAssignedIdentityID: "7*******-****-****-****-***********1"
keyvaultName: "*****-*****-kv" # the name of the AKV instance
objects: |
array:
- |
objectName: ingresscert
objectType: secret
tenantId: "e*******-****-****-****-***********f" # the tenant ID of the AKV instance
Deployed the nginx-ingress-controller helm chart in the same namespace, where certificates are binded with application.
Deployed the Busy Box deployment are as follows:
apiVersion: apps/v1
kind: Deployment
metadata:
name: busybox-one
labels:
app: busybox-one
spec:
replicas: 1
selector:
matchLabels:
app: busybox-one
template:
metadata:
labels:
app: busybox-one
spec:
containers:
- name: busybox
image: k8s.gcr.io/e2e-test-images/busybox:1.29-1
command:
- "/bin/sleep"
- "10000"
volumeMounts:
- name: secrets-store-inline
mountPath: "/mnt/secrets-store"
readOnly: true
volumes:
- name: secrets-store-inline
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: "azure-tls"
---
apiVersion: v1
kind: Service
metadata:
name: busybox-one
spec:
type: ClusterIP
ports:
- port: 80
selector:
app: busybox-one
Check secret is created or not by using command
kubectl get secret -n <namespaceName>
One thing to notice here is, if I attach shell with the busy box pod and go to the mount path which I provided to mount secrets I have seen that secrets are successfully fetched there. But this secrets are not showing in the AKS's secret list.
I have troubleshooted all the AKS,KV and manifest files but not found anything.
IF there is anything I have missed or anyone has solution for this please let me know.
Thanks in advance..!!!
i added this as a new answer, bcs the formatting was bad in the comments:
As you are using the Helm chart, you have to activate the secret sync in the values.yaml of the Helm Chart:
secrets-store-csi-driver:
syncSecret:
enabled: true
I would still recommend to use the csi-secrets-store-provider-azure as AKS Addon instead of the Helm-Chart
Your config looks good to me. One thing to consider is, that the User Assigned Managed Identity should not be the one you created for the AKS, it should be the managed identity from your nodepool (kubelet) and it also needs permission on the AKV.
I had the same issues while using the wrong Managed identity.
userAssignedIdentityID = Kubelet Client Id ( Nodepool Managed Idendity )
AZ CLI
export KUBE_ID=$(az aks show -g <resource group> -n <aks cluster name> --query identityProfile.kubeletidentity.objectId -o tsv)
export AKV_ID=$(az keyvault show -g <resource group> -n <akv name> --query id -o tsv)
az role assignment create --assignee $KUBE_ID --role "Key Vault Secrets Officer" --scope $AKV_ID

AKS Azure DevOps Build Agent

I'm trying to build a Azure DevOps Linux Build Agent in Azure Kubernetes Service.
I created the yaml file and created the secrets to use inside of the file.
I applied the file and have "CreateContainerConfigError" with my pod in a "waiting" state.
I run command
"kubectl get pod <pod name> -o yaml"
and it states the secret "vsts" could not be found.
I find this weird because I used "kubectl get secrets" and I see the secrets "vsts-account" and "vsts-token" listed.
You may check your kubernetes configuration, which is supposed to be like below:
apiVersion: v1
kind: ReplicationController
metadata:
name: vsts-agent
spec:
replicas: 1
template:
metadata:
labels:
app: vsts-agent
version: "0.1"
spec:
containers:
– name: vsts-agent
image: microsoft/vsts-agent:ubuntu-16.04-docker-18.06.1-ce-standard
env:
– name: VSTS_ACCOUNT
valueFrom:
secretKeyRef:
name: vsts
key: VSTS_ACCOUNT
– name: VSTS_TOKEN
valueFrom:
secretKeyRef:
name: vsts
key: VSTS_TOKEN
– name: VSTS_POOL
value: dockerized-vsts-agents
volumeMounts:
– mountPath: /var/run/docker.sock
name: docker-volume
volumes:
– name: docker-volume
hostPath:
path: /var/run/docker.sock
You may follow the blog below to see whether it helps you:
https://mohitgoyal.co/2019/01/10/run-azure-devops-private-agents-in-kubernetes-clusters/

Refer the docker image from another azure container repository (ACR) of different subscription

I am trying to pull the docker image in (QA-ACR) of subscription (QA-Subscription) from another Azure Container Registry (DEV-ACR) in subscription (DEV-Subscription).
Below are the steps in detail.
Created the docker image (example: docker-image-sample) in Subscription DEV-Subscription
Created the secret file by using the following command in Subscription DEV-Subscption
kubectl create secret docker-registry test-secret --docker-server=devsample.azurecr.io --docker-username=**** --docker-password=****
Pod is running in DEV-subscription by referring this secret. below is deployment file
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: test
spec:
replicas: 2
template:
metadata:
labels:
app: test
spec:
containers:
- image: devsample.azurecr.io/test_msdi:latest
imagePullPolicy: Always
name: test
ports:
- containerPort: 443
env:
- name: ASPNETCORE_ENVIRONMENT
value: dev
imagePullSecrets:
- name: test-secret
I am trying to pull the docker image from another ACR in different subscription.
Created the same secret here also like above.
Below is the content of the kubernetes deployment file
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: test
spec:
replicas: 2
template:
metadata:
labels:
app: test
spec:
containers:
- image: devsample.azurecr.io/test_msdi:latest
imagePullPolicy: Always
name: test
ports:
- containerPort: 443
env:
- name: ASPNETCORE_ENVIRONMENT
value: qa
imagePullSecrets:
- name: test-secret
Pod is failing from another ACR of different subscription. Issue is "Back off pulling the image ..."
Since your using an Azure Container Registry you might find it easier to assign the AKS Service Principal permissions on the container registry rather than rely on passing in credentials using a Kubernetes secret.
$Aks = Get-AzAks -ResourceGroupName QaSubscriptionAksResourceGroup -Name QaSubscriptionAks
New-AzRoleAssignment -ApplicationId $Aks.ServicePrincipalProfile.ClientId -RoleDefinitionName AcrPull -ResourceGroupName DevSubscriptionAcrResourceGroup
You might need to run Select-AzSubscription between the two commands to change from the QA subscription to the DEV subscription. Once that's set up remove
imagePullSecrets:
- name: test-secret
from your deployment file and rerun it.
Depending on how your AKS instances were deployed you might find that the AKS Service Principals already have the AcrPull role assigned within their own subscriptions, if that's the case you can remove imagePullSecrets completely.

Data from volumes as kubernetes secrets

I have an application that starts with docker-compose up. Some ssh credentials are provided with a json file, in a volume, in the host machine. I want to run the app in kubernetes, how can I provide the credentials using kubernetes secrets? my json file looks like:
{
"HOST_USERNAME"="myname",
"HOST_PASSWORD"="mypass",
"HOST_IP"="myip"
}
I created a file named mysecret.yml with base64 and I applied in kubernetes
apiVersion: v1
kind: Secret
metadata:
name: mysecret
type: Opaque
data:
HOST_USERNAME: c2gaQ=
HOST_PASSWORD: czMxMDIsdaf0NjcoKik=
HOST_IP: MTcyLjIeexLjAuMQ==
How I have to write the volumes in deployment.yml in order to use the secret properly?
apiVersion: v1
kind: Pod
metadata:
name: mypod
spec:
containers:
- name: mypod
image: redis
volumeMounts:
- name: foo
mountPath: "/etc/foo"
readOnly: true
volumes:
- name: foo
secret:
secretName: mysecret
This is the above example of using secret as volumes. You can use the same to define a deployment.
Please refer to official kubernetes documentation for further info:
https://kubernetes.io/docs/concepts/configuration/secret/

Mount Error for Block Storage on Azure kubernetes

I have been trying to mount a file share on Kubernetes pod hosted on AKS in Azure. So far, I have tried to:
1. Successfully created a secret by base64 encoding the name and the key
2. Create a yaml by specifying the correct configurations
3. Once I apply it using kubectl apply -f azure-file-pod.yaml, it gives me the following error:
Output: mount error: could not resolve address for
demo.file.core.windows.net: Unknown error
I have an Azure File Share by the name of demo.
Here is my yaml file:
apiVersion: v1
kind: Pod
metadata:
name: azure-files-pod
spec:
containers:
- image: microsoft/sample-aks-helloworld
name: azure
volumeMounts:
- name: azure
mountPath: /mnt/azure
volumes:
- name: azure
azureFile:
secretName: azure-secret
shareName: demo
readOnly: false
How can this possibly be resolved?

Resources