I am trying to sync an Azure Key Vault Secret with a Kubernetes Secret of type dockerconfigjson by applying the following yaml manifest with the 4 objects Pod, SecretProviderClass, AzureIdentity and AzureIdentityBinding.
All configuration around key vault access and managed identity RBAC rules have been done and proven to work, as I have access to the Azure Key Vault secret from within the running Pod.
But, when applying this manifest, and according to the documentation here, I expect to see the kubernetes secret regcred reflecting the Azure Key Vault Secret when I create the Pod with mounted secret volume, but the kubernetes secret remains unchanged. I have also tried to recreate the Pod in an attempt to trigger the sync but in vain.
Since this is a very declarative way of configuring this functionality, I am also confused where to look at logs for troubleshooting.
Can someone lead me to what may I be doing wrong?
apiVersion: v1
kind: Pod
metadata:
name: demo
labels:
aadpodidbinding: webapp
spec:
containers:
- name: demo
image: mcr.microsoft.com/oss/azure/aad-pod-identity/demo:v1.6.3
volumeMounts:
- name: web-app-secret
mountPath: "/mnt/secrets"
readOnly: true
nodeSelector:
kubernetes.io/os: linux
volumes:
- name: web-app-secret
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: web-app-secret-provide
---
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: web-app-secret-provide
spec:
provider: azure
parameters:
usePodIdentity: "true"
keyvaultName: <key-vault-name>
objects: |
array:
- |
objectName: registryPassword
objectType: secret
tenantId: <tenant-id>
secretObjects:
- data:
- key: .dockerconfigjson
objectName: registryPassword
secretName: regcred
type: kubernetes.io/dockerconfigjson
---
apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzureIdentity
metadata:
name: kv-managed-identity
spec:
type: 0
resourceID: <resource-id>
clientID: <client-id>
---
apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzureIdentityBinding
metadata:
name: kv-managed-binding
spec:
azureIdentity: kv-managed-identity
selector: web-app
Related
EDIT: Found the issue.I didnt installed the addon for secret driver. Once installed that i was able to make it work
I am facing an issue here and i have no idea what else i can try to figure out the issue.
I have an aks running with a single pod that runs a basic web app todo list. Nothing too fancy or complicated. what i am trying to do here, is to give permission to the aks cluster to access a keyvault and GET a secret to pass to the pod. the secret is just an ASPNETCORE_ENVIRONMENT: Development.
Following the documentations, i used helm to install the repo:
helm repo add csi-secrets-store-provider-azure https://azure.github.io/secrets-store-csi-driver-provider-azure/charts
helm install csi csi-secrets-store-provider-azure/csi-secrets-store-provider-azure
I created a Service Principle in azure:
SERVICE_PRINCIPLE_CLIENT_SECRET = az ad sp create-for-rbac --skip-assignment --name sp-aks-keyvault
i queried the clientId and Secret and passed them to my cluster as follow:
kubectl create secret generic secrets-store-creds --from-literal clientid="ClientID" --from-literal clientsecret="Password"
Once everything was set. I set those deployments.
Deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: webapp-deployment
namespace: default
labels:
app: webapp
spec:
replicas: 1
selector:
matchLabels:
app: webapp
template:
metadata:
labels:
app: webapp
spec:
containers:
- name: webapp
image: dockerimage-acr
ports:
- containerPort: 80
env:
- name: ASPNETCORE_ENVIRONMENT
valueFrom:
secretKeyRef:
name: aspenet-environment
key: environment
securityContext:
allowPrivilegeEscalation: false
volumeMounts:
- name: secrets-mount
mountPath: "/mnt/secrets-store"
readOnly: true
restartPolicy: Always
volumes:
- name: secrets-mount
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: "kv-name"
nodePublishSecretRef: # Only required when using service principal mode
name: secrets-store-creds
And my secretProvider.yaml
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: keyvault-secret-class
namespace: default
spec:
provider: azure
secretObjects:
- secretName: aspenet-environment
type: Opaque
data:
- objectName: aspnetcoreenvironment
key: environment
parameters:
usePodIdentity: "false"
useVMManagedIdentity: "false"
userAssignedIdentityID: ""
keyvaultName: "mykeyvault-name"
objects: |
array:
- |
objectName: aspnetcoreenvironment
objectType: secret
objectVersion: ""
tenantId: "<Tenant-Id>"
In my keyvault i gave access policy to the Service principle created and assigned Secret Permissions: GET and created a secret called
Name: aspnetcoreenvironment
value: Development
So far everything went ok, but when i run the deployment. and use the command kubectl describe pod <podname> i see the error, that prevents the container to start
Warning Failed 8s (x3 over 21s) kubelet Error: secret "aspenet-environment" not found
I tried different solutions but nothing works.
if i run the command kubectl get secretproviderclass i get back my provider i created.
As far as i understand, if no service is requiring a specific secret, i wont be able to find the secret i want to create if i run the command: kubectl get secret
And this is correct, i guess, because my pod is not starting.
Any help or enlightenment here about what i am doing wrong or how to fix it?
Thank you so much guys
EDIT:
Some extra debugging i came across the fact that the volume mount is still required. So i did add the volume to the deployment. But this is still giving an error.
The issue is, as i realized. Is when i run the command kubectl apply -f secretProviderClass.yml, no secret get created at all, reason why is failing.
So i think something is wrong here. Applying the SecretProviderClass shouldnt create automatically a secret service?
Hi Kubernetes Experts,
I was using the following ServiceAccount creation config:
apiVersion: v1
kind: ServiceAccount
metadata:
name: my-scheduler
namespace: kube-system
and the following Deployment config:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
component: scheduler
tier: control-plane
name: my-scheduler
namespace: kube-system
spec:
...
template:
metadata:
labels:
component: scheduler
tier: control-plane
version: second
k8s-custom-scheduler: my-scheduler
spec:
serviceAccountName: my-scheduler
Things were working fine, And now I want to make my pod more secure setting automountServiceAccountToken to false.
I changed my ServiceAccount creation and deployment config:
apiVersion: v1
kind: ServiceAccount
metadata:
name: my-scheduler
namespace: kube-system
automountServiceAccountToken: false
Deployment config:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
component: scheduler
tier: control-plane
name: my-scheduler
namespace: kube-system
spec:
...
template:
metadata:
labels:
component: scheduler
tier: control-plane
version: second
k8s-custom-scheduler: my-scheduler
spec:
serviceAccountName: my-scheduler
automountServiceAccountToken: false
After setting this my scheduler pod is not coming up and it says CrashLoopBackOff
Error:
I0325 17:37:50.304810 1 flags.go:33] FLAG: --write-config-to=""
I0325 17:37:50.891504 1 serving.go:319] Generated self-signed cert in-memory
W0325 17:37:51.168023 1 authentication.go:387] failed to read in-cluster kubeconfig for delegated authentication: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
W0325 17:37:51.168064 1 authentication.go:249] No authentication-kubeconfig provided in order to lookup client-ca-file in configmap/extension-apiserver-authentication in kube-system, so client certificate authentication won't work.
W0325 17:37:51.168072 1 authentication.go:252] No authentication-kubeconfig provided in order to lookup requestheader-client-ca-file in configmap/extension-apiserver-authentication in kube-system, so request-header client certificate authentication won't work.
W0325 17:37:51.168089 1 authorization.go:177] failed to read in-cluster kubeconfig for delegated authorization: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
W0325 17:37:51.168102 1 authorization.go:146] No authorization-kubeconfig provided, so SubjectAccessReview of authorization tokens won't work.
W0325 17:37:51.168111 1 options.go:298] Neither --kubeconfig nor --master was specified. Using default API client. This might not work.
invalid configuration: no configuration has been provided
I believe we need to configure something more along with automountServiceAccountToken: false.
Can someone point me to the additional configurations needed to use automountServiceAccountToken: false?
Configure Service Accounts for Pods
You can access the API from inside a pod using automatically mounted service account credentials.
In version 1.6+, you can opt out of automounting API credentials for a service account by setting automountServiceAccountToken: false on the service account or for a particular pod.
So, when you are creating a ServiceAccount and a Deployment like in your example yaml files, credentials for accessing the Kubernetes API are not automatically mounted to the Pod. But your k8s Deployment 'my-scheduler' requires them to access the API.
You can test your ServiceAccount with some dummy Deployment of nginx, for example. And it will work without mounting credentials.
Also, if you create a ServiceAccount like in your example:
apiVersion: v1
kind: ServiceAccount
metadata:
name: my-scheduler
namespace: kube-system
automountServiceAccountToken: false
You can manually mount the API credentials like this:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-scheduler
namespace: kube-system
...
spec:
...
template:
metadata:
labels:
app: my-scheduler
spec:
containers:
- image: <YOUR_IMAGE>
imagePullPolicy: Always
name: my-scheduler
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access
readOnly: true
serviceAccountName: my-scheduler
volumes:
- name: kube-api-access
projected:
defaultMode: 420
sources:
- serviceAccountToken:
expirationSeconds: 3607
path: token
- configMap:
items:
- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
See the Managing Service Accounts link for more information.
I'm trying to build a Azure DevOps Linux Build Agent in Azure Kubernetes Service.
I created the yaml file and created the secrets to use inside of the file.
I applied the file and have "CreateContainerConfigError" with my pod in a "waiting" state.
I run command
"kubectl get pod <pod name> -o yaml"
and it states the secret "vsts" could not be found.
I find this weird because I used "kubectl get secrets" and I see the secrets "vsts-account" and "vsts-token" listed.
You may check your kubernetes configuration, which is supposed to be like below:
apiVersion: v1
kind: ReplicationController
metadata:
name: vsts-agent
spec:
replicas: 1
template:
metadata:
labels:
app: vsts-agent
version: "0.1"
spec:
containers:
– name: vsts-agent
image: microsoft/vsts-agent:ubuntu-16.04-docker-18.06.1-ce-standard
env:
– name: VSTS_ACCOUNT
valueFrom:
secretKeyRef:
name: vsts
key: VSTS_ACCOUNT
– name: VSTS_TOKEN
valueFrom:
secretKeyRef:
name: vsts
key: VSTS_TOKEN
– name: VSTS_POOL
value: dockerized-vsts-agents
volumeMounts:
– mountPath: /var/run/docker.sock
name: docker-volume
volumes:
– name: docker-volume
hostPath:
path: /var/run/docker.sock
You may follow the blog below to see whether it helps you:
https://mohitgoyal.co/2019/01/10/run-azure-devops-private-agents-in-kubernetes-clusters/
I have a Statefulset with 3 instances on Azure Kubernetes 1.16, where I try to use Azure File storage to create a single file share for the 3 instances.
I use Azure Files dynamic where all is declarative i.e. storage account, secrets, pvc's and pv's are created automatically.
Manifest with VolumeClaimTemplate
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: xxx
spec:
replicas: 3
...
volumeClaimTemplates:
- metadata:
name: xxx-data-shared
spec:
accessModes: [ ReadWriteMany ]
storageClassName: azfile-zrs-sc
resources:
requests:
storage: 1Gi
The StorageClass:
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: azfile-zrs-sc
provisioner: kubernetes.io/azure-file
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
mountOptions:
- dir_mode=0777
- file_mode=0777
- uid=0
- gid=0
- mfsymlinks
- cache=strict
parameters:
resourceGroup: xxx
skuName: Standard_ZRS
shareName: data
Instead of one share, I end up with 3 pv's each referring to a separate created Azure Storage Account each with a share data.
Question: Can I use the Azure Files dynamic, with additional configuration in the manifest to get a single file share? Or will I have to do Static?
Turns out that volumeClaimTemplates is not the right place (reference).
Instead use persistentVolumeClaim.
For Azure File Storage this becomes:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: data-shared-claim
spec:
accessModes:
- ReadWriteMany
storageClassName: azfile-zrs-sc
resources:
requests:
storage: 1Gi
And refer to it in the manifest:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: xxx
spec:
replicas: 3
...
template:
spec:
containers:
...
volumeMounts:
- name: data-shared
mountPath: /data
volumes:
- name: data-shared
persistentVolumeClaim:
claimName: data-shared-claim
I have an application that starts with docker-compose up. Some ssh credentials are provided with a json file, in a volume, in the host machine. I want to run the app in kubernetes, how can I provide the credentials using kubernetes secrets? my json file looks like:
{
"HOST_USERNAME"="myname",
"HOST_PASSWORD"="mypass",
"HOST_IP"="myip"
}
I created a file named mysecret.yml with base64 and I applied in kubernetes
apiVersion: v1
kind: Secret
metadata:
name: mysecret
type: Opaque
data:
HOST_USERNAME: c2gaQ=
HOST_PASSWORD: czMxMDIsdaf0NjcoKik=
HOST_IP: MTcyLjIeexLjAuMQ==
How I have to write the volumes in deployment.yml in order to use the secret properly?
apiVersion: v1
kind: Pod
metadata:
name: mypod
spec:
containers:
- name: mypod
image: redis
volumeMounts:
- name: foo
mountPath: "/etc/foo"
readOnly: true
volumes:
- name: foo
secret:
secretName: mysecret
This is the above example of using secret as volumes. You can use the same to define a deployment.
Please refer to official kubernetes documentation for further info:
https://kubernetes.io/docs/concepts/configuration/secret/