Issue with Cert-manager ClusterIssuer in AKS - azure

Question

I am getting this error in clusterissuer (cert-manager version 1.7.1):
"Error getting keypair for CA issuer: error decoding certificate PEM block"
I have the ca.crt, tls.crt and tls.key stored in a Key Vault in Azure.
kubectl describe clusterissuer ca-issuer
Ca:
Secret Name: cert-manager-secret
Status:
Conditions:
Last Transition Time: 2022-02-25T11:40:49Z
Message: Error getting keypair for CA issuer: error decoding certificate PEM block
Observed Generation: 1
Reason: ErrGetKeyPair
Status: False
Type: Ready
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning ErrGetKeyPair 3m1s (x17 over 58m) cert-manager Error getting keypair for CA issuer: error decoding certificate PEM block
Warning ErrInitIssuer 3m1s (x17 over 58m) cert-manager Error initializing issuer: error decoding certificate PEM block
kubectl get clusterissuer
NAME READY AGE
ca-issuer False 69m
This is the clusterissuer yaml file:
ca-issuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: ca-issuer
namespace: cert-manager
spec:
ca:
secretName: cert-manager-secret
This is the KeyVault yaml file to retrieve the ca.crt, tls.crt and tls.key
keyvauls.yaml
apiVersion: spv.no/v2beta1
kind: AzureKeyVaultSecret
metadata:
name: secret-akscacrt
namespace: cert-manager
spec:
vault:
name: kv-xx # name of key vault
object:
name: akscacrt # name of the akv object
type: secret # akv object type
output:
secret:
name: cert-manager-secret # kubernetes secret name
dataKey: ca.crt # key to store object value in kubernetes secret
---
apiVersion: spv.no/v2beta1
kind: AzureKeyVaultSecret
metadata:
name: secret-akstlscrt
namespace: cert-manager
spec:
vault:
name: kv-xx # name of key vault
object:
name: akstlscrt # name of the akv object
type: secret # akv object type
output:
secret:
name: cert-manager-secret # kubernetes secret name
dataKey: tls.crt # key to store object value in kubernetes secret
---
apiVersion: spv.no/v2beta1
kind: AzureKeyVaultSecret
metadata:
name: secret-akstlskey
namespace: cert-manager
spec:
vault:
name: kv-xx # name of key vault
object:
name: akstlskey # name of the akv object
type: secret # akv object type
output:
secret:
name: cert-manager-secret # kubernetes secret name
dataKey: tls.key # key to store object value in kubernetes secret
---
and these are the certificates used:
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: argocd-xx
namespace: argocd
spec:
secretName: argocd-xx
issuerRef:
name: ca-issuer
kind: ClusterIssuer
commonName: "argocd.xx"
dnsNames:
- "argocd.xx"
privateKey:
size: 4096
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: sonarqube-xx
namespace: sonarqube
spec:
secretName: "sonarqube-xx"
issuerRef:
name: ca-issuer
kind: ClusterIssuer
commonName: "sonarqube.xx"
dnsNames:
- "sonarqube.xx"
privateKey:
size: 4096
I can see that I can retrive the secrets for the certificate from key vault:
kubectl get secret -n cert-manager cert-manager-secret -o yaml
apiVersion: v1
data:
ca.crt: XXX
tls.crt: XXX
tls.key: XXX
Also, another strange thing is that I am getting other secrets in sonarqube/argocd namespace which I deployed previously but are not any more in my deployment file. I cannot delete them, when I try to delete them, they are re-created automatically. Looks like they are stored in some kind of cache. Also I tried to delete the namespace akv2k8s/cert-manager and delete the cert-manager/akv2k8s controllers and re-install them again but same issue after re-installing and applying the deployment...
kubectl get secret -n sonarqube
NAME TYPE DATA AGE
cert-manager-secret Opaque 3 155m
default-token-c8b86 kubernetes.io/service-account-token 3 2d1h
sonarqube-xx-7v7dh Opaque 1 107m
sql-db-secret Opaque 2 170m
kubectl get secret -n argocd
NAME TYPE DATA AGE
argocd-xx-7b5kb Opaque 1 107m
cert-manager-secret-argo Opaque 3 157m
default-token-pjb4z kubernetes.io/service-account-token 3 3d15h
kubectl describe certificate sonarqube-xxx -n sonarqube
Status:
Conditions:
Last Transition Time: 2022-02-25T11:04:08Z
Message: Issuing certificate as Secret does not exist
Observed Generation: 1
Reason: DoesNotExist
Status: False
Type: Ready
Last Transition Time: 2022-02-25T11:04:08Z
Message: Issuing certificate as Secret does not exist
Observed Generation: 1
Reason: DoesNotExist
Status: True
Type: Issuing
Next Private Key Secret Name: sonarqube-xxx-7v7dh
Events: <none>
Any idea?
Thanks.

Answer 1

I figured it out just uploading the certificate info ca.crt. tls.crt and tls.key in plain text, without BASE64 encoding in the Key Vault secrets in Azure.
When AKV2K8S retrives the secrets from the Key Vault and stored in Kubernetes, automatically it is encoded in BASE64.
Regards,