Kubernetes cert-manager with letsencrypt waiting on certificate issuance - azure

Question

I am trying to set up an Azure Kubernetes cluster with an HTTPS ingress controller for separate dev, staging, and prod environments. I have followed the Microsoft Azure guide on how to Create an HTTPS ingress controller on Azure Kubernetes Service (AKS) which allows me to set up an HTTPS ingress controller for a single namespace, but my end goal is to have separate namespaces for the dev, staging, and prod environments. According to the answers to this question, the way to do this is to have the ingress controller on one namespace (ingress in my case), and then separate ingress rules for each namespace (dev in my case).
Hence I setup the nginx ingress controller and the cert-manager on the ingress namespace:
# Add the ingress-nginx repository
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
# Use Helm to deploy an NGINX ingress controller
helm install nginx-ingress ingress-nginx/ingress-nginx \
--namespace ingress \
--set controller.replicaCount=2 \
--set controller.nodeSelector."beta\.kubernetes\.io/os"=linux \
--set defaultBackend.nodeSelector."beta\.kubernetes\.io/os"=linux \
--set controller.admissionWebhooks.patch.nodeSelector."beta\.kubernetes\.io/os"=linux
# Label the ingress-basic namespace to disable resource validation
kubectl label namespace ingress cert-manager.io/disable-validation=true
# Add the Jetstack Helm repository
helm repo add jetstack https://charts.jetstack.io
# Update your local Helm chart repository cache
helm repo update
# Install the cert-manager Helm chart
helm install cert-manager jetstack/cert-manager \
--namespace ingress \
--version v0.16.1 \
--set installCRDs=true \
--set nodeSelector."kubernetes\.io/os"=linux \
--set webhook.nodeSelector."kubernetes\.io/os"=linux \
--set cainjector.nodeSelector."kubernetes\.io/os"=linux
I then create a cluster-issuer.yml file with the following:
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: letsencrypt
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: email#address.com
privateKeySecretRef:
name: letsencrypt
solvers:
- http01:
ingress:
class: nginx
podTemplate:
spec:
nodeSelector:
"kubernetes.io/os": linux
which I apply with
$ kubectl apply -f cluster-issuer.yml
Next I create an ingress rule on the dev namespace with the following ingress.yml file:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-dev
namespace: dev
annotations:
kubernetes.io/ingress.class: nginx
ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/rewrite-target: /$2
cert-manager.io/cluster-issuer: letsencrypt
spec:
tls:
- hosts:
- domain.azure.com
secretName: tls-secret-dev
rules:
- host: domain.azure.com
http:
paths:
- backend:
serviceName: my-service
servicePort: 80
path: /dev/my-service(/|$)(.*)
and apply it:
$ kubectl apply -f ingress.yml
Now I check to see whether a secret has been created:
$ kubectl get certificate -n dev
NAME READY SECRET AGE
tls-secret-dev False tls-secret-dev 61s
So it seems that something went wrong when creating the secret. If I look at the certificate, it seems that a certificate is requested, but it never gets further than that:
$ kubectl describe certificate tls-secret -n dev
Name: tls-secret-dev
Namespace: dev
Labels: <none>
Annotations: <none>
API Version: cert-manager.io/v1beta1
Kind: Certificate
...
Status:
Conditions:
Last Transition Time: 2021-02-16T13:47:33Z
Message: Issuing certificate as Secret does not exist
Reason: DoesNotExist
Status: False
Type: Ready
Last Transition Time: 2021-02-16T13:47:33Z
Message: Issuing certificate as Secret does not exist
Reason: DoesNotExist
Status: True
Type: Issuing
Next Private Key Secret Name: tls-secret-dev-6ngw8
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Issuing 70s cert-manager Issuing certificate as Secret does not exist
Normal Generated 70s cert-manager Stored new private key in temporary Secret resource "tls-secret-dev-6ngw8"
Normal Requested 70s cert-manager Created new CertificateRequest resource "tls-secret-dev-vtlbd"
Looking at the certificate request, an order is created:
$ kubectl describe certificaterequest tls-secret-dev-vtlbd -n dev
Name: tls-secret-dev-vtlbd
Namespace: dev
Labels: <none>
Annotations: cert-manager.io/certificate-name: tls-secret-dev
cert-manager.io/certificate-revision: 1
cert-manager.io/private-key-secret-name: tls-secret-dev-6ngw8
API Version: cert-manager.io/v1beta1
Kind: CertificateRequest
...
Status:
Conditions:
Last Transition Time: 2021-02-16T13:47:33Z
Message: Waiting on certificate issuance from order dev/tls-secret-dev-vtlbd-526778456: ""
Reason: Pending
Status: False
Type: Ready
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal OrderCreated 3m3s cert-manager Created Order resource dev/tls-secret-dev-vtlbd-526778456
Inspecting the order is where the trail seems to run cold:
$ kubectl describe order tls-secret-dev-vtlbd-526778456 -n dev
Name: tls-secret-dev-vtlbd-526778456
Namespace: dev
Labels: <none>
Annotations: cert-manager.io/certificate-name: tls-secret-dev
cert-manager.io/certificate-revision: 1
cert-manager.io/private-key-secret-name: tls-secret-dev-6ngw8
API Version: acme.cert-manager.io/v1beta1
Kind: Order
...
Status:
Events: <none>
Question: How do I get the certificate manager to stop waiting on certificate issuance so I can finish setting up my HTTPS ingress controller?