AKS Ingress-Nginx ingress controller failing to route by host - azure

I am configuring an ingress-nginx load balancer on Azure Kubernetes service. I have installed the load balancer using Helm, and set up ingress. Here is the behavior I'm encountering:
When I include a host in my pathing rules in my ingress config, I cannot access service at that host URL. The request times out
When I don't include a host in my pathing rules, I can access the service at that host URL with no issues
Regardless of whether or not the host is included in the pathing rules, I can successfully access the service at the host URL when I CURL it from any pod in the cluster.
Nslookup successfully resolves the host on my machine
I'm trying to figure out why I'm unable to reach my service when host is included in my ingress configuration. Any ideas? Technical details are below.
Note that the configuration is only pointing to one service currently, but filtering by host will eventually be necessary - I'm planning to have multiple services with different domains running through this load balancer.
Ingress controller configuration:
helm install --replace ingress-nginx ingress-nginx/ingress-nginx \
--create-namespace \
--namespace $NAMESPACE \
--set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path"=127.0.0.1 \
--set controller.service.annotations."service\.beta\.kubernetes\.io/azure-dns-label-name"=$DNS_LABEL \
--set controller.service.loadBalancerIP=$IP \
The ingress configuration:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: hello-world-ingress
annotations:
cert-manager.io/cluster-issuer: letsencrypt
spec:
ingressClassName: nginx
tls:
- hosts:
- my.host.com
secretName: tls-secret
rules:
- host: my.host.com //Removing this item makes the service reachable
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: xrcfrontend
port:
number: 80
This is the curl command I'm running. It returns the correct results when run inside the pod, and times out when run outside.
curl https://my.host.com --insecure

If you are using AKS v>=1.24, then try adding below annotation with path /healthz instead of 127.0.0.1 during nginx ingress controller installation or in nginx ingress controller service and use host based routing with nginx ingress routes -
service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path"=/healthz
If the above helps then
Why was it not working with host earlier?
because backend pool of LB goes unhealthy because of wrong health-probe path of ingress controller. Ingress route is only accepting traffic for the particular host name and hence health probe of ingress controller service(Azure LB) is failing because / or 127.0.0.1 for http protocol returns 404.
Github discussion on changes - https://github.com/Azure/AKS/issues/2903#issuecomment-1115720970
More details on installation - https://learn.microsoft.com/en-us/azure/aks/ingress-basic?tabs=azure-cli#basic-configuration

Related

Create ssl certificate in aks and where to add it in manifest file of loadbalancer having kind service in AKS?

I have kubernetes service of type loadbalancer on azure cluster.that service must accept https req.How to create ssl cert and add them in annotations?
I read different articals but i am very confused between ingress , cert-manager and other things
To answer your question, service does not accept any annotations. So to use ssl/tls for your applications, ingress is the best choice. Here is the official microsoft guide that will help through each step to setup ingress controller with LetsEncrypt cert-manager.
If you want to use your BYOC (bring your own certificates) from any known CA. Here are the steps.
Please follow the guide provided in above URL upto Use a dynamic IP address.
kubectl --namespace ingress-basic get services -o wide -w nginx-ingress-ingress-nginx-controller
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
nginx-ingress-ingress-nginx-controller LoadBalancer 10.0.74.133 EXTERNAL_IP 80:32486/TCP,443:30953/TCP 44s app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx-ingress,app.kubernetes.io/name=ingress-nginx
Create create secrets using .crt and .key file provided by CA. Assuming you are using default namespace for your application workload.
kubectl create secret tls TargetPods-tls --cert nameOfCertfile.crt --key privateKey.key --namespace default
Consume these secrets inside your ingress object and add annotations for http to https redirect
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
nginx.ingress.kubernetes.io/ssl-redirect: 'true' # Annotation to redirect http to https.
name: TargetPods-6dc98445c4-jr6pt
spec:
tls:
- hosts:
- test.domain.io
secretName: TargetPods-tls
rules:
- host: test.domain.io
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: TargetPod-6dc98445c4-jr6pt
port:
number: 80

AKS Ingress controller DNS gives 404 error

I have created aks cluster with 2 services exposed using Ingress controller
below is the yml file for ingress controller with TLS
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: xyz-office-ingress02
annotations:
nginx.ingress.kubernetes.io/use-regex: "true"
nginx.ingress.kubernetes.io/rewrite-target: /
cert-manager.io/cluster-issuer: letsencrypt
kubernetes.io/ingress.class: "nginx"
spec:
tls:
- hosts:
- office01.xyz.com
secretName: tls-office-secret
rules:
- host: office01.xyz.com
- http:
paths:
- path: /(/|$)(.*)
pathType: Prefix
backend:
service:
name: office-webapp
port:
number: 80
- path: /api/
pathType: Prefix
backend:
service:
name: xyz-office-api
port:
number: 80
kubenctl describe ing
Name: xyz-office-ingress02
Labels: <none>
Namespace: default
Address: <EXTERNAL Public IP>
Ingress Class: <none>
Default backend: <default>
TLS:
tls-office-secret terminates office01.xyz.com
Rules:
Host Path Backends
---- ---- --------
*
/(/|$)(.*) office-webapp:80 (10.244.1.18:80,10.244.2.16:80)
/api/ xyz-office-api:80 (10.244.0.14:8000,10.244.1.19:8000)
Annotations: cert-manager.io/cluster-issuer: letsencrypt
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/use-regex: true
Events: <none>
On IP i am able to access both services, however when using the DNS it is not working and gives 404 error
Cleaning up remarks from comments: basically, the issue is with the ingress rules definition. We have the following:
rules:
- host: office01.xyz.com
- http:
paths:
...
We know connecting to ingress directly does work, without using DNS. While when querying it through DNS: we get a 404.
The reason for this 404 is that, when entering with a DNS name, you enter the first rules. In which you did not define any backend.
One way to fix this would be to relocate the "host" part of that ingress with your http rules, eg:
spec:
tls:
...
rules:
- host: office01.xyz.com
http: #no "-", not a new entry => http & host belong to a single rule
paths:
- path: /(/|$)(.*)
...
- path: /api/
...
I tried to reproduce the same issue in my environment and got the below results
I have created the dns zone for the cluster
Created the namespace
kubectl create namespace ingress-basic
I have installed the helm repo and used the helm to deploy the ingress controller
helm install ingress-nginx ingress-nginx/ingress-nginx \
--namespace <namespace-name> \
--set controller.replicaCount=2
When I check in the logs I am able to see public IP with load balancer
I have created the some role assignments to connect the DNS zones
Assigned the managed identity of cluster node pool DNS contributor rights to the domain zone
az role assignment create --assignee $UserClientId --role 'DNS Zone Contributor' --scope $DNSID
I have run some helm commands to deploy the dns zones
helm install external-dns bitnami/external-dns --namespace ingress-basic --set provider=azure --set txtOwnerId=<cluster-name> --set policy=sync --set azure.resourceGroup=<rg-name> --set azure.tenantId=<tenant-id> --set azure.subscriptionId=<sub-id> --set azure.useManagedIdentityExtension=true --set azure.userAssignedIdentityID=<UserClient-Id>
I have installed the cert manager using helm
helm install cert-manager jetstack/cert-manager \
--namespace ingress-basic \
--version vXXXX
I have created and run the application
vi nginxfile.yaml
kubectl apply -f file.yaml
I have created the ingress route it will configure the traffic to the application
After that we have to verify the certificates it will create or not and wait for few minutes it will update the DNS zone
I have created the cert manager and deployed that cluster
kubectl apply -f file.yaml --namespace ingress-basic
Please find this url for Reference for more details

How we can access kubernetes ingress controller IP using https?

I have deployed application in Azure Kubernetes (AKS). I have used ingress-controller for my POC. Previously I was using domain (saurabh.com). I am able to access saurabh.com through https.
Now what I want is that I want to access my application using IP address with https.
My ingress controller yaml files looks like:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: saurabh-ingress
annotations:
kubernetes.io/ingress.class: nginx
cert-manager.io/cluster-issuer: letsencrypt
nginx.ingress.kubernetes.io/use-regex: "true"
spec:
tls:
- secretName: tls-secret
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: saurabh-ui
port:
number: 4200
By doing this, I am able to access my application using IP but its coming http not https. Can someone please help me with this. I want to access my application IP through https.
Note: I have installed the certificates. When I am trying to access domain using saurabh.com, its coming with https.
Thanks in advance.
I tried reproduce the issue in my environment and got the below results
Please use this link to access the files
I have created the namespace
kubectl create namespace namespace_name
Created the applications and deployed into the kubernetes
kubectl apply -f filename.yaml
To check the namespaces which are created and we can get the IP address using below command
kubectl get svc -n namespace_name
I have installed the helm chat for controller and deployed the ingress resource into the kubernetes
NOTE: After installing the nginx controller we have to change the Cluster IP to LoadBalancer.
Here I have enabled HTTPS in AKS using cert manager, it will automatically generate and configured
I have created the namespace for cert manager
kubectl create namespace namespace_name
kubectl get svc namespace namespace_name
I have installed the cert manager using helm using below command
helm install cert-manager jetstack/cert-manager \
--namespace cert-manager \
--version v0.14.0 \
--set installCRDs=true
To check the cert manager namespace
kubectl get pods --namespace cert-manager
I have created the cluster issuer and deployed
vi filename.yaml
kubectl apply --namespace app -f filename.yaml
I have created and installed the TLS or SSL certificates
kubectl apply --namespace app -f filename.yaml
We can verify that the certificate is created or not using below command
Here it will show the certificate is created or not
kubectl describe cert app-web-cert --namespace namespace_name
Check the service using below command
kubectl get services -n app
Test the app with HTTPS: https://hostname with IP address
Here we can also check the certificates which we have added.

K8 Ingress does not have an endpoint

I want to make services accessible from outside the K8 cluster using an ingress controller. Following 5.5 from the Kubernetes Cookbook, I ran this manifest:
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: nginx-public
annotations:
ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- host:
http:
paths:
- path: /web
backend:
serviceName: nginx
servicePort: 80
The Ingress object is visible in the Kubernetes dashboard; but it does not have an assigned endpoint:
Output of kubectl get ing:
NAME HOSTS ADDRESS PORTS AGE
nginx-public * 80 54m
update
Running kubectl describe ingress nginx-public gives:
Name: nginx-public
Namespace: default
Address:
Default backend: default-http-backend:80 (<none>)
Rules:
Host Path Backends
---- ---- --------
*
/web nginx:80 (<none>)
Annotations:
ingress.kubernetes.io/rewrite-target: /
Events: <none>
Actually this is an issue with Kubernetes Dashboard, we have the same issue.
Even if it isn't displayed it doesn't mean your ingress isn't working. First you should check the ingress with kubectl (kubectl describe ingress nginx-public) and verify that the output is smiliar to this:
Name: test-ingress
Namespace: test
Address:
Default backend: default-http-backend:80 (<none>)
TLS:
test-ssl-secret terminates test.myorg.com
Rules:
Host Path Backends
---- ---- --------
test.myorg.com
/ test-service:80 (<none>)
Afterwards you should verify your service is reachable via your specified host.
Update:
Depending on the service in front of your ingress-controller your service should be reachable via http://{serverip}:{nodeport-http-port}/web in case your service is of type NodePort(you will get 2 external ports in the 30000-39999 range, one is the http port the other the https port) or http://{address-from-external-loadbalancer}/web if the service is of type LoadBalancer.
2nd-Update
After some further investigation about the issue i stumbled upon a bug issue of kubernetes-dashboard stating that it's indeed possible to show the endpoints of ingress. The problem actually isn't caused by the dashboard, but a missing parameter on the ingress deployment.
For nginx-ingress-controller its the following:
NGINX Ingress CLI arguments
The missing option is --publish-service
If you used helm to deploy the controller you need to add the parameter --set controller.publishService.enabled=true

aks ingress address is empty

I created a service call portal, then I create ingress:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: portal-ingress
spec:
backend:
serviceName: portal
servicePort: 8080
but the address is empty:
NAME HOSTS ADDRESS PORTS AGE
portal-ingress * 80 33m
The address will remain empty in AKS ingress and that is not a problem. You can still use the external IP address of the ingress controller service as the IP.
kubectl get svc -n <namespaceinwhichnginxcontrollerisdeployed>
For example:kubectl get svc -n nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-nginx-ingress-controller LoadBalancer 10.23.145.21 13.15.230.190 80:31108/TCP,443:31753/TCP
You can access the ingress as http(s)://13.15.230.190/
I think there are probably ways to make the address be populated, but I did not have a need to make it populated. I hope that is not what you want but to use the exposed service.
Ok, 3 years after and k8s apis changed, but, for the records, in my (today's) case this was due to not having installed the ingress controller as described the doc:
https://learn.microsoft.com/en-us/azure/aks/ingress-basic?tabs=azure-cli
NAMESPACE=ingress-basic
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm install ingress-nginx ingress-nginx/ingress-nginx \
--create-namespace \
--namespace $NAMESPACE \
--set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path"=/healthz

Resources