nginx ingress setup up on Azure (not using helm) troubleshooting - azure

Question

I need some help trying to achieve this two things here:
Setting up my nginx ingress controller on AKS without using helm (I don't want to)
Make my ingress to use an already reserved IP address with resource name 'kubernetes-ip'
For the first step I'm following with no luck this documentation: https://kubernetes.github.io/ingress-nginx/deploy/#azure
And I didn't forgot the mandatory.yaml!
my step by step guide:
I have a basic kubernetes cluster with two pods, as follows:
NAME READY STATUS RESTARTS AGE
activemq-demo-7b769bcc4-jtsj5 1/1 Running 0 55m
ubuntu-dcb9c6ccb-wkz2w 1/1 Running 0 2d
At this point I want to add my ingress so I can reach the demo activemq using my public ip address a.b.c.d
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.30.0/deploy/static/mandatory.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.30.0/deploy/static/provider/cloud-generic.yaml
After doing it I run kubectl get services -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx LoadBalancer 10.0.186.143 zz.zz.zzz.zzz 80:32703/TCP,443:30584/TCP 17s
Which is fine! seems to be working right? at this point I should be able to connect to the external ip address to any of those ports but I can't :(
$ telnet zz.zz.zzz.zzz 80
Trying zz.zz.zzz.zzz...
(a long time later...)
telnet: Unable to connect to remote host: Resource temporarily unavailable
$ telnet zz.zz.zzz.zzz 443
Trying zz.zz.zzz.zzz...
(a long time later...)
telnet: Unable to connect to remote host: Resource temporarily unavailable
I know that the mandatory.yaml doesn't know nothing about my reserved IP address, but I'm ignoring it because I have a bigger problem, I can't connect.
I also ignore for a minute that I can't connect just to test if I need the actual ingress.yaml running. So I use kubectl apply -f ingress.html. (ingress.yaml contains the following code)
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress1
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
nginx.ingress.kubernetes.io/enable-cors: "true"
nginx.ingress.kubernetes.io/cors-allow-origin: "*"
nginx.ingress.kubernetes.io/cors-allow-credentials: "true"
nginx.ingress.kubernetes.io/cors-allow-methods: "*"
spec:
tls:
- hosts:
- amq-test.mydomain.com
secretName: my-certificate
rules:
- host: amq-test.mydomain.com
http:
paths:
- path: /*
backend:
serviceName: activemq-demo-service
servicePort: 8161
After that If I run kubectl get ing I get:
NAME HOSTS ADDRESS PORTS AGE
ingress1 amq-test.mydomain.com zz.zz.zzz.zzz 80, 443 66s
But its the same, I can't connect:
FROM WSL
$ telnet zz.zz.zzz.zzz 80
Trying zz.zz.zzz.zzz...
(a long time later...)
telnet: Unable to connect to remote host: Resource temporarily unavailable
$ telnet zz.zz.zzz.zzz 443
Trying zz.zz.zzz.zzz...
(a long time later...)
telnet: Unable to connect to remote host: Resource temporarily unavailable
Honestly I'm not sure what I'm missing, this should be very straight forward with the official documentation, I don't know if I have to enable something else in Azure ...
Thanks for reading, any help will be appreciated.
EDIT 3/7/2020: activemq-demo Deployment & Service #Jean-Philippe Bond
apiVersion: apps/v1
kind: Deployment
metadata:
name: activemq-demo
labels:
app: activemq-demo
tier: backend
spec:
revisionHistoryLimit: 1
replicas: 1
selector:
matchLabels:
app: activemq-demo
template:
metadata:
labels:
app: activemq-demo
tier: backend
spec:
containers:
- name: activemq-demo
image: myproject.azurecr.io/activemq-slim:5.15.9-3
imagePullPolicy: "Always"
command: ["/start.sh"]
args: ["somename"]
env:
- name: LANG
value: "C.UTF-8"
ports:
- containerPort: 8161
- containerPort: 61616
livenessProbe:
exec:
command:
- /isAlive.sh
- somename
initialDelaySeconds: 15
periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
name: activemq-demo-service
labels:
tier: controller
spec:
type: NodePort
ports:
- name: http
protocol: TCP
port: 8161
targetPort: 8161
- name: acceptor
protocol: TCP
port: 61616
targetPort: 61616
selector:
app: activemq-demo
tier: backend
Please note that the only thing I want to access from outside is the HTTP web service that ActiveMQ provides on port 8161 by default
EDIT 3/9/2020: #HelloWorld request
telnet from WSL
$ telnet zz.zz.zzz.zzz 80
Trying zz.zz.zzz.zzz...
(a long time later...)
telnet: Unable to connect to remote host: Resource temporarily unavailable
$ telnet zz.zz.zzz.zzz 443
Trying zz.zz.zzz.zzz...
(a long time later...)
telnet: Unable to connect to remote host: Resource temporarily unavailable
telnet from macos
$ telnet zz.zz.zzz.zzz 80
Trying zz.zz.zzz.zzz...
telnet: connect to address zz.zz.zzz.zzz: Operation timed out
telnet: Unable to connect to remote host
$ telnet zz.zz.zzz.zzz 443
Trying zz.zz.zzz.zzz...
telnet: connect to address zz.zz.zzz.zzz: Operation timed out
telnet: Unable to connect to remote host
curl from macos - HTTP
$ curl -v -X GET http://amq-test.mydomain.com
Note: Unnecessary use of -X or --request, GET is already inferred.
* Trying zz.zz.zzz.zzz:80...
* TCP_NODELAY set
* Connection failed
* connect to zz.zz.zzz.zzz port 80 failed: Operation timed out
* Failed to connect to amq-test.mydomain.com port 80: Operation timed out
* Closing connection 0
curl: (7) Failed to connect to amq-test.mydomain.com port 80: Operation timed out
curl from macos - HTTPS
$ curl -v -X GET https://amq-test.mydomain.com
Note: Unnecessary use of -X or --request, GET is already inferred.
* Trying zz.zz.zzz.zzz:443...
* TCP_NODELAY set
* Connection failed
* connect to zz.zz.zzz.zzz port 443 failed: Operation timed out
* Failed to connect to amq-test.mydomain.com port 443: Operation timed out
* Closing connection 0
curl: (7) Failed to connect to amq-test.mydomain.com port 443: Operation timed out
EDIT 3/10/2020: Starting from scratch (multiple times)
I deleted the whole thing to start "fresh" with no much more luck, but I have noticed some things that hopefully may trigger some thoughts out there...
The basics:
I already have a Resource Group that I'm using: MyResourceGroup
I already have a Virtual Network with a Subnet MyVirtualNet
I already have reserved a Public IP address I want to use with my ingress, It's an Static IP that I want to prevent from changing (or be deleted) until the end of time: A.B.C.D
I already have my own domain that I have routed to A.B.C.D: amq-test.mydomain.com
My procedure:
I'm creating a new Kubernetes Service using the Azure Web Interface, I'm making sure to select my Resource Group as well as my Virtual Network and Subnet
As soon I created the base Kubernetes Service I notice that it creates a LoadBalancer with a different Public IP address that I can't control, I assume that its because that IP address will be used as the main entry point for things like kubectl and remote management.
With the cluster live I create the basic ActiveMQ image that I shared previously
Now, I start with the ingress-nginx and deploy the mandatory.yaml
And then I add the Azure Service yaml here, but this time with two modifications to make it use my Public IP address. this modification was extracted from Microsoft documentation
kind: Service
apiVersion: v1
metadata:
name: ingress-nginx
namespace: ingress-nginx
annotations:
service.beta.kubernetes.io/azure-load-balancer-resource-group: MyResourceGroup
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
loadBalancerIP: A.B.C.D
externalTrafficPolicy: Local
type: LoadBalancer
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
ports:
- name: http
port: 80
protocol: TCP
targetPort: http
- name: https
port: 443
protocol: TCP
targetPort: https
With that running without errors, everything is as expect:
$ kubectl get svc -n ingress-nginx -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
ingress-nginx LoadBalancer 10.0.30.100 A.B.C.D 80:30682/TCP,443:31002/TCP 35m app.kubernetes.io/name=ingress-nginx,app.kubernetes.io/part-of=ingress-nginx
I deploy the final part right? my ingress.yaml:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress1
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/enable-cors: "true"
nginx.ingress.kubernetes.io/cors-allow-origin: "*"
nginx.ingress.kubernetes.io/cors-allow-credentials: "true"
nginx.ingress.kubernetes.io/cors-allow-methods: "*"
spec:
rules:
- http:
paths:
- path: /*
backend:
serviceName: activemq-demo-service
servicePort: 8161
This runs without issues and when is up I checked with:
$ kubectl describe ingress
Name: ingress1
Namespace: default
Address: A.B.C.D
Default backend: default-http-backend:80 (<none>)
Rules:
Host Path Backends
---- ---- --------
*
/* queue-callbacks-service:8161 (10.94.20.14:8161)
Annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/cors-allow-credentials: true
nginx.ingress.kubernetes.io/cors-allow-methods: *
nginx.ingress.kubernetes.io/cors-allow-origin: *
nginx.ingress.kubernetes.io/enable-cors: true
kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx","nginx.ingress.kubernetes.io/cors-allow-credentials":"true","nginx.ingress.kubernetes.io/cors-allow-methods":"*","nginx.ingress.kubernetes.io/cors-allow-origin":"*","nginx.ingress.kubernetes.io/enable-cors":"true"},"name":"ingress1","namespace":"default"},"spec":{"rules":[{"http":{"paths":[{"backend":{"serviceName":"queue-callbacks-service","servicePort":8161},"path":"/callbacks/*"}]}}]}}
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 35m nginx-ingress-controller Ingress default/ingress1
Normal UPDATE 34m nginx-ingress-controller Ingress default/ingress1
Everything seems to be perfect but I just can't connect to the IP or host using any (80, 443) port
Hope this helps
Just FYI I just tried with helm following this documentation and I got the same result
EDIT 4/15/2020: not done yet
I was working on another project so this one was paused for a moment, I´m returning to this now, unfortunately still not working, I opened a ticket with Microsoft and I'm waiting for it.
However, we noticed that port 80 are being filtered by some firewall or something, we are not sure what is causing this as we have inbound rules on our SG with port 80 and 443 open from * on any protocol
$ nmap -Pn zz.zz.zz.zzz -p 80,443
Starting Nmap 7.80 ( https://nmap.org ) at 2020-04-15 11:01 Pacific SA Standard Time
Nmap scan report for zz.zz.zz.zzz
Host is up.
PORT STATE SERVICE
80/tcp filtered http
443/tcp filtered https
Nmap done: 1 IP address (1 host up) scanned in 4.62 seconds

Answer 1

ActiveMQ is accessible via TCP, not HTTP and Kubernetes Ingress were not built to support TCP services. Whit that said, Nginx does support TCP load balancing if you really want to use it, but you'll not be able to use an Ingress rule based on the host like you did since this is reserved for HTTP/HTTPS. Your best bet would probably be to use the Azure L4 Load balancer directly instead of going through the ingress controller.
If you want to use Nginx, you'll need to modify the yaml in mandatory.yaml to expose the ActiveMQ port on the Nginx deployment.
kind: ConfigMap
apiVersion: v1
metadata:
name: tcp-services
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
data:
9000: "default/activemq-demo-service:8161"
You'll also need to add the tcp-services port on the Service resource. Ex :
apiVersion: v1
kind: Service
metadata:
name: ingress-nginx
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
type: LoadBalancer
ports:
- name: http
port: 80
targetPort: 80
protocol: TCP
- name: https
port: 443
targetPort: 443
protocol: TCP
- name: proxied-tcp-9000
port: 9000
targetPort: 9000
protocol: TCP
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
Here is the documentation for TCP/UDP support in the Nginx Ingress.

Answer 2

I wrote a story that might be useful to you some time ago: https://medium.com/cooking-with-azure/tcp-load-balancing-with-ingress-in-aks-702ac93f2246