dashboard not working with https - K8s Version- v1.19.6 - linux

I have deployed Kubernetes Dashboard with a command:
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.2.0/aio/deploy/recommended.yaml
and I've edited the Service as a Nodeport and configured Ingress object accordingly. I could able to login to dashboard with http but getting issue while login the same URL with https:
"TLS handshake error from 10.244.0.0:44950: remote error: tls: unknown certificate" .
When i configured ingress rule with ssl it is giving error:
"Client sent an HTTP request to an HTTPS server."
I have jenkins application running on same cluster with real certificate and i could able to login the jenkins url with https .
Cluster Information:
k8s cluster running on (Linux Server release 7.9)
kubernetes version (v1.19.6)
Request you to confirm if any suggestion to fix this issue
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: kube-system-ingress
namespace: kubernetes-dashboard
annotations:
kubernetes.io/ingress.class: "haproxy"
ingress.kubernetes.io/ssl-passthrough: "false"
spec:
tls:
- hosts:
- console.qa.test.com
secretName: qa-pss-dashboard
rules:
- host: console.qa.test.com
http:
paths:
- path: /
backend:
serviceName: kubernetes-dashboard
servicePort: 8443

I think you have to add the annotation
ingress.kubernetes.io/backend-protocol: "HTTPS"
Please not that the kubernetes dashboard service is exposed in 443 port, not 8443 that is related to the deployment (pod port).
so:
backend:
service:
name: kubernetes-dashboard
port:
number: 443

Related

AKS can't modify AGIC on ingress creation due to the policy

I've just finished setting up AKS with AGIC and using Azure CNI. I'm trying to deploy NGINX to test if I set the AKS up correctly with the following configuration:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx-ingress
annotations:
kubernetes.io/ingress.class: azure/application-gateway
kubernetes.io/ingress.allow-http: "false"
appgw.ingress.kubernetes.io/use-private-ip: "false"
appgw.ingress.kubernetes.io/override-frontend-port: "443"
spec:
tls:
- hosts:
- my.domain.com
secretName: aks-ingress-tls
rules:
- host: my.domain.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: nginx-service
port:
number: 80
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
replicas: 1
selector:
matchLabels:
component: nginx
template:
metadata:
labels:
component: nginx
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
apiVersion: v1
kind: Service
metadata:
name: nginx-service
spec:
type: ClusterIP
selector:
component: nginx
ports:
- port: 80
protocol: TCP
There's no error or any other log message on apply the above configurations.
> k apply -f nginx-test.yml
deployment.apps/nginx-deployment created
service/nginx-service created
ingress.networking.k8s.io/nginx-ingress created
But after a further investigation in the Application Gateway I found these entries in the Activity log popped up at the same time I applied the said configuration.
Further details in one of the entries is as follows:
Operation name: Create or Update Application Gateway
Error code: RequestDisallowedByPolicy
Message: Resource 'my-application-gateway' was disallowed by policy.
[
{
"policyAssignment": {
"name": "Encryption In Transit",
"id": "/providers/Microsoft.Management/managementGroups/***/providers/Microsoft.Authorization/policyAssignments/EncryptionInTransit"
},
"policyDefinition": {
"name": "HTTPS protocol only on Application Gateway listeners",
"id": "/providers/microsoft.management/managementgroups/***/providers/Microsoft.Authorization/policyDefinitions/HttpsOnly_App_Gateways"
},
"policySetDefinition": {
"name": "Encryption In Transit",
"id": "/providers/Microsoft.Management/managementgroups/***/providers/Microsoft.Authorization/policySetDefinitions/EncryptionInTransit"
}
}
]
My organization have a policy to enforce TLS but from my configuration I'm not sure what I did wrong as I have already configured the ingress to only use HTTPS and also have certificate (from the secret) installed.
I'm not sure where to look and wish someone could guide me in the correct direction. Thanks!
• As you said, your organization has a policy for enforcing TLS for securing encrypted communication over HTTPS. Therefore, when you create an ‘NGINX’ deployment through the ‘yaml’ file posted, you can see that the nginx application is trying to connect to the application gateway ingress controller over Port 80 which is reserved for HTTP communications. Thus, your nginx application has also disallowed the usage of private IPs with the AGIC due to which the nginx application is directly overriding the HTTPS 443 port for reaching out to the domain ‘my.domain.com’ over port 80 without using the SSL/TLS certificate-based port for communication.
Thus, would suggest you to please configure NGINX application for port 443 as the frontend port for the cluster IP and ensure ‘SSL redirection’ is set to enabled due to which when the NGINX application is deployed, it will be not face the policy restrictions and get failed. Also, refer to the below snapshot of the listeners in application gateway and load balancer when provisioning an AGIC for an AKS cluster.
Also, for more detailed information on deploying the NGINX application in AKS cluster on ports, kindly refer to the below documentation link: -
https://learn.microsoft.com/en-us/azure/aks/ingress-basic?tabs=azure-cli

Is it safe to have an internal ClusterIp backend service using HTTP behind an Ngynx Ingress controller accessible via HTTPS?

I have a Service configured to be accessible via HTTP.
kind: Service
apiVersion: v1
metadata:
name: myservice
spec:
ports:
- name: http
protocol: TCP
port: 80
targetPort: 8080
type: ClusterIP
And an Ngynx Ingress configured to make that internal service accessible from a specific secure subdomain.domain
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: myservice-ingress
selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/myservice-ingress
annotations:
certmanager.k8s.io/issuer: letsencrypt-prod
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/backend-protocol: HTTP
spec:
tls:
- hosts:
- myservice.mydomain.com
secretName: myservice-ingress-secret-tls
rules:
- host: myservice.mydomain.com
http:
paths:
- path: /
backend:
serviceName: myservice
servicePort: 80
status:
loadBalancer:
ingress:
- {}
So when I reach https://myservice.mydomain.com I can access to my service through HTTPS.
Is it safe enough or should I configure my service and pods to communicate only through HTTPS?
It's expected behaviour since you've set TLS in your Ingress.
Note that by default the controller redirects (308) to HTTPS if TLS is enabled for that ingress. If you want to disable this behavior globally, you can use ssl-redirect: "false" in the NGINX ConfigMap.
To configure this feature for specific ingress resources, you can use the nginx.ingress.kubernetes.io/ssl-redirect: "false" annotation in the particular resource.
About your question: "Is it safe enough.." - it's opinion based question, so I can answer to use better HTTPS, rather than HTTP, but it's just my opinion. You can always find the difference between HTTP and HTTPS

Exposing service from Kubernetes NGINX Ingress controller always return 502 Bad Gateway

I am having issues with request to my NodeJS app running in my kubernetes cluster in digital ocean. Every request returns a 502 Bad Gateway Error. I am not sure what I am missing.
This is what the service config looks like
apiVersion: v1
kind: Service
metadata:
name: service-api
namespace: default
labels:
app: service-api
spec:
type: ClusterIP
ports:
- name: http
protocol: TCP
port: 80
targetPort: 3000
selector:
app: service-api
The Ingress.yml looks like this
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: service-api-ingress
annotations:
kubernetes.io/ingress.class: nginx
cert-manager.io/cluster-issuer: letsencrypt-prod
nginx.ingress.kubernetes.io/limit-connections: '2'
nginx.ingress.kubernetes.io/limit-rpm: '60'
service.beta.kubernetes.io/do-loadbalancer-protocol: "http"
service.beta.kubernetes.io/do-loadbalancer-algorithm: "round_robin"
service.beta.kubernetes.io/do-loadbalancer-http2-ports: "443,80"
service.beta.kubernetes.io/do-loadbalancer-tls-ports: "443"
service.beta.kubernetes.io/do-loadbalancer-tls-passthrough: "true"
spec:
tls:
- hosts:
- dev-api.service.com
secretName: service-api-tls
rules:
- host: "dev-api.service.com"
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: service-api
port:
number: 80
Whenever I visit the host url I get a 502 error.
This is what appears in the nginx ingress log
2021/01/13 08:41:34 [error] 319#319: *31338 connect() failed (111: Connection refused) while connecting to upstream, client: IP, server: dev-api.service.com, request: "GET /favicon.ico HTTP/2.0", upstream: "http://10.244.0.112:3000/favicon.ico", host: "dev-api.service.com", referrer: "https://dev-api.service.com/status"
As we ( with #Emmanuel Amodu ) have discussed in comment:
mistake was to connect to app using wrong port, port 4000 instead of 3000 as defined in service-api.
For community which will have similar problem please - most important steps for debugging:
Checking netstat -plant output table
Checking your Nginx Configuration: $ kubectl exec -it -n <namespace-of-ingress-controller> <nginx-ingress-controller-pod> -- cat /etc/nginx/nginx.conf
Checking service: $ kubectl describe svc service-api
Could it be the annotation that configures SSL passthru?
If SSL passthru has been configured on your ingress controller then your service needs to expose port 443 in addition to port 80. You're basically saying the pod is terminating the secure connection not nginx.
If this is the issue would explain 50X error which indicates a problem with the backend
No special setup is needed, that 404 is most likely coming from your actual backend.

Getting 502 Bad Gateway nginx/1.13.9 for my angular app in k8's

I am getting 502 Bad Gateway nginx/1.13.9 for my angular app when accessing in the browser in k8's. My service and ingress is as below.
Angular app pod logs show all successful and in-fact port forwarding is working fine. This same image is also working fine in my local machine with docker.
From k8s logs, I could see this:
[error] 1534#1534: *32272457 SSL_do_handshake() failed (SSL:
error:1408F10B:SSL routines:ssl3_get_record:wrong version number)
while SSL handshaking to upstream
Service:
Name: test-portal
Namespace: testproject
Labels: app=test-portal
chart=test-portal-1.0.0
environment=dev
heritage=Tiller
release=test-portal
version=dev
Annotations: <none>
Selector: app=test-portal,release=test-portal
Type: ClusterIP
IP: x.x.x.x
Port: <unset> 80/TCP
TargetPort: 4200/TCP
Endpoints: x.x.x.x:4200
Session Affinity: None
Events: <none>
Ingress:
Name: test-portal
Namespace: testproject
Address:
Default backend: default-http-backend:80 (<none>)
TLS:
SNI routes test-portal.us-west-2.xxxxx.xxxxxx.delivery
Rules:
Host Path Backends
---- ---- --------
test-portal.us-west-2.xxxxx.xxxxxx.delivery
/ test-portal:80 (<none>)
Annotations:
secure-backends: true
ssl-redirect: true
Events: <none>
Ingress Config map
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
ingress.kubernetes.io/secure-backends: "false"
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"ingress.kubernetes.io/secure-backends":"false","kubernetes.io/ingress.class":"nginx"},"labels":{"app":"test-portal","chart":"test-portal-1.0.0","environment":"dev","heritage":"test-portal","release":"Helm","version":"1.0.0"},"name":"test-portal","namespace":"testproject"},"spec":{"rules":[{"host":"test-portal.us-west-2.xxxx.xxxxxxxxx.delivery","http":{"paths":[{"backend":{"serviceName":"test-portal","servicePort":80},"path":"/"}]}}],"tls":[{"hosts":["test-portal.us-west-2.xxxx.xxxxxxxxx.delivery"]}]}}
kubernetes.io/ingress.class: nginx
creationTimestamp: 2020-01-08T11:24:18Z
generation: 1
labels:
app: test-portal
chart: test-portal-1.0.0
environment: dev
heritage: test-portal
release: Helm
version: 1.0.0
name: test-portal
namespace: testproject
resourceVersion: "2379156945"
selfLink: /apis/extensions/v1beta1/namespaces/testproject/ingresses/test-portal
uid: 6925819b-3209-11ea-80fb-02fb0c9060d8
spec:
rules:
- host: test-portal.us-west-2.xxxx.xxxxxxxxx.delivery
http:
paths:
- backend:
serviceName: test-portal
servicePort: 80
path: /
tls:
- hosts:
- test-portal.us-west-2.xxxx.xxxxxxxxx.delivery
status:
loadBalancer:
ingress:
- {}
You seem to be running the angular dev server on port 4200 in your pod.
The angular app is served using http, not https, therefore you must configure the ingress to not use https (secure-backends: false) for backend communication.
Besides, the angular dev server should not be used for prod serving. Build a container image with the angular prod build to benefit from much increased performance.

K8 Ingress does not have an endpoint

I want to make services accessible from outside the K8 cluster using an ingress controller. Following 5.5 from the Kubernetes Cookbook, I ran this manifest:
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: nginx-public
annotations:
ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- host:
http:
paths:
- path: /web
backend:
serviceName: nginx
servicePort: 80
The Ingress object is visible in the Kubernetes dashboard; but it does not have an assigned endpoint:
Output of kubectl get ing:
NAME HOSTS ADDRESS PORTS AGE
nginx-public * 80 54m
update
Running kubectl describe ingress nginx-public gives:
Name: nginx-public
Namespace: default
Address:
Default backend: default-http-backend:80 (<none>)
Rules:
Host Path Backends
---- ---- --------
*
/web nginx:80 (<none>)
Annotations:
ingress.kubernetes.io/rewrite-target: /
Events: <none>
Actually this is an issue with Kubernetes Dashboard, we have the same issue.
Even if it isn't displayed it doesn't mean your ingress isn't working. First you should check the ingress with kubectl (kubectl describe ingress nginx-public) and verify that the output is smiliar to this:
Name: test-ingress
Namespace: test
Address:
Default backend: default-http-backend:80 (<none>)
TLS:
test-ssl-secret terminates test.myorg.com
Rules:
Host Path Backends
---- ---- --------
test.myorg.com
/ test-service:80 (<none>)
Afterwards you should verify your service is reachable via your specified host.
Update:
Depending on the service in front of your ingress-controller your service should be reachable via http://{serverip}:{nodeport-http-port}/web in case your service is of type NodePort(you will get 2 external ports in the 30000-39999 range, one is the http port the other the https port) or http://{address-from-external-loadbalancer}/web if the service is of type LoadBalancer.
2nd-Update
After some further investigation about the issue i stumbled upon a bug issue of kubernetes-dashboard stating that it's indeed possible to show the endpoints of ingress. The problem actually isn't caused by the dashboard, but a missing parameter on the ingress deployment.
For nginx-ingress-controller its the following:
NGINX Ingress CLI arguments
The missing option is --publish-service
If you used helm to deploy the controller you need to add the parameter --set controller.publishService.enabled=true

Resources