I'm looking for any working samples of applying different certificates on AKS with Application Gateway as Ingress Controller.
I have Key Vault with a certificate that is used imported in ApGw/Ingress as sitecomcert and here is Ingress manifest:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: site-agic-ig
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/appgw-ssl-certificate: sitecomcert
appgw.ingress.kubernetes.io/ssl-redirect: "true"
appgw.ingress.kubernetes.io/request-timeout: "180"
appgw.ingress.kubernetes.io/cookie-based-affinity: "true"
spec:
rules:
- host: "site.com"
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: site-svc
port:
number: 80
...
Everything works perfect here.
Now I have a second certificate in Key Vault for site2.com and this cert is already imported in Ap Gw as site2comcert and I have container that should serve requests coming to site2.com which point to Ap Gw Public IP.
So I'm about to add
- host: "site2.com" <--- How can I attach **site2comcert** cert?
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: site2-svc
port:
number: 80
but with this setup I receive Untrusted Connection warning in browser because sitecomcert is used. How to configure ApGw / Ingress in a way that allows to use site2comcert for site2.com host specified above?
You can have multiple ingress resource definitions (snipped for brevity):
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: site-agic-ig
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/appgw-ssl-certificate: sitecomcert
spec:
rules:
- host: "site.com"
and
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: site-agic-ig-site2
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/appgw-ssl-certificate: site2comcert
spec:
rules:
- host: "site2.com"
Related
Im looking for the way to configure OAuth2 and Azure provider for Nginx Ingress with multiple hosts definitions.
I need add authentication over for my application dev.example.com on AKS which has internal auth based on Identity service.
I've read examples like this:
https://kristhecodingunicorn.com/post/k8s_nginx_oauth/#setting-up-authentication-with-oauth-20
I've created Application in Azure AD and configure it like described above:
Redirect URIs - https://dev.example.com/oauth2/callback
Front-channel logout - URL https://dev.example.com/oauth2/sign_out
As far I can see almost all works ok - when I open dev.example.com in browser it redirect me to MS Sign In form, then runs 2FA, and opens dev.example.com.
But there is one error during opening web site:
Access to XMLHttpRequest at
'https://api.dev.example.com/identity/.well-known/openid-configuration'
from origin 'https://dev.example.com' has been blocked by CORS policy:
Response to preflight request doesn't pass access control check:
Redirect is not allowed for a preflight request.
the main application tries to connect to internal Identity service and gets an error because of CORS.
Is there a way to fix this?
here is my Nginx ingress for application:
kind: Ingress
apiVersion: networking.k8s.io/v1
metadata:
name: dev-ingress
annotations:
cert-manager.io/issuer: letsencrypt-cert
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/auth-signin: https://$host/oauth2/start?rd=$escaped_request_uri
nginx.ingress.kubernetes.io/auth-url: https://$host/oauth2/auth
nginx.ingress.kubernetes.io/ssl-redirect: 'false'
nginx.ingress.kubernetes.io/use-regex: 'true'
spec:
tls:
- hosts:
- dev.example.com
- api.dev.example.com
secretName: letsencrypt-cert
rules:
- host: dev.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: dev-service
port:
number: 80
- host: api.dev.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: api-service
port:
number: 80
and OAuth2 Configuration:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
k8s-app: oauth2-proxy
name: oauth2-proxy
namespace: dev
spec:
replicas: 1
selector:
matchLabels:
k8s-app: oauth2-proxy
template:
metadata:
labels:
k8s-app: oauth2-proxy
spec:
containers:
- args:
- --provider=oidc
#- --provider=azure
- --azure-tenant=xxxxxxxxxxxxxx
- --skip-jwt-bearer-tokens=true
- --skip-auth-preflight=true
- --email-domain=*
- --http-address=0.0.0.0:4180
- --cookie-domain=.example.com
- --whitelist-domain=.example.com
- --oidc-issuer-url=https://login.microsoftonline.com/xxxxxxxx/v2.0
env:
- name: OAUTH2_PROXY_CLIENT_ID
valueFrom:
secretKeyRef:
name: client-id
key: oauth2_proxy_client_id
- name: OAUTH2_PROXY_CLIENT_SECRET
valueFrom:
secretKeyRef:
name: client-secret
key: oauth2_proxy_client_secret
- name: OAUTH2_PROXY_COOKIE_SECRET
valueFrom:
secretKeyRef:
name: cookie-secret
key: oauth2_proxy_cookie_secret
image: quay.io/oauth2-proxy/oauth2-proxy:latest
imagePullPolicy: Always
name: oauth2-proxy
ports:
- containerPort: 4180
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
labels:
k8s-app: oauth2-proxy
name: oauth2-proxy
namespace: dev
spec:
ports:
- name: http
port: 4180
protocol: TCP
targetPort: 4180
selector:
k8s-app: oauth2-proxy
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: oauth2-proxy
namespace: dev
spec:
ingressClassName: nginx
rules:
- host: dev.example.com
http:
paths:
- path: /oauth2
pathType: Prefix
backend:
service:
name: oauth2-proxy
port:
number: 4180
- host: api.dev.example.com
http:
paths:
- path: /oauth2
pathType: Prefix
backend:
service:
name: oauth2-proxy
port:
number: 4180
You seem to have an SPA and API architecture here. Therefore you need to deal with CORS on the API side of things. As a first step you could implement CORS in your api-service, by plugging in a library.
A more elegant solution is to do so in the ingress, via a CORS plugin. This example is from Kong but maybe there is an equivalent plugin in NGINX. The approach is to build a custom ingress controller image and then add a configuration snippet to inject settings into the nginx.conf file, though it may prove difficult.
Be aware that in oauth2-proxy you are using a website security solution, which is unlikely to be SPA friendly during Ajax calls to APIs, eg when you need to refresh an expired access token. A backend for frontend (BFF) with a more complete design is preferred.
I'm trying to make Ingress use external IP i have created in Azure
First I have created an IP in the portal and added my AKS service as network contributor, then added it in the values file used by HELM
# -- List of IP addresses at which the controller services are available
## Ref: https://kubernetes.io/docs/user-guide/services/#external-ips
##
externalIPs: ["20.124.63.xxx"]
# -- Used by cloud providers to connect the resulting `LoadBalancer` to a pre-existing static IP according to https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer
loadBalancerIP: ""
loadBalancerSourceRanges: []
enableHttp: true
enableHttps: true
But after deployment, my ingress gets two external IPs, and the one set by me does not work at all, only automatically generated works:
My config looks like this, so I think running this as loadbalancer is not exactly possible:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: hello-world-ingress
namespace: default
annotations:
nginx.ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/use-regex: "true"
nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
ingressClassName: nginx
rules:
- host: xxx.com
http:
paths:
- path: /(.*)
pathType: Prefix
backend:
service:
name: aks-one
port:
number: 80
- host: xxx.com
http:
paths:
- path: /(.*)
pathType: Prefix
backend:
service:
name: aks-two
port:
number: 80
I would like to use static IP I have created to access my Ingress, what should I do to achieve that?
Exposing the Service of your ingress controller with your public ip can be done like this:
apiVersion: v1
kind: Service
metadata:
annotations:
service.beta.kubernetes.io/azure-load-balancer-resource-group: myResourceGroup # only needed if the LB is in another RG
name: ingress-nginx-controller
spec:
loadBalancerIP: <YOUR_STATIC_IP>
type: LoadBalancer
Azure now will spin-up a LoadBalancer with your public IP.
The Ingress Controller then will route incoming traffic to your apps with an Ingress resource:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: minimal-ingress
spec:
ingressClassName: nginx # ingress-nginx specifix
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: test
port:
number: 80
I am facing the 502 Bad gateway issue in my Application Gateway.
I am using Azure Kubernetes Service to deploy my cluster which is connected to Ingress Application Gateway.
Configuration Files:
kube-deployment.yml
apiVersion: apps/v1
kind: Deployment
metadata:
name: myApp
namespace: en02
labels:
app: myApp
spec:
selector:
matchLabels:
app: myApp
replicas: 1
template:
metadata:
labels:
app: myApp
spec:
containers:
- name: myApp
image: somecr.azurecr.io/myApp:1.0.0.30
resources:
limits:
memory: "64Mi"
cpu: "100m"
ports:
- containerPort: 5100
env:
- name: ASPNETCORE_HOSTINGSTARTUPASSEMBLIES
value: "Microsoft.AspNetCore.ApplicationInsights.HostingStartup"
- name: "ApplicationInsights__ConnectionString"
value: "myKey"
---
apiVersion: v1
kind: Service
metadata:
namespace: en02
name: myApp
spec:
selector:
app: myApp
ports:
- port: 30153
targetPort: 5100
protocol: TCP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
namespace: en02
name: etopia
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/health-probe-path: "/api/home"
spec:
rules:
- http:
paths:
- path: /myApp/
backend:
service:
name: myApp
port:
number: 30153
pathType: Exact
Result of
kubectl describe ingress -n en02
Name: ingress
Labels: <none>
Namespace: en02
Address: public-ip
Ingress Class: <none>
Default backend: <default>
Rules:
Host Path Backends
---- ---- --------
*
/myApp/ myApp:30153 (10.0.0.106:5100)
Annotations: appgw.ingress.kubernetes.io/health-probe-path: /api/home
kubernetes.io/ingress.class: azure/application-gateway
Events: <none>
I am getting expected results from 10.0.0.106:5100/api/home and Application Gateway health status is 200.
No matter what I do, I always get Bad Gateway error, I was able to access a sample app on port 80 (where the ingress path was /) but if I specify anything in ingress path (/cashify/) it always give me bad gateway.
I tried adding readinessProbe to container but it doesn't work (However I am already getting 200 under application gateway health status).
Please help.
Please check if below can be worked around.
Please try to update deployment yaml to use wild card path specification to access apis with different paths.
deployment.yml
apiVersion: extensions/v1beta1
kind: Ingress
....
annotations:
kubernetes.io/ingress.class: azure/application-gateway
spec:
rules:
- host: xxx
http:
paths:
- path: /api/* #wild card path
backend:
serviceName: apiservice
servicePort: 80
- backend:
.....
servicePort: 80
Note from MS docs: If you want Application Gateway to probe on a different protocol, host name, or path and to recognize a
different status code as Healthy, configure a custom probe and
associate it with the HTTP settings.
As you said you have defined readinessProbe , please check if path of those probes is correct.
Check same with 1. livenessProbe 2. readinessProbe
Also please note that readinessProbe and livenessProbe are supported when configured with httpGet.
References:
bad request - path based routing · kubernetes-ingress · GitHub
application-gateway-troubleshooting-502
I am following MS documentation to create ingress NGINX controller
https://learn.microsoft.com/en-us/learn/modules/aks-workshop/07-deploy-ingress
But below yaml file is giving me error : The Ingress "ratings-web-ingress" is invalid: spec.rules[0].http.paths[0].pathType: Required value: pathType must be specified
Command : kubectl apply --namespace ratingsapp -f ratings-web-ingress.yaml --validate=false
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ratings-web-ingress
annotations:
kubernetes.io/ingress.class: nginx
spec:
rules:
- host: frontend.20-83-140-186.nip.io # IMPORTANT: update <ingress ip> with the dashed public IP of your ingress, for example frontend.13-68-177-68.nip.io
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: ratings-web
port:
number: 80
I tried to reproduce creating ingress NGINX controller with the same code and got the same error:
To resolve the error "spec.rules[0].http.paths[0].pathType: Required value: pathType must be specified" replace pathType: ImplementationSpecific after pathType like below:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ratings-web-ingress
annotations:
kubernetes.io/ingress.class: nginx
spec:
rules:
- host: frontend.10.0.0.1.nip.io # IMPORTANT: update <ingress ip> with the dashed public IP of your ingress, for example frontend.13-68-177-68.nip.io
http:
paths:
- backend:
service:
name: ratings-web
port:
number: 80
path: /
pathType: ImplementationSpecific
After modifying the yaml file, I was able to create ingress successfully:
For more in detail, please refer below link:
Lab10.1 - Error creating ingress rule: "pathType must be specified" — Linux Foundation Forums
I have one service and a single ingress resource with kubenetes nginx ingress controller. I want the /student path of my url to go to the root of the application and match any other url segments which follow the student.
For example: http://example.com/student/ver should match the /ver route of my application.
However, my ingress always hit the application with the /student url path prefixing the other url segments. If I call http://example.com/student/ver, my application is hit with the same url (student/ver).
My ingress:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
ingress.kubernetes.io/rewrite-target: /
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: "false"
name: ingress-resource
spec:
rules:
- host: example.com
http:
paths:
- backend:
serviceName: lesson-streaming
servicePort: 80
path: /student
I spent days with this and was not successful once.
Edit:
The ingress is changed to the following - not my requests say http 404
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
ingress.kubernetes.io/rewrite-target: /$2
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: "false"
name: ingress-resource
namespace: default
spec:
rules:
- host: example.com
http:
paths:
- backend:
serviceName: lesson-streaming
servicePort: 80
path: /student(/|$)(.*)
You can follow the link to use the rewrite-target annotation correctly and keep the right key nginx.ingress.kubernetes.io/rewrite-target.
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$2
name: rewrite
namespace: default
spec:
rules:
- host: rewrite.bar.com
http:
paths:
- backend:
serviceName: http-svc
servicePort: 80
path: /something(/|$)(.*)
Im using k8 - v1.25.2 along with nginx-Ingress v2.4.1 https://docs.nginx.com/nginx-ingress-controller/installation/installation-with-manifests/
nginx.ingress.kubernetes.io/rewrite-target: /$2 - doesnt work
instead use :
nginx.org/rewrites: "serviceName=app1-svc rewrite=/;serviceName=app2-svc rewrite=/"
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: path-ingress
namespace: default
annotations:
nginx.org/rewrites: "serviceName=app1-svc rewrite=/;serviceName=app2-svc rewrite=/"
spec:
ingressClassName: nginx
rules:
- host: apps.relo.com
http:
paths:
- backend:
service:
name: app1-svc
port:
number: 2041
path: /app1
pathType: Prefix
- backend:
service:
name: app2-svc
port:
number: 2042
path: /app2
pathType: Prefix
I have tried many times to execute properly path based routing in nginx ingress controller but I have found the main reason behind the path based routing not working.
replace the (nginx.ingress.kubernetes.io/rewrite-target: /) to (nginx.org/rewrites: "serviceName=servicename rewrite=/")
must add ingressClassName in spec
in my case
I have kubernetes 1.23.0 & nginx/1.21.6
I hope this will resolve the above problem.
Best of luck