I am facing the 502 Bad gateway issue in my Application Gateway.
I am using Azure Kubernetes Service to deploy my cluster which is connected to Ingress Application Gateway.
Configuration Files:
kube-deployment.yml
apiVersion: apps/v1
kind: Deployment
metadata:
name: myApp
namespace: en02
labels:
app: myApp
spec:
selector:
matchLabels:
app: myApp
replicas: 1
template:
metadata:
labels:
app: myApp
spec:
containers:
- name: myApp
image: somecr.azurecr.io/myApp:1.0.0.30
resources:
limits:
memory: "64Mi"
cpu: "100m"
ports:
- containerPort: 5100
env:
- name: ASPNETCORE_HOSTINGSTARTUPASSEMBLIES
value: "Microsoft.AspNetCore.ApplicationInsights.HostingStartup"
- name: "ApplicationInsights__ConnectionString"
value: "myKey"
---
apiVersion: v1
kind: Service
metadata:
namespace: en02
name: myApp
spec:
selector:
app: myApp
ports:
- port: 30153
targetPort: 5100
protocol: TCP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
namespace: en02
name: etopia
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/health-probe-path: "/api/home"
spec:
rules:
- http:
paths:
- path: /myApp/
backend:
service:
name: myApp
port:
number: 30153
pathType: Exact
Result of
kubectl describe ingress -n en02
Name: ingress
Labels: <none>
Namespace: en02
Address: public-ip
Ingress Class: <none>
Default backend: <default>
Rules:
Host Path Backends
---- ---- --------
*
/myApp/ myApp:30153 (10.0.0.106:5100)
Annotations: appgw.ingress.kubernetes.io/health-probe-path: /api/home
kubernetes.io/ingress.class: azure/application-gateway
Events: <none>
I am getting expected results from 10.0.0.106:5100/api/home and Application Gateway health status is 200.
No matter what I do, I always get Bad Gateway error, I was able to access a sample app on port 80 (where the ingress path was /) but if I specify anything in ingress path (/cashify/) it always give me bad gateway.
I tried adding readinessProbe to container but it doesn't work (However I am already getting 200 under application gateway health status).
Please help.
Please check if below can be worked around.
Please try to update deployment yaml to use wild card path specification to access apis with different paths.
deployment.yml
apiVersion: extensions/v1beta1
kind: Ingress
....
annotations:
kubernetes.io/ingress.class: azure/application-gateway
spec:
rules:
- host: xxx
http:
paths:
- path: /api/* #wild card path
backend:
serviceName: apiservice
servicePort: 80
- backend:
.....
servicePort: 80
Note from MS docs: If you want Application Gateway to probe on a different protocol, host name, or path and to recognize a
different status code as Healthy, configure a custom probe and
associate it with the HTTP settings.
As you said you have defined readinessProbe , please check if path of those probes is correct.
Check same with 1. livenessProbe 2. readinessProbe
Also please note that readinessProbe and livenessProbe are supported when configured with httpGet.
References:
bad request - path based routing · kubernetes-ingress · GitHub
application-gateway-troubleshooting-502
Related
This is my ingress yaml file
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: example
namespace: ingress-nginx
spec:
ingressClassName: nginx
rules:
- host: ticketing.dev
http:
paths:
- pathType: Prefix
backend:
service:
name: auth-srv
port:
number: 5000
path: /
when ever I go to ticketing.dev it shows
As all of the services are working as expected
**All of the pods are also working just fine"
Following is my Service and Deployment yaml code
apiVersion: apps/v1
kind: Deployment
metadata:
name: auth-depl
spec:
replicas: 1
selector:
matchLabels:
app: auth
template:
metadata:
labels:
app: auth
spec:
containers:
- name: auth
image: 9862672975/auth
env:
- name: JWT_KEY
valueFrom:
secretKeyRef:
name: jwt-secret
key: JWT_KEY
---
apiVersion: v1
kind: Service
metadata:
name: auth-srv
spec:
selector:
app: auth
ports:
- name: auth
protocol: TCP
port: 5000
targetPort: 5000
I am trying build microservices with Nodejs and Nextjs. As I try to add both frontend and backend to ingress it did not respond, and I tried removing frontend and just running backend with following code it is not working.
You have not specified a path in your ingress-file, only a pathType.
Below paths you want to add path: "/".
If you look at the Ingress reference, you may see that the path field is not marked as "required", but with this note:
Paths must begin with a '/' and must be present when using PathType with value "Exact" or "Prefix".
Since you have specified your pathType as "Prefix", you need to include the path. In general, I would advise explicitly specifying both path and prefix whenever possible, rather than relying on defaults.
I have a AKS cluster deployed with an Application Gateway. These are all docker images running on the AKS cluster with a simple ingress. They all run on the same default namespace. One is a Vue frontend, the second is a Java spring backend, the last being a Fullstack Tomcat image. All three services ran completely fine without any issues; however, today all of the 3 services gave back a 404 error (without any changes to the cluster to my knowledge).
When testing for the health of these services, I was still able to kubectl to all the services. In addition, the Health Probe on Azure Application Gateway returned a healthy 200 status code for all three services.
I have tried removing the workloads and adding them back again. I have tried removing the services then adding them back again. I have done the same for the ingress as well.
I have also tried setting up the entire cluster from a fresh AKS cluster from scratch from a new AKS cluster, all three images perform without any issues.
The Yaml for a application on the AKS network looks like:
- apiVersion: apps/v1
kind: Deployment
metadata:
name: testvuefe
spec:
replicas: 2
selector:
matchLabels:
app: testvuefe
template:
metadata:
labels:
app: testvuefe
spec:
nodeSelector:
kubernetes.io/os: linux
containers:
- name: testvuefe
image: testdocker.azurecr.io/testvuefe:latest
ports:
- containerPort: 80
resources:
requests:
cpu: '0'
memory: '0'
limits:
cpu: '128'
memory: 512G
volumeMounts:
- mountPath: "/fileShare"
name: volume
volumes:
- name: volume
persistentVolumeClaim:
claimName: aks-azurefile
- apiVersion: v1
kind: Service
metadata:
name: testvuefe-service
spec:
type: ClusterIP
ports:
- targetPort: 80
name: port80
port: 80
protocol: TCP
selector:
app: testvuefe
- apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: testvuefe-ingress
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/health-probe-path: /user
appgw.ingress.kubernetes.io/cookie-based-affinity: "true"
spec:
rules:
- host: test.wbsoft.co.kr
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: testvuefe-service
port:
number: 80
As this issue as occurred once on a cluster, I need to understand what is causing these issues so that it does not happen during production. It would also be great if a solution can be found for the issue.
I'm trying to access a simple Asp.net core application deployed on Azure AKS but I'm doing something wrong.
This is the deployment .yml
apiVersion: apps/v1
kind: Deployment
metadata:
name: aspnetapp
spec:
replicas: 1
selector:
matchLabels:
app: aspnet
template:
metadata:
labels:
app: aspnet
spec:
containers:
- name: aspnetapp
image: <my_image>
resources:
limits:
cpu: "0.5"
memory: 64Mi
ports:
- containerPort: 8080
and this is the service .yml
apiVersion: v1
kind: Service
metadata:
name: aspnet-loadbalancer
spec:
type: LoadBalancer
ports:
- protocol: TCP
port: 80
targetPort: 8080
selector:
name: aspnetapp
Everything seems deployed correctly
Another check I did was to enter the pod and run
curl http://localhost:80,
and the application is running correctly, but if I try to access the application from the browser using http://20.103.147.69 a timeout is returned.
What else could be wrong?
Seems that you do not have an Ingress Controller deployed on your AKS as you have your application exposed directly. You will need that in order to get ingress to work.
To verify if your application is working your can use port-forward and then access http://localhost:8080 :
kubectl port-forward aspnetapp 8080:8080
But you should def. install a ingress-controller: Here is a Workflow from MS to install ingress-nginx as IC on your Cluster.
You will then only expose the ingress-controller to the internet and could also specify the loadBalancerIP statically if you created the PublicIP in advance:
apiVersion: v1
kind: Service
metadata:
annotations:
service.beta.kubernetes.io/azure-load-balancer-resource-group: myResourceGroup # only needed if the LB is in another RG
name: ingress-nginx-controller
spec:
loadBalancerIP: <YOUR_STATIC_IP>
type: LoadBalancer
The Ingress Controller then will route incoming traffic to your application with an Ingress resource:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: minimal-ingress
spec:
ingressClassName: nginx # ingress-nginx specifix
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: test
port:
number: 80
PS: Never expose your application directly to the internet, always use the ingress controller
In your Deployment, you configured your container to listen on port 8080. You need to add targetport value set to 8080 in the Service definition.
Documentation
I need help in AGIC configuration. I am using Loadbalancer service for my existing AKS cluster and below is the sample yaml file that works and I can access application using LB public IP :
apiVersion: apps/v1
kind: Deployment
metadata:
name: aspnetapp
namespace: asp-test
labels:
app: asp-frontend
spec:
selector:
matchLabels:
app: asp-frontend
template:
metadata:
labels:
app: asp-frontend
spec:
containers:
- name: aspnetapp
image: "mcr.microsoft.com/dotnet/core/samples:aspnetapp"
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: aspnetapp-load
namespace: asp-test
labels:
app: asp-frontend
annotations:
service.beta.kubernetes.io/azure-load-balancer-resource-group: mc_asp-onef-dev_rg_asp_aks_eastus2
spec:
loadBalancerIP: 10.10.10.10
type: LoadBalancer
ports:
- port: 80
targetPort: 80
selector:
app: asp-frontend
==================
Now I would like to use AGIC instead of LB and I am just adding below section in the file but I get "502 bad gateway" error. My AKS and AG vnets are peered. I don't have NSG to block connection. Deployment is successful and pods are running. I can access it using LB IP but not using AGIC.
I have tried editing this file and use normal AKS service instead of LB but I still get same error.
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: aspnetapp
namespace: asp-test
annotations:
kubernetes.io/ingress.class: azure/application-gateway
spec:
rules:
- http:
paths:
- path: /
backend:
serviceName: aspnetapp-load
servicePort: 80
I am trying to setup a K8s deployment where ingress's controllers can define a service as a subdomain. i.e. app1 can define itself to get traffic from app1.sub.domain.io in its ingress config.
I have a DNS A Record *.sub.domain.io that points to a Load Balancer. That load balancer is pointing to the cluster's instance group.
So if I am right all traffic that goes to anything at sub.domain.io will land inside the cluster and just need to route said traffic.
Below are the k8 configs, which has a pod, a service and an ingress. The pods are healthy and working, I believe the service isn't required but will want other pods to talk to it via internal DNS so it's added.
The ingress rules have a host app1.sub.domain.io, so in theory, curl'ing app1.sub.domain.io should follow:
DNS -> Load Balancer -> Cluster -> Ingress Controller -> Pod
At the moment when I try to hit app1.sub.domain.io it just hangs. I have tried not having service, making external-name service and doesn't work.
I don't want to go down the route of using the loadBalancer ingress as that makes a new external IP that needs to be applied to DNS records manually, or with a nasty bash script that waits for services external IP and runs GCP command, and we don't want to do this for each service.
Ref links:https://kubernetes.io/docs/concepts/services-networking/ingress/#name-based-virtual-hosting
Deployment
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: app1
namespace: default
labels:
app: app1
spec:
replicas: 3
selector:
matchLabels:
app: app1
template:
metadata:
labels:
app: app1
spec:
containers:
- image: xxxx:latest
name: app1
ports:
- containerPort: 80
env:
- name: NODE_ENV
value: production
Service
---
kind: Service
apiVersion: v1
metadata:
name: app1
labels:
app: app1
spec:
ports:
- port: 80
targetPort: 80
selector:
app: app1
type: ClusterIP
Ingress
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: app1-ingress
labels:
app: app1
spec:
rules:
- host: app1.sub.domain.io
http:
paths:
- backend:
serviceName: app1
servicePort: 80
Once everything is deployed if you query
kubectl get pods,services,ingresses -l app=app1
NAME READY STATUS RESTARTS AGE
po/app1-6d4b9d8c5-4gcz5 1/1 Running 0 20m
po/app1-6d4b9d8c5-m4kwq 1/1 Running 0 20m
po/app1-6d4b9d8c5-rpm9l 1/1 Running 0 20m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
svc/app1 ClusterIP x.x.x.x <none> 80/TCP 20m
NAME HOSTS ADDRESS PORTS AGE
ing/app1-ingress app1.sub.domain.io 80 20m
----------------------------------- Update -----------------------------------
Currently doing this, not ideal. Have global static IP that's assigned to a DNS record.
---
kind: Service
apiVersion: v1
metadata:
name: app1
labels:
app: app1
spec:
type: NodePort
selector:
app: app1
ports:
- port: 80
targetPort: 80
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: app1-ingress
annotations:
kubernetes.io/ingress.global-static-ip-name: app1-static-ip
labels:
app: app1-static-ip
spec:
backend:
serviceName: app1
servicePort: 80
*.sub.domain.io should point to the IP of the Ingress.
You can use a static IP for the Ingress by following the instructions in the tutorial here: https://cloud.google.com/kubernetes-engine/docs/tutorials/http-balancer#step_5_optional_configuring_a_static_ip_address
Try adding path to your Ingress:
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: app1-ingress
labels:
app: app1
spec:
rules:
- host: app1.sub.domain.io
http:
paths:
- backend:
serviceName: app1
servicePort: 80
path: /
If that doesn't work, please post the output of describe service and describe ingress.
Do you have an Ingress Controller?
Traffic should go LB-> Ingress Controller-> Ingress-> Service ClusterIP-> Pods