I have AKS cluster up and running and on a heavy user load I get some 502 bad gateway responses. This only happens when the request load is high. I used the Azure DevOps load testing to achieve this behavior. I believe that it has something to do with the Load Balancer timeouts but I am not too sure how to go about debugging this. Perhaps I should be checking logs somehwere? Searching around google tells me that i should be checking nginx logs but not sure where to find those. Sorry I am newbie in kubernettes world.
These are all the pods that are in the cluster. apsever-api-... are my actual apps that serve the request:
The YAML file used to generate this:
# DS for AP
kind: DaemonSet
apiVersion: extensions/v1beta1
metadata:
name: apserver-api
spec:
updateStrategy:
type: RollingUpdate
selector:
template:
metadata:
labels:
app: apserver-api
spec:
containers:
- name: apserver-api
image: IMAGE
env:
- name: APP_SVC
value: apserver-api
ports:
- containerPort: 80
imagePullPolicy: IfNotPresent
# Service for AP
kind: Service
apiVersion: v1
metadata:
labels:
app: apserver-api
name: apserver-api
spec:
type: ClusterIP
ports:
- name: http
port: 80
- name: https
port: 443
targetPort: 80
selector:
app: apserver-api
type: "LoadBalancer"
and screenshot of the load test:
Related
I have a AKS cluster deployed with an Application Gateway. These are all docker images running on the AKS cluster with a simple ingress. They all run on the same default namespace. One is a Vue frontend, the second is a Java spring backend, the last being a Fullstack Tomcat image. All three services ran completely fine without any issues; however, today all of the 3 services gave back a 404 error (without any changes to the cluster to my knowledge).
When testing for the health of these services, I was still able to kubectl to all the services. In addition, the Health Probe on Azure Application Gateway returned a healthy 200 status code for all three services.
I have tried removing the workloads and adding them back again. I have tried removing the services then adding them back again. I have done the same for the ingress as well.
I have also tried setting up the entire cluster from a fresh AKS cluster from scratch from a new AKS cluster, all three images perform without any issues.
The Yaml for a application on the AKS network looks like:
- apiVersion: apps/v1
kind: Deployment
metadata:
name: testvuefe
spec:
replicas: 2
selector:
matchLabels:
app: testvuefe
template:
metadata:
labels:
app: testvuefe
spec:
nodeSelector:
kubernetes.io/os: linux
containers:
- name: testvuefe
image: testdocker.azurecr.io/testvuefe:latest
ports:
- containerPort: 80
resources:
requests:
cpu: '0'
memory: '0'
limits:
cpu: '128'
memory: 512G
volumeMounts:
- mountPath: "/fileShare"
name: volume
volumes:
- name: volume
persistentVolumeClaim:
claimName: aks-azurefile
- apiVersion: v1
kind: Service
metadata:
name: testvuefe-service
spec:
type: ClusterIP
ports:
- targetPort: 80
name: port80
port: 80
protocol: TCP
selector:
app: testvuefe
- apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: testvuefe-ingress
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/health-probe-path: /user
appgw.ingress.kubernetes.io/cookie-based-affinity: "true"
spec:
rules:
- host: test.wbsoft.co.kr
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: testvuefe-service
port:
number: 80
As this issue as occurred once on a cluster, I need to understand what is causing these issues so that it does not happen during production. It would also be great if a solution can be found for the issue.
I have hosted Docker Images in a VM of Azure and I'm trying to access the Service outside VM. This is not working because of External IP is not generated for the Service.
After building the Docker image, I've applied yml file for creating Deployment and Service. My yml file looks as below:
apiVersion: apps/v1
kind: Deployment
metadata:
name: planservice-deployment
labels:
app: planservice-deploy
spec:
selector:
matchLabels:
run: planservice-deploy
replicas: 2
template:
metadata:
labels:
run: planservice-deploy
spec:
containers:
- name: planservice-deploy
image: planserviceimage
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8086
---
apiVersion: v1
kind: Service
metadata:
name: planservice-service
labels:
app: planservice-deploy
spec:
type: NodePort
ports:
- port: 80
protocol: TCP
targetPort: 8086
selector:
run: planservice-deploy
---
After I ran the following command to look running services:
kubectl get pods --output=wide
This command returned all the running services and it's external IP information. But, when I saw the list, all the services are generated with blank external IPs.
How to set external IP for all the services, so that I can access my web services outside VM?
you need to change type to LoadBalancer:
apiVersion: v1
kind: Service
metadata:
name: planservice-service
labels:
app: planservice-deploy
spec:
type: LoadBalancer
ports:
- port: 80
protocol: TCP
targetPort: 8086
selector:
run: planservice-deploy
https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer
I have deployed angular frontend and python backend in kubernetes via microk8s as separate pods and they are running. I have given backend url as 'http://backend-service.default.svc.cluster.local:30007' in my angular file in order to link frontend with backend. But this is raising ERR_NAME_NOT_RESOLVED. Can someone help me in understanding the issue?
Also, I have a config file which specifies the ip's ports and other configurations in my backend. Do I need to make any changes(value of database host?, flask host?, ports? ) to that file before deploying t to kubernetes?
Shown below is my deployment and service files of angular and backend.
apiVersion: v1
kind: Service
metadata:
name: angular-service
spec:
type: NodePort
selector:
app: angular
ports:
- protocol: TCP
nodePort: 30042
targetPort: 4200
port: 4200
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: angular-deployment
labels:
name: angular
spec:
replicas: 1
selector:
matchLabels:
name: angular
template:
metadata:
labels:
name: angular
spec:
containers:
- name: angular
image: angular:local
ports:
- containerPort: 4200
apiVersion: v1
kind: Service
metadata:
name: backend-service
spec:
type:ClusterIP
selector:
name: backend
ports:
- protocol: TCP
targetPort: 7000
port: 7000
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: backend-deployment
labels:
name: backend
spec:
replicas: 1
selector:
matchLabels:
name: backend
template:
metadata:
labels:
name: backend
spec:
containers:
- name: backend
image: flask:local
ports:
- containerPort: 7000
Is your cluster in a healthy state ? DNS are resolved by object coredns in kube-system namespace.
In a classic way your angular app should show up your API Url in your browser so they must exposed and public. It is not your case and I have huge doubts about this.
Expose us your app architecture?
Moreover if you expose your service though NodePort you must not use it for internal access because you never know the node you will access.
When exose a service your apps need to use the port attribute (not the nodeport) to access pod generated in backend.
I have a true roadblock here and I have not found any solutions so far. Ultimately, my deployed NodeJS + Express server is not reachable when deploying to a Kubernete cluster on GCP. I followed the guide & example, nothing seems to work.
The cluster, node and service are running just fine and don't have any issues. Furthermore, it works just fine locally when running it with Docker.
Here's my Node YAML:
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
creationTimestamp: 2019-08-06T04:13:29Z
generation: 1
labels:
run: nodejsapp
name: nodejsapp
namespace: default
resourceVersion: "23861"
selfLink: /apis/apps/v1/namespaces/default/deployments/nodejsapp
uid: 8b6b7ac5-b800-11e9-816e-42010a9600de
spec:
progressDeadlineSeconds: 2147483647
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
run: nodejsapp
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
run: nodejsapp
spec:
containers:
- image: gcr.io/${project}/nodejsapp:latest
imagePullPolicy: Always
name: nodejsapp
ports:
- containerPort: 5000
protocol: TCP
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
status:
availableReplicas: 1
conditions:
- lastTransitionTime: 2019-08-06T04:13:29Z
lastUpdateTime: 2019-08-06T04:13:29Z
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
observedGeneration: 1
readyReplicas: 1
replicas: 1
updatedReplicas: 1
Service YAML:
apiVersion: v1
kind: Service
metadata:
creationTimestamp: 2019-08-06T04:13:34Z
labels:
run: nodejsapp
name: nodejsapp
namespace: default
resourceVersion: "25444"
selfLink: /api/v1/namespaces/default/services/nodejsapp
uid: 8ef81536-b800-11e9-816e-42010a9600de
spec:
clusterIP: XXX.XXX.XXX.XXX
externalTrafficPolicy: Cluster
ports:
- nodePort: 32393
port: 80
protocol: TCP
targetPort: 5000
selector:
run: nodejsapp
sessionAffinity: None
type: LoadBalancer
status:
loadBalancer:
ingress:
- ip: XXX.XXX.XXX.XXX
The NodeJS server is configured to run on Port 5000. I tried doing no port-forwarding as well but not a difference in the result.
Any help is much appreciated.
UPDATE:
I used this guide and followed the instructions: https://cloud.google.com/kubernetes-engine/docs/tutorials/hello-app
UPDATE 2:
FINALLY - figured it out. I'm not sure why this is not mentioned anywhere but you have to create an Ingress that routes the traffic to the pod accordingly.
Here's the example config:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
ingress.kubernetes.io/backends: '{"k8s-be-32064--abfe1f07378017e9":"HEALTHY"}'
ingress.kubernetes.io/forwarding-rule: k8s-fw-default-nodejsapp--abfe1f07378017e9
ingress.kubernetes.io/target-proxy: k8s-tp-default-nodejsapp--abfe1f07378017e9
ingress.kubernetes.io/url-map: k8s-um-default-nodejsapp--abfe1f07378017e9
creationTimestamp: 2019-08-06T18:59:15Z
generation: 1
name: nodejsapp
namespace: default
resourceVersion: "171168"
selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/versapay-api
uid: 491cd248-b87c-11e9-816e-42010a9600de
spec:
backend:
serviceName: nodejsapp
servicePort: 80
status:
loadBalancer:
ingress:
- ip: XXX.XXX.XXX
Adding it as an answer as need to include image (But not necessarily an answer):
As shown in the image, besides your backend service, a green tick should be visible
Probable Solution:
In your NodeJsApp, please add the following base URL .i.e.,
When the application is started locally, http://localhost:5000/ should return a 200 status code (With ideally Server is running... or some message)
And also, if path based routing is enabled, another base URL is also required:
http://localhost:5000/<nodeJsAppUrl>/ should also return 200 status code.
Above URLs are required for health check of both LoadBalancer and Backend Service and redeploy the service.
Please let me know if the above solution doesn't fix the said issue.
You need an intermediate service to internally expose your deployment.
Right now, you have a set of pods grouped in a deployment and a load balancer exposed in your cluster but you need to link them with an additional service.
You can try using a NodePort like the following:
apiVersion: v1
kind: Service
metadata:
name: nodejsapp-nodeport
spec:
selector:
run: nodejsapp
ports:
- name: default
protocol: TCP
port: 32393
targetPort: 5000
type: NodePort
This NodePort service is in between your Load Balancer and the pods in your deployment, targeting them in port 5000 and exposing port 32393 (as per your settings in the original question, you can change it).
From here, you can redeploy your Load Balancer to target the previous NodePort. This way, you can reach your NodeJS app via port 80 from your load balancer public address.
apiVersion: v1
kind: Service
metadata:
name: nodejs-lb
spec:
selector:
run: nodejsapp
ports:
- name: default
protocol: TCP
port: 80
targetPort: 32393
type: LoadBalancer
The whole scenario would look like this:
publicy exposed address --> LoadBalancer --> | NodePort --> Deployment --> Pods
I am new to K8s and this is my first time trying to get to grips with it. I am trying to set up a basic Nodejs Express API using this deployment.yml
kind: Service
apiVersion: v1
metadata:
name: ${GCP_PROJECT_NAME}
spec:
selector:
app: ${GCP_PROJECT_NAME}
ports:
- protocol: TCP
port: 80
targetPort: 3000
type: LoadBalancer
loadBalancerIP: ${STATIC_IP_ADDRESS}
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: ${GCP_PROJECT_NAME}
labels:
app: ${GCP_PROJECT_NAME}
spec:
replicas: 1
selector:
matchLabels:
app: ${GCP_PROJECT_NAME}
template:
metadata:
labels:
app: ${GCP_PROJECT_NAME}
spec:
containers:
- name: ${GCP_PROJECT_NAME}
image: gcr.io/${GCP_PROJECT_ID}/${GCP_PROJECT_NAME}:${CIRCLE_SHA1}
ports:
- name: http
containerPort: 3000
protocol: TCP
env:
- name: MONGO_URL_PROD
value: $MONGO_URL_PROD
Everything works great with this setup and deploys to Kubernetes. When I hit my endpoint i.e. http://123.345.333.123 as expected there is no SSL.
I generated my SSL certificates and tried to follow this tutorial [https://vorozhko.net/kubernetes-sidecar-pattern-nginx-ssl-proxy-for-nodejs] but I wasn't able to. Could anyone point me in the right direction, what am I doing wrong or what am I missing?
You can use nginx ingress controller to handle all your SSL setup and usage. Following is a step by step guide to do so:
https://dgkanatsios.com/2017/07/07/using-ssl-for-a-service-hosted-on-a-kubernetes-cluster/
Hope this helps.
This approach didnt work for me. Ingress was not able to get the Cluster IP, it shows <none>