i use google k8s as a service with preemptible instances.
i faced with problem when google preempt one of node which serving kube-dns pod i get 5-7 mins failures in all another pods with "Cannot resolve" error.
I tried run second kube-dns pod but sometime both dns is running on the same node and i get failures again. I tried define nodeSelector for kube-dns pod but got error
Pod "kube-dns-2185667875-8b42l" is invalid: spec: Forbidden: pod updates may not change fields other than `containers[*].image` or `spec.activeDeadlineSeconds`
There is a possibility to run dns pods on different nodes redundantly? Maybe there are any best practice?
You can not modify POD like this, you need to modify your Deployment. Also you might want to look into pod anti-affinity to separate your pods in the same deployment in a way that will never schedule them on the same node. Alternatively, you can also switch from Deployment to DaemonSet to get exactly one pod running per node in cluster.
Related
Before I could use kubectl logs devops2-pdf-xxx to check the log of the pods.
But after I upgraded the kubectl version, I could not do that. Thus, seems the service is not running well.
But when I run kubectl describe node, the resource allocation is less than 100%.
kubectl logs xxx:
Error from server: Get "https://aks-agentpool-123456-1:10250/containerLogs/default/devops2-deployment-123456-456/devops2-pdf": dial tcp 10.240.0.5:10250: i/o timeout
There are several options to solve this problem. It is probably related to a closed port:
First, check that your port 10250 is open. Similar problem is described here
You are using AKS, so check solution described here:
Make sure that the default network security group isn't modified and that both port 22 and 9000 are open for connection to the API server. Check whether the tunnelfront pod is running in the kube-system namespace using the kubectl get pods --namespace kube-system command. If it isn't, force deletion of the pod and it will restart.
You can also check official Microsoft help page
These timeouts may be related to internal traffic between nodes being blocked. Verify that this traffic is not being blocked, such as by network security groups on the subnet for your cluster's nodes.
or this one.
I am going to deploy an application on Azure kubernetes with the following configuration
Kind : service
Replicas : 4
Horizontal pod scaling : ~8
Ingress : Nginx Ingress controller
As I have the Horizontal Pod scaling, when the load increases number of service Pod will increase automatically. However, should I also need to increase the frontier Nginx controller, if so, how would I do?
The short answer is: no, you don't have to add more Ingress controllers when autoscaling.
One Nginx Ingress controller is enough and you don't have to add/configure more in a situation when more application pods will be created. You just need one controller.
If you need to know more details regarding the HPA consider visiting the official sources here and here.
We have a k8s deployment of several services including Apache Spark. All services seem to be operational. Our application connects to the Spark master to submit a job using the k8s DNS service for the cluster where the master is called spark-api so we use master=spark://spark-api:7077 and we use spark.submit.deployMode=cluster. We submit the job through the API not by the spark-submit script.
This will run the "driver" and all "executors" on the cluster and this part seems to work but there is a callback to the launching code in our app from some Spark process. For some reason it is trying to connect to harness-64d97d6d6-4r4d8, which is the pod ID, not the k8s cluster IP or DNS.
How could this pod ID be getting into the system? Spark somehow seems to think it is the address of the service that called it. Needless to say any connection to the k8s pod ID fails and so does the job.
Any idea how Spark could think the pod ID is an IP address or DNS name?
BTW if we run a small sample job with master=local all is well, but the same job executed with the above config tries to connect to the spurious pod ID.
BTW2: the k8s DNS for the calling pod is harness-api
You can consider to use Headless service for harness-64etcetc Pod in order to accomplish backward DNS discovery. Actually, it will create particular endpoint for the relevant service by matching appropriate selector inside your application Pod and as result A record expects to be added into Kubernetes DNS configuration.
Eventually, I've found related #266 Github issue, which probably can bring some useful information for further investigation.
I am new to Kubernetes and looking for a better understanding of the difference between Kube-DNS and CoreDNS.
As I understand it the recommendation is to use the newer CoreDNS rather than the older Kube-DNS.
I have setup a small cluster using kubeadm and now I am a little confused about the difference between CoreDNS and Kube-DNS.
Using kubectl get pods --all-namespaces I can see that I have two CoreDNS pods running.
However using kubectl get svc --all-namespaces I also see that I have a service named kube-dns running in the kube-system namespace. When I inspect that with kubectl describe svc/kube-dns -n kube-system I can see that the kube-dns service links to coredns.
I am now wondering if I am actually running both kube-dns and coredns. Or else, why is that service called kube-dns and not core-dns?
I have K8S 1.12. Do a describe of the dns pod.
kubectl describe pod coredns-576cbf47c7-hhjrs --namespace=kube-system | grep -i "image:"
Image: k8s.gcr.io/coredns:1.2.2
Looks like coredns is running. According to the documentation CoreDNS is default from K8S 1.11. For previous installations it's kube-dns.
The image is what important, rest are metadata (names, labels etc).
According to the K8S blog here.
In Kubernetes 1.11, CoreDNS has reached General Availability (GA) for DNS-based service discovery, as an alternative to the kube-dns addon. This means that CoreDNS will be offered as an option in upcoming versions of the various installation tools. In fact, the kubeadm team chose to make it the default option starting with Kubernetes 1.11.
Also, see this link for more info.
First of all, I am not very expert of K8s, I understand some of the concepts and made already my hands dirty in the configurations.
I correctly set up the cluster configured by my company but I have this issue
I am working on a cluster with 2 pods, ingress rules are correctly configured for www.my-app.com and dashboard.my-app.com.
Both pods runs on the same VM.
If I enter in the dashboard pod (kubectl exec -it $POD bash) and try to curl http://www.my-app.com I land on the dashboard pod again (the same happens all the way around, from www to dashboard).
I have to use http://www-svc.default.svc.cluster.local and http://dashboard-svc.default.svc.cluster.local to land on the correct pods but this is a problem (links generated by the other app will contain internal k8s host, instead of the "public url").
Is there a way to configure routing so I can access pods with their "public" hostnames, from the pods themselves?
So what should happen when you curl is the external DNS record (www.my-app.com in this case) will resolve to your external IP address, usually a load balancer that then sends traffic to a kubernetes service. That service then should send traffic to the appropriate pod. It would seem that you have a misconfigured service. Make sure your service has an external IP that is different between dashboard and www. To see this a simple kubectl get svc should suffice. My guess is that the external IP is wrong, or the service is pointing to the wrong podm which you can see with a kubectl describe svc <name of service>.