kubernetes skydns failure to forward request - dns

Question

I am creating a cluster of 1 master 2 nodes kubernetes. I am trying to create the skydns based on the following:
apiVersion: v1
kind: ReplicationController
metadata:
name: kube-dns-v11
namespace: kube-system
labels:
k8s-app: kube-dns
version: v11
kubernetes.io/cluster-service: "true"
spec:
replicas: 1
selector:
k8s-app: kube-dns
version: v11
template:
metadata:
labels:
k8s-app: kube-dns
version: v11
kubernetes.io/cluster-service: "true"
spec:
containers:
- name: etcd
image: gcr.io/google_containers/etcd-amd64:2.2.1
# resources:
# # TODO: Set memory limits when we've profiled the container for large
# # clusters, then set request = limit to keep this container in
# # guaranteed class. Currently, this container falls into the
# # "burstable" category so the kubelet doesn't backoff from restarting it.
# limits:
# cpu: 100m
# memory: 500Mi
# requests:
# cpu: 100m
# memory: 50Mi
command:
- /usr/local/bin/etcd
- -data-dir
- /var/etcd/data
- -listen-client-urls
- http://127.0.0.1:2379,http://127.0.0.1:4001
- -advertise-client-urls
- http://127.0.0.1:2379,http://127.0.0.1:4001
- -initial-cluster-token
- skydns-etcd
volumeMounts:
- name: etcd-storage
mountPath: /var/etcd/data
- name: kube2sky
image: gcr.io/google_containers/kube2sky:1.14
# resources:
# # TODO: Set memory limits when we've profiled the container for large
# # clusters, then set request = limit to keep this container in
# # guaranteed class. Currently, this container falls into the
# # "burstable" category so the kubelet doesn't backoff from restarting it.
# limits:
# cpu: 100m
# # Kube2sky watches all pods.
# memory: 200Mi
# requests:
# cpu: 100m
# memory: 50Mi
livenessProbe:
httpGet:
path: /healthz
port: 8080
scheme: HTTP
initialDelaySeconds: 60
timeoutSeconds: 5
# successThreshold: 1
# failureThreshold: 5
readinessProbe:
httpGet:
path: /readiness
port: 8081
scheme: HTTP
# we poll on pod startup for the Kubernetes master service and
# only setup the /readiness HTTP server once that's available.
initialDelaySeconds: 30
timeoutSeconds: 5
args:
# command = "/kube2sky"
- --domain=cluster.local
- name: skydns
image: gcr.io/google_containers/skydns:2015-10-13-8c72f8c
resources:
# TODO: Set memory limits when we've profiled the container for large
# clusters, then set request = limit to keep this container in
# guaranteed class. Currently, this container falls into the
# "burstable" category so the kubelet doesn't backoff from restarting it.
limits:
cpu: 100m
memory: 200Mi
requests:
cpu: 100m
memory: 50Mi
args:
# command = "/skydns"
- -machines=http://127.0.0.1:4001
- -addr=0.0.0.0:53
- -ns-rotate=false
- -domain=cluster.local
ports:
- containerPort: 53
name: dns
protocol: UDP
- containerPort: 53
name: dns-tcp
protocol: TCP
- name: healthz
image: gcr.io/google_containers/exechealthz:1.0
# resources:
# # keep request = limit to keep this container in guaranteed class
# limits:
# cpu: 10m
# memory: 20Mi
# requests:
# cpu: 10m
# memory: 20Mi
args:
- -cmd=nslookup kubernetes.default.svc.cluster.local 127.0.0.1 >/dev/null
- -port=8080
ports:
- containerPort: 8080
protocol: TCP
volumes:
- name: etcd-storage
emptyDir: {}
dnsPolicy: Default # Don't use cluster DNS.
However, skydns is spitting out the following:
> $kubectl logs kube-dns-v11-k07j9 --namespace=kube-system skydns
> 2016/04/18 12:47:05 skydns: falling back to default configuration,
> could not read from etcd: 100: Key not found (/skydns) [1] 2016/04/18
> 12:47:05 skydns: ready for queries on cluster.local. for
> tcp://0.0.0.0:53 [rcache 0] 2016/04/18 12:47:05 skydns: ready for
> queries on cluster.local. for udp://0.0.0.0:53 [rcache 0] 2016/04/18
> 12:47:11 skydns: failure to forward request "read udp
> 192.168.122.1:53: i/o timeout" 2016/04/18 12:47:15 skydns: failure to forward request "read udp 192.168.122.1:53: i/o timeout" 2016/04/18
> 12:47:19 skydns: failure to forward request "read udp
> 192.168.122.1:53: i/o timeout" 2016/04/18 12:47:23 skydns: failure to forward request "read udp 192.168.122.1:53: i/o timeout" 2016/04/18
> 12:47:27 skydns: failure to forward request "read udp
> 192.168.122.1:53: i/o timeout" 2016/04/18 12:47:31 skydns: failure to forward request "read udp 192.168.122.1:53: i/o timeout" 2016/04/18
> 12:47:35 skydns: failure to forward request "read udp
> 192.168.122.1:53: i/o timeout" 2016/04/18 12:47:39 skydns: failure to forward request "read udp 192.168.122.1:53: i/o timeout" 2016/04/18
> 12:47:43 skydns: failure to forward request "read udp
> 192.168.122.1:53: i/o timeout" 2016/04/18 12:47:47 skydns: failure to forward request "read udp 192.168.122.1:53: i/o timeout" 2016/04/18
> 12:47:51 skydns: failure to forward request "read udp
> 192.168.122.1:53: i/o timeout" 2016/04/18 12:47:55 skydns: failure to forward request "read udp 192.168.122.1:53: i/o timeout" 2016/04/18
> 12:47:59 skydns: failure to forward request "read udp
> 192.168.122.1:53: i/o timeout" 2016/04/18 12:48:03 skydns: failure to forward request "read udp 192.168.122.1:53: i/o timeout"
After looking further, I just realized what is a 192.168.122.1? It is virtual switch on kvm. Why is skydns trying to hit my virtual switch or dns server of virtual machine?

Answer 1

SkyDNS defaults its forwarding nameservers to the one listed in /etc/resolv.conf. Since SkyDNS runs inside the kube-dns pod as a cluster addon, it inherits its /etc/resolv.conf from its host as described in the kube-dns doc.
From your question, it looks like your host's /etc/resolv.conf is configured to use 192.168.122.1 as its nameserver and hence that becomes the forwarding server in your SkyDNS config. I believe 192.168.122.1 is not routable from your Kubernetes cluster and that's why you are seeing "failure to forward request" errors in the kube-dns logs.
The simplest solution to this problem is to supply a reachable DNS server as a flag to SkyDNS in your RC config. Here is an example (it's just your RC config, but adds the -nameservers flag in the SkyDNS container spec):
apiVersion: v1
kind: ReplicationController
metadata:
name: kube-dns-v11
namespace: kube-system
labels:
k8s-app: kube-dns
version: v11
kubernetes.io/cluster-service: "true"
spec:
replicas: 1
selector:
k8s-app: kube-dns
version: v11
template:
metadata:
labels:
k8s-app: kube-dns
version: v11
kubernetes.io/cluster-service: "true"
spec:
containers:
- name: etcd
image: gcr.io/google_containers/etcd-amd64:2.2.1
# resources:
# # TODO: Set memory limits when we've profiled the container for large
# # clusters, then set request = limit to keep this container in
# # guaranteed class. Currently, this container falls into the
# # "burstable" category so the kubelet doesn't backoff from restarting it.
# limits:
# cpu: 100m
# memory: 500Mi
# requests:
# cpu: 100m
# memory: 50Mi
command:
- /usr/local/bin/etcd
- -data-dir
- /var/etcd/data
- -listen-client-urls
- http://127.0.0.1:2379,http://127.0.0.1:4001
- -advertise-client-urls
- http://127.0.0.1:2379,http://127.0.0.1:4001
- -initial-cluster-token
- skydns-etcd
volumeMounts:
- name: etcd-storage
mountPath: /var/etcd/data
- name: kube2sky
image: gcr.io/google_containers/kube2sky:1.14
# resources:
# # TODO: Set memory limits when we've profiled the container for large
# # clusters, then set request = limit to keep this container in
# # guaranteed class. Currently, this container falls into the
# # "burstable" category so the kubelet doesn't backoff from restarting it.
# limits:
# cpu: 100m
# # Kube2sky watches all pods.
# memory: 200Mi
# requests:
# cpu: 100m
# memory: 50Mi
livenessProbe:
httpGet:
path: /healthz
port: 8080
scheme: HTTP
initialDelaySeconds: 60
timeoutSeconds: 5
# successThreshold: 1
# failureThreshold: 5
readinessProbe:
httpGet:
path: /readiness
port: 8081
scheme: HTTP
# we poll on pod startup for the Kubernetes master service and
# only setup the /readiness HTTP server once that's available.
initialDelaySeconds: 30
timeoutSeconds: 5
args:
# command = "/kube2sky"
- --domain=cluster.local
- name: skydns
image: gcr.io/google_containers/skydns:2015-10-13-8c72f8c
resources:
# TODO: Set memory limits when we've profiled the container for large
# clusters, then set request = limit to keep this container in
# guaranteed class. Currently, this container falls into the
# "burstable" category so the kubelet doesn't backoff from restarting it.
limits:
cpu: 100m
memory: 200Mi
requests:
cpu: 100m
memory: 50Mi
args:
# command = "/skydns"
- -machines=http://127.0.0.1:4001
- -addr=0.0.0.0:53
- -ns-rotate=false
- -domain=cluster.local
- -nameservers=8.8.8.8:53,8.8.4.4:53 # Adding this flag. Dont use double quotes.
ports:
- containerPort: 53
name: dns
protocol: UDP
- containerPort: 53
name: dns-tcp
protocol: TCP
- name: healthz
image: gcr.io/google_containers/exechealthz:1.0
# resources:
# # keep request = limit to keep this container in guaranteed class
# limits:
# cpu: 10m
# memory: 20Mi
# requests:
# cpu: 10m
# memory: 20Mi
args:
- -cmd=nslookup kubernetes.default.svc.cluster.local 127.0.0.1 >/dev/null
- -port=8080
ports:
- containerPort: 8080
protocol: TCP
volumes:
- name: etcd-storage
emptyDir: {}
dnsPolicy: Default # Don't use cluster DNS.