KubeDNS not injecting nameservers, Kubernetes 1.5.2 on RHEL 7- - dns

Question

I've created a cluster with a master and 5 nodes with flannel for the POD network and that is working fine.
What is not working is that after installing kubeDNS (kubedns, dnsmasq and sidecar) but I can't get the new nameserver to be injected into HOST /etc/resolv.conf, because of that I can't resolve any hostnames.
Everything else works fine, all KubeDNS containers are running and with no errors
My kube-proxy ARGS
KUBE_PROXY_ARGS="--cluster-cidr=10.254.0.0/16"
My Kubelet configs
KUBELET_DNS="--cluster-dns=10.254.0.253"
KUBELET_DOMAIN="--cluster-domain=cluster.local"
Here are my configs for the DNS POD:
{
"kind": "Pod",
"apiVersion": "v1",
"metadata": {
"name": "kube-dns-4073989832-f7g5g",
"generateName": "kube-dns-4073989832-",
"namespace": "kube-system",
"selfLink": "/api/v1/namespaces/kube-system/pods/kube-dns-4073989832-f7g5g",
"uid": "6f76055c-5b1e-11e7-b0c5-0050568fc023",
"resourceVersion": "3974782",
"creationTimestamp": "2017-06-27T09:53:13Z",
"labels": {
"k8s-app": "kube-dns",
"pod-template-hash": "4073989832"
},
"annotations": {
"kubernetes.io/created-by": "{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicaSet\",\"namespace\":\"kube-system\",\"name\":\"kube-dns-4073989832\",\"uid\":\"8afa7fce-5a9e-11e7-b714-0050568fc023\",\"apiVersion\":\"extensions\",\"resourceVersion\":\"3974404\"}}\n",
"scheduler.alpha.kubernetes.io/critical-pod": ""
},
"ownerReferences": [
{
"apiVersion": "extensions/v1beta1",
"kind": "ReplicaSet",
"name": "kube-dns-4073989832",
"uid": "8afa7fce-5a9e-11e7-b714-0050568fc023",
"controller": true
}
]
},
"spec": {
"volumes": [
{
"name": "kube-dns-config",
"configMap": {
"name": "kube-dns",
"defaultMode": 420
}
}
],
"containers": [
{
"name": "kubedns",
"image": "vvcelparti01:443/k8s-dns-kube-dns-amd64:1.14.2",
"args": [
"--domain=cluster.local",
"--dns-port=10053",
"--config-dir=/kube-dns-config",
"--kube-master-url=http://10.64.146.26:8080",
"--v=2"
],
"ports": [
{
"name": "dns-local",
"containerPort": 10053,
"protocol": "UDP"
},
{
"name": "dns-tcp-local",
"containerPort": 10053,
"protocol": "TCP"
},
{
"name": "metrics",
"containerPort": 10055,
"protocol": "TCP"
}
],
"env": [
{
"name": "PROMETHEUS_PORT",
"value": "10055"
}
],
"resources": {
"limits": {
"memory": "170Mi"
},
"requests": {
"cpu": "100m",
"memory": "70Mi"
}
},
"volumeMounts": [
{
"name": "kube-dns-config",
"mountPath": "/kube-dns-config"
}
],
"livenessProbe": {
"httpGet": {
"path": "/healthcheck/kubedns",
"port": 10054,
"scheme": "HTTP"
},
"initialDelaySeconds": 60,
"timeoutSeconds": 5,
"periodSeconds": 10,
"successThreshold": 1,
"failureThreshold": 5
},
"readinessProbe": {
"httpGet": {
"path": "/readiness",
"port": 8081,
"scheme": "HTTP"
},
"initialDelaySeconds": 3,
"timeoutSeconds": 5,
"periodSeconds": 10,
"successThreshold": 1,
"failureThreshold": 3
},
"terminationMessagePath": "/dev/termination-log",
"imagePullPolicy": "IfNotPresent"
},
{
"name": "dnsmasq",
"image": "vvcelparti01:443/k8s-dns-dnsmasq-nanny-amd64:1.14.2",
"args": [
"-v=2",
"-logtostderr",
"-configDir=/etc/k8s/dns/dnsmasq-nanny",
"-restartDnsmasq=true",
"--",
"-k",
"--cache-size=1000",
"--log-facility=-",
"--server=/cluster.local/127.0.0.1#10053",
"--server=/in-addr.arpa/127.0.0.1#10053",
"--server=/ip6.arpa/127.0.0.1#10053"
],
"ports": [
{
"name": "dns",
"containerPort": 53,
"protocol": "UDP"
},
{
"name": "dns-tcp",
"containerPort": 53,
"protocol": "TCP"
}
],
"resources": {
"requests": {
"cpu": "150m",
"memory": "20Mi"
}
},
"volumeMounts": [
{
"name": "kube-dns-config",
"mountPath": "/etc/k8s/dns/dnsmasq-nanny"
}
],
"livenessProbe": {
"httpGet": {
"path": "/healthcheck/dnsmasq",
"port": 10054,
"scheme": "HTTP"
},
"initialDelaySeconds": 60,
"timeoutSeconds": 5,
"periodSeconds": 10,
"successThreshold": 1,
"failureThreshold": 5
},
"terminationMessagePath": "/dev/termination-log",
"imagePullPolicy": "IfNotPresent"
},
{
"name": "sidecar",
"image": "vvcelparti01:443/k8s-dns-sidecar-amd64:1.14.2",
"args": [
"--v=2",
"--logtostderr",
"--probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local,5,A",
"--probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local,5,A"
],
"ports": [
{
"name": "metrics",
"containerPort": 10054,
"protocol": "TCP"
}
],
"resources": {
"requests": {
"cpu": "10m",
"memory": "20Mi"
}
},
"livenessProbe": {
"httpGet": {
"path": "/metrics",
"port": 10054,
"scheme": "HTTP"
},
"initialDelaySeconds": 60,
"timeoutSeconds": 5,
"periodSeconds": 10,
"successThreshold": 1,
"failureThreshold": 5
},
"terminationMessagePath": "/dev/termination-log",
"imagePullPolicy": "IfNotPresent"
}
],
"restartPolicy": "Always",
"terminationGracePeriodSeconds": 30,
"dnsPolicy": "Default",
"serviceAccountName": "kube-dns",
"serviceAccount": "kube-dns",
"nodeName": "gopher01",
"securityContext": {}
},
"status": {
"phase": "Running",
"conditions": [
{
"type": "Initialized",
"status": "True",
"lastProbeTime": null,
"lastTransitionTime": "2017-06-27T09:52:45Z"
},
{
"type": "Ready",
"status": "True",
"lastProbeTime": null,
"lastTransitionTime": "2017-06-27T09:52:55Z"
},
{
"type": "PodScheduled",
"status": "True",
"lastProbeTime": null,
"lastTransitionTime": "2017-06-27T09:53:13Z"
}
],
"hostIP": "10.64.146.24",
"podIP": "172.30.18.4",
"startTime": "2017-06-27T09:52:45Z",
"containerStatuses": [
{
"name": "dnsmasq",
"state": {
"running": {
"startedAt": "2017-06-27T09:52:47Z"
}
},
"lastState": {},
"ready": true,
"restartCount": 0,
"image": "vvcelparti01:443/k8s-dns-dnsmasq-nanny-amd64:1.14.2",
"imageID": "docker-pullable://vvcelparti01:443/k8s-dns-dnsmasq-nanny-amd64#sha256:5a9dda0fdf5bf548eb6a63260c3f5e6f5cdc3d0917279e38a435c00967c6c57c",
"containerID": "docker://682fa7e0ffb28f26aee97a8ac7fe564096ece3ef3d7fe14fd9ed6857526d2d2f"
},
{
"name": "kubedns",
"state": {
"running": {
"startedAt": "2017-06-27T09:52:47Z"
}
},
"lastState": {},
"ready": true,
"restartCount": 0,
"image": "vvcelparti01:443/k8s-dns-kube-dns-amd64:1.14.2",
"imageID": "docker-pullable://vvcelparti01:443/k8s-dns-kube-dns-amd64#sha256:c78ed83587e42e7fc21f07756364c568c5c0fe10289f4f7f19d03a97f15b7a60",
"containerID": "docker://20b729004655a43efd384f8dded1f97d898a3b54092e190aba3d2031e72da056"
},
{
"name": "sidecar",
"state": {
"running": {
"startedAt": "2017-06-27T09:52:47Z"
}
},
"lastState": {},
"ready": true,
"restartCount": 0,
"image": "vvcelparti01:443/k8s-dns-sidecar-amd64:1.14.2",
"imageID": "docker-pullable://vvcelparti01:443/k8s-dns-sidecar-amd64#sha256:8d8c0e03e5f91ae85be7402ac88f804c52431dac32491c7a2557fd462fd2695b",
"containerID": "docker://bbaec6e9d0aa933daaee7c33b6d64d0f37f1a57213fabd2aa1c686c61a356f7f"
}
]
}
}
Here is my throubleshooting session:
$ kubectl get svc --namespace=kube-system
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns 10.254.0.253 <none> 53/UDP,53/TCP 24d
kubernetes-dashboard 10.254.170.86 <none> 80/TCP 29d
$ kubectl get ep kube-dns --namespace=kube-system
NAME ENDPOINTS AGE
kube-dns 172.30.18.4:53,172.30.18.4:53 24d