I have faced an issue with resolving services host between with kubernetes on minikube.
So from the inside the pod I cannot wget web-server:8081/endpoint. But I can access the same server directly by IP address like this wget 10.0.0.81:8081/endpoint.
After troubleshooting the issue I have found that inside of the pod /etc/resolve.conf file the nameserver is set to 10.96.0.10. Here is how it looks:
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
When the Cluster IP of kube-dns service is 10.0.0.10.
After changing manually the nameserver on the pod to 10.0.0.10 I cat do wget web-server:8081/endpoint.
Why is it set to the wrong IP address and how fix it?
The problem was that I updated minikube without minikube delete.
After minikube delete and minikube start the DNS service is getting the IP address 10.96.0.10 the same as is set in the pod's /etc/resolve.conf.
Related
I am trying to access a host that sits in another server (but on my network) from inside the pod of deployment and I am using microk8s.
The thing is that on the server where I have microk8s installed I can easily ping it by ping my-network-host.qa.local. But when I go inside the pod with microk8s kubectl exec -it pod_name -- /bin/bash and I do ping my-network-host.qa.local it says: Name or service not known.
And when I connect to a VPN on my computer to be on that network and I deploy it locally using docker-desktop kubernetes I can ping that host from within the pod. So I think the problem sits in microk8s which is not letting my pod use my network.
Is there any way to tell microk8s to use my hosts from my network?
p.s. I can ping the ip of that server from the pod, but I am not being able to ping the host from the pod
Based on another answer that I found on StackOverflow, i managed to fixed it.
There were 2 changes needed to make it work:
Update kubelet configuration to use resolv-conf:
sudo echo "--resolv-conf=/run/systemd/resolve/resolv.conf" >> /var/snap/microk8s/current/args/kubelet
Restart kubelet service:
sudo service snap.microk8s.daemon-kubelet restart
Then change the CoreDNS forward to point to your nameserver:
First open coredns config map so you can edit it
sudo microk8s.kubectl edit configmap coredns -n kube-system
and update the file at
forward . 8.8.8.8 8.8.4.4 #REMOVE THIS LINE
forward . xxx.xxx.xxx.xxx #ADD THIS WITH YOUR IP
You can get eth0 DNS address:
nmcli dev show 2>/dev/null | grep DNS | sed 's/^.*:\s*//'
On my case I already had the ip as nameserver on /run/systemd/resolve/resolv.conf.
Now just save the changes, and go inside your pods so you will be able to access them.
There is a comment in another post that suggest adding
forward . /etc/resolv.conf
But that didnt work on my case.
I am on ubuntu, and I am running a docker default bridge network. I have containerized versions of zookeeper, kafka, and an app that I wrote that talks to kafka.
I do a:
docker exec -it <my-app id> /bin/bash
Then inside my app's container I run nslookup kafka
/go # nslookup schmafka
nslookup: can't resolve '(null)': Name does not resolve
Name: schmafka
Address 1: 172.20.0.8 docker_kafka_1.docker_default
I do not understand why I get the output "can't resolve '(null)'" and then I get the expected ip address printed out later. I tried to google nslookup and this output message but I cannot figure why this happens.
My /etc/resolv.conf file looks like this:
/go # cat /etc/resolv.conf
search valhalla.local valhalla v
nameserver 127.0.0.11
options ndots:0
This is a bug/oddity in nslookup. The "can't resolve" message is actually about the DNS server in use, not the site you are trying to look up.
For example this query (which tells nslookup to lookup google.com using the 8.8.8.8 DNS server) has no error message:
nslookup google.com 8.8.8.8
Server: 8.8.8.8
Address 1: 8.8.8.8 dns.google
Name: google.com
Address 1: 172.217.164.110 sfo03s18-in-f14.1e100.net
Address 2: 2607:f8b0:4005:80b::200e sfo03s18-in-x0e.1e100.net
But this query (in which the DNS server is "null") does show the "error":
UAP-AC-LR1-BZ.v4.0.42# nslookup google.com
nslookup: can't resolve '(null)': Name does not resolve
Name: google.com
Address 1: 172.217.164.110 sfo03s18-in-f14.1e100.net
Address 2: 2607:f8b0:4005:80b::200e sfo03s18-in-x0e.1e100.net
Admittedly this is misleading/confusing, and really should be fixed in nslookup.
I try create docker container with custom network and dos settings.
docker network create --driver=bridge --opt "com.docker.network.bridge.enable_ip_masquerade"="true" --opt "com.docker.network.bridge.enable_icc"="true" --opt="com.docker.network.driver.mtu"="1500" --opt="com.docker.network.bridge.host_binding_ipv4"="0.0.0.0" net
--
docker run --dns 10.0.0.2 --network=net busybox cat /etc/resolv.conf
nameserver 127.0.0.11
options ndots:0
Else if I use standard network all work fine
docker run --dns 10.0.0.2 --network=bridge busybox cat /etc/resolv.conf
nameserver 10.0.0.2
As of Docker 1.10, DNS is managed differently for user-defined networks. DNS for the default bridge network is unchanged for backwards compatibility. In a user-defined network, docker daemon uses the embedded DNS server. According to the documentation found here:
https://docs.docker.com/engine/userguide/networking/configure-dns/
--dns=[IP_ADDRESS...] The IP addresses passed via the --dns option is used by the embedded
DNS server to forward the DNS query if embedded DNS server is unable
to resolve a name resolution request from the containers. These
--dns IP addresses are managed by the embedded DNS server and will not
be updated in the container’s /etc/resolv.conf file.
So, the DNS nameserver will be used, it just is not visible in the container's /etc/resolv.conf.
I'm using Kubernetes v1.0.6 on AWS that has been deployed using kube-up.sh.
Cluster is using kube-dns.
$ kubectl get svc kube-dns --namespace=kube-system
NAME LABELS SELECTOR IP(S) PORT(S)
kube-dns k8s-app=kube-dns,kubernetes.io/cluster-service=true,kubernetes.io/name=KubeDNS k8s-app=kube-dns 10.0.0.10 53/UDP
Which works fine.
$ kubectl exec busybox -- nslookup kubernetes.default
Server: 10.0.0.10
Address 1: 10.0.0.10 ip-10-0-0-10.eu-west-1.compute.internal
Name: kubernetes.default
Address 1: 10.0.0.1 ip-10-0-0-1.eu-west-1.compute.internal
This is the resolv.conf of a pod.
$ kubectl exec busybox -- cat /etc/resolv.conf
nameserver 10.0.0.10
nameserver 172.20.0.2
search default.svc.cluster.local svc.cluster.local cluster.local eu-west-1.compute.internal
Is it possible to have the containers use an additional nameserver?
I have a secondary DNS based service discovery Oon let's say 192.168.0.1) that I would like my kubernetes containers to be able to use for dns resolution.
ps. A kubernetes 1.1 solution would also be acceptable :)
Thank you very much in advance,
George
The DNS addon README has some details on this. Basically, the pod will inherit the resolv.conf setting of the node it is running on, so you could add your extra DNS server to the nodes' /etc/resolv.conf. The kubelet also takes a --resolv-conf argument that may provide a more explicit way for you to inject the extra DNS server. I don't see that flag documented anywhere yet, however.
In Kuberenetes (probably) 1.2 we'll be moving to a model where nameservers are assumed to be fungible. There are too many resolvers that break when different nameservers serve different subsets of DNS, and there is no real specification here that we can point to.
In other words, we'll start dropping the host's nameserver records from the container's merged resolv.conf and making our own DNS server the only nameserver line. Our DNS will be able to forward requests to upstream nameservers.
I eventually managed to solve this pretty easily by configuring SkyDNS to add an additional nameserver, you can just add the environmental variable SKYDNS_NAMESERVERS as defined in the SkyDNS docs in your SkyDNS replication controller. It has minimal impact and does not depend on node changes etc.
env:
- name: SKYDNS_NAMESERVERS
value: 10.0.0.254:53,10.0.64.254:53
For those usign Kubernetes kube-dns, flag -nameservers nor environment variable SKYDNS_NAMESERVERS are no longer avaiable.
Usage of /kube-dns:
--alsologtostderr log to standard error as well as files
--config-map string config-map name. If empty, then the config-map will not used. Cannot be used in conjunction with federations flag. config-map contains dynamically adjustable configuration.
--config-map-namespace string namespace for the config-map (default "kube-system")
--dns-bind-address string address on which to serve DNS requests. (default "0.0.0.0")
--dns-port int port on which to serve DNS requests. (default 53)
--domain string domain under which to create names (default "cluster.local.")
--healthz-port int port on which to serve a kube-dns HTTP readiness probe. (default 8081)
--kube-master-url string URL to reach kubernetes master. Env variables in this flag will be expanded.
--kubecfg-file string Location of kubecfg file for access to kubernetes master service; --kube-master-url overrides the URL part of this; if neither this nor --kube-master-url are provided, defaults to service account tokens
--log-backtrace-at traceLocation when logging hits line file:N, emit a stack trace (default :0)
--log-dir string If non-empty, write log files in this directory
--log-flush-frequency duration Maximum number of seconds between log flushes (default 5s)
--logtostderr log to standard error instead of files (default true)
--stderrthreshold severity logs at or above this threshold go to stderr (default 2)
-v, --v Level log level for V logs
--version version[=true] Print version information and quit
--vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging
Now, either you put your name servers on the hosts resolv.conf, so DNS is inherited from the node, or you use custom resolv.conf and add it to Kubelet with the flag --resolv-conf as explained here
You need to know the IP of your Core DNS to set it as a secondary DNS
Run this command to get the CoreDNS IP:
kubectl -n kube-system get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 172.20.0.10 <none> 53/UDP,53/TCP 43d
metrics-server ClusterIP 172.20.232.147 <none> 443/TCP 43d
This is how I setup DNS in my deployment yaml.
I posted the Google DNS IP (for clarity) and my CoreDNS ip, but you should use your VPC DNS and your CoreDNS server.
containers:
- name: nginx
image: nginx
ports:
- containerPort: 8080
dnsPolicy: None
dnsConfig:
nameservers:
- 8.8.8.8
- 172.20.0.10
searches:
- 1b.svc.cluster.local
- svc.cluster.local
- cluster.local
- ec2.internal
options:
- name: ndots
value: "5"
On host machine, it's very fast to lookup a domain. But inside docker container, it's much
slower and sometimes timeout.
The host machine is a virtual host, and it's dns server address is 127.0.0.1 (weird but true). So I've tried to modify /etc/resolv.conf inside container and set the dns server to be 172.x (host's address). As a result, I didn't see any good effect.
I've also tried to set the container's dns server to be a self-built one (101.x), but still, it's slow to look up a domain. Another weird thing is that ping 101.x is very fast.
I'm confused about this phenomenon, anyone can explain and help?
I am not sure of why resolving DNS is slow in the containers, but I have procedure which I follow to resolve the DNS in the docker containers.
To verify DNS resolution issue:
# docker run busybox nslookup google.com
Server: 8.8.8.8
Address 1: 8.8.8.8
nslookup: can't resolve 'google.com'
Find out the DNS server used in your machine :
# nm-tool |grep DNS
DNS: 172.24.100.50
DNS: 10.1.100.50
Run it again using DNS IP found in the above step which resolves the DNS issue:
# docker run --dns 172.24.100.50 busybox nslookup google.com
Server: 172.24.100.50
Address 1: 172.24.100.50 indc01.radisys.com
Name: google.com
Address 1: 2607:f8b0:4009:80c::200e ord36s01-in-x0e.1e100.net
Address 2: 172.217.4.110 ord36s04-in-f14.1e100.net
To resolve it permanently add the following content as below to a new file:
root#labadmin-VirtualBox:/home/labadmin# cat /etc/docker/daemon.json
{
"dns" : ["172.24.100.50", "8.8.8.8"]
}
More info on Docker DNS configuration.
Restart the docker service and verify it again:
# docker run busybox nslookup google.com
Server: 172.24.100.50
Address 1: 172.24.100.50 indc01.radisys.com
Name: google.com
Address 1: 2607:f8b0:4009:801::200e ord30s31-in-x0e.1e100.net
Address 2: 172.217.4.238 ord30s31-in-f14.1e100.net
Check it by running the container:
# docker run -it e02e811dd08f
/ # ping google.com
PING google.com (172.217.4.238): 56 data bytes
64 bytes from 172.217.4.238: seq=0 ttl=47 time=251.506 ms
64 bytes from 172.217.4.238: seq=1 ttl=47 time=245.621 ms
Hope this helps.