I have created a kubernetes v1.2 running in Azure cloud with one master(Master) and two nodes(Node1 and Node2). I have deployed an Nginx and Tomcat application. Both the containers are deployed in individual pods with RC and they have a SERVICE for each.
Nginx pod is deployed in the Node1 and Tomcat pod is deployed in Node2. Now Nginx from Node1 is trying to access Tomcat via tomcat's ServiceIP(clusterIP) which is in Node2. But its unreachable.
Nginx serviceIP: 10.16.0.2 Node1
Tomcat serviceIP: 10.16.0.4 Node2
I tried curl 10.16.0.4:8080 from Node2, it works. But same from Node1 fails with curl: (52) Empty reply from server
So communication to serviceIP across nodes fails. Is this the problem with kube v1.2?
Note: ClusterIP for the Service will be specified at the time of creating the service.
Since you are able to reach the cluster ip from the Node2, it looks like the service selector is properly defined.
Kube-proxy is the component that watches the services and creates iptable rules for end points. I would check if kube-proxy is running properly on Node1. Then check if iptable rules are set properly for the cluster ip you are trying to reach.
You can see these with iptables -L -t nat | grep namespace/servicename
Here is an example:
bash-4.3# iptables -L -t nat | grep kube-system/heapster
KUBE-MARK-MASQ all -- 172.168.16.182 anywhere /* kube-system/heapster: */
DNAT tcp -- anywhere anywhere /* kube-system/heapster: */ tcp to:172.168.16.182:8082
KUBE-SVC-BJM46V3U5RZHCFRZ tcp -- anywhere 192.168.172.66 /* kube-system/heapster: cluster IP */ tcp dpt:http
KUBE-SEP-KNJP5BBKUOCH7NDB all -- anywhere anywhere /* kube-system/heapster: */
In this example I looked up heapster running in kube-system namespace. It is showing that the cluster ip is 192.168.172.66 DNATs to the endpoint 172.168.16.182, which is pods ip (You should cross check this with the endpoints listed in kubectl describe service.
If is it not there, restarting kube-proxy might help.
Related
I am trying to access a host that sits in another server (but on my network) from inside the pod of deployment and I am using microk8s.
The thing is that on the server where I have microk8s installed I can easily ping it by ping my-network-host.qa.local. But when I go inside the pod with microk8s kubectl exec -it pod_name -- /bin/bash and I do ping my-network-host.qa.local it says: Name or service not known.
And when I connect to a VPN on my computer to be on that network and I deploy it locally using docker-desktop kubernetes I can ping that host from within the pod. So I think the problem sits in microk8s which is not letting my pod use my network.
Is there any way to tell microk8s to use my hosts from my network?
p.s. I can ping the ip of that server from the pod, but I am not being able to ping the host from the pod
Based on another answer that I found on StackOverflow, i managed to fixed it.
There were 2 changes needed to make it work:
Update kubelet configuration to use resolv-conf:
sudo echo "--resolv-conf=/run/systemd/resolve/resolv.conf" >> /var/snap/microk8s/current/args/kubelet
Restart kubelet service:
sudo service snap.microk8s.daemon-kubelet restart
Then change the CoreDNS forward to point to your nameserver:
First open coredns config map so you can edit it
sudo microk8s.kubectl edit configmap coredns -n kube-system
and update the file at
forward . 8.8.8.8 8.8.4.4 #REMOVE THIS LINE
forward . xxx.xxx.xxx.xxx #ADD THIS WITH YOUR IP
You can get eth0 DNS address:
nmcli dev show 2>/dev/null | grep DNS | sed 's/^.*:\s*//'
On my case I already had the ip as nameserver on /run/systemd/resolve/resolv.conf.
Now just save the changes, and go inside your pods so you will be able to access them.
There is a comment in another post that suggest adding
forward . /etc/resolv.conf
But that didnt work on my case.
We are trying to implement something like bump in the wire sceario with the kubernetes pod running as bump.
so basically we have traffic for a paticular destination we intercept the traffic in kubernetes pod and throw it again towards destination .
To run this scenario we started of with squid proxy which would redirect the outgoing traffic to kubernetes pod(bump) and once inside pod we would run iptables rule to redirect the processed packets again to destination ip.
The iptables rule used : iptables -t nat -A PREROUTING -p tcp --dport 3124 -j DNAT --to-destination 1.1.1.1
however this iptable rule does not work(also i need the returning traffic to be redirected to original destination ie after getting processed in pod)
so say destination is 172.172.172.172(some ip outside the cluster) squid has been setup to redirect this destination traffic to service which has node port enabled(hence new destination is minionip:nodeport). when the traffic reaches the pod.. after processing the data outgoing traffic has to be set to original destination(172.172.172.172) instead of going back
I cant seem to find a way. is there any iptable rule to do so taking in consideration of kubertes setup iptables rules.
also is there any other elegant way to do so.
please help.
I have an unsecured Postfix instance in a container that listens to port 25. This port is not exposed using a Service. The idea is that only a PHP container that runs inside the same pod should be able to connect to Postfix and there is no need for additional Postfix configuration .
Is there any way for other processes that run in the same network or Kubernetes cluster to connect to this hidden port?
From what I know, only other containers in the same Pod can connect to an unexposed port, via localhost.
I'm interested from a security point of view.
P.S. I now that one should make sure it has multiple levels of security in place but I'm interested only theoretically if there is some way to connect to this port from outside the pod.
From what I know, only other containers in the same Pod can connect to an unexposed port, via localhost.
Not exactly.
How this is implemented is a detail of the particular container runtime in use.
...I'm interested only theoretically if there is some way to connect to this port from outside the pod.
So here we go :)
For example on GKE you can easily access Pod from other Pod if you know Target Pod's IP.
I have used the following setup on GKE:
apiVersion: v1
kind: Pod
metadata:
annotations:
run: fake-web
name: fake-default-knp
spec:
containers:
- image: mendhak/http-https-echo
imagePullPolicy: IfNotPresent
name: fake-web
The Docker file for that image can be found here.
It specifies EXPOSE 80 443
So, container listens on these 2 Ports.
$kubectl exec fake-default-knp -- netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 :::443 :::* LISTEN 1/node
tcp 0 0 :::80 :::* LISTEN 1/node
I have no services:
$kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 40d
and only 2 Pods.
$kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP
busybox-sleep-less 1/1 Running 3476 40d 10.52.1.6
fake-default-knp 1/1 Running 0 13s 10.52.0.50
And I can connect to
$kubectl exec busybox-sleep-less -- telnet 10.52.0.50 80
Connected to 10.52.0.50
$kubectl exec busybox-sleep-less -- telnet 10.52.0.50 443
Connected to 10.52.0.50
As you can see, container is accessible on POD_IP:container_port from other pod (located on another node)
P.S> It worth checking "Inter-process communications (IPC)" if you really would like to continue using unsecured Postfix and prefer avoiding "unauthorized access from outside of Pod". It is described here.
Hope that helps!
Edit 30-Jan-2020
I decided to play with it a little bit. Technically, you can achieve what you want with the help of iptables. You need to specifically ACCEPT all traffic from localhost on port25 and DROP from everywhere else.
something like:
cat iptab.txt
# Generated by xtables-save v1.8.2 on Thu Jan 30 16:37:27 2020
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -s 127.0.0.1/32 -p 6 -m tcp --dport 80 -j ACCEPT
-A INPUT -p 6 -m tcp --dport 80 -j DROP
COMMIT
I've tested it and can't telnet on port 80 from anywhere except that very Pod. Please note that I had to run my container in privileged mode in order to be able editing iptables rules directly from Pod. But that is going beyond initial question. :)
Yes, you can use kubectl port-forward to set up a tunnel directly to it for testing purposes.
I have an OpenShift, cluster, and periodically when accessing logs, I get:
worker1-sass-on-prem-origin-3-10 on 10.1.176.130:53: no such host" kube doing a connection to 53 on a node.
I also tend to see tcp: lookup postgres.myapp.svc.cluster.local on 10.1.176.136:53: no such host errors from time to time in pods, again, this makes me think that, when accessing internal service endpoints, pods, clients, and other Kubernetes related services actually talk to a DNS server that is assumed to be running on the given node that said pods are running on.
Update
Looking into one of my pods on a given node, I found the following in resolv.conf (I had to ssh and run docker exec to get this output - since oc exec isn't working due to this issue).
/etc/cfssl $ cat /etc/resolv.conf
nameserver 10.1.176.129
search jim-emea-test.svc.cluster.local svc.cluster.local cluster.local bds-ad.lc opssight.internal
options ndots:5
Thus, it appears that in my cluster, containers have a self-referential resolv.conf entry. This cluster is created with openshift-ansible. I'm not sure if this is infra-specific, or if its actually a fundamental aspect of how openshift nodes work, but i suspect the latter, as I haven't done any major customizations to my ansible workflow from the upstream openshift-ansible recipes.
Yes, DNS on every node is normal in openshift.
It does appear that its normal for an openshift ansible deployment to deploy dnsmasq services on every node.
Details.
As an example of how this can effect things, the following https://github.com/openshift/openshift-ansible/pull/8187 is instructive. In any case, if a local node's dnsmasq is acting flakey for any reason, it will prevent containers running on that node from properly resolving addresses of other containers in a cluster.
Looking deeper at the dnsmasq 'smoking gun'
After checking on an individual node, I found that in fact, there was a process indeed bounded to port 53, and it is dnsmasq. Hence,
[enguser#worker0-sass-on-prem-origin-3-10 ~]$ sudo netstat -tupln | grep 53
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 675/openshift
And, dnsmasq is running locally:
[enguser#worker0-sass-on-prem-origin-3-10 ~]$ ps -ax | grep dnsmasq
4968 pts/0 S+ 0:00 grep --color=auto dnsmasq
6994 ? Ss 0:22 /usr/sbin/dnsmasq -k
[enguser#worker0-sass-on-prem-origin-3-10 ~]$ sudo ps -ax | grep dnsmasq
4976 pts/0 S+ 0:00 grep --color=auto dnsmasq
6994 ? Ss 0:22 /usr/sbin/dnsmasq -k
The final clue, resolv.conf itself is even adding the local IP address as a nameserver... And this is obviously borrowed into containers that start.
nameserver updated by /etc/NetworkManager/dispatcher.d/99-origin-dns.sh
Generated by NetworkManager
search cluster.local bds-ad.lc opssight.internal
NOTE: the libc resolver may not support more than 3 nameservers.
The nameservers listed below may not be recognized.
nameserver 10.1.176.129
The solution (in my specific case)
In my case , this was happening because the local nameserver was using an ifcfg (you can see these files in /etc/sysconfig/network-scripts/) with
[enguser#worker0-sass-on-prem-origin-3-10 network-scripts]$ cat ifcfg-ens192
TYPE=Ethernet
BOOTPROTO=dhcp
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens192
UUID=50936212-cb5e-41ff-bec8-45b72b014c8c
DEVICE=ens192
ONBOOT=yes
However, my internally configured Virtual Machines could not resolve IPs provided to them by the PEERDNS records.
Ultimately the fix was to work with our IT department to make sure our authoritative domain for our kube clusters had access to all IP addresses in our data center.
The Generic Fix to :53 lookup errors...
If youre seeing the :53 record errors are coming up when you try to kubectl or oc logs / exec, then there is likely that your apiserver is not able to connect with kubelets via their IP address.
If youre seeing :53 record errors in other places, for example, inside of pods, then this is because your pod, using its own local DNS, isnt able to resolve internal cluster IP addresses. This might simply be because you have an outdated controller that is looking for services that don't exist anymore, or else, you have flakiness at your kubernetes dns implementation level.
It appears that local-up-cluster in kubernetes, on ubuntu, isn't able to resolve DNS queries when relying on cluster DNS.
setup
I'm running an ubuntu box, with environmental variables for DNS set in local-up-cluster:
# env | grep KUBE
KUBE_ENABLE_CLUSTER_DNS=true
KUBE_DNS_SERVER_IP=172.17.0.1
running information
sky-dns seems happy:
I0615 00:04:13.563037 1 server.go:198] Skydns metrics enabled (/metrics:10055)
I0615 00:04:13.563051 1 dns.go:147] Starting endpointsController
I0615 00:04:13.563054 1 dns.go:150] Starting serviceController
I0615 00:04:13.563125 1 logs.go:41] skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053 [rcache 0]
I0615 00:04:13.563141 1 logs.go:41] skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053 [rcache 0]
I0615 00:04:13.589840 1 dns.go:264] New service: kubernetes
I0615 00:04:13.589971 1 dns.go:462] Added SRV record &{Host:kubernetes.default.svc.cluster.local. Port:443 Priority:10 Weight:10 Text: Mail:false Ttl:30 TargetStrip:0 Group: Key:}
I0615 00:04:14.063246 1 dns.go:171] Initialized services and endpoints from apiserver
I0615 00:04:14.063267 1 server.go:129] Setting up Healthz Handler (/readiness)
I0615 00:04:14.063274 1 server.go:134] Setting up cache handler (/cache)
I0615 00:04:14.063288 1 server.go:120] Status HTTP port 8081
kube-proxy seems happy:
I0615 00:03:53.448369 5706 proxier.go:864] Setting endpoints for "default/kubernetes:https" to [172.31.44.133:6443]
I0615 00:03:53.545124 5706 controller_utils.go:1001] Caches are synced for service config controller
I0615 00:03:53.545146 5706 config.go:210] Calling handler.OnServiceSynced()
I0615 00:03:53.545208 5706 proxier.go:979] Not syncing iptables until Services and Endpoints have been received from master
I0615 00:03:53.545125 5706 controller_utils.go:1001] Caches are synced for endpoints config controller
I0615 00:03:53.545224 5706 config.go:110] Calling handler.OnEndpointsSynced()
I0615 00:03:53.545274 5706 proxier.go:309] Adding new service port "default/kubernetes:https" at 10.0.0.1:443/TCP
I0615 00:03:53.545329 5706 proxier.go:991] Syncing iptables rules
I0615 00:03:53.993514 5706 proxier.go:991] Syncing iptables rules
I0615 00:03:54.008738 5706 bounded_frequency_runner.go:221] sync-runner: ran, next possible in 0s, periodic in 30s
I0615 00:04:24.008904 5706 proxier.go:991] Syncing iptables rules
I0615 00:04:24.023057 5706 bounded_frequency_runner.go:221] sync-runner: ran, next possible in 0s, periodic in 30s
result
However, I don't seem to be able to resolve anything inside the cluster, same result with docker exec or kube exec:
➜ kubernetes git:(master) kc exec --namespace=kube-system kube-dns-2673147055-4j6wm -- nslookup kubernetes.default.svc.cluster.local
Defaulting container name to kubedns.
Use 'kubectl describe pod/kube-dns-2673147055-4j6wm' to see all of the
containers in this pod.
nslookup: can't resolve '(null)': Name does not resolve
nslookup: can't resolve 'kubernetes.default.svc.cluster.local': Name does not resolve
question
Whats the simplest way to further debug a system created using local-up-cluster where the DNS pods are running, but kubernetes.default.svc.cluster.local is not resolved ? Note that all other aspects of this cluster appear to be working perfectly.
System info : Linux ip-172-31-44-133 4.4.0-1018-aws #27-Ubuntu SMP Fri May 19 17:20:58 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux.
Example of resolv.conf that is being placed in my containers...
/etc/cfssl # cat /etc/resolv.conf
nameserver 172.17.0.1
search default.svc.cluster.local svc.cluster.local cluster.local dc1.lan
options ndots:5
I can't comment on your post so I'll attempt to answer this.
First of all, certain Alpine images have trouble resolving using nslookup. DNS might in fact be working normally in your cluster.
To validate this, read the logs of the pods (eg. traefik, heapster, calico) that communicates with kube-apiserver. If no errors are observed, what you have is probably a non-problem.
If you want to be doubly-sure, deploy a non-Alpine pod and try nslookup.
If it really is a DNS issue, I will debug in this sequence.
kubectl exec into the kube-dns pod. Run nslookup kubernetes.default.svc.cluster.local localhost. If this works, DNS is in fact running. If it doesn't, kube-dns should have entered a CrashLoopbackOff state by now.
kubectl exec into a deployed pod. Run nslookup kubernetes.default.svc.cluster.local <cluster-ip>. If this works, you're good to go. If it doesn't, something is up with the pod network. Without details, I can't recommend further steps.
Bonne chance!
I figured I'd post a systematic answer that usually works for me. I was hoping for something more elegant, and this isn't ideal, but I think its the best place to start.
1) Make sure your DNS nanny and your SkyDNS are running. The nanny and Sky DNS should both show in their docker logs that they've bound to a port.
2) When you create new services, make sure that SkyDNS is writing them to the logs and showing the creation of SRV and so on.
3) Look in /etc/resolv.conf in your docker containers. Make sure the nameserver looks like something on your internal docker IP addresses (i.e. 10.... in a regular docker0 config on fedora)
There are specific env variables you need to export correctly: API_HOST=true and KUBE_ENABLE_CLUSTER_DNS=true.
Theres alot of deeper tools you can use, like route -n and so on to debug container networking even more, but local up cluster should generally 'just work' and if the above steps surface something supsicious, its worth mentioning in the kubernetes community as a possible bug.