When executing into a pod I get following message
Error from server: error dialing backend: dial tcp: lookup k8s-worker-node-a on 192.168.1.1:53: no such host
Is it possible to tell kubectl to use /etc/hosts to lookup machines as I have saved them inside there.
Can the API server resolve that hostname? It's insufficient for just your workstation to know the hostname, since the API server must be able to contact kubelet on the Node to construct the URL used to view the logs
Turning up the verbosity of kubectl will show the actual handshaking that goes on between your machine, the API server, and the kubelet on the Node, which is how we discovered a similar misconfiguration in our cluster.
Related
I'm trying to setup a cluster of one machine for now. I know that I can get the API server running and listening to some ports.
I am looking to issue commands against the master machine from my laptop.
KUBECONFIG=/home/slackware/kubeconfig_of_master kubectl get nodes should send a request to the master machine, hit the API server, and get a response of the running nodes.
However, I am hitting issues with permissions. One is similar to x509: certificate is valid for 10.61.164.153, not 10.0.0.1. Another is a 403 if I hit the kubectl proxy --port=8080 that is running on the master machine.
I think two solutions are possible, with a preferable one (B):
A. Add my laptop's ip address to the list of accepted ip addresses that API server or certificates or certificate agents holds. How would I do that? Is that something I can set in kubeadm init?
B. Add 127.0.0.1 to the list of accepted ip addresses that API server or certificates or certificate agents holds. How would I do that? Is that something I can set in kubeadm init?
I think B would be better, because I could create an ssh tunnel from my laptop to the remote machine and allow my teammates (if I ever have any) to do similarly.
Thank you,
Slackware
You shoud add --apiserver-cert-extra-sans 10.0.0.1 to your kubeadm init command.
Refer to https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#options
You should also use a config file:
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.16.2
apiServer:
certSANs:
- 10.0.0.1
You can find all relevant info here: https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2
We're using k8s 1.9.3 managed via kops 1.9.3 in AWS with Gossip based DNS using the weave cni network plugin.
I was doing a rolling-update of the master IG's to enable a some additional admission controllers. (PodNodeSelector and PodTolerationRestriction) I did this in two other clusters with no problems. When the cluster got to rolling the third master (we run our cluster in a 3 master setup) it brought down the instance and tried to bring up the new master instance but the new master instance failed to join the cluster. Upon further research and subsequent attempts to roll the third master to bring it into the cluster I found that the third, failing to join master, keeps trying to join the cluster as the old masters ip address. Even though it's ip address is something different. Watching a kubectl get nodes | grep master shows that the cluster thinks it's the old ip address and it fails because it's not that ip anymore. It seems that for some reason the cluster gossip based DNS is not getting notified about the new master's ip address.
This is causing problems because the kubernetes svc still has the old master's ip address in it, which is causing any api requests that get directed to that non-existent backend master to fail. It is also causing problems for etcd which keeps trying to contact it on the old ip address. Lots of logs like this:
018-10-29 22:25:43.326966 W | etcdserver: failed to reach the peerURL(http://etcd-events-f.internal.kops-prod.k8s.local:2381) of member 3b7c45b923efd852 (Get http://etcd-events-f.internal.kops-prod.k8s.local:2381/version: dial tcp 10.34.6.51:2381: i/o timeout)
2018-10-29 22:25:43.327088 W | etcdserver: cannot get the version of member 3b7c45b923efd852 (Get http://etcd-events-f.internal.kops-prod.k8s.local:2381/version: dial tcp 10.34.6.51:2381: i/o timeout)
One odd thing is that if I run etcdctl cluster-health on the available masters etcd instances they all show the unhealthy member id as f90faf39a4c5d077 but when I look at the etcd-events logs I see that it sees the unhealth member id as 3b7c45b923efd852. So there seems to be some inconsistency with etcd.
Since we are running in a three node master setup with one master down we don't want to restart any of the other masters to try to fix the problem because we're afraid to lose quorum on the etcd cluster.
We use weave 2.3.0 as our network CNI provider.
Noticed on the failing master that the weave cni config /etc/cni/net.d/10-weave.conf isn't getting created and the /etc/hosts files on the working masters isn't properly getting updated with the new master ip address. It seems like kube-proxy isn't getting the update for some reason.
Running the default debian 8 (jessie) image that is provided with kops 1.9.
How can we get the master to properly update DNS with it's new ip address?
My co-worker found that the fix was restarting the kube-dns and kube-dns-autoscaler pods. We're still not sure why they were failing to update dns with the new master ip but after restarting them adding the new master to the cluster worked fine.
I have successfully created a Kubernetes POD/Service using MiniKube on windows. But I would now like to ensure that DNS is working correctly.
The DNS service is shown as running
.\kubectl get pod -n kube-system
Which shows me the kube-dns pod is running
I also have the DNS add on shown as running
So I then want to verify that DNS is working, Ideally I want to test that PODs that have a service on top of them can lookup the service by DNS name.
But I started simple like this, where I get my running POD
So now that I have my POD name, I want to try do simple DNS lookup in it using the following commmand
.\kubectl exec simple-sswebapi-pod-v1-f7f8764b9-xs822 -- nslookup google.com
Where I am using the kubectl exec to try and run this nslookup in the POD that was found (running I should point out above).
But I get this error
Why would it not be able to find nslookup inside POD. All the key things seem to be ok
Kube-DNS pod is running (as shown above)
DNS AddOn is installed and running (as shown above)
What am I missing, is there something else I need to enable for DNS lookups to work inside my PODs?
To do it like this your container needs to include the command you want to use inside of the built image.
Sidenote: kubectl debug is coming to kube in near future https://github.com/kubernetes/kubernetes/issues/45922 which will help solve things like that by enabling you to attach a custom container to existing pod and debug in it
So more on this I installed busybox into a POD to allow me to use nslookup and this enabled me to do this
So this looks cool, but should I not be able to ping that service either by its IP address or by its DNS name which seems to be resolving just fine as shown above.
If I ping google.com inside busybox command prompt all is ok, but when I do a ping for either this IP address of this service of the DNS names, it never gets anywhere.
DNS lookup is clearly working. What am I missing?
I have a project at http://github.com/KPB-US/dispatch-display and in it I determine what [fire] station a display [client] belongs to based on it's ip address when the socket connects.
I have a docker composition set up so it can be up and running easily using docker-compose build and docker-compose up. This builds an nginx server that points to the upstream node application running on a node server.
Everything runs fine when I specify network_mode: "host" in both services in the docker-compose.yml. I'd rather have it run more securely using dockers default network mode-- but when I do that, I get NATed [I guess, see example message below] ip addresses from the clients instead of their actual ip addresses, so I don't know which stations [clients] they are coming from.
A connection attempt was made but that remote address is not
registered in the STATIONS list - 172.18.0.1
I am determining the incoming IP address based on the x-forwarded-for header and if missing, then the socket.conn.remoteAddress.
Any ideas on how I can resolve this? I'd like to keep the containers isolated from the network (as mentioned above) but yet still be able to know where the incoming connection is really coming from.
i'm trying to create restcomm cluster: sip-balancer + a few restcomm instances. But i can't connect restcomm node and sip-loadbalancer.
i used this tutorial - http://docs.telestax.com/sip-servlets-clustering-high-availability/ however haven't got any result.
Seems it should be 2 steps
change path-name attribute in
standalone/configuration/standalone-sip.xml
add org.mobicents.ha.javax.sip.BALANCERS to
standalone/configuration/mss-sip-stack.properties
as i understand node and loadbalancer use rmi as channel. i see(i used netstat) that server listens port 2000 and node establishes connection to it.
but when i try to use loadbalancer from sip client it returns "error 500 - no available nodes".
also i used remote debugged - nodes list is empty.
have i missed something?
p.s. i used docker restromm instance and sip-loadbalancer on the same machine.
thanks,
so i have found my issue.
According to the log file on restcomm node - it can't connect to balancer by RMI.
Connection error is very strange - Connection refused to host: 127.0.0.1 and sometimes Connection refused to host: 127.0.1.1
yesterday i tired to specify java.rmi.server.hostname but it did not help me
today i created small RMI client to balancer and it worked from my local machine(balancer is hosted on it too). however this app did work from virtual machine. so i added more logs to code and found:
app can lookup remote been
remote endpoint of this been is 127.0.0.1, but should be ip address of remote machine
After that i specified externalHost and public-ip for my sip-balancer and got bean endpoint address with 127.0.1.1
so issue was found - ubuntu uses this "local" ip address for your "machine name".
you can find it in /etc/hosts.
sip-balancer(java application) gets it as ip address of endpoint for services
My fix is - change 127.0.1.1 to 127.0.0.1 in /etc/hosts. after that sip-balancer provides real ip address of your machine for remote objects.
Conclusion: my issue - wrong operation system :)
Common solution: developer should check address type and don't use loopback addresses.