K8s Issue connecting to Cassandra on Mac OS (via Node.js) - node.js

Question

While trying to setup Cassandra database in a local Kubernetes cluster on a Mac OS (via Minikube), I am getting connection issues.
It seems like Node.js is not able to resolve DNS settings correctly, but resolving via command line DOES work.
The setup is as following (simplified):
Cassandra Service
apiVersion: v1
kind: Service
metadata:
labels:
app: cassandra
name: cassandra
spec:
type: NodePort
ports:
- port: 9042
targetPort: 9042
protocol: TCP
name: http
selector:
app: cassandra
In addition, there's a PersistentVolume and a StatefulSet.
The application itself is very basic
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: app1
labels:
app: app1
spec:
replicas: 1
selector:
matchLabels:
app: app1
template:
metadata:
labels:
app: app1
spec:
containers:
- name: app1
image: xxxx.dkr.ecr.us-west-2.amazonaws.com/acme/app1
imagePullPolicy: "Always"
ports:
- containerPort: 3003
And a service
apiVersion: v1
kind: Service
metadata:
name: app1
namespace: default
spec:
selector:
app: app1
type: NodePort
ports:
- port: 3003
targetPort: 3003
protocol: TCP
name: http
there also a simple ingress setup
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress
annotations:
kubernetes.io/ingress.class: nginx
spec:
rules:
- host: dev.acme.com
http:
paths:
- path: /app1
backend:
serviceName: app1
servicePort: 3003
And added to /etc/hosts the minikube ip address
192.xxx.xx.xxx dev.acme.com
So far so good.
When trying to call dev.acme.com/app1 via Postman, the node.js app itself is being called correctly (can see in the logs), HOWEVER, the app can not connect to Cassandra and times out with the following error:
"All host(s) tried for query failed. First host tried,
92.242.140.2:9042: DriverError: Connection timeout. See innerErrors."
The IP 92.242.140.2 seems to be just a public IP that is related to my ISP, I believe since the app is not able to resolve the service name.
I created a simple node.js script to test dns:
var dns = require('dns')
dns.resolve6('cassandra', (err, res) => console.log('ERR:', err, 'RES:', res))
and the response is
ERR: { Error: queryAaaa ENOTFOUND cassandra
at QueryReqWrap.onresolve [as oncomplete] (dns.js:197:19) errno: 'ENOTFOUND', code: 'ENOTFOUND', syscall: 'queryAaaa', hostname:
'cassandra' } RES: undefined
However, and this is where it gets confusing - when I ssh into the pod (app1), I am able to connect to cassandra service using:
cqlsh cassandra 9042 --cqlversion=3.4.4
So it seems as the pod is "aware" of the service name, but node.js runtime is not.
Any idea what could cause the node.js to not being able to resolve the service name/dns settings?
UPDATE
After re-installing the whole cluster, including re-installing docker, kubectl and minikube I am getting the same issue.
While running ping cassandra from app1 container via ssh, I am getting the following
PING cassandra.default.svc.cluster.local (10.96.239.137) 56(84) bytes
of data. 64 bytes from cassandra.default.svc.cluster.local
(10.96.239.137): icmp_seq=1 ttl=61 time=27.0 ms
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
Which seems to be fine.
However, when running from Node.js runtime I am still getting the same error -
"All host(s) tried for query failed. First host tried,
92.242.140.2:9042: DriverError: Connection timeout. See innerErrors."
These are the services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
app1 ClusterIP None <none> 3003/TCP 11m
cassandra NodePort 10.96.239.137 <none> 9042:32564/TCP 38h
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 38h
And these are the pods (all namespaces)
NAMESPACE NAME READY STATUS RESTARTS AGE
default app1-85d889db5-m977z 1/1 Running 0 2m1s
default cassandra-0 1/1 Running 0 38h
kube-system calico-etcd-ccvs8 1/1 Running 0 38h
kube-system calico-node-thzwx 2/2 Running 0 38h
kube-system calico-policy-controller-5bb4fc6cdc-cnhrt 1/1 Running 0 38h
kube-system coredns-86c58d9df4-z8pr4 1/1 Running 0 38h
kube-system coredns-86c58d9df4-zcn6p 1/1 Running 0 38h
kube-system default-http-backend-5ff9d456ff-84zb5 1/1 Running 0 38h
kube-system etcd-minikube 1/1 Running 0 38h
kube-system kube-addon-manager-minikube 1/1 Running 0 38h
kube-system kube-apiserver-minikube 1/1 Running 0 38h
kube-system kube-controller-manager-minikube 1/1 Running 0 38h
kube-system kube-proxy-jj7c4 1/1 Running 0 38h
kube-system kube-scheduler-minikube 1/1 Running 0 38h
kube-system kubernetes-dashboard-ccc79bfc9-6jtgq 1/1 Running 4 38h
kube-system nginx-ingress-controller-7c66d668b-rvxpc 1/1 Running 0 38h
kube-system registry-creds-x5bhl 1/1 Running 0 38h
kube-system storage-provisioner 1/1 Running 0 38h
UPDATE 2
The code to connect to Cassandra from Node.js:
const cassandra = require('cassandra-driver');
const client = new cassandra.Client({ contactPoints: ['cassandra:9042'], localDataCenter: 'datacenter1', keyspace: 'auth_server' });
const query = 'SELECT * FROM user';
client.execute(query, [])
.then(result => console.log('User with email %s', result.rows[0].email));
It DOES work when replacing cassandra:9042 with 10.96.239.137:9042 (10.69.239.137 is the ip address received from pinging cassandra via cli).

Answer 1

The Cassandra driver for Node.js uses resolve4/resolve6 to do its dns lookup, which bypasses your resolv.conf file. A program like ping uses resolv.conf to resolve 'cassandra' to 'cassandra.default.svc.cluster.local', the actual dns name assigned to your Cassandra service. For a more detailed explanation of name resolution in node.js see here.
The fix is simple, just pass in the full service name to your client:
const client = new cassandra.Client({ contactPoints: ['cassandra.default.svc.cluster.local:9042'], localDataCenter: 'datacenter1', keyspace: 'auth_server' });