ZMQ sockets do not work as expected on Kubernetes - node.js

Question

Short summary: My ZMQ sockets do not receive (probably also send) messages when I deploy the code on kubernetes.
I have an application that involves multiple clients and servers. It is developed on node and uses ZeroMQ as communication layer. It works on my local machine, it works on docker and I am trying to deploy the application using kubernetes.
When deploying the application, the pods, the deployments and the kubernete's services launch. Apparently everything goes ok, but the clients send an initial message that never arrives to the server. The deployments are in the same namespace and I use Flannel as CNI. To the best of my knowledge, the cluster was properly initialized but the messages never arrive.
I read this question about ZMQ sockets having problems binding on kubernetes. I have tried playing with the ZMQ_CONNECT_TIMEOUT parameter but it does not do anything. Also, unlike the question I cite, my messages never arrive.
I could provide some code, but there is a lot of code and I don't think the application is the problem. I think I am missing something on the Kubernetes configuration since it's my firts time using it . Let me know if you need more information.
Edit 1. 12/01/2021
As #anemyte suggests I will try to provide a simplified version of the code:
Client Side:
initiate () {
return new Promise(resolve => {
this.N_INCOMING = 0;
this.N_OUTGOING = 0;
this.rrCounter = 0;
this.PULL_SOCKET.bind("tcp://*:" + this.MY_PORT, error => {
utils.handleError(error);
this.PUB_SOCKET.bind("tcp://*:" + (this.MY_PORT + 1), error => {
utils.handleError(error);
this.SUB_SOCKET.subscribe("");
this.SUB_SOCKET.connect(this.SERVER + ":" + (this.SERVER_PORT + 1),
error => {utils.handleError(error)});
this.PULL_SOCKET.on("message", (m) => this.handlePullSocket(m));
this.SUB_SOCKET.on("message", (m) => this.handleSubSocket(m));
this.SERVER_PUSH_SOCKET = zmq.socket("push");
this.SERVER_PUSH_SOCKET.connect(this.SERVER + ":" + this.SERVER_PORT,
error => {utils.handleError(error)});
this.sendHello();
resolve();
});
});
});
Server side:
initiate () {
return new Promise(resolve => {
this.PULL_SOCKET.bind(this.MY_IP + ":" + this.MY_PORT, err => {
if (err) {
console.log(err);
process.exit(0);
}
this.PUB_SOCKET.bind(this.MY_IP + ":" + (this.MY_PORT + 1), err => {
if (err) {
console.log(err);
process.exit(0);
}
this.PULL_SOCKET.on("message", (m) => this.handlePullSocket(m));
resolve();
});
});
});
}
Client initiates the connection by sending the Hello Message. Server's listener function handlePullSocket should handle those messages.
Edit 2. 12/01/2021
As requested, I am adding the deployment/service configurations.
Client-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
kompose.cmd: kompose convert -f docker-compose-resolved.yml
kompose.version: 1.19.0 (f63a961c)
creationTimestamp: null
labels:
io.kompose.service: c1
name: c1
namespace: fishtrace
spec:
replicas: 1
selector:
matchLabels:
app: c1
strategy:
type: Recreate
template:
metadata:
annotations:
kompose.cmd: kompose convert -f docker-compose-resolved.yml
kompose.version: 1.19.0 (f63a961c)
creationTimestamp: null
labels:
app: c1
spec:
containers:
- env:
- name: NODO_ADDRESS
value: 0.0.0.0
- name: NODO_PUERTO
value: "9999"
- name: NODO_PUERTO_CADENA
value: "8888"
- name: SERVER_ADDRESS
value: tcp://servidor
- name: SERVER_PUERTO
value: "7777"
image: registrogeminis.com/docker_c1_rpi:latest
name: c1
ports:
- containerPort: 8888
- containerPort: 9999
resources: {}
volumeMounts:
- mountPath: /app/vol
name: c1-volume
imagePullPolicy: Always
restartPolicy: Always
imagePullSecrets:
- name: myregistrykey
volumes:
- name: c1-volume
persistentVolumeClaim:
claimName: c1-volume
status: {}
Client-service.yaml
apiVersion: v1
kind: Service
metadata:
annotations:
kompose.cmd: ./kompose convert
kompose.version: 1.22.0 (955b78124)
creationTimestamp: null
labels:
io.kompose.service: c1
name: c1
spec:
ports:
- name: "9999"
port: 9999
targetPort: 9999
- name: "8888"
port: 8888
targetPort: 8888
selector:
app: c1
type: ClusterIP
Server-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
kompose.cmd: kompose convert -f docker-compose-resolved.yml
kompose.version: 1.19.0 (f63a961c)
creationTimestamp: null
labels:
io.kompose.service: servidor
name: servidor
namespace: fishtrace
spec:
replicas: 1
selector:
matchLabels:
app: servidor
strategy:
type: Recreate
template:
metadata:
annotations:
kompose.cmd: kompose convert -f docker-compose-resolved.yml
kompose.version: 1.19.0 (f63a961c)
creationTimestamp: null
labels:
app: servidor
spec:
containers:
- env:
- name: SERVER_ADDRESS
value: tcp://*
- name: SERVER_PUERTO
value: "7777"
image: registrogeminis.com/docker_servidor_rpi:latest
name: servidor
ports:
- containerPort: 7777
- containerPort: 7778
resources: {}
volumeMounts:
- mountPath: /app/vol
name: servidor-volume
imagePullPolicy: Always
restartPolicy: Always
imagePullSecrets:
- name: myregistrykey
volumes:
- name: servidor-volume
persistentVolumeClaim:
claimName: servidor-volume
status: {}
Server-service.yaml
apiVersion: v1
kind: Service
metadata:
annotations:
kompose.cmd: ./kompose convert
kompose.version: 1.22.0 (955b78124)
creationTimestamp: null
labels:
io.kompose.service: servidor
name: servidor
spec:
ports:
- name: "7777"
port: 7777
targetPort: 7777
selector:
app: servidor
type: ClusterIP

Answer 1

At the end it was a DNS problem. It was always a DNS problem. Thanks to #Matt for pointing out the problem.
In the official Kubernetes DNS doc they state that there is a known issue with systems that use /etc/resolv.conf as a link to the real configuration file, /run/systemd/resolve/resolv.conf in my case. It is a well-known problem, and the recommended solution is to update the kubelet's configuration to point to /run/systemd/resolve/resolv.conf.
To do so, I added the line resolvConf:/run/systemd/resolve/resolv.conf in /var/lib/kubelet/config.yaml. I also edited /etc/kubernetes/kubelet.conf just to be sure. Finally, you are supposed to reload the service executing sudo systemctl daemon-reload && sudo systemctl restart kubelet to propagate the changes.
However, I already did that before asking in SE. And it did not seem to work. I had to restart the whole cluster to make the changes take effect. Then the DNS worked perfectly and the ZMQ sockets behaved as expected.
Update 31/04/2021: I discovered that you have to forcefully restart the coredns kubernetes' service to actually propagate the changes. So in the end kubectl rollout restart coredns -n kube-system after restarting the kubelet service was enough.