Docker listening inside the docker host for RabbitMQ but not from outside, why? - linux

Question

This is how I run the rabbitMQ image:
docker run -d --restart always --hostname host-rabbit --name cg-rabbit -p 5029:5672 -p 5020:15672 -e RABBITMQ_DEFAULT_VHOST=sample_vhost -e RABBITMQ_DEFAULT_USER=sampleuser -e RABBITMQ_DEFAULT_PASS=samplepass rabbitmq:3-management
Now in netstat -nltp:
ubuntu#infra:~$ netstat -nltp
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -
tcp6 0 0 :::22 :::* LISTEN -
tcp6 0 0 :::5020 :::* LISTEN -
tcp6 0 0 :::5029 :::* LISTEN -
I'm not sure why I see tcp6 when docker exposes ports to host and if it makes issues!
Now when I telnet from within the server I can see that port is open:
ubuntu#infra:~$ telnet MY-SERVER-IP-ADDRESS 5029
Trying MY-SERVER-IP-ADDRESS...
Connected to MY-SERVER-IP-ADDRESS.
Escape character is '^]'.
^]
telnet> Connection closed.
But in my machine when I try to telnet (or from another server):
$ telnet MY-SERVER-IP-ADDRESS 5020
Trying MY-SERVER-IP-ADDRESS...
^C
iptables -L reports:
sudo iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:5020
ACCEPT tcp -- anywhere anywhere state NEW tcp dpt:5029
ACCEPT tcp -- anywhere anywhere
Chain FORWARD (policy DROP)
target prot opt source destination
DOCKER-USER all -- anywhere anywhere
DOCKER-ISOLATION-STAGE-1 all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain DOCKER (1 references)
target prot opt source destination
ACCEPT tcp -- anywhere 172.17.0.2 tcp dpt:amqp
ACCEPT tcp -- anywhere 172.17.0.2 tcp dpt:15672
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target prot opt source destination
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
RETURN all -- anywhere anywhere
Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target prot opt source destination
DROP all -- anywhere anywhere
RETURN all -- anywhere anywhere
Chain DOCKER-USER (1 references)
target prot opt source destination
RETURN all -- anywhere anywhere
It is good to note that I have installed a redis server in server (non-docker) and I am able to telnet to it form outside.
EDIT-1:
sudo iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
DOCKER all -- anywhere anywhere ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
DOCKER all -- anywhere !localhost/8 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
MASQUERADE all -- 172.17.0.0/16 anywhere
MASQUERADE tcp -- 172.17.0.2 172.17.0.2 tcp dpt:15672
MASQUERADE tcp -- 172.17.0.2 172.17.0.2 tcp dpt:amqp
Chain DOCKER (2 references)
target prot opt source destination
RETURN all -- anywhere anywhere
DNAT tcp -- anywhere anywhere tcp dpt:15672 to:172.17.0.2:15672
DNAT tcp -- anywhere anywhere tcp dpt:amqp to:172.17.0.2:5672
EDIT-2:
Docker configuration:
ubuntu#infra:~$ sudo cat /var/snap/docker/796/config/daemon.json
{
"log-level": "error",
"storage-driver": "overlay2"
}

Answer 1

This is really odd. By flushing the NAT in iptables everything works as expected:
iptables -t nat -F
My nat before flushing:
ubuntu#infra:~$ sudo iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
DOCKER all -- anywhere anywhere ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
DOCKER all -- anywhere !localhost/8 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
MASQUERADE all -- 172.17.0.0/16 anywhere
MASQUERADE tcp -- 172.17.0.2 172.17.0.2 tcp dpt:15672
MASQUERADE tcp -- 172.17.0.2 172.17.0.2 tcp dpt:amqp
Chain DOCKER (2 references)
target prot opt source destination
RETURN all -- anywhere anywhere
DNAT tcp -- anywhere anywhere tcp dpt:15672 to:172.17.0.2:15672
DNAT tcp -- anywhere anywhere tcp dpt:amqp to:172.17.0.2:5672
And now after flushing everything is gone:
ubuntu#infra:~$ sudo iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
Chain DOCKER (0 references)
target prot opt source destination
NOTE: by restarting docker via sudo snap restart docker net rules are back again and I had to flush NATs again!