Docker on AWS ElasticBeanstalk: "host.docker.internal: host-gateway": Connection refused. Iptables issue? - linux

Question

So I have a multicontainer project running on AWS ElasticBeanstalk (latest Docker platform).
The containers share the same bridge network since they need to be able to communicate with each other (ports 3000, 8080, 9090, 9100). But one of those containers needs to be able to communicate with the host too (ports 55113, 56113).
This is normally possible by just adding extra_hosts: "host.docker.internal:host-gateway" to that container and then using host.docker.internal instead of localhost inside the container, i.e. host.docker.internal:55113. I've successfully used it before on a Debian.
But this is not working on AWS ElasticBeanstalk, the connection is refused.
Am I doing something wrong? Is there any workaround?
AWS ElasticBeanstalk Platform:
Docker running on 64bit Amazon Linux 2/3.4.2
Host's Docker & Docker-Compose versions:
$ sudo docker --version
Docker version 20.10.4, build d3cb89e
$ sudo docker-compose --version
docker-compose version 1.29.2, build unknown
Project's docker-compose.yml: (showing only the relevant parts)
version: "3.9"
services:
prometheus:
image: prom/prometheus:v2.28.1
container_name: prometheus
ports:
- 9090:9090
extra_hosts:
- host.docker.internal:host-gateway # <-------- EXTRA HOST
networks:
default:
name: monitoring
Container debug:
$ sudo docker exec -it prometheus cat /etc/hosts
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.17.0.1 host.docker.internal # <-------- the mapping is present
172.25.0.4 f01ffc6b2a8a
$ sudo docker exec -it prometheus wget host.docker.internal
Connecting to host.docker.internal (172.17.0.1:80)
wget: can't connect to remote host (172.17.0.1): Connection refused
$ sudo docker exec -it prometheus wget host.docker.internal:55113 # <-------- the port I want to reach
Connecting to host.docker.internal:55113 (172.17.0.1:55113)
wget: can't connect to remote host (172.17.0.1): Connection refused
Host's iptables (configured by AWS and/or Docker, I didn't edit it):
$ sudo iptables -S
-P INPUT ACCEPT
-P FORWARD DROP
-P OUTPUT ACCEPT
-N DOCKER
-N DOCKER-ISOLATION-STAGE-1
-N DOCKER-ISOLATION-STAGE-2
-N DOCKER-USER
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o br-a8a58f19c627 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-a8a58f19c627 -j DOCKER
-A FORWARD -i br-a8a58f19c627 ! -o br-a8a58f19c627 -j ACCEPT
-A FORWARD -i br-a8a58f19c627 -o br-a8a58f19c627 -j ACCEPT
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER -d 172.25.0.2/32 ! -i br-a8a58f19c627 -o br-a8a58f19c627 -p tcp -m tcp --dport 8080 -j ACCEPT
-A DOCKER -d 172.25.0.3/32 ! -i br-a8a58f19c627 -o br-a8a58f19c627 -p tcp -m tcp --dport 9100 -j ACCEPT
-A DOCKER -d 172.25.0.4/32 ! -i br-a8a58f19c627 -o br-a8a58f19c627 -p tcp -m tcp --dport 9090 -j ACCEPT
-A DOCKER -d 172.25.0.5/32 ! -i br-a8a58f19c627 -o br-a8a58f19c627 -p tcp -m tcp --dport 3000 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i br-a8a58f19c627 ! -o br-a8a58f19c627 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o br-a8a58f19c627 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
Host's open ports:
$ netstat -tuplen
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State User Inode PID/Program name
tcp 0 0 0.0.0.0:9090 0.0.0.0:* LISTEN 0 200315 -
tcp 0 0 127.0.0.1:55113 0.0.0.0:* LISTEN 1000 119886 -
tcp 0 0 0.0.0.0:9100 0.0.0.0:* LISTEN 0 200280 -
tcp 0 0 127.0.0.1:22221 0.0.0.0:* LISTEN 1001 197452 -
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 0 14121 -
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 0 201081 -
tcp 0 0 127.0.0.1:56113 0.0.0.0:* LISTEN 1000 119890 -
tcp 0 0 127.0.0.1:38581 0.0.0.0:* LISTEN 0 20325 -
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 0 18585 -
tcp 0 0 0.0.0.0:3000 0.0.0.0:* LISTEN 0 201164 -
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 0 16235 -
tcp6 0 0 ::1:55113 :::* LISTEN 1000 119885 -
tcp6 0 0 :::111 :::* LISTEN 0 14124 -
tcp6 0 0 ::1:56113 :::* LISTEN 1000 119889 -
tcp6 0 0 :::22 :::* LISTEN 0 18594 -
udp 0 0 0.0.0.0:68 0.0.0.0:* 0 15762 -
udp 0 0 0.0.0.0:111 0.0.0.0:* 0 14119 -
udp 0 0 127.0.0.1:323 0.0.0.0:* 0 14990 -
udp 0 0 0.0.0.0:979 0.0.0.0:* 0 14120 -
udp6 0 0 fe80::ed:19ff:fe84::546 :::* 0 16539 -
udp6 0 0 :::111 :::* 0 14122 -
udp6 0 0 ::1:323 :::* 0 14991 -
udp6 0 0 :::979 :::* 0 14123 -

Answer 1

I'm new in the field.
Your logs shows that your port is listening on localhost (127.0.0.1:56113) instead of all IPs (eg 0.0.0.0:8080).
In my case the port forward rule on the host machine has to be modified to -L0.0.0.0:<lport>:<ip>:<rport> .