docker-compose v3 + apache spark, connection refused on port 7077 - apache-spark

Question

I'm not sure if this is 100% programming or sysadmin related question.
I'm trying to setup a docker-compose file, in the version 3, for docker-swarm, docker version 1.13 to test spark for my local workflow.
Sadly the port 7077 does only get bound to localhost on my swarm cluster and so is not reachable from the outside world, where my spark app is trying to connect to it.
Does anyone have an idea, how to get docker-compose in swarm mode to bind to all interfaces?
I publish my ports and this works fine for say 8080, but not for 7070.
nmap output:
Starting Nmap 7.01 ( https://nmap.org ) at 2017-03-02 11:27 PST
Nmap scan report for localhost (127.0.0.1)
Host is up (0.000096s latency).
Other addresses for localhost (not scanned): ::1
Not shown: 994 closed ports
PORT STATE SERVICE
22/tcp open ssh
80/tcp open http
443/tcp open https
8080/tcp open http-proxy
8081/tcp open blackice-icecap
8888/tcp open sun-answerbook
Explanation of ports
8081 is my spark worker
8080 is my spark master frontend
8888 is the spark hue frontend
nmap does not list 7077
Using netstat:
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1641/sshd
tcp6 0 0 :::4040 :::* LISTEN 1634/dockerd
tcp6 0 0 :::2377 :::* LISTEN 1634/dockerd
tcp6 0 0 :::7946 :::* LISTEN 1634/dockerd
tcp6 0 0 :::80 :::* LISTEN 1634/dockerd
tcp6 0 0 :::8080 :::* LISTEN 1634/dockerd
tcp6 0 0 :::8081 :::* LISTEN 1634/dockerd
tcp6 0 0 :::6066 :::* LISTEN 1634/dockerd
tcp6 0 0 :::22 :::* LISTEN 1641/sshd
tcp6 0 0 :::8888 :::* LISTEN 1634/dockerd
tcp6 0 0 :::443 :::* LISTEN 1634/dockerd
tcp6 0 0 :::7077 :::* LISTEN 1634/dockerd
And I can connect to 7077 over telnet on localhost without any issues, but outside of localhost I'm receiving a connection refused error.
At this point in time (please bear with me, I'm not a sysadmin, I'm a software guy), I'm starting to have the feeling this is somehow related to the docker mesh network.
Docker compose section for my master configuration:
#the spark master, having to run on the frontend of the cluster
master:
image: eros.fiehnlab.ucdavis.edu/spark
command: bin/spark-class org.apache.spark.deploy.master.Master -h master
hostname: master
environment:
MASTER: spark://master:7077
SPARK_CONF_DIR: /conf
SPARK_PUBLIC_DNS: blonde.fiehnlab.ucdavis.edu
ports:
- 4040:4040
- 6066:6066
- 8080:8080
- 7077:7077
volumes:
- /tmp:/tmp/data
networks:
- spark
- frontends
deploy:
placement:
#only run on manager node
constraints:
- node.role == manager
The networks spark and frontend are both overlay networks

Answer 1

The issue was a configuration error in the docker-compose file. The -h master in the original configuration always bound to the local host interface.
Even after specifying the SPARK_LOCAL_IP value
master:
image: eros.fiehnlab.ucdavis.edu/spark:latest
command: bin/spark-class org.apache.spark.deploy.master.Master
hostname: master
environment:
SPARK_CONF_DIR: /conf
SPARK_PUBLIC_DNS: blonde.fiehnlab.ucdavis.edu
SPARK_LOCAL_IP: 0.0.0.0
ports:
- 4040:4040
- 6066:6066
- 8080:8080
- 7077:7077
volumes:
- /tmp:/tmp/data
deploy:
placement:
#only run on manager node
constraints:
- node.role == manager