Spark 2 broadcast inside Docker uses random port - apache-spark

Question

I'm trying to run Spark 2 inside Docker containers and it is being kind of hard for me. So long I think have come a long way, having been able to deploy a Standalone master to host A and a worker to host B. I configured the /etc/hosts of the Docker container so master and worker can access their respective hosts. They see each other and everything looks to be fine.
I deploy master with this set of opened ports:
docker run -ti --rm \
--name sparkmaster \
--hostname=host.a \
--add-host host.a:xx.xx.xx.xx \
--add-host host.b:xx.xx.xx.xx \
-p 18080:18080 \
-p 7001:7001 \
-p 7002:7002 \
-p 7003:7003 \
-p 7004:7004 \
-p 7005:7005 \
-p 7006:7006 \
-p 4040:4040 \
-p 7077:7077 \
malkab/spark:ablative_alligator
Then I submit this Spark config options:
export SPARK_MASTER_OPTS="-Dspark.driver.port=7001
-Dspark.fileserver.port=7002
-Dspark.broadcast.port=7003 -Dspark.replClassServer.port=7004
-Dspark.blockManager.port=7005 -Dspark.executor.port=7006
-Dspark.ui.port=4040 -Dspark.broadcast.blockSize=4096"
export SPARK_MASTER_PORT=7077
export SPARK_MASTER_WEBUI_PORT=18080
I deploy the worker on its host with the SPARK_WORKER_XXX version of the aforementioned env variables, and with an analogous docker run.
Then I enter into the master container and spark-submit a job:
spark-submit --master spark://host.a:7077 /src/Test06.py
Everything starts fine: I can see the job being distributed to the worker. But when the Block Manager tries to register the block, it seems to be using a random port, which is not accesible outside the container:
INFO BlockManagerMasterEndpoint: Registering block manager host.b:39673 with 366.3 MB RAM, BlockManagerId(0, host.b, 39673, None)
Then I get this error:
WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, host.b, executor 0): java.io.IOException: org.apache.spark.SparkException: Failed to get broadcast_0_piece0 of broadcast_0
And the worker reports:
java.io.IOException: Failed to connect to host.a/xx.xx.xx.xx:55638
I've been able so far to avoid the usage of random ports with the previous settings, but this Block Manager port in particular seems to be random. I though it was controlled by the spark.blockManager.port, but this seems not to be the case. I've reviewed all configuration options to no avail.
So, final question: what is this port and can be avoided it to be random?
Thanks in advance.
EDIT:
This is the executor launched. As you can see, random ports are open both on the driver (which is in the same host as the master) and the worker:
Random ports for executors
I understand that a worker instantiates many executors, and that they should have a port assigned. Is there any way to, at least, limit the range of ports for this communications? How do the people handle Spark behind tight firewalls then?
EDIT 2:
I finally got it working. If I pass, as I mentioned before, the property spark.driver.host=host.a in the SPARK_MASTER_OPTS and SPARK_WORKER_OPTS it won't work, however, if I configure it in my code at the configuration of the context:
conf = SparkConf().setAppName("Test06") \
.set("spark.driver.port", "7001") \
.set("spark.driver.host", "host.a") \
.set("spark.fileserver.port", "7002") \
.set("spark.broadcast.port", "7003") \
.set("spark.replClassServer.port", "7004") \
.set("spark.blockManager.port", "7005") \
.set("spark.executor.port", "7006") \
.set("spark.ui.port", "4040") \
.set("spark.broadcast.blockSize", "4096") \
.set("spark.local.dir", "/tmp") \
.set("spark.driver.extraClassPath", "/classes/postgresql.jar") \
.set("spark.executor.extraClassPath", "/classes/postgresql.jar")
it somehow worked. Why is the setting not honored in SPARK_MASTER_OPTS or SPARK_WORKER_OPTS?