When SSL is enabled, Spark UI doesn't use HTTPS, but forwards to port 0 over HTTP - apache-spark

Question

We are using Spark 2.0.2 and Hadoop 3.2.1. I have gotten SSL configured across Hadoop without any trouble. But Spark is having some trouble.
Without SSL, I can launch a job and view the Spark UI, proxied through Yarn.
When I enable SSL, I can still launch a job and run to completion, but I can not access the Spark Web UI. Yarn links to an ApplicationMaster proxy that fails with java.net.ConnectException: Connection refused (Connection refused)
Digging into the job's stdout, I see lines like this:
2020-04-14 16:21:57,258 INFO server.ServerConnector: Started ServerConnector#1484944f{HTTP/1.1}{0.0.0.0:40809}
2020-04-14 16:21:57,303 INFO server.ServerConnector: Started ServerConnector#799e8b3a{SSL-HTTP/1.1}{0.0.0.0:36015}
2020-04-14 16:21:57,303 INFO server.Server: Started #5614ms
2020-04-14 16:21:57,304 INFO util.Utils: Successfully started service 'SparkUI' on port 40809.
2020-04-14 16:21:57,309 INFO ui.SparkUI: Bound SparkUI to 0.0.0.0, and started at https://10.10.75.40:40809
If I call up https://10.10.75.40:40809 I get an error that it isn't SSL.
If I call up http://10.10.75.40:40809 I get served a bogus redirect to port 10.10.75.40:0.
If I call up https://10.10.75.40:36015 and ignore certificate errors, or if I call via the correct hostname, I get redirected to the RM's :8090/proxy/redirect which throws the 500 error again.
My configuration consists of:
spark.ssl.enabled true
spark.ssl.keyStore /etc/sslmate/STAR.mtv.qxxxxxxxxd.com.jks
spark.ssl.keyStorePassword _the password_
spark.ssl.trustStore /etc/sslmate/STAR.mtv.qxxxxxxxxd.com.jks
spark.ssl.trustStorePassword _the password_
spark.ssl.protocol TLSv1.2
I have also experimented with trying to set ports explicitly, to no avail:
spark.ssl.ui.port 4440
spark.ssl.history.port 18480
spark.ui.driver.port 2020
spark.ssl.ui.driver.port 2420

Answer 1

I could not find this documented via Apache, but I found a note from MapR that got me working:
Spark UI SSL is not needed when running Spark on YARN because encryption is provided by the YARN protocol. To disable Spark SSL, add spark.ssl.ui.enabled false to the spark-defaults.conf file on each spark node.
When configured in this manner, the ResourceManager correctly provides the Spark UI via a :8090/proxy/redirect type URL. In the stderr I see output along these lines:
2020-04-14 17:36:38,098 INFO server.ServerConnector: Started ServerConnector#7ef020c8{HTTP/1.1}{0.0.0.0:45951}
2020-04-14 17:36:38,098 INFO server.Server: Started #7871ms
2020-04-14 17:36:38,107 INFO util.Utils: Successfully started service 'SparkUI' on port 45951.
2020-04-14 17:36:38,113 INFO ui.SparkUI: Bound SparkUI to 0.0.0.0, and started at http://10.10.31.230:45951