How to scrape master and driver metrics of Spark 3 using Prometheus - apache-spark

Question

I'd like to use Prometheus to monitor Spark 3.
Firstly, I deploy the Prometheus and Spark 3 via helm, and they both up and running.
Then, I followed this blog
Spark 3.0 Monitoring with Prometheus to get spark 3 to expose its metrics by uncommenting these lines from the metrics.properties
*.sink.prometheusServlet.class=org.apache.spark.metrics.sink.PrometheusServlet
*.sink.prometheusServlet.path=/metrics/prometheus
master.sink.prometheusServlet.path=/metrics/master/prometheus
applications.sink.prometheusServlet.path=/metrics/applications/prometheus
My first spark-submit job works fine, and I am able to see all metrics_executor_* from Prometheus UI
spark-submit
…
--conf spark.ui.prometheus.enabled=true \
--conf spark.kubernetes.driver.annotation.prometheus.io/scrape=true \
--conf spark.kubernetes.driver.annotation.prometheus.io/path=/metrics/executors/prometheus/ \
--conf spark.kubernetes.driver.annotation.prometheus.io/port=4040 \
…
However, when I tried to submit another job to get metrics from master and worker, the desired metrics_master* or metrics_worker* metrics were not shown in Prometheus
spark-submit
…
--conf spark.ui.prometheus.enabled=true \
--conf spark.kubernetes.driver.annotation.prometheus.io/scrape=true \
--conf spark.kubernetes.driver.annotation.prometheus.io/path=/metrics/executors/prometheus/ \
--conf spark.kubernetes.driver.annotation.prometheus.io/port=4040 \
--conf spark.kubernetes.driver.annotation.prometheus.io/scrape=true \
--conf spark.kubernetes.driver.annotation.prometheus.io/path=/metrics/prometheus/ \
--conf spark.kubernetes.driver.annotation.prometheus.io/port=8081 \
--conf spark.kubernetes.driver.annotation.prometheus.io/scrape=true \
--conf spark.kubernetes.driver.annotation.prometheus.io/path=/metrics/master/prometheus/ \
--conf spark.kubernetes.driver.annotation.prometheus.io/port=8080 \
--conf spark.kubernetes.driver.annotation.prometheus.io/scrape=true \
--conf spark.kubernetes.driver.annotation.prometheus.io/path=/metrics/applications/prometheus/ \
--conf spark.kubernetes.driver.annotation.prometheus.io/port=8080 \
…
It seems the relevant spark services in Kubernetes only expose port 4040, although I nominated port 8080 and 8081 in the spark-sbumit. That could be the reason whymetrics_master* and metrics_worker* are absent from Prometheus.
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-spark-py-042f06864dacaf4d-driver-svc ClusterIP None <none> 7078/TCP,7079/TCP,4040/TCP 4m46s
my-spark-py84edeb8643cde90e-driver-svc ClusterIP None <none> 7078/TCP,7079/TCP,4040/TCP 46h
Is there any fix for it?