Spark Client Pod in Kubernetes Getting 401 Error After One Hour - apache-spark

Question

Kubernetes Version: 1.21
Spark Version: 3.0.0
I am using a container in a Kubernetes pod (client pod) to invoke Spark Submit which then starts a Driver pod. The client pod which did the Spark Submit starts to watch the Driver pod via LoggingPodStatusWatcherImpl. After approximately 1 hour, the client pod experiences 401 error
22/11/03 13:05:44 WARN WatchConnectionManager: Exec Failure: HTTP 401, Status: 401 - Unauthorized
java.net.ProtocolException: Expected HTTP 101 response but was '401 Unauthorized'
at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229)
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196)
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203)
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
22/11/03 13:05:46 INFO LoggingPodStatusWatcherImpl: Application status for spark-blahblahblah (phase: Running)
I think Spark on Kubernetes usually looks in /var/run/secrets/kubernetes.io/serviceaccount/token so I would get the warning below when starting the client pod.
22/11/03 13:13:13 WARN Config: Error reading service account token from: [/var/run/secrets/kubernetes.io/serviceaccount/token]. Ignoring.
However, since I provide another oauth token file via the conf below in Spark Submit command the client pod was able to connect to the Kubernetes API and start the Driver pod.
--conf spark.kubernetes.authenticate.submission.oauthTokenFile=/mytokendir/token
The token is provided to the client pod via projected volume (new to Kubernete versions 1.20+), the token expiration duration can be specified in the yaml manifest as shown below:
See this doc for reference on how this is implemented:
https://cloud.google.com/blog/products/containers-kubernetes/kubernetes-bound-service-account-tokens
spec:
serviceAccountName: my-serviceaccount
volumes:
- name: token-vol
projected:
sources:
- serviceAccountToken:
expirationSeconds: 7200
path: token
containers:
-name: my-container
image: some-image
volumeMounts:
-name: token-vol
mountPath: /mytokendir
I then exec into the client pod to get the JWT token in /mytokendir and decoded it.
It showed valid for 2 hours; however, coming back to the original question, my client pod is still getting 401 error after 1 hour.
Sometimes I would get this error:
22/11/03 14:10:57 INFO LoggingPodStatusWatcherImpl: Application my-application with submission ID my-namespace:my-driver finished
22/11/03 14:10:57 INFO ShutdownHookManager: Shutdown hook called
22/11/03 14:10:57 INFO ShutdownHookManager: Deleting directory /tmp/spark-blahblah
The connection to the server localhost:8080 was refused - did you specify the right host or port?