Futures timed out after [5 seconds] - databricks

Question

On a job cluster with the following configuration:
Driver: Standard_E8ds_v5
Workers: Standard_E8ds_v5
30 workers
11.3 LTS Photon (includes Apache Spark 3.3.0, Scala 2.12)
We are about 5% of the time getting a Futures timed out after [5 seconds] error with the stack trace seen at the bottom. I was hoping the stack trace would be enough for someone to tell me what spark config I should adjust to extend this 5 second timeout.
The notebook for the job does this:
def RunChild(s):
dbutils.notebook.run("./ProcessChild", 0, {"param": s})
scenarios = [ some array with 107 items]
with ThreadPoolExecutor(max_workers=20) as executor:
final = executor.map(RunChild, scenarios)
The ProcessChild notebook fails fairly frequently on a different bit of Spark code each time with the following stack trace and error:
java.util.concurrent.TimeoutException: Futures timed out after [5 seconds]
      at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:259)
      at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:263)
      at scala.concurrent.Await$.$anonfun$result$1(package.scala:223)
      at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:57)
      at scala.concurrent.Await$.result(package.scala:146)
      at com.databricks.backend.daemon.driver.JupyterDriverLocal$RequestStatus.waitForReply(JupyterDriverLocal.scala:209)
      at com.databricks.backend.daemon.driver.JupyterDriverLocal.repl(JupyterDriverLocal.scala:971)
      at com.databricks.backend.daemon.driver.DriverLocal.$anonfun$execute$23(DriverLocal.scala:725)
      at com.databricks.unity.EmptyHandle$.runWith(UCSHandle.scala:103)
      at com.databricks.backend.daemon.driver.DriverLocal.$anonfun$execute$20(DriverLocal.scala:708)
      at com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:398)
      at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
      at com.databricks.logging.AttributionContext$.withValue(AttributionContext.scala:147)
      at com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:396)
      at com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:393)
      at com.databricks.backend.daemon.driver.DriverLocal.withAttributionContext(DriverLocal.scala:62)
      at com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:441)
      at com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:426)
      at com.databricks.backend.daemon.driver.DriverLocal.withAttributionTags(DriverLocal.scala:62)
      at com.databricks.backend.daemon.driver.DriverLocal.execute(DriverLocal.scala:685)
      at com.databricks.backend.daemon.driver.DriverWrapper.$anonfun$tryExecutingCommand$1(DriverWrapper.scala:622)
      at scala.util.Try$.apply(Try.scala:213)
      at com.databricks.backend.daemon.driver.DriverWrapper.tryExecutingCommand(DriverWrapper.scala:614)
      at com.databricks.backend.daemon.driver.DriverWrapper.executeCommandAndGetError(DriverWrapper.scala:533)
      at com.databricks.backend.daemon.driver.DriverWrapper.executeCommand(DriverWrapper.scala:568)
      at com.databricks.backend.daemon.driver.DriverWrapper.runInnerLoop(DriverWrapper.scala:438)
      at com.databricks.backend.daemon.driver.DriverWrapper.runInner(DriverWrapper.scala:381)
      at com.databricks.backend.daemon.driver.DriverWrapper.run(DriverWrapper.scala:232)
      at java.lang.Thread.run(Thread.java:750)
The cluster is certainly busy with all the parallel threads and operations and we would like to know which spark config can extend that 5s timeout.

Answer 1

We're having the same issue, though more annoyingly in our instance we are running absolutely fine on 10.4LTS. Simply upgrading the runtime and re-firing results in Futures timing out after five seconds.
For us, we were able to use the following to increase the broadcast join timeout from -1000 to 300000 (5 minutes).
spark.conf.get("spark.sql.broadcastTimeout")
spark.conf.set("spark.sql.broadcastTimeout", '300000ms')
This has to be a temporary fix (at least for us haha) but hopefully it will get you out of immediate trouble. We also ended up increasing the size of our cluster to assist with the memory being used.
Given this is caused by broadcast joins, as part of our testing we were able to disable automatic broadcast joins also to see if that helped (it did but it was flaky):
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", -1)
Ref https://spark.apache.org/docs/latest/sql-performance-tuning.html