Cross validation fails in Spark-ML - multithreading

Question

I have an execution of Spark-ML with a decision tree and a cross validation inside.
It fails for an unknown reason with this stack trace during the cross validation :
org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:205)
org.apache.spark.ml.tuning.CrossValidator$$anonfun$4$$anonfun$6.apply(CrossValidator.scala:164)
org.apache.spark.ml.tuning.CrossValidator$$anonfun$4$$anonfun$6.apply(CrossValidator.scala:164)
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186)
org.apache.spark.ml.tuning.CrossValidator$$anonfun$4.apply(CrossValidator.scala:164)
org.apache.spark.ml.tuning.CrossValidator$$anonfun$4.apply(CrossValidator.scala:144)
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186)
org.apache.spark.ml.tuning.CrossValidator.fit(CrossValidator.scala:144)
decisionTree.DecisionTreeDisplay.process(DecisionTreeDisplay.scala:151)
Followed by some thread stack traces:
2019-01-23 16:26:21 ERROR TaskSchedulerImpl:91 - Exception in
statusUpdate java.util.concurrent.RejectedExecutionException: Task
org.apache.spark.scheduler.TaskResultGetter$$anon$3#764726a7 rejected
from java.util.concurrent.ThreadPoolExecutor#783b07b9[Shutting down,
pool size = 2, active threads = 2, queued tasks = 0, completed tasks =
4914] at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
at
org.apache.spark.scheduler.TaskResultGetter.enqueueSuccessfulTask(TaskResultGetter.scala:61)
at
org.apache.spark.scheduler.TaskSchedulerImpl.liftedTree2$1(TaskSchedulerImpl.scala:413)
at
org.apache.spark.scheduler.TaskSchedulerImpl.statusUpdate(TaskSchedulerImpl.scala:394)
at
org.apache.spark.scheduler.local.LocalEndpoint$$anonfun$receive$1.applyOrElse(LocalSchedulerBackend.scala:67)
at
org.apache.spark.rpc.netty.Inbox$$anonfun$process$1.apply$mcV$sp(Inbox.scala:117)
at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:205) at
org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:101) at
org.apache.spark.rpc.netty.Dispatcher$MessageLoop.run(Dispatcher.scala:221)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
My cross validation code is:
// define Cross-Validation
val cv = new CrossValidator()
.setEstimator(pipeline)
.setEvaluator(evaluator)
.setEstimatorParamMaps(paramGrid)
.setNumFolds(3)
.setSeed(seed)
.setCollectSubModels(true) // requires version of spark >= 2.3.0
.setParallelism(8) // requires version of spark >= 2.3.0
val cvModel = cv.fit(trainInfile) //Fail here
In the ML library it seems to fail at line:
val foldMetrics = foldMetricFutures.map(ThreadUtils.awaitResult(_, Duration.Inf))
Any idea?