Running Spark Mllib Decision Tree gives block size error - apache-spark

Question

I'm running a decision tree on a dataframe of about 2000 points and 500 features. Maxbins is 182. No matter how i increase the shuffling block size from 200 up to 4000 i keep getting a failure at stage 3 of the decision tree training saying "max integer reached" referring to Spark block size shuffling size. Note my dataframes are not rdds but spark sql dataframes.
Here is the error:
...
Caused by: java.lang.RuntimeException: java.lang.IllegalArgumentException: Size exceeds Integer.MAX_VALUE
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:828)
at org.apache.spark.storage.DiskStore$$anonfun$getBytes$2.apply(DiskStore.scala:125)
at org.apache.spark.storage.DiskStore$$anonfun$getBytes$2.apply(DiskStore.scala:113)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1206)
at org.apache.spark.storage.DiskStore.getBytes(DiskStore.scala:127)
at org.apache.spark.storage.DiskStore.getBytes(DiskStore.scala:134)
at org.apache.spark.storage.BlockManager.doGetLocal(BlockManager.scala:522)
at org.apache.spark.storage.BlockManager.getBlockData(BlockManager.scala:312)
at org.apache.spark.network.netty.NettyBlockRpcServer$$anonfun$2.apply(NettyBlockRpcServer.scala:58)
at org.apache.spark.network.netty.NettyBlockRpcServer$$anonfun$2.apply(NettyBlockRpcServer.scala:58)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
...
Here is the code producing it:
val assembled = assembler.transform(features)
val dt = new DecisionTreeClassifier().setLabelCol("indexedLabel").setFeaturesCol("indexedFeatures").setImpurity(impurity).setMaxBins(maxBins).setMaxDepth(maxDepth)
val pipeline = new Pipeline().setStages(Array(labelIndexer, dt))
val model = pipeline.fit(assembled)
Thank you for any pointers on what might be causing this and how to fix it.
Thank you.

Answer 1

Try to increase number of partitions - try repartition() method.
The reason for this error is that spark uses memory mapped files to handle partition data blocks and it's currently not possible to memory mapped something that's more, than 2GB (Integer.MAX_VALUE) - not the Spark issue, though.
The workaround would be increase number of partitions. That would reduce block size of particular partition and might help with the issue.
And there is also some activity to workaround that in Spark itself - to process partitions blocks in chunks.