Spark and SparklyR error "grows beyond 64 KB" - apache-spark

Question

I am getting the following error on Spark after calling logistic regression using SparklyR and Spark 2.0.2.
ml_logistic_regression(Data, ml_formula)
The dataset that I read into Spark is relatively large (2.2GB). Here is the error message:
Error: org.apache.spark.SparkException: Job aborted due to stage failure: Task
13 in stage 64.0 failed 1 times, most recent failure:
Lost task 13.0 in stage 64.0 (TID 1132, localhost):
java.util.concurrent.ExecutionException:
java.lang.Exception:
failed to compile: org.codehaus.janino.JaninoRuntimeException:
Code of method "(Lorg/apache/spark/sql/catalyst/InternalRow;)Z"
of class "org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificPredicate"
grows beyond 64 KB
Others have had a similar issue: https://github.com/rstudio/sparklyr/issues/298 but I cannot find a resolution. Any ideas?

Answer 1

What happens when you subset the data and try running the model? You might need to change your configuration settings to deal with the size of the data:
library(dplyr)
library(sparklyr)
#configure the spark session and connect
config <- spark_config()
config$`sparklyr.shell.driver-memory` <- "XXG" #change depending on the size of the data
config$`sparklyr.shell.executor-memory` <- "XXG"
sc <- spark_connect(master='yarn-client', spark_home='/XXXX/XXXX/XXXX',config = config)
There are other settings in spark_config() that you could change as well to deal with performance. This is just an example of a couple.