Why does Word2Vec.fit run out of heap space - apache-spark

Question

I am tokenizing a month's worth of URLs. I am unable to get through fit without running out of heap space. The cached dataframe looks like it is taking up less than 400MB of RAM and have 1.9TB total available over 11 nodes.
As you see in the code below, I have thrown a ton of space at it in the form of executor RAM, driver RAM, and have created a number of partitions both in the dataframe as well as in the Word2Vec process. I am not sure how to provide more RAM or divide the task into smaller portions. I feel like I must be missing something fundamental about this error.
myconf = SparkConf().setAll([
("spark.driver.memory", "200G"),
("spark.driver.cores", "10"),
("spark.executor.cores", "10"),
("spark.executor.memory", "200G"),
("spark.driver.maxResultSize","100G"),
("spark.yarn.executor.memoryOverhead","20G"),
("spark.kryoserializer.buffer.max", "1400mb"),
("spark.maxRemoteBlockSizeFetchToMem","1400mb"),
("spark.executor.extraJavaOptions","-XX:+UseConcMarkSweepGC"),
("spark.sql.shuffle.partitions","3000"),
])
appName = "word2vec-test2"
spark = (SparkSession.builder.config(conf=myconf)
.getOrCreate())
interval_df = (spark.read.parquet("/output/ddrml/preprocessed/201906*/")
.select("request")
.withColumnRenamed("request", "uri")
.where(F.col("uri").isNotNull())
.dropDuplicates()
.withColumn("uniqueID",F.monotonically_increasing_id())
.cache())
tokenizer = Tokenizer(inputCol="uri", outputCol="words")
regexTokenizer = RegexTokenizer(inputCol="uri", outputCol="words", pattern="\\W")
regexTokenized = regexTokenizer.transform(interval_df)
regexTokenized = regexTokenized.repartition(2000)
word2Vec = Word2Vec(vectorSize=15, minCount=0, inputCol="words", outputCol="uri_vec", numPartitions=500, maxSentenceLength=10)
model = word2Vec.fit(regexTokenized)
print("Made it through fit")
It appears to have some problems keeping up. I presume that something is doing a bunch of garbage collection. Then it dies in fit with a java heap error.
19/07/17 12:28:43 ERROR scheduler.LiveListenerBus: Dropping SparkListenerEvent because no remaining room in event queue. This likely means one of the SparkListeners is too slow and cannot keep up with the rate at which tasks are being started by the scheduler.
19/07/17 12:28:43 WARN scheduler.LiveListenerBus: Dropped 1 SparkListenerEvents since Wed Dec 31 19:00:00 EST 1969
Exception in thread "dispatcher-event-loop-4" java.lang.OutOfMemoryError: Java heap space
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "dispatcher-event-loop-4"
19/07/17 12:30:48 WARN scheduler.LiveListenerBus: Dropped 794 SparkListenerEvents since Wed Jul 17 12:28:43 EDT 2019
19/07/17 12:32:00 WARN server.TransportChannelHandler:
Exception in connection from /192.168.7.29:46960
java.lang.OutOfMemoryError: Java heap space
at sun.reflect.ByteVectorImpl.trim(ByteVectorImpl.java:70)
at sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:386)
at sun.reflect.MethodAccessorGenerator.generateSerializationConstructor(MethodAccessorGenerator.java:112)
at sun.reflect.ReflectionFactory.generateConstructor(ReflectionFactory.java:398)
at sun.reflect.ReflectionFactory.newConstructorForSerialization(ReflectionFactory.java:360)
at java.io.ObjectStreamClass.getSerializableConstructor(ObjectStreamClass.java:1520)
at java.io.ObjectStreamClass.access$1500(ObjectStreamClass.java:79)
at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:507)
at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:482)
at java.security.AccessController.doPrivileged(Native Method)
at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:482)
at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:379)
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:669)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1876)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1745)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2033)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1567)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2278)
at java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:557)
at org.apache.spark.rpc.netty.NettyRpcEndpointRef.readObject(NettyRpcEnv.scala:495)
at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1158)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2060)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1567)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2278)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2202)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2060)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1567)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:427)
Update 1:
I was able to make this run over 20 days of data by changing the conf as follows below. Still not able to make it run over 30 days. Also, when looking at jobs in UI via history server, I don't see any failed executors in the stage that the job is in when it dies. Not sure what to make of that.
myconf = SparkConf().setAll([
("spark.driver.memory", "32G"),
("spark.driver.cores", "10"),
("spark.executor.cores", "5"),
("spark.executor.memory", "15G"),
("spark.rpc.message.maxSize","800"),
("spark.default.parallelism","2000"),
])