Spark: executor heartbeat timed out - apache-spark

Question

I am working in a databricks cluster that have 240GB of memory and 64 cores. This the settings I defined.
import pyspark
from pyspark.sql import SparkSession
from pyspark.sql.types import *
import pyspark.sql.functions as fs
from pyspark.sql import SQLContext
from pyspark import SparkContext
from pyspark.sql.functions import count
from pyspark.sql.functions import col, countDistinct
from pyspark import SparkContext
from geospark.utils import GeoSparkKryoRegistrator, KryoSerializer
from geospark.register import upload_jars
from geospark.register import GeoSparkRegistrator
spark.conf.set("spark.sql.shuffle.partitions", 1000)
#Recommended settings for using GeoSpark
spark.conf.set("spark.driver.memory", "20g")
spark.conf.set("spark.network.timeout", "1000s")
spark.conf.set("spark.driver.maxResultSize", "10g")
spark.conf.set("spark.serializer", KryoSerializer.getName)
spark.conf.set("spark.kryo.registrator", GeoSparkKryoRegistrator.getName)
upload_jars()
SparkContext.setSystemProperty("geospark.global.charset","utf8")
spark.conf.set
I am working with large datasets and this is the error I get after hours of running.
org.apache.spark.SparkException: Job aborted due to stage failure: Task 3 in stage 10.0 failed 4 times, most recent failure: Lost task 3.3 in stage 10.0 (TID 6054, 10.17.21.12, executor 7):
ExecutorLostFailure (executor 7 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 170684 ms

Answer 1

Let the heartbeat Interval be default(10s) and increase the network time out interval(default -120 s) to 300s (300000ms) and see. Use set and get .
spark.conf.set("spark.sql.<name-of-property>", <value>)
spark.conf.set("spark.network.timeout", 300000 )
or run this script in the notebook .
%scala
dbutils.fs.put("dbfs:/databricks/init/set_spark_params.sh","""
|#!/bin/bash
|
|cat << 'EOF' > /databricks/driver/conf/00-custom-spark-driver-defaults.conf
|[driver] {
| "spark.network.timeout" = "300000"
|}
|EOF
""".stripMargin, true)

Answer 2

The error tells you that the worker has timed out because it took too long.
There is probably some bottleneck happening in the background. Check the spark UI for executor 7, task 3 and stage 10. You also want to check the query that you have been running.
You also want to check these setting for better configuration:
spark.conf.set("spark.databricks.io.cache.enabled", True) # delta caching
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", True) # adaptive query execution for skewed data
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", -1) # setting treshhold on broadcasting
spark.conf.set("spark.databricks.optimizer.rangeJoin.binSize", 20) #range optimizer
Feel free to give us more info on the Spark UI, we can better help you find the problem that way. Also, what kind of query were you doing?

Answer 3

Can you please try the following options ,
repartition the dataframe you work in more numbers for example df.repartition(1000)
--conf spark.network.timeout 10000000
--conf spark.executor.heartbeatInterval=10000000