Hudi Job failing with ExecutorDeadException in spark - apache-spark

Question

I have recently started using Hudi so not very experience on hudi internals.
I am trying to run a hudi job with around 50GB of base data on spark in upsert mode.
from pyspark.sql import SparkSession spark = SparkSession \
    .builder \
    .appName("hudi_test") \
    .enableHiveSupport().getOrCreate()  
tableName = "hudi_test12"
basePath = "/tmp/rahul/hudi_test12" 
df = spark.read.parquet("/user/data/input")
df = df.repartition(500)
#df.show() 
hudi_options = {
    'hoodie.table.name': tableName,
    'hoodie.datasource.write.recordkey.field': 'row_id',
    'hoodie.datasource.write.partitionpath.field': 'rpt_partition_id',
    'hoodie.datasource.write.table.name': tableName,
    'hoodie.datasource.write.operation': 'upsert',
    'hoodie.datasource.write.precombine.field': 'created',
    'hoodie.upsert.shuffle.parallelism': 500,
    'hoodie.insert.shuffle.parallelism': 500
} 
df.write.format("hudi"). \
    options(**hudi_options). \
    mode("overwrite"). \
    save(basePath)
I creates around 3000 task at stage SparkUpsertCommitActionExecutor and after around 1000 tasks completion, all other tasks start failing and job eventually fails.
On executors, i see following error logs getting printed multiple times before executor gets killed
23/01/24 12:37:55 ERROR client.TransportResponseHandler: Still have 1 requests outstanding when connection from mnplld-shddn02.india.airtel.itm/10.240.8.108:42916 is closed
23/01/24 12:37:55 INFO shuffle.RetryingBlockTransferor: Retrying fetch (1/3) for 1 outstanding blocks after 5000 ms
23/01/24 12:38:01 INFO client.TransportClientFactory: Found inactive connection to mnplld-shddn02.india.airtel.itm/10.240.8.108:42916, creating a new one.
23/01/24 12:38:01 ERROR shuffle.RetryingBlockTransferor: Exception while beginning fetch of 1 outstanding blocks (after 1 retries)
org.apache.spark.ExecutorDeadException: The relative remote executor(Id: 19), which maintains the block data to fetch is dead.
at org.apache.spark.network.netty.NettyBlockTransferService$$anon$2.createAndStart(NettyBlockTransferService.scala:136)
at org.apache.spark.network.shuffle.RetryingBlockTransferor.transferAllOutstanding(RetryingBlockTransferor.java:154)
at org.apache.spark.network.shuffle.RetryingBlockTransferor.lambda$initiateRetry$0(RetryingBlockTransferor.java:184)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
23/01/24 12:38:01 ERROR storage.ShuffleBlockFetcherIterator: Failed to get block(s) from mnplld-shddn02.india.airtel.itm:42916
org.apache.spark.ExecutorDeadException: The relative remote executor(Id: 19), which maintains the block data to fetch is dead.
at org.apache.spark.network.netty.NettyBlockTransferService$$anon$2.createAndStart(NettyBlockTransferService.scala:136)
at org.apache.spark.network.shuffle.RetryingBlockTransferor.transferAllOutstanding(RetryingBlockTransferor.java:154)
at org.apache.spark.network.shuffle.RetryingBlockTransferor.lambda$initiateRetry$0(RetryingBlockTransferor.java:184)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
spark configurations given are as follows :
spark3-submit --master yarn --jars avro-1.10.0.jar,hudi-spark3.2-bundle_2.12-0.13.0-SNAPSHOT.jar --deploy-mode cluster --conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer'   --conf 'spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog'   --conf 'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension' --driver-memory 5g  --executor-memory 22g --num-executors 20 --executor-cores 12 --queue ocp test.py
I think numbers of executros and cores are sufficient for the data. What is reason for the failures and how to fix this ?