Spark-submit in cluster mode - apache-spark

Question

I'm facing a problem launching a Spark Application in cluster mode
This is the .sh :
export SPARK_MAJOR_VERSION=2
spark-submit \
--master yarn \
--deploy-mode cluster \
--driver-memory 8G \
--executor-memory 8G \
--total-executor-cores 4 \
--num-executors 4 \
/home/hdfs/spark_scripts/ETL.py &> /home/hdfs/spark_scripts/log_spark.txt
In YARN logs, I found out that there's an Import Error related to a .py file that I need in "ETL.py". In other words, in "ETL.py", I',ve got a line in which I do this :
import AppUtility
AppUtilit.py is in the same path of ETL.py
In local mode,it works
This is the YARN log:
20/04/28 10:59:59 INFO compress.CodecPool: Got brand-new decompressor [.deflate]
Container: container_e64_1584554814241_22431_02_000001 on ftpandbit02.carte.local_45454
LogAggregationType: AGGREGATED
LogType:stdout
LogLastModifiedTime:Tue Apr 28 10:57:10 +0200 2020
LogLength:138
LogContents:
Traceback (most recent call last):
File "ETL.py", line 8, in
import AppUtility
ImportError: No module named AppUtility
End of LogType:stdout
End of LogType:prelaunch.err

Answer 1

It depends on either client mode or cluster mode.
If you use Spark in Yarn client mode,
you'll need to install any dependencies to the machines on which Yarn starts the executors. That's the only surefire way to make this work.
Using Spark with Yarn cluster mode, is a different story. You can distribute python dependencies with
spark-submit ./bin/spark-submit --py-files AppUtility.py
/home/hdfs/spark_scripts/ETL.py
The --py-files directive sends the file to the Spark workers but does not add it to the PYTHONPATH.
To add the dependencies to the PYTHONPATH to fix the ImportError, add the following line to the Spark job, ETL.py
sc.addPyFile(PATH)
PATH: AppUtility.py (It can be either a local file, a file in HDFS,zip or an HTTP, HTTPS or FTP URI)