How to run Spark structured streaming using local JAR files - apache-spark

Question

I'm using one of the Docker images of EMR on EKS (emr-6.5.0:20211119) and investigating how to work on Kafka with Spark Structured Programming (pyspark). As per the integration guide, I run a Python script as following.
$SPARK_HOME/bin/spark-submit \
--deploy-mode client \
--master local \
--packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.1.2 \
<myscript>.py
It download the package from Maven central and I see some JAR files are downloaded into ~/.ivy2/jars.
com.github.luben_zstd-jni-1.4.8-1.jar org.apache.spark_spark-sql-kafka-0-10_2.12-3.1.2.jar org.slf4j_slf4j-api-1.7.30.jar
org.apache.commons_commons-pool2-2.6.2.jar org.apache.spark_spark-token-provider-kafka-0-10_2.12-3.1.2.jar org.spark-project.spark_unused-1.0.0.jar
org.apache.kafka_kafka-clients-2.6.0.jar org.lz4_lz4-java-1.7.1.jar org.xerial.snappy_snappy-java-1.1.8.2.jar
However I find the main JAR file is already download into $SPARK_HOME/external/lib and I wonder how to make use of it instead of downloading it.
spark-avro_2.12-3.1.2-amzn-1.jar spark-ganglia-lgpl.jar spark-streaming-kafka-0-10-assembly_2.12-3.1.2-amzn-1.jar spark-streaming-kinesis-asl-assembly.jar
spark-avro.jar **spark-sql-kafka-0-10_2.12-3.1.2-amzn-1.jar spark-streaming-kafka-0-10-assembly.jar spark-token-provider-kafka-0-10_2.12-3.1.2-amzn-1.jar
spark-ganglia-lgpl_2.12-3.1.2-amzn-1.jar **spark-sql-kafka-0-10.jar spark-streaming-kinesis-asl-assembly_2.12-3.1.2-amzn-1.jar spark-token-provider-kafka-0-10.jar
UPDATE 2022-03-09
I tried after updating spark-defaults.conf as shown below - added the external lib folder.
spark.driver.extraClassPath /usr/lib/spark/external/lib/*:...
spark.driver.extraLibraryPath ...
spark.executor.extraClassPath /usr/lib/spark/external/lib/*:...
spark.executor.extraLibraryPath ...
I can run without --packages but it fails with the following error.
22/03/09 05:37:25 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.NoClassDefFoundError: org/apache/commons/pool2/impl/GenericKeyedObjectPoolConfig
at org.apache.spark.sql.kafka010.consumer.KafkaDataConsumer$.<init>(KafkaDataConsumer.scala:623)
at org.apache.spark.sql.kafka010.consumer.KafkaDataConsumer$.<clinit>(KafkaDataConsumer.scala)
at org.apache.spark.sql.kafka010.KafkaBatchPartitionReader.<init>(KafkaBatchPartitionReader.scala:52)
at org.apache.spark.sql.kafka010.KafkaBatchReaderFactory$.createReader(KafkaBatchPartitionReader.scala:40)
at org.apache.spark.sql.execution.datasources.v2.DataSourceRDD.compute(DataSourceRDD.scala:60)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:131)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.pool2.impl.GenericKeyedObjectPoolConfig
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 33 more
It doesn't help although I tried with adding --packages org.apache.commons:commons-pool2:2.6.2.

Answer 1

You would use --jars to refer to local filesystem in-place of --packages

Answer 2

Unfortunately I cannot submit an app only with the JAR files in $SPARK_HOME/external/lib due to an error. The details of the error are updated to the question. Instead I ended up pre-downloading the package JAR files and using those.
I first ran with the following command. Here foo.py is an empty file and it'll download the package JAR files into /home/hadoop/.ivy2/jars.
$SPARK_HOME/bin/spark-submit \
--deploy-mode client \
--master local \
--packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.1.2 \
foo.py
Then I updated spark-defaults.conf as following.
spark.driver.extraClassPath /home/hadoop/.ivy2/jars/*:...
spark.driver.extraLibraryPath ...
spark.executor.extraClassPath /home/hadoop/.ivy2/jars/*:...
spark.executor.extraLibraryPath ...
After that, I ran the submit command without --packages and it worked without an error.
$SPARK_HOME/bin/spark-submit \
--deploy-mode client \
--master local \
<myscript>.py
This approach is likely to be useful when it takes long to download package JAR files as they can be pre-downloaded. Note EMR on EKS supports using a custom image from ECR.