Apache Spark : How to add default and specific dependency? - apache-spark

Question

My spark-defaults.conf :
#a package I need everytime
spark.jars.packages org.influxdb:influxdb-java:2.14
When I launch a job :
spark-shell --master yarn --num-executors 6 --packages "a random package that I need only for this job specifically"
I get this error :
java.lang.NoClassDefFoundError: org/influxdb/InfluxDBFactory
at ch.cern.sparkmeasure.InfluxDBSink.<init>(influxdbsink.scala:53)
at ch.cern.sparkmeasure.InfluxDBSinkExtended.<init>(influxdbsink.scala:232)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.spark.util.Utils$$anonfun$loadExtensions$1.apply(Utils.scala:2688)
at org.apache.spark.util.Utils$$anonfun$loadExtensions$1.apply(Utils.scala:2680)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
at org.apache.spark.util.Utils$.loadExtensions(Utils.scala:2680)
at org.apache.spark.SparkContext$$anonfun$setupAndStartListenerBus$1.apply(SparkContext.scala:2387)
at org.apache.spark.SparkContext$$anonfun$setupAndStartListenerBus$1.apply(SparkContext.scala:2386)
at scala.Option.foreach(Option.scala:257)
at org.apache.spark.SparkContext.setupAndStartListenerBus(SparkContext.scala:2386)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:555)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
at org.apache.spark.repl.Main$.createSparkSession(Main.scala:106)
... 62 elided
Caused by: java.lang.ClassNotFoundException: org.influxdb.InfluxDBFactory
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 88 more
I think --packages overriding packages in spark-default.conf so I get this error
I don't want to include every time influxdb packages.
I want to have fixe packages in spark-default.conf and dynamic package when I launch a job with --packages but it seems incompatible.
Any idea ?

Answer 1

That class is indeed part of influxdb-java, so you shouldn't be getting that error. Especially if your code isn't even trying to use Influx packages
However, if you have packages specific to a certain application, you should actually package them as part of that application. Then others will be able to run it as well and not remember the specific invocation of the spark submit options
If you are using a build system for your code, look into creating an uber jar
I'm not familiar with the semantics of spark-defaults or if it's overridden, but if you do have packages that always need to be part of Spark applications, you should just download the JARs directly into the Spark workers classpaths, not make applications download them every time