I am trying to migrate our application to spark running on yarn. I use cmdline as spark-submit --master yarn --deploy-mode cluster -jars ${my_jars}...
But yarn throws Expections with the following log:
Container id: container_1462875359170_0171_01_000002
Exit code: 1
Exception message: .../launch_container.sh: line 4145: /bin/bash: Argument list too long
I think the reason may be that we have too many jars (684 jars separated by comma) specified by option --jars ${my_jars}, my question is what is the graceful way to specify all our jars? Or how can we avoid this yarn error?
Check if you can use spark.driver.extraClassPath extraClassPath Spark Documentation
spark.driver.extraClassPath /fullpath/firs.jar:/fullpath/second.jar
spark.executor.extraClassPath /fullpath/firs.jar:/fullpath/second.jar
Just found the threadspark-submit-add-multiple-jars-in-classpath
I'd try these two things
Build a fat jar for spark submit application or
Build a thin jar with maven and install unavailable jars in the maven repo. so that it will be available to load at runtime in the cluster.
Try sbt-assembly which packages all your classes and dependency classes into an uber jar.
It is very easy and comfortable to use, but you have to take care of two things:
version conflict
the jar would be a little bit large
Related
I have a pyspark application, which I submit using spark-submit like this:
spark-submit --deploy-mode cluster --master yarn --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.0 app.py
This works. But only because the cluster has internet access, and can download the dependency specified by "--packages".
The goal
Now I would like to bundle my pyspark application with its dependencies, so they would not be downloaded.
I find some tutorials about how to bundle python dependencies, but that is not what I need.
What I tried:
I used the maven-shade-plugin, creating a maven package like this where I specified org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.0 as a dependency and bundled it using
mvn package
I submitted the resulting jar using:
spark-submit --deploy-mode cluster --master yarn app.jar --py-files app.py
But I got:
Exception in thread "main" org.apache.spark.SparkException: No main class set in JAR; please specify one with --class.
I do not have a main class, because my main is python packge, or not?!?
So how do I do this?
I have spark job that writes data to Cassandra(Cassandra is on GCP). When I run this from IntelliJIDEA (my IDE) it works perfectly fine. The data is perfectly sent and written to Cassandra. However, this fails when I package my project into a fat jar and run it.
Here is an example of how I run it.
spark-submit --class com.testing.Job --master local out/artifacts/SparkJob_jar/SparkJob.jar 1 0
However, this fails for me and gives me the following errors
Caused by: java.io.IOException: Failed to open native connection to Cassandra at {X.X.X:9042} :: 'com.datastax.oss.driver.api.core.config.ProgrammaticDriverConfigLoaderBuilder com.datastax.oss.driver.api.core.config.DriverConfigLoader.programmaticBuilder()'
Caused by: java.lang.NoSuchMethodError: 'com.datastax.oss.driver.api.core.config.ProgrammaticDriverConfigLoaderBuilder com.datastax.oss.driver.api.core.config.DriverConfigLoader.programmaticBuilder()'
My artifacts file does include the spark-Cassandra files
spark-cassandra-connector-driver_2.12-3.0.0-beta.jar
spark-cassandra-connector_2.12-3.0.0-beta.jar
I'm wondering why this is happening and how I can fix it?
The problem is that besides that 2 things, you need to have more jars - full Java driver, and its dependencies. You have following possibilities to fix that:
You need to make sure that these artifact is packaged into the resulting jar (so-called "fat jar" or an "assembly") using Maven or SBT, or anything else
you can can specify maven coordinates com.datastax.spark:spark-cassandra-connector_2.12:3.0.0-beta with --packages like this --packages com.datastax.spark:spark-cassandra-connector_2.12:3.0.0-beta
you can download spark-cassandra-connector-assembly artifact to the node from which you're doing spark-submit, and then use that file name with --jars
See the documentation for Spark Cassandra Connector for more details.
I have a Mesos Cluster on which I want to run Spark jobs.
I have downloaded the spark precompiled package and I can use the spark-shell by simply decompressing the archive.
So far, I haven't managed to run spark jobs on the Mesos Cluster.
First question : Do I need to install and build Spark from source to get it work on Mesos? And Does this precompiled package used only for Spark on Yarn and Hadoop?
Second question : Can anyone provide the best way to build spark. I have found many ways like :
sbt clean assembly
./build/mvn -Pmesos -DskipTests clean package
./build/sbt package
I don't know which one to use, and whether they are all correct or not.
I have a Hortonworks yarn cluster with Spark 2.1.
However I want to run my application with spark 2.3+ (because an essential third-party ML library in use needs it).
Do we have to use spark-submit from the Spark 2.1 version or we have to submit job to yarn using Java or Scala with a FAT jar? Is this even possible? What about Hadoop libraries?
On a Hortonworks cluster, running a custom spark version in yarn client/cluster mode needs following steps:
Download Spark prebuilt file with appropriate hadoop version
Extract and unpack into a spark folder. eg. /home/centos/spark/spark-2.3.1-bin-hadoop2.7/
Copy jersey-bundle 1.19.1 jar into spark jar folder [Download from here][1]
Create a zip file containing all the jars in spark jar folder. Spark-jar.zip
Put this spark-jar.zip file in a world accessible hdfs location such as (hdfs dfs -put spark-jars.zip /user/centos/data/spark/)
get hdp version (hdp-select status hadoop-client): eg output. hadoop-client - 3.0.1.0-187
Use the above hdp version in export commands below
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-/usr/hdp/3.0.1.0-187/hadoop/conf}
export HADOOP_HOME=${HADOOP_HOME:-/usr/hdp/3.0.1.0-187/hadoop}
export SPARK_HOME=/home/centos/spark/spark-2.3.1-bin-hadoop2.7/
Edit the spark-defaults.conf file in spark_home/conf directory, add following entries
spark.driver.extraJavaOptions -Dhdp.version=3.0.1.0-187
spark.yarn.am.extraJavaOptions -Dhdp.version=3.0.1.0-187
create java-opts file in spark_home/conf directory, add below entries and use the above mentioned hdp version
-Dhdp.version=3.0.1.0-187
export LD_LIBRARY_PATH=/usr/hdp/3.0.1.0-187/hadoop/lib/native:/usr/hdp/3.0.1.0-187/hadoop/lib/native/Linux-amd64-64
spark-shell --master yarn --deploy-mode client --conf spark.yarn.archive=hdfs:///user/centos/data/spark/spark-jars.zip
I assume you use sbt as the build tool in your project. The project itself could use Java or Scala. I also think that the answer in general would be similar if you used gradle or maven, but the plugins would simply be different. The idea is the same.
You have to use an assembly plugin (e.g. sbt-assembly) that is going to bundle all non-Provided dependencies together, including Apache Spark, in order to create a so-called fat jar or uber-jar.
If the custom Apache Spark version is part of the application jar that version is going to be used whatever spark-submit you use for deployment. The trick is to trick the classloader so it loads the jars and classes of your choice not spark-submit's (and hence whatever is used in the cluster).
I am trying to do spark-submit of the Apache beam word-count example by giving the below command
spark-submit --class org.apache.beam.examples.WordCount word-count-beam-0.1.jar --inputFile=pom.xml --output=counts --runner=SparkRunner
I get the below Exception:
Exception in thread "main" java.lang.IllegalArgumentException: Unknown
'runner' specified 'SparkRunner', supported pipeline runners
[DirectRunner]
Your pom.xml needs to include a dependency on the Spark runner. The documentation on using the Spark runner includes more details about what is necessary.
Looks like you're not building an Uber-jar with the necessary Spark dependencies.
Re-run your Maven package as follows:
mvn package -Pspark-runner
This will build a Jar in target containing the wordcount classes as well as all of the necessary spark dependencies called something like:
word-count-beam-bundled-0.1.jar
Then use that jar in the spark-submit command