How to run Apache spark Java program in standalone - apache-spark

Question

I have written a java program for spark, but I am not able to run it from the command line.
I have followed the steps given in the Quick start guide, but I am getting the following error. Please help me out with this problem.
Here is the error :
hadoopnod#hadoopnod:~/spark-1.2.1/bin$ ./run-example "SimpleApp " --master local /home/hadoopnod/Spark_Java/target/simple-project-1.0.jarjava.lang.ClassNotFoundException: org.apache.spark.examples.SimpleApp
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:342)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties

Answer 1

Create a JAR file using following command. You can find the SimpleApp.class file in "target/classes" folder. cd to this directory.
jar cfve file.jar SimpleApp.class
Put this JAR file into your project in target directory.
This JAR file contains the dependency of your SimpleApp class while submitting your job to Spark.
cd to your spark directory. I am using spark-1.4.0-bin-hadoop2.6. Your cmd looks like this.
spark-1.4.0-bin-hadoop2.6>
Submit your spark program using Spark Submit. If you have structure like Harsha has explained in another answer then provide
--class org.apache.spark.examples.SimpleApp
else
--class SimpleApp
Finally submit your spark program.
spark-1.4.0-bin-hadoop2.6>./bin/spark-submit --class SimpleApp --master local[2] /home/hadoopnod/Spark_Java/target/file.jar

Answer 2

The script ./run-example.sh is used to execute the examples included in the distribution. To run the example "SparkPi" do this...
> cd /apps/spark-1.2.0
> ./bin/run-example SparkPi
If you look at how this script executes its just a new user friendly wrapper which actually calls spark-submit.
Here's an example that executes the same "SparkPi" example from above, but using spark-submit
> .bin/spark-submit --class org.apache.spark.examples.SparkPi --master local examples/target/spark-examples_2.10-1.2.0.jar
You should use spark-submit to run your own code.

Answer 3

ClassNotFoundException: org.apache.spark.examples.SimpleApp
From the above error, it is clear the reason that it cannot find the class you are trying to execute. Have you bundled your java project into a jar file. If you have any other dependencies while creating your jar file, you need to include them as well.
Assume if you have a project structure like this
simpleapp
- src/main/java
- org.apache.spark.examples
-SimpleApp.java
- lib
- dependent.jars (you can put all dependent jars inside lib directory)
- target
- simpleapp.jar (after compiling your source)
You can use any build tool or any IDE to bundle your source into a Jar file. After that if you have added spark/bin directory into your path. you can execute below command from your project directory. you need to add --jars $(echo lib/*.jar | tr ' ' ',' ) only if you have dependent libraries in your SimpleApp.java
spark-submit --jars $(echo lib/*.jar | tr ' ' ',' ) --class org.apache.spark.examples.SimpleApp --master local[2] target/simpleapp.jar

Answer 4

I had the same issue. If you want to use the command provided by the Spark Quickstart, be sure your project has the same architecture:
find .
./pom.xml
./src
./src/main
./src/main/java
./src/main/java/SimpleApp.java
It may be not the case for you but my pom.xml built my architecture like
./src/main/java/myGroupId/myArtifactId/SimpleApp.java
I moved my class in default package and it worked fine after.