Spark 1.5.1 Create RDD from Cassandra (ClassNotFoundException: com.datastax.spark.connector.japi.rdd.CassandraTableScanJavaRDD) - apache-spark

Question

I am trying to fetch records from cassandra and create rdd.
JavaRDD<Encounters> rdd = javaFunctions(ctx).cassandraTable("kesyspace1", "employee", mapRowTo(Employee.class));
I am getting this error on submitting job on Spark 1.5.1
Exception in thread "main" java.lang.NoClassDefFoundError: com/datastax/spark/connector/japi/rdd/CassandraTableScanJavaRDD
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:274)
at org.apache.spark.util.Utils$.classForName(Utils.scala:173)
at org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:56)
at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
Caused by: java.lang.ClassNotFoundException: com.datastax.spark.connector.japi.rdd.CassandraTableScanJavaRDD
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
Current Dependencies:
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>1.5.1</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>1.5.1</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.7.1</version>
</dependency>
<dependency>
<groupId>com.datastax.spark</groupId>
<artifactId>spark-cassandra-connector-java_2.11</artifactId>
<version>1.5.0-M2</version>
</dependency>
<dependency>
<groupId>com.datastax.cassandra</groupId>
<artifactId>cassandra-driver-core</artifactId>
<version>3.0.0-alpha4</version>
</dependency>
Java Code:
import com.tempTable.Encounters;
import org.apache.spark.SparkContext;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.SparkConf;
import static com.datastax.spark.connector.japi.CassandraJavaUtil.javaFunctions;
import static com.datastax.spark.connector.japi.CassandraJavaUtil.mapRowTo;
Long now = new Date().getTime();
SparkConf conf = new SparkConf(true)
.setAppName("SparkSQLJob_" + now)
set("spark.cassandra.connection.host", "192.168.1.75")
set("spark.cassandra.connection.port", "9042");
SparkContext ctx = new SparkContext(conf);
JavaRDD<Encounters> rdd = javaFunctions(ctx).cassandraTable("keyspace1", "employee", mapRowTo(Employee.class));
System.out.println("rdd count = "+rdd.count());
Is there issue with version in dependencies?
Please help to resolve this error.
Thanks in advance.

Answer 1

you need to add jar file with SparkConf
.setJars(Seq(System.getProperty("user.dir") + "/target/scala-2.10/sparktest.jar"))
For more information refer http://www.datastax.com/dev/blog/common-spark-troubleshooting

Answer 2

The simple answer is "
you need all the dependencies bundled inside jar file
or
the executor machine should contain all your dependent jar files in
their classpath
Solution for building a fatJar using gradle:
buildscript {
dependencies {
classpath 'com.github.jengelman.gradle.plugins:shadow:1.2.2'
}
repositories {
jcenter()
}
}
apply plugin: 'com.github.johnrengelman.shadow'
Then call "gradle shadowJar" to build your jar file. After that submit your job it should resolve your problem.