Exception while submit spark job on yarn cluster with remote jvm - apache-spark

Question

I am using below java code to submit job on yarn-cluster.
public ApplicationId submitQuery(String requestId, String query,String fileLocations) {
String driverJar = getDriverJar();
String driverClass = propertyService.getAppPropertyValue(TypeString.QUERY_DRIVER_CLASS);
String driverAppName = propertyService.getAppPropertyValue(TypeString.DRIVER_APP_NAME);
String extraJarsNeeded = propertyService.getAppPropertyValue(TypeString.DRIVER_EXTRA_JARS_NEEDED);
String[] args = new String[] {
// the name of your application
"--name",
driverAppName,
// memory for driver (optional)
"--driver-memory",
"1000M",
// path to your application's JAR file
// required in yarn-cluster mode
"--jar",
"local:/home/ankit/Repository/Personalization/rtis/Cust360QueryDriver/target/SnapdealCustomer360QueryDriver-jar-with-selective-dependencies.jar",
"--addJars",
"local:/home/ankit/Downloads/lib/spark-assembly-1.3.1-hadoop2.4.0.jar,local:/home/ankit/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar,local:/home/ankit/.m2/repository/org/slf4j/slf4j-log4j12/1.7.5/slf4j-log4j12-1.7.5.jar",
// name of your application's main class (required)
"--class",
driverClass,
"--arg",
requestId,
"--arg",
query,
"--arg",
fileLocations,
"--arg",
"yarn-client"
};
System.setProperty("HADOOP_CONF_DIR", "/home/hduser/hadoop-2.7.0/etc/hadoop");
Configuration config = new Configuration();
config.set("yarn.resourcemanager.address", propertyService.getAppPropertyValue(TypeString.RESOURCE_MANGER_URL));
config.set("fs.default.name", propertyService.getAppPropertyValue(TypeString.FS_DEFAULT_NAME));
System.setProperty("SPARK_YARN_MODE", "true");
SparkConf sparkConf = new SparkConf();
ClientArguments cArgs = new ClientArguments(args, sparkConf);
// create an instance of yarn Client client
Client client = new Client(cArgs, config, sparkConf);
ApplicationId id = client.submitApplication();
return id;
}
Job is getting submitted to yarn-cluster and i am able to retrieve application id but i am getting below Exception while running job on spark cluster.
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/Logging
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.Logging
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 13 more
though mentioned class in /home/ankit/Downloads/lib/spark-assembly-1.3.1-hadoop2.4.0.jar. looks like jar mentioned in --addJars is not getting added in driver's spark context.
Am i doing something wrong?? Any help would be appreciated.

Answer 1

Are you deploying on Cloudera's distribution ? spark.yarn.jar in the CDH 5.4 config has a 'local:' prefix for local files, but Spark version >= 1.5 does not like this, you should just use the full path name for your spark assembly. See also here.

Answer 2

Try building JAR without spark dependency and pass dependent jars with --jars in spark submit. Most of the times ClassNotFoundException is due to spark and application itself is dependent on same jar.
Suggested solution:
package without dependency and add dependent jars with --jars during
spark-submit
Modify application to use same version of third-party
library as spark has
Use shading in your build tool