kind: YARN_CLIENT_TOKEN, User class threw exception: java.io.FileNotFoundException: /home/username/config.properties (No such file or directory) - apache-spark

I am trying to run the below spark-submit command reading config file from command line args, when I run the code on local it runs fine, but when I run using yarn, it fails with the below error
Spark Submit :
time spark-submit --files /etc/hive/conf/hive-site.xml --master yarn --deploy-mode cluster --class IntegrateAD /home/username/s3ReadWrite-assembly-1.1.jar "day0" "/home/username/config.properties"
Error :
INFO yarn.Client:
client token: Token { kind: YARN_CLIENT_TOKEN, service: }
diagnostics: User class threw exception: java.io.FileNotFoundException: /home/username/config.properties (No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.(FileInputStream.java:138)
Code I am running :
val propFile = new File(args(1))
val properties: Properties = new Properties()
if (propFile != null) {
// val source = Source.fromFile(propFile)
//properties.load(source.bufferedReader())
properties.load(new FileInputStream(propFile))
properties
}
else {
logger.error("properties file cannot be loaded at path " + propFile)
throw new FileNotFoundException("Properties file cannot be loaded")
}
Please help me with what maybe wrong here, in my code or my spark-submit or is it something else.
Thanks for your help in advance.

Related

Spark on Kubernetes with Minio - Postgres -> Minio -unable to create executor due to

Hi I am facing an error with providing dependency jars for spark-submit in kubernetes.
/usr/middleware/spark-3.1.1-bin-hadoop3.2/bin/spark-submit --master k8s://https://112.23.123.23:6443 --deploy-mode cluster --name spark-postgres-minio-kubernetes --jars file:///AirflowData/kubernetes/externalJars/postgresql-42.2.14.jar --driver-class-path file:///AirflowData/kubernetes/externalJars/postgresql-42.2.14.jar --conf spark.executor.instances=1 --conf spark.kubernetes.namespace=spark --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark --conf spark.kubernetes.file.upload.path=s3a://daci-dataintegration/spark-operator-on-k8s/code --conf spark.hadoop.fs.s3a.fast.upload=true --conf spark.kubernetes.container.image=hostname:5000/spark-py:spark3.1.2 file:///AirflowData/kubernetes/python/postgresminioKube.py
Below is the code to execute. The jars needed for the S3 minio and configurations are placed in the spark_home/conf and spark_home/jars and the docker image is created.
from pyspark.sql import SparkSession
from pyspark.sql.functions import *
from pyspark.sql import functions as F
spark = SparkSession.builder.appName("Postgres-Minio-Kubernetes").getOrCreate()
import json
#spark = SparkSession.builder.config('spark.driver.extraClassPath', '/hadoop/externalJars/db2jcc4.jar').getOrCreate()
jdbcUrl = "jdbc:postgresql://{0}:{1}/{2}".format("hosnamme", "port", "db")
connectionProperties = {
"user" : "username",
"password" : "password",
"driver": "org.postgresql.Driver",
"fetchsize" : "100000"
}
pushdown_query = "(select * from public.employees) emp_als"
df = spark.read.jdbc(url=jdbcUrl, table=pushdown_query, column="employee_id", lowerBound=1, upperBound=100, numPartitions=2, properties=connectionProperties)
df.write.format('csv').options(delimiter=',').mode('overwrite').save('s3a://daci-dataintegration/spark-operator-on-k8s/data/postgres-minio-csv/')
df.write.format('parquet').options(delimiter='|').options(header=True).mode('overwrite').save('s3a://daci-dataintegration/spark-operator-on-k8s/data/postgres-minio-csv/')
Error is below . It is trying to execute the jar for some reason
21/11/09 17:05:44 INFO SparkContext: Added JAR file:/tmp/spark-d987d7e7-9d49-4523-8415-1e438da1730e/postgresql-42.2.14.jar at spark://spark-postgres-minio-kubernetes-49d7d77d05a980e5-driver-svc.spark.svc:7078/jars/postgresql-42.2.14.jar with timestamp 1636477543573
21/11/09 17:05:49 ERROR TaskSchedulerImpl: Lost executor 1 on 192.168.216.12: Unable to create executor due to ./postgresql-42.2.14.jar
The external jars are getting added to the /opt/spark/work-dir and it didnt had access. So i changed the dockerfile to have access to the folder and then it worked.
RUN chmod 777 /opt/spark/work-dir

Spark, kerberos, yarn-cluster -> connection to hbase

Facing one issue with Kerberos enabled Hadoop cluster.
We are trying to run a streaming job on yarn-cluster, which interacts with Kafka (direct stream), and hbase.
Somehow, we are not able to connect to hbase in the cluster mode. We use keytab to login to hbase.
This is what we do:
spark-submit --master yarn-cluster --keytab "dev.keytab" --principal "dev#IO-INT.COM" --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j_executor_conf.properties -XX:+UseG1GC" --conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=log4j_driver_conf.properties -XX:+UseG1GC" --conf spark.yarn.stagingDir=hdfs:///tmp/spark/ --files "job.properties,log4j_driver_conf.properties,log4j_executor_conf.properties" service-0.0.1-SNAPSHOT.jar job.properties
To connect to hbase:
def getHbaseConnection(properties: SerializedProperties): (Connection, UserGroupInformation) = {
val config = HBaseConfiguration.create();
config.set("hbase.zookeeper.quorum", HBASE_ZOOKEEPER_QUORUM_VALUE);
config.set("hbase.zookeeper.property.clientPort", 2181);
config.set("hadoop.security.authentication", "kerberos");
config.set("hbase.security.authentication", "kerberos");
config.set("hbase.cluster.distributed", "true");
config.set("hbase.rpc.protection", "privacy");
config.set("hbase.regionserver.kerberos.principal", “hbase/_HOST#IO-INT.COM”);
config.set("hbase.master.kerberos.principal", “hbase/_HOST#IO-INT.COM”);
UserGroupInformation.setConfiguration(config);
var ugi: UserGroupInformation = null;
if (SparkFiles.get(properties.keytab) != null
&& (new java.io.File(SparkFiles.get(properties.keytab)).exists)) {
ugi = UserGroupInformation.loginUserFromKeytabAndReturnUGI(properties.kerberosPrincipal,
SparkFiles.get(properties.keytab));
} else {
ugi = UserGroupInformation.loginUserFromKeytabAndReturnUGI(properties.kerberosPrincipal,
properties.keytab);
}
val connection = ConnectionFactory.createConnection(config);
return (connection, ugi);
}
and we connect to hbase:
….
foreachRDD { rdd =>
if (!rdd.isEmpty()) {
//var ugi: UserGroupInformation = Utils.getHbaseConnection(properties)._2
rdd.foreachPartition { partition =>
val connection = Utils.getHbaseConnection(propsObj)._1
val table = …
partition.foreach { json =>
}
table.put(puts)
table.close()
connection.close()
}
}
}
Keytab file is not getting copied to yarn staging/temp directory, we are not getting that in SparkFiles.get… and if we pass keytab with --files, spark-submit is failing because it’s there in --keytab already.
error is:
This server is in the failed servers list: myserver.test.com/120.111.25.45:60020
RpcRetryingCaller{globalStartTime=1497943263013, pause=100, retries=5}, org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: myserver.test.com/120.111.25.45:60020
RpcRetryingCaller{globalStartTime=1497943263013, pause=100, retries=5}, org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: myserver.test.com/120.111.25.45:60020 at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:147)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:935)

where to check my yarn+spark's applications' log?

I wrote an application with yarn+spark, for simplicity, I list the following
object testKafkaSparkStreaming extends Logging {
private class Parser extends Logging{
def parse(row: String): Row =
{
val row = "/_dc.gif| |20160616063934| |39.190.5.69| |729252016040907094857083a3c7c62e"
logInfo("pengcz starting parse " + row)
}
}
def main(args: Array[String]) {
...
logInfo("main starting parse " + row)
...
}
}
When I executed:
spark-submit --master local[*] --class $CLASS $JAR ...
I can see the two log infos in the console
But when I executed:
spark-submit --master yarn --class $CLASS $JAR ...
And I opened the yarn web ui of my own application:
http://192.168.36.172:8088/cluster/app/application_1465894400511_3624
And I click logs under the page, I got:
But the page has not information including my two logs
What should I do to find my own logs?
Any advice will be appreciated!
You can use this command to see the Yarn Logs of an Application :
yarn logs -applicationId <Your Yarn Application ID> // i.e. application_1465894400511_3624

DSE spark-submit failing with SHUTDOWN_HOOK_PRIORITY, I do not have hadoop2 in cp

I am trying run the following java driver pgm in my local mac env, and i'm pretty sure I do not have hadoop2 in my class path and not sure why it still fails with shutdown-hook-priority error ?. Any insight will be of gr8 help, and I can run pyspark job with no exception
I am running dse 484 on my local and following is invocation
$SPARKBINFOLDER/dse spark-submit --master local[2] --class com.sample.driver.SampleLoader SampleLoader.jar $#
Following is code snippet I am using
public class SampleLoader implements Serializable {
private transient SparkConf sconf;
private SampleLoader(SparkConf sconf) {
this.sconf = sconf;
}
private void run() {
//
ClassLoader cl = ClassLoader.getSystemClassLoader();
URL[] urls = ((URLClassLoader)cl).getURLs();
for(URL url: urls){
System.out.println(url.getFile());
}
//
JavaSparkContext jsc = new JavaSparkContext(sconf);
runSparkJob(jsc);
jsc.stop();
}
private void runSparkJob(JavaSparkContext jsc) {
}
}
Following is classloader cp which I printed just before failed line of code ( JavaSparkContext jsc = new JavaSparkContext(sconf);)
########Printing the Classloader class path ........ /Users/xxxxxx/cassandra/dse484/resources/spark/conf/ /Users/xxxxxx/cassandra/dse484/lib/dse-core-4.8.4.jar /Users/xxxxxx/cassandra/dse484/lib/dse-hadoop-4.8.4.jar /Users/xxxxxx/cassandra/dse484/lib/dse-hive-4.8.4.jar /Users/xxxxxx/cassandra/dse484/lib/dse-search-4.8.4.jar /Users/xxxxxx/cassandra/dse484/lib/dse-spark-4.8.4.jar /Users/xxxxxx/cassandra/dse484/lib/dse-sqoop-4.8.4.jar /Users/xxxxxx/cassandra/dse484/resources/spark/conf/ /Users/xxxxxx/cassandra/dse484/resources/spark/lib/JavaEWAH-0.3.2.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/RoaringBitmap-0.4.5.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/ST4-4.0.4.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/activation-1.1.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/akka-actor_2.10-2.3.4-spark.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/akka-remote_2.10-2.3.4-spark.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/akka-slf4j_2.10-2.3.4-spark.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/ant-1.9.1.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/ant-launcher-1.9.1.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/antlr-2.7.7.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/antlr-runtime-3.4.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/arpack_combined_all-0.1.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/asm-3.1.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/asm-4.0.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/asm-commons-3.1.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/asm-tree-3.1.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/avro-1.7.7.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/avro-ipc-1.7.7.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/avro-mapred-1.7.7-hadoop1.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/avro-mapred-1.7.7.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/bonecp-0.8.0.RELEASE.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/breeze-macros_2.10-0.11.2.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/breeze_2.10-0.11.2.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/chill-java-0.5.0.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/chill_2.10-0.5.0.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/commons-cli-1.2.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/commons-codec-1.9.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/commons-collections-3.2.2.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/commons-compress-1.4.1.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/commons-httpclient-3.1.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/commons-io-2.4.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/commons-lang-2.4.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/commons-lang3-3.3.2.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/commons-logging-1.2.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/commons-math3-3.4.1.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/commons-net-2.2.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/compress-lzf-1.0.3.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/config-1.2.1.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/core-1.1.2.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/guava-16.0.1.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/hamcrest-core-1.3.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/httpclient-4.4.1.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/httpcore-4.4.1.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/ivy-2.4.0.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/jackson-annotations-2.3.5.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/jackson-core-2.3.5.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/jackson-core-asl-1.9.13.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/jackson-databind-2.3.5.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/jackson-mapper-asl-1.9.13.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/jackson-module-scala_2.10-2.3.5.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/jansi-1.4.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/javax.servlet-3.0.0.v201112011016.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/javolution-5.5.1.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/jaxb-api-2.2.7.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/jaxb-core-2.2.7.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/jaxb-impl-2.2.7.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/jdo-api-3.0.1.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/jersey-core-1.9.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/jersey-server-1.9.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/jets3t-0.7.1.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/jetty-all-7.6.0.v20120127.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/jetty-continuation-8.1.14.v20131031.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/jetty-http-8.1.14.v20131031.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/jetty-io-8.1.14.v20131031.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/jetty-security-8.1.14.v20131031.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/jetty-server-8.1.14.v20131031.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/jetty-servlet-8.1.14.v20131031.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/jetty-util-6.1.26.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/jetty-util-8.1.14.v20131031.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/jline-0.9.94.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/jline-2.10.5.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/joda-convert-1.2.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/joda-time-2.3.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/jodd-core-3.6.3.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/jpam-1.1.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/json-20090211.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/json4s-ast_2.10-3.2.10.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/json4s-core_2.10-3.2.10.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/json4s-jackson_2.10-3.2.10.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/jsr166e-1.1.0.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/jsr305-2.0.1.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/jta-1.1.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/jtransforms-2.4.0.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/junit-4.12.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/kryo-2.21.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/libfb303-0.9.3.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/libthrift-0.9.3.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/lz4-1.2.0.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/mail-1.4.1.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/mesos-0.21.1-shaded-protobuf.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/metrics-core-3.1.2.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/metrics-graphite-3.1.2.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/metrics-json-3.1.2.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/metrics-jvm-3.1.2.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/minlog-1.2.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/objenesis-1.2.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/opencsv-2.3.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/oro-2.0.8.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/paranamer-2.6.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/parquet-column-1.6.0rc3.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/parquet-common-1.6.0rc3.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/parquet-encoding-1.6.0rc3.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/parquet-format-2.2.0-rc1.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/parquet-generator-1.6.0rc3.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/parquet-hadoop-1.6.0rc3.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/parquet-hadoop-bundle-1.3.2.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/parquet-jackson-1.6.0rc3.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/pmml-agent-1.1.15.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/pmml-model-1.1.15.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/pmml-schema-1.1.15.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/protobuf-java-2.5.0-spark.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/py4j-0.8.1.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/pyrolite-4.4.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/quasiquotes_2.10-2.0.1.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/reflectasm-1.07-shaded.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/scala-compiler-2.10.5.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/scala-library-2.10.5.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/scala-reflect-2.10.5.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/scalap-2.10.5.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/servlet-api-2.5.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/slf4j-api-1.7.12.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/snappy-0.2.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/snappy-java-1.0.5.3.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/spark-bagel_2.10-1.4.2.2.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/spark-cassandra-connector-java_2.10-1.4.1.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/spark-cassandra-connector_2.10-1.4.1.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/spark-catalyst_2.10-1.4.2.2.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/spark-core_2.10-1.4.2.2.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/spark-graphx_2.10-1.4.2.2.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/spark-hive_2.10-1.4.2.2.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/spark-launcher_2.10-1.4.2.2.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/spark-mllib_2.10-1.4.2.2.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/spark-network-common_2.10-1.4.2.2.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/spark-network-shuffle_2.10-1.4.2.2.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/spark-repl_2.10-1.4.2.2.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/spark-sql_2.10-1.4.2.2.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/spark-streaming_2.10-1.4.2.2.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/spark-unsafe_2.10-1.4.2.2.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/spire-macros_2.10-0.7.4.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/spire_2.10-0.7.4.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/stax-api-1.0.1.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/stream-2.7.0.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/stringtemplate-3.2.1.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/tachyon-0.6.4.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/tachyon-client-0.6.4.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/uncommons-maths-1.2.2a.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/unused-1.0.0.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/velocity-1.7.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/xz-1.0.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/datanucleus-api-jdo-3.2.6.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/datanucleus-core-3.2.10.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/datanucleus-rdbms-3.2.9.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/hive-0.13-metastore-cassandra-connector-0.2.11.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/hive-0.13.1-cassandra-connector-0.2.11.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/hive-ant-0.13.1a.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/hive-beeline-0.13.1a.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/hive-cli-0.13.1a.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/hive-common-0.13.1a.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/hive-exec-0.13.1a.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/hive-hwi-0.13.1a.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/hive-jdbc-0.13.1a.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/hive-metastore-0.13.1a.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/hive-serde-0.13.1a.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/hive-service-0.13.1a.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/hive-shims-0.13.1a.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/hive-shims-0.20-0.13.1a.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/hive-shims-0.20S-0.13.1a.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/hive-shims-0.23-0.13.1a.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/hive-shims-common-0.13.1a.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/hive-shims-common-secure-0.13.1a.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/spark-hive-thriftserver_2.10-1.4.2.2.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/HdrHistogram-1.2.1.1.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/antlr-2.7.7.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/antlr-3.2.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/antlr-runtime-3.2.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/aopalliance-1.0.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/api-asn1-api-1.0.0-M24.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/api-asn1-ber-1.0.0-M24.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/api-i18n-1.0.0-M24.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/api-ldap-client-api-1.0.0-M24.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/api-ldap-codec-core-1.0.0-M24.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/api-ldap-codec-standalone-1.0.0-M24.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/api-ldap-extras-codec-1.0.0-M24.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/api-ldap-extras-codec-api-1.0.0-M24.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/api-ldap-model-1.0.0-M24.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/api-ldap-net-mina-1.0.0-M24.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/api-util-1.0.0-M24.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/asm-5.0.3.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/commons-beanutils-1.7.0.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/commons-beanutils-core-1.8.0.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/commons-codec-1.9.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/commons-collections-3.2.2.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/commons-compiler-2.6.1.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/commons-configuration-1.6.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/commons-digester-1.8.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/commons-io-2.4.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/commons-lang-2.6.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/commons-logging-1.1.1.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/commons-pool-1.6.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/guava-16.0.1.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/guice-3.0.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/guice-multibindings-3.0.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/jackson-annotations-2.2.2.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/jackson-core-2.2.2.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/jackson-databind-2.2.2.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/janino-2.6.1.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/java-uuid-generator-3.1.3.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/javassist-3.18.2-GA.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/javax.inject-1.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/jbcrypt-0.4d.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/jcl-over-slf4j-1.7.10.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/jline-1.0.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/journalio-1.4.2.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/jsr305-2.0.1.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/kmip-1.7.1e.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/log4j-1.2.13.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/mina-core-2.0.7.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/reflections-0.9.10.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/slf4j-api-1.7.10.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/stringtemplate-3.2.jar /Users/xxxxxx/cassandra/dse484/resources/dse/lib/validation-api-1.1.0.Final.jar /Users/xxxxxx/cassandra/dse484/resources/dse/conf/ /Users/xxxxxx/cassandra/dse484/resources/hadoop/ /Users/xxxxxx/cassandra/dse484/resources/hadoop/conf/ /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/airline-0.6.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/ant-1.6.5.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/automaton-1.11-8.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/commons-beanutils-1.7.0.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/commons-beanutils-core-1.8.0.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/commons-cli-1.2.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/commons-codec-1.4.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/commons-collections-3.2.2.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/commons-configuration-1.6.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/commons-digester-1.8.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/commons-el-1.0.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/commons-httpclient-3.0.1.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/commons-lang-2.4.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/commons-logging-1.1.1.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/commons-math-2.1.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/commons-net-1.4.1.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/core-3.1.1.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/elephant-bird-hadoop-compat-4.3.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/ftplet-api-1.0.0.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/ftpserver-core-1.0.0.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/ftpserver-deprecated-1.0.0-M2.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/hadoop-core-1.0.4.18.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/hadoop-examples-1.0.4.18.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/hadoop-fairscheduler-1.0.4.18.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/hadoop-streaming-1.0.4.18.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/hadoop-test-1.0.4.18.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/hadoop-tools-1.0.4.18.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/hsqldb-1.8.0.10.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/jackson-core-asl-1.8.8.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/jackson-mapper-asl-1.8.8.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/javax.inject-1.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/jets3t-0.7.1.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/jetty-6.1.26.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/jetty-util-6.1.26.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/jsp-2.1-6.1.14.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/jsp-api-2.1-6.1.14.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/kfs-0.3.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/mina-core-2.0.0-M5.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/netty-3.9.8.Final.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/oro-2.0.8.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/servlet-api-2.5-20081211.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/servlet-api-2.5-6.1.14.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/snappy-java-1.0.5.3.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/lib/xmlenc-0.52.jar /Users/xxxxxx/cassandra/dse484/resources/driver/lib/cassandra-driver-core-2.1.7.1.jar /Users/xxxxxx/cassandra/dse484/resources/driver/lib/cassandra-driver-dse-2.1.7.1.jar /Users/xxxxxx/cassandra/dse484/resources/driver/lib/metrics-core-3.0.2.jar /Users/xxxxxx/cassandra/dse484/resources/driver/lib/slf4j-api-1.7.5.jar /Users/xxxxxx/cassandra/dse484/resources/cassandra/lib/antlr-runtime-3.5.2.jar /Users/xxxxxx/cassandra/dse484/resources/cassandra/conf/ /Users/xxxxxx/cassandra/dse484/resources/cassandra/tools/lib/stress.jar /Users/xxxxxx/cassandra/dse484/resources/cassandra/lib/ST4-4.0.8.jar /Users/xxxxxx/cassandra/dse484/resources/cassandra/lib/antlr-3.5.2.jar /Users/xxxxxx/cassandra/dse484/resources/cassandra/lib/cassandra-all-2.1.12.1046.jar /Users/xxxxxx/cassandra/dse484/resources/cassandra/lib/cassandra-clientutil-2.1.12.1046.jar /Users/xxxxxx/cassandra/dse484/resources/cassandra/lib/cassandra-thrift-2.1.12.1046.jar /Users/xxxxxx/cassandra/dse484/resources/cassandra/lib/commons-cli-1.1.jar /Users/xxxxxx/cassandra/dse484/resources/cassandra/lib/commons-codec-1.9.jar /Users/xxxxxx/cassandra/dse484/resources/cassandra/lib/commons-lang-2.6.jar /Users/xxxxxx/cassandra/dse484/resources/cassandra/lib/commons-lang3-3.1.jar /Users/xxxxxx/cassandra/dse484/resources/cassandra/lib/commons-logging-1.2.jar /Users/xxxxxx/cassandra/dse484/resources/cassandra/lib/commons-math3-3.2.jar /Users/xxxxxx/cassandra/dse484/resources/cassandra/lib/compress-lzf-0.8.4.jar /Users/xxxxxx/cassandra/dse484/resources/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar /Users/xxxxxx/cassandra/dse484/resources/cassandra/lib/disruptor-3.0.1.jar /Users/xxxxxx/cassandra/dse484/resources/cassandra/lib/elephant-bird-hadoop-compat-4.3.jar /Users/xxxxxx/cassandra/dse484/resources/cassandra/lib/fastutil-6.5.7.jar /Users/xxxxxx/cassandra/dse484/resources/cassandra/lib/guava-16.0.1.jar /Users/xxxxxx/cassandra/dse484/resources/cassandra/lib/high-scale-lib-1.0.6.jar /Users/xxxxxx/cassandra/dse484/resources/cassandra/lib/httpclient-4.4.1.jar /Users/xxxxxx/cassandra/dse484/resources/cassandra/lib/httpcore-4.4.1.jar /Users/xxxxxx/cassandra/dse484/resources/cassandra/lib/jackson-core-asl-1.9.2.jar /Users/xxxxxx/cassandra/dse484/resources/cassandra/lib/jackson-mapper-asl-1.9.2.jar /Users/xxxxxx/cassandra/dse484/resources/cassandra/lib/jamm-0.3.0.jar /Users/xxxxxx/cassandra/dse484/resources/cassandra/lib/jbcrypt-0.4d.jar /Users/xxxxxx/cassandra/dse484/resources/cassandra/lib/jna-4.0.0.jar /Users/xxxxxx/cassandra/dse484/resources/cassandra/lib/joda-time-1.6.2.jar /Users/xxxxxx/cassandra/dse484/resources/cassandra/lib/json-simple-1.1.jar /Users/xxxxxx/cassandra/dse484/resources/cassandra/lib/libthrift-0.9.3.jar /Users/xxxxxx/cassandra/dse484/resources/cassandra/lib/logback-classic-1.1.2.jar /Users/xxxxxx/cassandra/dse484/resources/cassandra/lib/logback-core-1.1.2.jar /Users/xxxxxx/cassandra/dse484/resources/cassandra/lib/lz4-1.2.0.jar /Users/xxxxxx/cassandra/dse484/resources/cassandra/lib/metrics-core-2.2.0.jar /Users/xxxxxx/cassandra/dse484/resources/cassandra/lib/netty-all-4.0.33.dse.jar /Users/xxxxxx/cassandra/dse484/resources/cassandra/lib/reporter-config-2.1.0.jar /Users/xxxxxx/cassandra/dse484/resources/cassandra/lib/slf4j-api-1.7.12.jar /Users/xxxxxx/cassandra/dse484/resources/cassandra/lib/snakeyaml-1.12.jar /Users/xxxxxx/cassandra/dse484/resources/cassandra/lib/snappy-java-1.0.5.3.jar /Users/xxxxxx/cassandra/dse484/resources/cassandra/lib/stream-2.5.2.jar /Users/xxxxxx/cassandra/dse484/resources/cassandra/lib/super-csv-2.1.0.jar /Users/xxxxxx/cassandra/dse484/resources/cassandra/lib/thrift-server-0.3.7.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/datanucleus-api-jdo-3.2.6.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/datanucleus-core-3.2.10.jar /Users/xxxxxx/cassandra/dse484/resources/spark/lib/datanucleus-rdbms-3.2.9.jar /Users/xxxxxx/cassandra/dse484/resources/hadoop/conf/
Following is Exception:
Exception in thread "main" java.lang.ExceptionInInitializerError
at org.apache.spark.util.Utils$.createTempDir(Utils.scala:225)
at org.apache.spark.util.Utils$$anonfun$getOrCreateLocalRootDirsImpl$2.apply(Utils.scala:653)
at (JavaSparkContext.scala:61)
at com.walmart.gis.spark.uber.ExtractCatalogItems.run(ExtractCatalogItems.java:60)
at com.walmart.gis.spark.uber.ExtractCatalogItems.main(ExtractCatalogItems.java:285)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
Caused by: java.lang.NoSuchFieldException: SHUTDOWN_HOOK_PRIORITY
at java.lang.Class.getField(Class.java:1584)
at org.apache.spark.util.SparkShutdownHookManager.install(ShutdownHookManager.scala:222)
at org.apache.spark.util.ShutdownHookManager$.shutdownHooks$lzycompute(ShutdownHookManager.scala:50)
at org.apache.spark.util.ShutdownHookManager$.shutdownHooks(ShutdownHookManager.scala:48)
at org.apache.spark.util.ShutdownHookManager$.addShutdownHook(ShutdownHookManager.scala:191)
at org.apache.spark.util.ShutdownHookManager$.<init>(ShutdownHookManager.scala:58)
at org.apache.spark.util.ShutdownHookManager$.<clinit>(ShutdownHookManager.scala)
... 32 more
I was able to work around this issue by setting scoped=provided on the spark dependency in the pom file.
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>${spark.version}</version>
<scope>provided</scope> <!--To fix the SHUTDOWN_HOOK_PRIORITY error, add this line -->
</dependency>
This forced Spark to use the Spark library included with DSE rather than the one packaged in my JAR file.

Yarn support on Ooyala Spark JobServer

Just started experimenting with the JobServer and would like to use it in our production environment.
We usually run spark jobs individually in yarn-client mode and would like to shift towards the paradigm offered by the Ooyala Spark JobServer.
I am able to run the WordCount examples shown in the official page.
I tried running submitting our custom spark job to the Spark JobServer and I got this error:
{
"status": "ERROR",
"result": {
"message": "null",
"errorClass": "scala.MatchError",
"stack": ["spark.jobserver.JobManagerActor$$anonfun$spark$jobserver$JobManagerActor$$getJobFuture$4.apply(JobManagerActor.scala:220)",
"scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)",
"scala.concurrent.impl.Future $PromiseCompletingRunnable.run(Future.scala:24)",
"akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)",
"akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)",
"scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)",
"scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java 1339)",
"scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)",
"scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)"]
}
I had made the necessary code modifications like extending SparkJob and implementing the runJob() method.
This is the dev.conf file that I used:
# Spark Cluster / Job Server configuration
spark {
# spark.master will be passed to each job's JobContext
master = "yarn-client"
# Default # of CPUs for jobs to use for Spark standalone cluster
job-number-cpus = 4
jobserver {
port = 8090
jar-store-rootdir = /tmp/jobserver/jars
jobdao = spark.jobserver.io.JobFileDAO
filedao {
rootdir = /tmp/spark-job-server/filedao/data
}
context-creation-timeout = "60 s"
}
contexts {
my-low-latency-context {
num-cpu-cores = 1
memory-per-node = 512m
}
}
context-settings {
num-cpu-cores = 2
memory-per-node = 512m
}
home = "/data/softwares/spark-1.2.0.2.2.0.0-82-bin-2.6.0.2.2.0.0-2041"
}
spray.can.server {
parsing.max-content-length = 200m
}
spark.driver.allowMultipleContexts = true
YARN_CONF_DIR=/home/spark/conf/
Also how can I give run-time parameters for the spark job, such as --files, --jars ?
For example, I usually run our custom spark job like this:
./spark-1.2.0.2.2.0.0-82-bin-2.6.0.2.2.0.0-2041/bin/spark-submit --class com.demo.SparkDriver --master yarn-cluster --num-executors 3 --jars /tmp/api/myUtil.jar --files /tmp/myConfFile.conf,/tmp/mySchema.txt /tmp/mySparkJob.jar
Number of executors and extra jars are passed in a different way, through the config file (see dependent-jar-uris config setting).
YARN_CONF_DIR should be set in the environment and not in the .conf file.
As for other issues, the google group is the right place to ask. You may want to search it for yarn-client issues, as several other folks have figured out how to get it to work.

Resources