Spark NoSuchMethodError - apache-spark

Question

I'm running spark using the following Docker command:
docker run -it \
-p 8088:8088 -p 8042:8042 -p 50070:50070 \
-v "$(PWD)"/log4j.properties:/usr/local/spark/conf/log4j.properties \
-v "$(PWD)":/app -h sandbox sequenceiq/spark:1.6.0 bash
Running spark-submit --version reports version 1.6.0
My spark-submit command is the following:
spark-submit --class io.jobi.GithubDay \
--master local[*] \
--name "Daily Github Push Counter" \
/app/min-spark_2.11-1.0.jar \
"file:///app/data/github-archive/*.json" \
"/app/data/ghEmployees.txt" \
"file:///app/data/emp-gh-push-output" "json"
build.sbt
name := """min-spark"""
version := "1.0"
scalaVersion := "2.11.7"
lazy val sparkVersion = "1.6.0"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % sparkVersion % "provided",
"org.apache.spark" %% "spark-sql" % sparkVersion % "provided"
)
// Change this to another test framework if you prefer
libraryDependencies += "org.scalatest" %% "scalatest" % "2.2.4" % "test"
GithubDay.scala
package io.jobi
import org.apache.spark.sql.SQLContext
import org.apache.spark.{SparkConf, SparkContext}
import scala.io.Source.fromFile
/**
* Created by hammer on 7/15/16.
*/
object GithubDay {
def main(args: Array[String]): Unit = {
println("Application arguments: ")
args.foreach(println)
val conf = new SparkConf()
val sc = new SparkContext(conf)
val sqlContext = new SQLContext(sc)
try {
println("args(0): " + args(0))
val ghLog = sqlContext.read.json(args(0))
val pushes = ghLog.filter("type = 'PushEvent'")
val grouped = pushes.groupBy("actor.login").count()
val ordered = grouped.orderBy(grouped("count").desc)
val employees = Set() ++ (
for {
line <- fromFile(args(1)).getLines()
} yield line.trim
)
val bcEmployees = sc.broadcast(employees)
import sqlContext.implicits._
println("register function")
val isEmployee = sqlContext.udf.register("SetContainsUdf", (u: String) => bcEmployees.value.contains(u))
println("registered udf")
val filtered = ordered.filter(isEmployee($"login"))
println("applied filter")
filtered.write.format(args(3)).save(args(2))
} finally {
sc.stop()
}
}
}
I build using sbt clean package but the output when I run it is:
Application arguments:
file:///app/data/github-archive/*.json
/app/data/ghEmployees.txt
file:///app/data/emp-gh-push-output
json
args(0): file:///app/data/github-archive/*.json
imported implicits
defined isEmp
register function
Exception in thread "main" java.lang.NoSuchMethodError: scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader;)Lscala/reflect/api/JavaUniverse$JavaMirror;
at io.jobi.GithubDay$.main(GithubDay.scala:53)
at io.jobi.GithubDay.main(GithubDay.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
From what I've read NoSuchMethodError is a result of version incompatibilities, but I'm building with 1.6.0 and deploying to 1.6.0 so I don't understand what's happening.

Answer 1

Unless you've compiled Spark yourself, out of the box version 1.6.0 is compiled with Scala 2.10.x. This is stated in the docs (which says 1.6.2, but is also relevant to 1.6.0):
Spark runs on Java 7+, Python 2.6+ and R 3.1+. For the Scala API,
Spark 1.6.2 uses Scala 2.10. You will need to use a compatible Scala
version (2.10.x).
You want:
scalaVersion := "2.10.6"
One hint to that is that the error is in a Scala class: scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader;)