Spark ClassNotFoundException with the simplest rdd.map - apache-spark

Question

edit
This question is not a duplicate of Resolving dependency problems in Apache Spark:
the answers above are generalities about sharing code. I don't see how it applies to my issue
Here I do not use spark-submit but simply call spark from an app running on my computer
There is no dependency whatsoever. The only code to share is the simplistic function i => i
if it is a problem of spark/java/scala version, I don't understand why it would produce a ClassNotFoundException for the function in question
I run this code
object Tmp {
def main(args: Array[String]): Unit = {
val sparkConf = new SparkConf()
.setAppName("Spark Test")
.setMaster("spark://[ip-of-spark-master]:7077")
val sc = new SparkContext(sparkConf)
sc.parallelize(0 until 10000).map(i => i).count()
}
}
Which fails with java.lang.ClassNotFoundException: Tmp$$anonfun$main$1 when I simply run it on my computer, without sending any jar. It works if I don't do the map(i => i) (i.e. it is the function i => i that is not found).
To be clear:
I do not use spark-submit
my computer is the driver and is not on the spark cluster
I use the exact same spark version on my computer that is used in the cluster (2.0.2)
Now I thought that for simple function, spark would serialize it and there were no need to send the jar. I am pretty sure I have done it before.
Did I miss something ?
Looking at the stack trace, it fails on the driver, so on my side:
Driver stacktrace:
[error] at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1454)
[error] at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1442)
[error] at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1441)
...
[error] Caused by: java.lang.ClassNotFoundException: Tmp$$anonfun$1
[error] at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
[error] at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
[error] at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
[error] at java.lang.Class.forName0(Native Method)
[error] at java.lang.Class.forName(Class.java:348)
[error] at org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:67)
...