org.apache.spark.SparkException: Task not serializable - When using an argument - apache-spark

Question

I get a Task not serializable error when attempting to use an input parameter in a map:
val errors = inputRDD.map {
case (itemid, itemVector, userid, userVector, rating) =>
(itemid, itemVector, userid, userVector, rating,
(
(rating - userVector.dot(itemVector)) * itemVector)
- h4 * userVector
)
}
I pass h4 in with the arguments for the Class.
The map is in a method and it works fine if before the map transformation I put:
val h4 = h4
If I don't do this, or put this outside the method then it doesn't work and I get Task not serialisable. Why is this occurring? Other val's I generate for the Class outside the method work within the method, so how come when the val is instantiated from an input parameter/argument it does not?

Answer 1

The error indicates that the class to which h4 belongs is not Serializable.
Here is a similar example:
class ABC(h: Int) {
def test(s:SparkContext) = s.parallelize(0 to 5).filter(_ > h).collect
}
new ABC(3).test(sc)
//org.apache.spark.SparkException: Job aborted due to stage failure:
// Task not serializable: java.io.NotSerializableException:
// $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$ABC
When this.h is used in a rdd-transformation, this becomes part of the closure which gets serialized.
Making the class Serializable works as expected:
class ABC(h: Int) extends Serializable {
def test(s:SparkContext) = s.parallelize(0 to 5).filter(_ > h).collect
}
new ABC(3).test(sc)
// Array[Int] = Array(4, 5)
So does removing reference to this in the rdd-transformation, by defining a local variable in the method:
class ABC(h: Int) {
def test(s:SparkContext) = {
val x = h;
s.parallelize(0 to 5).filter(_ > x).collect
}
}
new ABC(3).test(sc)
// Array[Int] = Array(4, 5)

Answer 2

You can use Broadcast variable. Its Broadcast data from your variable to all your workers. For more Detail visit this link.