Spark error using a variable in map operation [duplicate] - apache-spark

Question

This question already has answers here:
Resolving dependency problems in Apache Spark
(7 answers)
Closed 4 years ago.
I am trying to iterate over a DataFrame and apply map operation over it's rows.
import spark.implicits._
import org.apache.spark.sql.Row
case class SomeData(name:String, value: Int)
val input = Seq(SomeData("a",2), SomeData("b", 3)).toDF
val SOME_STRING = "some_string"
input.map(row =>
SOME_STRING
).show
The above code fails with following exception,
ERROR TaskSetManager: Task 0 in stage 4.0 failed 4 times; aborting job
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 4.0 failed 4 times, most recent failure: Lost task 0.3 in stage 4.0 (TID 14, ip-xxxx, executor 4): java.lang.NoClassDefFoundError: L$iw;
at java.lang.Class.getDeclaredFields0(Native Method)
at java.lang.Class.privateGetDeclaredFields(Class.java:2583)
at java.lang.Class.getDeclaredField(Class.java:2068)
However, if the variable is replaced with string, the code works.
input.map(row =>
"some_string"
).show
+-----------+
| value|
+-----------+
|some_string|
|some_string|
+-----------+
Is there anything wrong with the above code? Is it possible to use variables and function calls inside map operation.

Answer 1

it's normal, your variable is define inside the driver then use inside a worker, so your worker dont know the variable.
what you can do is :
input.map(row =>
val SOME_STRING = "some_string"
).show
you can also check broadcast variable : https://spark.apache.org/docs/2.2.0/rdd-programming-guide.html#broadcast-variables