How to pass configuration from driver to executors in Spark? - apache-spark

Question

For example.
object App {
var confValue: String = ""
def main(args: Array[String]): Unit = {
// set conf by cmd args
confValue = args.head
// do some context init
val dataset: Dataset[Int] = ???
dataset.foreach { row =>
// get conf from executor
println(confValue)
}
}
}
I want to get conf on executors, but actually it can't be done because confValue only has been modified on the driver
I know I can pass confValue to executors by local variable like this.
def main(args: Array[String]): Unit = {
// set conf by cmd args
val confValue = args[0]
// do some context init
val dataset: Dataset[Int] = ???
dataset.foreach { row =>
// get conf from executor
println(confValue)
}
}
But my spark job is huge. It has so many functions. I can't pass confValue everywhere as a local variable. For example:
def main(args: Array[String]): Unit = {
// set conf by cmd args
val confValue = args[0]
// do some context init
val dataset: Dataset[Int] = ???
dataset.foreach { row =>
doSomeLogic(row)
}
}
private def doSomeLogic(row: Int): Unit = {
// get conf from executor
println(confValue)
}
There is so many doSomeLogic. So I can't pass confValue to all of them.
Is there some way to pass confValue to every executors automatically?
updated 1
My spark code may like below
object App {
/** env flag, will be inited by cmd args, and be used in executors */
var env: String = ""
val spark: SparkSession = ???
import spark.implicits._
def main(args: Array[String]): Unit = {
// read env from args
env = args.head
var ds: Dataset[Int] = ???
ds = doLogic1(ds)
ds = doLogic2(ds)
doLogic3(ds)
}
private def doLogic1(ds: Dataset[Int]): Dataset[Int] = {
ds.map { row =>
// env will be used here
???
}
}
private def doLogic2(ds: Dataset[Int]): Dataset[Int] = {
ds.map { row =>
// env will be used here
???
}
}
private def doLogic3(ds: Dataset[Int]): Dataset[Int] = {
ds.map { row =>
// env will be used here
???
}
}
}
env will be inited in main, and will be used in some of doLogicN functions. My spark project is a large project with many doLogicN functions, so passing the env flag to every doLogicN function will change too many codes.
What is the simplest way to pass the env flag to all doLogicN functions?
The most difficult point is that the env will be used in executors. If it will only be used in drivers I can pass it to everywhere by global env variable. But it won't work well in executors because the global env variable hasn't been inited. It only be inited in the driver side.

Answer 1

You could do something like below to broadcast the value to all the executors and then based on your requirement and can use it as you want. Also instead of using for each you should use for each partition if you want to process data for each partition in parallel.
Below is the sample code of how you can broadcast a value and use it:
//Sample data created
val df = Seq(("a","2020-01-16 08:55:50"),("b","2020-01-16 08:57:37"),("c","2020-01-16 09:00:13"),("d","2020-01-16 09:01:32"),("e","2020-01-16 09:03:32"),("f","2020-01-16 09:06:56")).toDF("ID","timestamp")
//check the partitions that a datframe has
df.rdd.partitions.size
//broadcast the value that you want to broadcast
val confValue = "Test"
val bdct_confvalue = spark.sparkContext.broadcast(confValue);
//using the broadcasted value on each executors nodes as required
df.foreachPartition(partition => {
println("Confvalue partition =" +bdct_confvalue.value)
}
)
Also to see the value printed in the logs you would have to see the executors logs and not the driver logs as you would not be able to see this print statement in the driver logs. you would also not able to see this in any notebook like Jupyter or Databricks notebook as they show driver logs on the UI.

Answer 2

I found a way to solve my question.
The conf can be set by Spark Conf while committing the spark job, such as spark.my.env=env_1
It can be read by SparkEnv.get.conf.get("spark.my.env"), which has the same effect between the driver and executors, after sparkContext has been initialized.