getting & setting spark.driver/executor.extraClassPath on EMR - apache-spark

Question

As far as I can tell, when setting / using spark.driver.extraClassPath and spark.executor.extraClassPath on AWS EMR within the spark-defaults.conf or elsewhere as a flag, I would have to first get the existing value that [...].extraClassPath is set to, then append :/my/additional/classpath to it in order for it to work.
is there a function in Spark that allows me to to just append an additional class path to it where it retains/respects the existing paths set by EMR in /etc/spark/conf/spark-defaults.conf?

Answer 1

No such "function" in Spark but:
On EMR AMI's you can write a bootstrap that will append/set whatever you want in spark-defaults, will of course affect all Spark jobs.
When EMR moved to the newer "release-label" this stopped working as bootstrap-steps were replaced with configuration JSONs and the manual bootstraps run before applications are installed ( At least when I tried it )

Answer 2

We use SageMaker Studio connected to EMR clusters via Apache Livy and none of the approaches suggested so far worked for me - we need to add a custom JAR on the classpath for a custom filesystem.
Following this guide:
https://aws.amazon.com/premiumsupport/knowledge-center/emr-spark-classnotfoundexception/
I made it work by adding an EMR Step to cluster set up with the following code (this is TypeScript CDK, passed into steps parameter of CfnCluster CDK construct:
//We are using this approach to add CkpFS JAR (in Docker image additional-jars) onto the class path because
// the spark.jars configuration in spark-defaults is not respected by Apache Livy on EMR when it creates a Spark session
// AWS guide followed: https://aws.amazon.com/premiumsupport/knowledge-center/emr-spark-classnotfoundexception/
private createStepToAddCkpFSJAROntoEMRSparkClassPath() {
return {
name: "AddCkpFSJAROntoEMRSparkClassPath",
hadoopJarStep: {
jar: "command-runner.jar",
args: [
"bash",
"-c",
"while [ ! -f /etc/spark/conf/spark-defaults.conf ]; do sleep 1; done;" +
"sudo sed -i '/spark.*.extraClassPath/s/$/:\\/additional-jars\\/\\*/' /etc/spark/conf/spark-defaults.conf",
],
},
actionOnFailure: "CONTINUE",
};
}