spark-submit python app with named argument via spark REST api - apache-spark

Question

Using the following as the body for a request to spark native REST api that executes a python app with a named argument, --filename, via spark-submit, the driver is successfully created, but the execution fails with very little information. Is the --filename argument causing this? Any ideas where/how to get more information about the failure?
{ "action": "CreateSubmissionRequest",
"appArgs": [
"/opt/bitnami/spark/hostfiles/bronze.py --filename 'filename.json'"
],
"appResource": "file:/opt/bitnami/spark/hostfiles/bronze.py",
"clientSparkVersion": "3.2.0",
"environmentVariables": {
"SPARK_ENV_LOADED": "1"
},
"mainClass": "org.apache.spark.deploy.SparkSubmit",
"sparkProperties": {
"spark.driver.supervise": "false",
"spark.app.name": "Spark REST API - Bronze Load",
"spark.submit.deployMode": "client",
"spark.master": "spark://spark:6066",
"spark.eventLog.enabled":"true"
}
}```
These are the logs from the worker container for the driver.
> 22/01/18 23:37:48 INFO Worker: Asked to launch driver driver-20220118233748-0006
> 22/01/18 23:37:48 INFO DriverRunner: Copying user jar file:/opt/bitnami/spark/hostfiles/bronze.py to /opt/bitnami/spark/work/driver-20220118233748-0006/bronze.py
> 22/01/18 23:37:48 INFO Utils: Copying /opt/bitnami/spark/hostfiles/bronze.py to /opt/bitnami/spark/work/driver-20220118233748-0006/bronze.py
> 22/01/18 23:37:48 INFO DriverRunner: Launch Command: "/opt/bitnami/java/bin/java" "-cp" "/opt/bitnami/spark/conf/:/opt/bitnami/spark/jars/*" "-Xmx1024M" "-Dspark.eventLog.enabled=true" "-Dspark.app.name=Spark REST API - Bronze Load" "-Dspark.driver.supervise=false" "-Dspark.master=spark://spark:7077" "-Dspark.submit.deployMode=client" "org.apache.spark.deploy.worker.DriverWrapper" "spark://Worker#192.168.32.4:35803" "/opt/bitnami/spark/work/driver-20220118233748-0006/bronze.py" "org.apache.spark.deploy.SparkSubmit" "/opt/bitnami/spark/hostfiles/bronze.py --filename 'filename.json'"
> 22/01/18 23:37:51 WARN Worker: Driver driver-20220118233748-0006 exited with failure
Meanwhile, the spark-submit successfully executes the app.
``` bin/spark-submit --deploy-mode "client" file:///opt/bitnami/spark/hostfiles/bronze.py --filename "filename.json"```

Answer 1

The following change to the body of the api request and processing the python application argument value without requiring '--filename' in the python script results the python app being executed.
updated appArgs and appResource to use file:/// for referencing the python app.
"appArgs": [
"file:///opt/bitnami/spark/hostfiles/bronze.py", "filename.json"
],
"appResource": "file:///opt/bitnami/spark/hostfiles/bronze.py",
"clientSparkVersion": "3.2.0",
"environmentVariables": {
"SPARK_ENV_LOADED": "1"
},
"mainClass": "org.apache.spark.deploy.SparkSubmit",
"sparkProperties": {
"spark.driver.supervise": "false",
"spark.app.name": "Spark REST API - Bronze Load",
"spark.submit.deployMode": "client",
"spark.master": "spark://spark:7077",
"spark.eventLog.enabled":"true"
}
}