How to get driver-Id in spark submission - apache-spark

Question

Spark cluster information:
Spark version: 2.2.0
Cluster contains a master node with 2 worker nodes
Cluster Manager Type: standalone
I submit a jar to spark cluster from one of the workers and I want to receive the driver-Id from the submission so that I can use that id for later application status checking. The problem is that I am not getting any output in the console. I use port 6066 for submission and set deploy mode to cluster.
By running
spark-submit --deploy-mode cluster --supervise --class "path/to/class" --master "spark://spark-master-headless:6066" path/to/app.jar
in the spark log file I am able to see the json response of the submission below, which is the exact thing that I want:
[INFO] 2018-07-18 12:48:40,030 org.apache.spark.deploy.rest.RestSubmissionClient logInfo - Submitting a request to launch an application in spark://spark-master-headless:6066.
[INFO] 2018-07-18 12:48:41,074 org.apache.spark.deploy.rest.RestSubmissionClient logInfo - Submission successfully created as driver-20180718124840-0023. Polling submission state...
[INFO] 2018-07-18 12:48:41,077 org.apache.spark.deploy.rest.RestSubmissionClient logInfo - Submitting a request for the status of submission driver-20180718124840-0023 in spark://spark-master-headless:6066.
[INFO] 2018-07-18 12:48:41,092 org.apache.spark.deploy.rest.RestSubmissionClient logInfo - State of driver driver-20180718124840-0023 is now RUNNING.
[INFO] 2018-07-18 12:48:41,093 org.apache.spark.deploy.rest.RestSubmissionClient logInfo - Driver is running on worker worker-20180707104934-<some-ip-was-here>-7078 at <some-ip-was-here>:7078.
[INFO] 2018-07-18 12:48:41,114 org.apache.spark.deploy.rest.RestSubmissionClient logInfo - Server responded with CreateSubmissionResponse:
{
"action" : "CreateSubmissionResponse",
"message" : "Driver successfully submitted as driver-20180718124840-0023",
"serverSparkVersion" : "2.2.0",
"submissionId" : "driver-20180718124840-0023",
"success" : true
}
[INFO] 2018-07-18 12:48:46,572 org.apache.spark.executor.CoarseGrainedExecutorBackend initDaemon - Started daemon with process name: 31605#spark-worker-662224983-4qpfw
[INFO] 2018-07-18 12:48:46,580 org.apache.spark.util.SignalUtils logInfo - Registered signal handler for TERM
[INFO] 2018-07-18 12:48:46,583 org.apache.spark.util.SignalUtils logInfo - Registered signal handler for HUP
[INFO] 2018-07-18 12:48:46,583 org.apache.spark.util.SignalUtils logInfo - Registered signal handler for INT
[WARN] 2018-07-18 12:48:47,293 org.apache.hadoop.util.NativeCodeLoader <clinit> - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[INFO] 2018-07-18 12:48:47,607 org.apache.spark.SecurityManager logInfo - Changing view acls to: root
[INFO] 2018-07-18 12:48:47,608 org.apache.spark.SecurityManager logInfo - Changing modify acls to: root
...
But I want to have this information in the console so that I can direct it to a separate file than the spark logs. I assume that some messages should be get printed when the above command gets run. I even used --verbose mode in the command so that maybe that helps but still the output in the console is empty.
The only thing that gets printed to console is
Running Spark using the REST application submission protocol. While in this page's question section, the user is able to see more than this.
I even tried to change the Logger level in my application code but that also didn't help. (based on some ideas from here)
Then the question would be, how come that I am not getting any output in the console and what can I do to get the info that I want to get printed to console?
P.S. I have developed and tweaked the cluster and the jar file to a good amount that maybe I have something somewhere causing the output not to get printed. What are the possible places I can check to fix this.
Update:
I found out that the default log4j.properties of spark has been edited. Here is the content:
# Set everything to be logged to the console
log4j.rootCategory=INFO, RollingAppender
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
log4j.appender.RollingAppender=org.apache.log4j.DailyRollingFileAppender
log4j.appender.RollingAppender.File=/var/log/spark.log
log4j.appender.RollingAppender.DatePattern='.'yyyy-MM-dd
log4j.appender.RollingAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.RollingAppender.layout.ConversionPattern=[%p] %d %c %M - %m%n
# Set the default spark-shell log level to WARN. When running the spark-shell, the
# log level for this class is used to overwrite the root logger's log level, so that
# the user can have different defaults for the shell and regular Spark apps.
log4j.logger.org.apache.spark.repl.Main=INFO
# Settings to quiet third party logs that are too verbose
log4j.logger.org.spark_project.jetty=INFO
log4j.logger.org.spark_project.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
log4j.logger.org.apache.parquet=ERROR
log4j.logger.parquet=ERROR
# SPARK-9183: Settings to avoid annoying messages when looking up nonexistent UDFs in SparkSQL with Hive support
log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL
log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR
I assume that this is not letting the verbose command to work. How can I change this to have get some content with --verbose?

Answer 1

As you are running the job in cluster mode the driver can be any node in cluster, so whatever you do print/console redirect may not return to client/edge node/worker node where the console is opened.
Try submitting the apllication in client mode