How to terminate a job in standalone and client mode? - apache-spark

Question

I use spark-submit to run a job, which has some exceptions, it blocked, so I tried to use ctrl + c to stop the process.
I would like to know if this job is still running on the cluster on not ?
If it is not the right way to kill the job, what's a right way ?
^C18/09/03 19:03:01 INFO SparkContext: Invoking stop() from shutdown hook
18/09/03 19:03:01 INFO SparkUI: Stopped Spark web UI at http://x.x.x.x:4040
18/09/03 19:03:01 INFO DAGScheduler: Job 2 failed: count at xxx.scala:155, took 773.555554 s
18/09/03 19:03:01 INFO DAGScheduler: ShuffleMapStage 2 (count at xxx.scala:155) failed in 773.008 s
18/09/03 19:03:01 ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerStageCompleted(org.apache.spark.scheduler.StageInfo#7f6a32f)
18/09/03 19:03:01 ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerJobEnd(2,1535994181627,JobFailed(org.apache.spark.SparkException: Job 2 cancelled because SparkContext was shut down))
18/09/03 19:03:01 ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerSQLExecutionEnd(0,1535994181630)
18/09/03 19:03:01 INFO StandaloneSchedulerBackend: Shutting down all executors
Exception in thread "main" org.apache.spark.SparkException: Job 2 cancelled because SparkContext was shut down
at org.apache.spark.scheduler.DAGScheduler$$anonfun$cleanUpAfterSchedulerStop$1.apply(DAGScheduler.scala:818)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$cleanUpAfterSchedulerStop$1.apply(DAGScheduler.scala:816)
at scala.collection.mutable.HashSet.foreach(HashSet.scala:78)
at org.apache.spark.scheduler.DAGScheduler.cleanUpAfterSchedulerStop(DAGScheduler.scala:816)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onStop(DAGScheduler.scala:1685)
at org.apache.spark.util.EventLoop.stop(EventLoop.scala:83)
at org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:1604)
at org.apache.spark.SparkContext$$anonfun$stop$8.apply$mcV$sp(SparkContext.scala:1781)
at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1290)
at org.apache.spark.SparkContext.stop(SparkContext.scala:1780)
at org.apache.spark.SparkContext$$anonfun$2.apply$mcV$sp(SparkContext.scala:559)
at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:215)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:187)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:187)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:187)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1953)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:187)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:187)
at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:187)
at scala.util.Try$.apply(Try.scala:192)
at org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:187)
at org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:177)
at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:632)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1873)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1886)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1899)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1913)
at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:912)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:358)
at org.apache.spark.rdd.RDD.collect(RDD.scala:911)
at org.apache.spark.sql.execution.SparkPlan.executeCollect(SparkPlan.scala:290)
at org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$execute$1$1.apply(Dataset.scala:2193)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:57)
at org.apache.spark.sql.Dataset.withNewExecutionId(Dataset.scala:2546)
at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$execute$1(Dataset.scala:2192)
at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$collect(Dataset.scala:2199)
at org.apache.spark.sql.Dataset$$anonfun$count$1.apply(Dataset.scala:2227)
at org.apache.spark.sql.Dataset$$anonfun$count$1.apply(Dataset.scala:2226)
at org.apache.spark.sql.Dataset.withCallback(Dataset.scala:2559)
at org.apache.spark.sql.Dataset.count(Dataset.scala:2226)
at xx.xx.xx.weekLyLoadingIDFA(xx.scala:155)
at xx.xx.xx.retrieve(xx.scala:171)
at xx.xx.xx.run(xx.scala:65)
at xx.xx.xxRunner$.delayedEndpoint$io$xxx$CellRunner$1(xx.scala:12)
at xx.xx.xxRunner$delayedInit$body.apply(xx.scala:11)
at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main$1.apply(App.scala:76)
at scala.App$$anonfun$main$1.apply(App.scala:76)
at scala.collection.immutable.List.foreach(List.scala:381)
at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
at scala.App$class.main(App.scala:76)
at xx.xx.xxRunner$.main(xx.scala:11)
at xx.xx.xxRunner.main(xx.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:736)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
18/09/03 19:03:01 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each executor to shut down
18/09/03 19:03:01 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
18/09/03 19:03:01 INFO MemoryStore: MemoryStore cleared
18/09/03 19:03:01 INFO BlockManager: BlockManager stopped
18/09/03 19:03:01 INFO BlockManagerMaster: BlockManagerMaster stopped
18/09/03 19:03:01 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
18/09/03 19:03:01 ERROR TransportResponseHandler: Still have 1 requests outstanding when connection from xxxxx/xxxx:7077 is closed
18/09/03 19:03:01 INFO SparkContext: Successfully stopped SparkContext
18/09/03 19:03:01 INFO ShutdownHookManager: Shutdown hook called
18/09/03 19:03:01 INFO ShutdownHookManager: Deleting directory /tmp/spark/spark-xxxxxxxxxx

Answer 1

If you are running on yarn you can kill the spark app by below command
yarn application -kill applicationId
For spark on stand alone mode use
spark-submit — kill applicationId — master masterurl

Answer 2

It depends on the resource manager. In my case ctrl+c works fine on yarn, and the job is killed and you still stay in spark-shell. Also you can kill job from the Spark WEB UI or from YARN.

Answer 3

The logs above shows that SparkContext was shutdown. This means that the Spark job is not running anymore on the cluster.
Since you are running the application in Client mode, So ctrl+c should kill the application in general.

Answer 4

When you start the Spark StandAlone Cluster, It's master have a UI on 8080 port.
On the Master UI you will see your application in Running Application Tab.
Corresponding to every application there is a (KILL) option button associated with that.
Just press that button & it will ask you to confirm it. Confirm it to close.
In the Image, you can see a running application & there is a kill option associated with it.
Happy Sparking....