How do I restart a stopped Spark Context? - apache-spark

Question

I'm running Spark with apache zeppelin and hadoop. My understanding is that Zeppelin is like a kube app that sends commands to a remote machine that's running Spark and accessing files with Hadoop.
I often run into a situation where the Spark Context gets stopped. In the past, I believed it was because I overloaded the system with a data pull that required too much data, but now I'm less enthusiastic about that theory. I've frequently had it happen after running totally reasonable and normal queries.
In order to restart the Spark Context, I've gone to the interpreter binding settings and restarted spark.
I've also run this command
%python
JSESSIONID="09123q-23se-12ae-23e23-dwtl12312
YOURFOLDERNAME="[myname]"
import requests
import json
cookies = {"JSESSIONID": JSESSIONID}
notebook_response = requests.get('http://localhost:8890/api/notebook/jobmanager', cookies=cookies)
body = json.loads(notebook_response.text)["body"]["jobs"]
notebook_ids = [(note["noteId"]) for note in body if note.get("interpreter") == "spark" and YOURFOLDERNAME in note.get("noteName", "")]
for note_id in notebook_ids:
requests.put("http://localhost:8890/api/interpreter/setting/restart/spark", data=json.dumps({"noteId": note_id}), cookies=cookies)
I've also gone to the machine running spark and entered yarn top and I don't see my username listed within the list of running applications.
I know that I can get it working if I restart the machine, but that'll also restart the machine for everyone else using it.
What other ways can I restart a Spark Context?

Answer 1

I assume that you have configured you spark interpreter to run in isolated mode:
In this case you get separate instances for each user:
You can restart your own instance and get a new SparkContext from the interpreter binding menu of a notebook by pressing the refresh button (tested with zeppelin 0.82):