Connecting Jupyter notebook to Spark - apache-spark

Question

I have a machine with Hadoop and Spark installed. Below is my current environment.
python3.6
spark1.5.2
Hadoop 2.7.1.2.3.6.0-3796
I was trying to connect jupyter notebook to connect to spark by building up ipython kernel.
2 new files written.
/root/.ipython/profile_pyspark/ipython_notebook_config.py
/root/.ipython/profile_pyspark/startup/00-pyspark-setup.py
/root/anaconda3/share/jupyter/kernels/pyspark/kernel.json
kernel.json
{
"display_name": "PySpark (Spark 2.0.0)",
"language": "python",
"argv": [
"/root/anaconda3/bin/python3",
"-m",
"ipykernel",
"--profile=pyspark"
],
"env": {
"CAPTURE_STANDARD_OUT": "true",
"CAPTURE_STANDARD_ERR": "true",
"SEND_EMPTY_OUTPUT": "false",
"PYSPARK_PYTHON" : "/root/anaconda3/bin/python3",
"SPARK_HOME": "/usr/hdp/current/spark-client/"
}
}
00-pyspark-setup.py
import os
import sys
os.environ["PYSPARK_PYTHON"] = "/root/anaconda3/bin/python"
os.environ["SPARK_HOME"] = "/usr/hdp/current/spark-client"
os.environ["PYLIB"] = os.environ["SPARK_HOME"] + "/python/lib"
spark_home = os.environ.get('SPARK_HOME', None)
sys.path.insert(0, os.environ["PYLIB"] +"/py4j-0.8.2.1-src.zip")
sys.path.insert(0, os.environ["PYLIB"] +"/pyspark.zip")
exec(open(os.path.join(spark_home, 'python/pyspark/shell.py')).read())
ipython_notebook_config.py
c = get_config()
c.NotebookApp.port = 80
Then, when i run the following
jupyter notebook --profile=pyspark
The notebook is running well. Then, i change kernel to 'PySpark (Spark 2.0.0)', and it suppose to use the 'sc' spark context . However, when i type 'sc', it does not shows anything.
So, since sc cannot be initiliazed, if i want to run the following, it failed!
nums = sc.parallelize(xrange(1000000))
Can anybody help me how to configure jupyter notebook to talk to Spark?

Answer 1

Just FYI, python 3.6 isnt supported until version spark 2.1.1. See JIRA https://issues.apache.org/jira/browse/SPARK-19019

Answer 2

There is a number of issues with your question...
1) On top of the answer by Punskr above - Spark 1.5 only works with Python 2; Python 3 support was introduced in Spark 2.0.
2) Even if you switch to Python 2 or upgrade Spark, you will still need to import the relevant modules of Pyspark and initialize the sc variable manually in the notebook
3) You also seem to use an old version of Jupyter, since the profiles functionality is not available in Jupyter >= 4.
To initialise sc "automatically" in Jupyter >=4, see my answer here.

Answer 3

You can make a few environment changes to have pyspark default ipython or a jupyter notebook.
Put the following in your ~/.bashrc
export PYSPARK_PYTHON=python3 ## for python3
export PYSPARK_DRIVER_PYTHON=ipython
export PYSPARK_DRIVER_PYTHON_OPTS="notebook --no-browser --port=7000"
See: pyspark on GitHub
Next, run source ~/.bashrc
Then, when you launch pyspark (or with YARN) it will open up a server for you to connect to.
On a local terminal that has ssh capabilities, run
ssh -N -f -L localhost:8000:localhost:7000 <username>#<host>
If you're on Windows, I recommend MobaXterm or Cygwin.
Open up a web browser, and enter the address localhost:8000 to tunnel into your notebook with Spark
Now some precautions, I've never tried this with Python 3 so this may or may not work for you. Regardless, you should really be using Python 2 on Spark 1.5. My company uses Spark 1.5 as well, as NO ONE uses Python 3 because of it.
Update:
Per #desertnaut's comments, setting
export PYSPARK_DRIVER_PYTHON=ipython
may cause issues if the user ever needs to use spark-submit. A work around, if you want to have both notebooks and spark-submit available is to create two new environment variables. Here is an example of what you may create
export PYSPARK_PYTHON=python3 ## for python3
export ipyspark='PYSPARK_DRIVER_PYTHON=ipython pyspark'
export pynb='PYSPARK_DRIVER_PYTHON_OPTS="notebook --no-browser --port=7000"'
where ipyspark and pynb are new commands on a bash terminal.