Do pyspark need a local Spark installation? - python-3.x

Question

I'm trying to get going with spark. Trying to create a simple SQL connection to a database while running Spark in a docker container.
I do not have Spark installed on my laptop. Only inside my docker container.
I got the following code on my laptop:
spark = SparkSession \
.builder \
.master("spark://localhost:7077") \ # <-- Docker container with master and worker
.appName("sparktest") \
.getOrCreate()
jdbcDF = spark.read.format("jdbc") \
.option("driver", "com.microsoft.sqlserver.jdbc.SQLServerDriver") \
.option("url", "jdbc:sqlserver://xxx") \
.option("dbtable", "xxx") \
.option("user", "xxx") \
.option("password", "xxx").load()
I can't get it to work.
I either get java.sql.SQLException: No suitable driver or ClassNotFoundException from Java.
I've moved the files to the container and everything seems fine over there.
I've made sure the mssql jar files are on the SPARK_CLASSPATH on both driver and executor.
Am I supposed to have Spark installed locally for me to use PySpark against the remote master running in my docker container?
It looks like its trying to find the SQL driver on my laptop?
Everything is fine if i run code using spark-submit from inside the docker container.
I was trying to avoid going the route of jupyter hosted inside the docker container, but was hoping to not having to install Spark on my Windows laptop and keeping it in my linux container.

Answer 1

I faced it before and for a solution you can download jdbc driver and set the driver configuration manually with giving jdbc driver path
from pyspark.context import SparkConf
conf = SparkConf()
conf.set('spark.jars', '/PATH_OF_DRIVER/driver.jar')
conf.set('spark.executor.extraClassPath', '/PATH_OF_DRIVER/driver.jar')