I am running Pyspark and with ipython notebook and I am trying to add a jar file to the spark env.
I tried Doing the sc.addjar to add the jar but that dint work. Any other suggestions?
I know you can do spark-submit --jars , but I am doing spark submit
I also tried env variable SPARK_CLASS_PATH and placed the jar in that folder and restarted the ipython/spark , it picked it up as a jar but dint do anything.
Related
I have used this command and it works fine:
spark = SparkSession.builder.appName('Apptest')\
.config('spark.jars.packages', 'org.mongodb.spark:mongo-spark-connector_2.11:2.3.5').getOrCreate()
But I'd like to download the jar file and always start with:
spark = SparkSession.builder.appName('Apptest').getOrCreate()
How can I do it? I have tried:
Move to SPARK_HOME jar dir:
cd /de/spark-2.4.6-bin-hadoop2.7/jars
Download jar file
curl https://repo1.maven.org/maven2/org/mongodb/spark/mongo-spark-connector_2.11/2.3.5/mongo-spark-connector_2.11-2.3.5.jar --output mongo-spark-connector_2.11-2.3.5.jar
But spark don't see it. I got the following error:
Py4JJavaError: An error occurred while calling o66.save.
: java.lang.NoClassDefFoundError: com/mongodb/ConnectionString
I know there is ./spark-shell --jar command, but I am using jupyter notebook. Is there some step missing?
Since you're using SparkSession in the jupyter notebook, unfortunately you have to use the .config('spark.jars.packages', '...') to add the jars that you want when you're creating the spark object.
Instead, if you want to add the jar in "default" mode when you launch the notebook, I would recommend you to create a custom kernel, so that every time when you create a new notebook, you even don't need to create the spark. If you're using Anaconda, you can check the docs: https://docs.anaconda.com/ae-notebooks/admin-guide/install/config/custom-pyspark-kernel/
What I was looking for is .config("spark.jars",".."):
spark = SparkSession.builder.appName('Test')\
.config("spark.jars", "/root/mongo-spark-connector_2.11-2.3.5.jar,/root/mongo-java-driver-3.12.5.jar") \
.getOrCreate()
Or:
import os
os.environ["PYSPARK_SUBMIT_ARGS"]="--jars /root/mongo-spark-connector_2.11-2.3.5.jar,/root/mongo-java-driver-3.12.5.jar pyspark-shell"
Also, seems that only put the jar files in $SPARK_HOME/jars works fine as well, but in my case and question example was missing the dependency mongo-java-driver-3.12.5.jar. After download all dependencies in $SPARK_HOME/jars I was able to run only with:
spark = SparkSession.builder.appName('Test').getOrCreate()
I have find out the dependencies in: https://mvnrepository.com/artifact/org.mongodb.spark/mongo-spark-connector_2.11/2.3.5
I have installed pyspark in local mac using homebrew. I am able to see spark under /usr/local/Cellar/apache-spark/3.2.1/
but not able to see hadoop folder. If I run pyspark in terminal it is running spark shell.
Where can I see its path?
I a trying to connect S3 to pyspark and I have dependency jars
You do not need to know the location of Hadoop to do this.
You should use a command like spark-submit --packages org.apache.hadoop:hadoop-aws:3.3.1 app.py instead, which will pull all necessary dependencies rather than download all JARs (with their dependencies) locally.
I am trying to connect to extract data from Teradata using Spark JDBC. I have created a "lib" directory on the main parent directory and placed the external Teradata jars and ran the sbt package. In addition,I am also providing the "--jars" option on my spark-shell command to provide the jar. However, when I run the spark-shell, it does not seem to find the class
Exception in thread "main" java.lang.ClassNotFoundException: com.teradata.hadoop.tool.TeradataImportTool
However, when I do "jar tvf" on the jar file, I see the class. Somehow the Spark utility is unable to find the jar. Is there anything else I need to do so Spark could find it? Please help
This particular class com.teradata.hadoop.tool.TeradataImportTool is in teradata-hadoop-connector.jar
you can try to pass while submitting job like below example :
--conf spark.driver.extraClassPath complete path of teradata-hadoop-connector.jar
--conf spark.executor.extraClassPath complete path of teradata-hadoop-connector.jar
OR
import jars to both driver & executor. So, you need to edit conf/spark-defaults.conf adding both lines below.
spark.driver.extraClassPath complete path of teradata-hadoop-connector.jar
spark.executor.extraClassPath complete path of teradata-hadoop-connector.jar
NOTE : You can use uber jar is also known as fat jar i.e. jar
with dependencies. as well as alternative approach to avoid this kind
of issue
I went through this post setting python path for workers/drivers in standalone spark mode. Apparently, the straightforward way is to direct PYSPARK_PATh environment variable in ./conf/spark-env.sh file located in the conf folder of spark such as /opt/cloudera/parcels/CDH/lib/spark/conf/ in my case. However, I was finding to repeat it for spark in YARN cluster mode. Tried playing around for quite some time. I found this cloudera blog to add Anaconda package.
Now all that is left to do, is add the Anaconda path in the spark-env.sh file instead of the standard python path. It finally worked. Please share if there is a better/alternative way for python setup/update in SPARK and pyspark.
I'm trying to automatically include jars to my PySpark classpath. Right now I can type the following command and it works:
$ pyspark --jars /path/to/my.jar
I'd like to have that jar included by default so that I can only type pyspark and also use it in IPython Notebook.
I've read that I can include the argument by setting PYSPARK_SUBMIT_ARGS in env:
export PYSPARK_SUBMIT_ARGS="--jars /path/to/my.jar"
Unfortunately the above doesn't work. I get the runtime error Failed to load class for data source.
Running Spark 1.3.1.
Edit
My workaround when using IPython Notebook is the following:
$ IPYTHON_OPTS="notebook" pyspark --jars /path/to/my.jar
You can add the jar files in the spark-defaults.conf file (located in the conf folder of your spark installation). If there is more than one entry in the jars list, use : as separator.
spark.driver.extraClassPath /path/to/my.jar
This property is documented in https://spark.apache.org/docs/1.3.1/configuration.html#runtime-environment
As far as I know, you have to import jars to both driver AND executor. So, you need to edit conf/spark-defaults.conf adding both lines below.
spark.driver.extraClassPath /path/to/my.jar
spark.executor.extraClassPath /path/to/my.jar
When I went through this, I did not need any other parameters. I guess you will not need them too.
Recommended way since Spark 2.0+ is to use
spark.driver.extraLibraryPath
and spark.executor.extraLibraryPath
https://spark.apache.org/docs/2.4.3/configuration.html#runtime-environment
ps. spark.driver.extraClassPath and spark.executor.extraClassPath are still there,
but deprecated and will be removed in a future release of Spark.