How to run Spark SQL JDBC/ODBC server and pyspark at the same time? - apache-spark

Question

I have a one node deployment of Spark. I am running JDBC/ODBC server on it. Which works fine. However, if at the same time I use pyspark to save a table (df.write.saveAsTable()) I get a very long error message. I think the core part to it is this:
Caused by: ERROR XSDB6: Another instance of Derby may have already booted the database /root/spark/bin/metastore_db.
Doing some research, I've found that this is caused by Spark creating a new session which tries to create another instance of Derby which causes an error. The solution offered is to shut down all other spark-shell processes. However if I do that then ODBC server stops running.
What can I do to have both running at the same time?

Answer 1

You might want to use derby network server instead of the default embedded version so it can be shared by multiple processes. Or you use another datastore such as MySQL.
After installing derby network server, you can copy the derby-client.jar file into the spark jars directory and then edit the file conf/hive-site.xml with something like:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
  <name>javax.jdo.option.ConnectionURL</name>
  <value>jdbc:derby://localhost:1527/metastore_db;create=true</value>
  <description>JDBC connect string for a JDBC metastore</description>
</property>
<property>
  <name>javax.jdo.option.ConnectionDriverName</name>
  <value>org.apache.derby.jdbc.ClientDriver</value>
  <description>Driver class name for a JDBC metastore</description>
</property>
</configuration>