Apache Spark - Unable to read data from MS Access tables into Spark dataset - apache-spark

Question

When I try to read .accdb data into my spark dataset I am getting
Exception in thread "main" java.lang.NoClassDefFoundError: Could not initialize class net.ucanaccess.jdbc.UcanaccessDriver
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:398)
at java.sql/java.sql.DriverManager.isDriverAllowed(DriverManager.java:555)
at java.sql/java.sql.DriverManager.isDriverAllowed(DriverManager.java:547)
at java.sql/java.sql.DriverManager.getDriver(DriverManager.java:280)
at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.$anonfun$driverClass$2(JDBCOptions.scala:105)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.<init>(JDBCOptions.scala:105)
at org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions.<init>(JDBCOptions.scala:35)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:32)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:318)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:167)
at business.extract.DataExtractorImpl.loadFromAccessTable(DataExtractorImpl.java:62)
at application.Orchestrator.initializeJob(Orchestrator.java:52)
at application.ETLEngine.main(ETLEngine.java:15)
Here is my code:
//DataExtractorImpl.java
public Dataset<Row> loadFromAccessTable(String url, String tableName) throws IOException, CustomValidationException {
return ETLContext.getETLContext().getSession()
.read()
.format("jdbc")
.option("URL", "jdbc:ucanaccess://C:/Users/KE926ES/Documents/db/Creditcard_default.accdb")
.option("dbtable", "CC_SOURCE_1")
.load();
I have the following jars
ucanaccess-5.0.0.jar
jackcess-3.0.1.jar
commons-lang3-3.10.jar
commons-logging-1.2.jar
I also tried adding the following to the options list
.option("driver", "net.ucanaccess.jdbc.UcanaccessDriver")

Answer 1

Libraries might not be available in class path or fat jar.
Try to pass required jars to your application using spark-submit like below.
spark-submit --conf spark.driver.extraClassPath=ucanaccess-5.0.0.jar:jackcess-3.0.1.jar:commons-lang3-3.10.jar:commons-logging-1.2.jar --conf spark.executor.extraClassPath=ucanaccess-5.0.0.jar:jackcess-3.0.1.jar:commons-lang3-3.10.jar:commons-logging-1.2.jar
Also If you are running application from IDE, Check those jars are added properly to your class path or not.