How to connect to Hbase using spark-3.3.1 - apache-spark

Question

I have hbase-2.2.6 and spark-3.3.1 installed on my cluster. and i want to perform operations on hbase using spark. But i am unable to do so. The error talks about classnotfoundexcepetion tableDiscriptor.
i have tried to create hbase-spark connector using https://github.com/LucaCanali/Miscellaneous/blob/master/Spark_Notes/Spark_HBase_Connector.md
but the connectors are not working
/opt/spark/spark-3.3.1-bin-hadoop3$ bin/pyspark --master yarn --num-executors 1 --executor-cores 2 --jars /opt/spark/spark-3.3.1-bin-hadoop3/jars/hbase-spark-1.0.1-SNAPSHOT.jar --packages org.apache.hbase:hbase-shaded-mapreduce:2.2.6
Python 3.8.10 (default, Nov 14 2022, 12:59:47)
\[GCC 9.4.0\] on linux
Welcome to
\___\_ \_\_
/ __/__ \__\_ ___/ /__
\\ / \_ / \_ \`/ \_\_/ '_/
/ / .\_/\_,_/_/ /_/\_\\ version 3.3.1
/_/
Using Python version 3.8.10 (default, Nov 14 2022 12:59:47)
Spark context Web UI available at http://node2.ellicium.com:4040
Spark context available as 'sc' (master = yarn, app id = application_1675083498730_0062).
SparkSession available as 'spark'.
> > > df = spark.sql("select id, 'myline\_'||id name from range(20)")
> > > df.show()
> > > \+---+---------+
> > > | id| name|
> > > \+---+---------+
> > > | 0| myline_0|
> > > | 1| myline_1|
> > > | 2| myline_2|
> > > | 3| myline_3|
> > > | 4| myline_4|
> > > | 5| myline_5|
> > > | 6| myline_6|
> > > | 7| myline_7|
> > > | 8| myline_8|
> > > | 9| myline_9|
> > > | 10|myline_10|
> > > | 11|myline_11|
> > > | 12|myline_12|
> > > | 13|myline_13|
> > > | 14|myline_14|
> > > | 15|myline_15|
> > > | 16|myline_16|
> > > | 17|myline_17|
> > > | 18|myline_18|
> > > | 19|myline_19|
> > > \+---+---------+
> > > df.write.format("org.apache.hadoop.hbase.spark").option("hbase.columns.mapping","id INT :key, name STRING cf:name").option("hbase.namespace", "default").option("hbase.table", "testspark").option("hbase.spark.use.hbasecontext", False).save()
> > > Traceback (most recent call last):
> > > File "\<stdin\>", line 1, in \<module\>
> > > File "/opt/spark/spark-3.3.1-bin-hadoop3/python/pyspark/sql/readwriter.py", line 966, in save
> > > self.\_jwrite.save()
> > > File "/opt/spark/spark-3.3.1-bin-hadoop3/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py", line 1321, in __call__
> > > File "/opt/spark/spark-3.3.1-bin-hadoop3/python/pyspark/sql/utils.py", line 190, in deco
> > > return f(\*a, \*\*kw)
> > > File "/opt/spark/spark-3.3.1-bin-hadoop3/python/lib/py4j-0.10.9.5-src.zip/py4j/protocol.py", line 326, in get_return_value
> > > py4j.protocol.Py4JJavaError: An error occurred while calling o56.save.
> > > : java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/client/TableDescriptor
> > > at org.apache.hadoop.hbase.spark.DefaultSource.createRelation(DefaultSource.scala:78)
> > > at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:47)
> > > at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
> > > at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
> > > at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
> > > at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:98)
> > > at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:109)
> > > at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:169)
> > > at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:95)
> > > at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
> > > at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
> > > at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:98)
> > > at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:94)
> > > at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:584)
> > > at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:176)
> > > at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:584)
> > > at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
> > > at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
> > > at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
> > > at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
> > > at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
> > > at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:560)
> > > at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:94)
> > > at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:81)
> > > at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:79)
> > > at org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:116)
> > > at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:860)
> > > at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:390)
> > > at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:363)
> > > at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:247)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > at java.lang.reflect.Method.invoke(Method.java:498)
> > > at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
> > > at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
> > > at py4j.Gateway.invoke(Gateway.java:282)
> > > at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
> > > at py4j.commands.CallCommand.execute(CallCommand.java:79)
> > > at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
> > > at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
> > > at java.lang.Thread.run(Thread.java:750)
> > > Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.client.TableDescriptor
> > > at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
> > > at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> > > at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> > > ... 42 more