Spark3.0 failed to write stream from cosmos changefeed to snowflake - apache-spark

Question

I use:
<dependency>
<groupId>com.azure.cosmos.spark</groupId>
<artifactId>azure-cosmos-spark_3-1_2-12</artifactId>
<version>4.4.0</version>
</dependency>
to read stream from cosmos changefeed, but I run faild because of
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.execution.streaming.MetadataVersionUtil$
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
I found that MetadataVersionUtil is a new class in spark3.1, and I can't find a jar suitable for spark3.0, such as azure-cosmos-spark_3-0_2-12.
So I clone https://github.com/Azure/azure-cosmosdb-spark.git and checkout 3.0 branch, and use mvn install to release the jar. But when I use this new jar, it failed because of:
Exception in thread "main" java.lang.ClassNotFoundException: Failed to find data source: cosmos.oltp.changeFeed. Please find packages at http://spark.apache.org/third-party-projects.html
at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:689)
at org.apache.spark.sql.streaming.DataStreamReader.loadInternal(DataStreamReader.scala:209)
at org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:195)
at com.fti.cosmos.StreamToSnowflake$.main(StreamToSnowflake.scala:79)
at com.fti.cosmos.StreamToSnowflake.main(StreamToSnowflake.scala)
Caused by: java.lang.ClassNotFoundException: cosmos.oltp.changeFeed.DefaultSource
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at org.apache.spark.sql.execution.datasources.DataSource$.$anonfun$lookupDataSource$5(DataSource.scala:663)
at scala.util.Try$.apply(Try.scala:213)
at org.apache.spark.sql.execution.datasources.DataSource$.$anonfun$lookupDataSource$4(DataSource.scala:663)
at scala.util.Failure.orElse(Try.scala:224)
at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:663)
... 4 more
22/02/25 17:15:56 INFO SparkContext: Invoking stop() from shutdown hook
the scala code is:
val readConfig = Map(
"spark.cosmos.accountEndpoint" -> s"${cosmosHost}:${cosmosPort}",
"spark.cosmos.accountKey" -> cosmosPassword,
"spark.cosmos.database" -> cosmosSourceDB,
"spark.cosmos.container" -> cosmosSourceCollection,
"spark.cosmos.read.partitioning.strategy" -> "Default",
"spark.cosmos.read.inferSchema.enabled" -> "false",
"spark.cosmos.changeFeed.startFrom" -> cosmosCfStartFrom,
"spark.cosmos.changeFeed.mode" -> "Incremental"
)
// init spark
val ss = SparkSession.builder
.appName(s"${this.getClass.getName}-${cosmosSourceDB}-${cosmosSourceCollection}")
.master("local")
.getOrCreate()
val df = ss.readStream.format("cosmos.oltp.changeFeed").options(readConfig).load()
val sfOptions = Map(
"sfURL" -> "",
"sfUser" -> "",
"sfRole" -> s"",
"pem_private_key" -> "",
"sfDatabase" -> s"",
"sfSchema" -> sfSchema,
"sfWarehouse" -> sfWarehouse
)
df.writeStream.trigger(Trigger.ProcessingTime("30 seconds"))
.foreachBatch((ds: DataFrame, batchId: Long) => {
ds.toDF().show(false)
ds.toDF().write
.format(SNOWFLAKE_SOURCE_NAME)
.options(sfOptions)
.option("dbtable", targetTable)
.mode(SaveMode.Append)
.save()
})
.option("checkpointLocation", f"wasbs://....")
.outputMode("append")
.start()
.awaitTermination()
Has anyone successfully synced cosmos's changefeed with spark3.0, I don't want to upgrade HdInsight to 3.1 anymore.