azure pyspark register udf from jar Failed UDFRegistration - azure

Question

I'm having trouble registering some udfs that are in a java file. I've a couple approaches but they all return :
Failed to execute user defined function(UDFRegistration$$Lambda$6068/1550981127: (double, double) => double)
First I tried this approach:
from pyspark.context import SparkContext, SparkConf
from pyspark.sql import SparkSession
from pyspark.sql.types import *
conf=SparkConf()
conf.set('spark.driver.extraClassPath', 'dbfs:/FileStore/jars/4b129434_12cd_4f2a_ab27_baaefe904857-scala_udf_similarity_0_0_7-35e3b.jar')
conf.set('spark.jars', 'dbfs:/FileStore/jars/4b129434_12cd_4f2a_ab27_baaefe904857-scala_udf_similarity_0_0_7-35e3b.jar')
spark = SparkSession(sc)
sc = SparkContext.getOrCreate(conf=conf)
#spark.sparkContext.addPyFile("dbfs:/FileStore/jars/4b129434_12cd_4f2a_ab27_baaefe904857-scala_udf_similarity_0_0_7-35e3b.jar")
udfs = [
('jaro_winkler_sim', 'JaroWinklerSimilarity',DoubleType()),
('jaccard_sim', 'JaccardSimilarity',DoubleType()),
('cosine_distance', 'CosineDistance',DoubleType()),
('Dmetaphone', 'DoubleMetaphone',StringType()),
('QgramTokeniser', 'QgramTokeniser',StringType())
]
for a,b,c in udfs:
spark.udf.registerJavaFunction(a, 'uk.gov.moj.dash.linkage.'+ b, c)
linker = Splink(settings, spark, df_l=df_l, df_r=df_r)
df_e = linker.get_scored_comparisons()
next I tried to move the jars and extraClassPath to the cluster config.
spark.jars dbfs:/FileStore/jars/4b129434_12cd_4f2a_ab27_baaefe904857-scala_udf_similarity_0_0_7-35e3b.jar
spark.driver.extraClassPath dbfs:/FileStore/jars/4b129434_12cd_4f2a_ab27_baaefe904857-scala_udf_similarity_0_0_7-35e3b.jar
The I registered them in my script as follows:
from pyspark.context import SparkContext, SparkConf
from pyspark.sql import SparkSession, udf
from pyspark.sql.types import *
# java path to class uk.gov.moj.dash.linkage.scala-udf-similarity.CosineDistance
udfs = [
('jaro_winkler_sim', 'JaroWinklerSimilarity',DoubleType()),
('jaccard_sim', 'JaccardSimilarity',DoubleType()),
('cosine_distance', 'CosineDistance',DoubleType()),
('Dmetaphone', 'DoubleMetaphone',StringType()),
('QgramTokeniser', 'QgramTokeniser',StringType())
]
for a,b,c in udfs:
spark.udf.registerJavaFunction(a, 'uk.gov.moj.dash.linkage.'+ b, c)
linker = Splink(settings, spark, df_l=df_l, df_r=df_r)
df_e = linker.get_scored_comparisons()
Thanks

Answer 1

Looking into the source code of the UDFs, I see that it's compiled with Scala 2.11, and uses Spark 2.2.0 as a base. The most probable reason for the error is that you're using this jar with DBR 7.x that is compiled with Scala 2.12 and based on Spark 3.x that are binary incompatible with your jar. You have following choices:
Recompile the library with Scala 2.12 and Spark 3.0
Use DBR 6.4 that uses Scala 2.11 and Spark 2.4
P.S. Overwriting classpath on Databricks sometimes could be tricky, so it's better to use other approaches:
Install your jar as library into cluster - this could be done via UI, or via REST API, or via some other automation, like, terraform
Use [init script][2] to copy your jar into default location of the jars. In simplest case it could look like as following:
#!/bin/bash
cp /dbfs/FileStore/jars/4b129434_12cd_4f2a_ab27_baaefe904857-scala_udf_similarity_0_0_7-35e3b.jar /databricks/jars/