Issue while converting Scala Map to Object on Databricks Notebook - azure

Question

Issue
I have a scenario where I need to convert a scala Map to a case class object and with help of the following references I was able to achieve it locally (scala version 2.12.13):
Scala: convert map to case class
Convert a Map into Scala object
But when I tried running the same block of code in Databricks notebook it throws an error:
IllegalArgumentException: Cannot construct instance of '$line23851bc084ae4df7a16bf9c475868d9265.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$Test' (although at least one Creator exists): can only instantiate non-static inner class by using default, no-argument constructor at [Source: UNKNOWN; line: -1, column: -1]
Cluster configuration: Databricks runtime 8.2(Includes Spark 3.1.1, Scala 2.12). Please refer to the screenshot for the complete code.
Workaround (not advisable):
def workaround(map: Map[String, Any]): Test = {
Test(
map("k1").asInstanceOf[Int],
map("k2").asInstanceOf[String],
map("k3").asInstanceOf[String],
)
}
val result = workaround(myMap)
Any thoughts on how to resolve this issue?

Answer 1

This smells to me like one of two possibilities.
First, we should double check that there is no version mis-matching between something in your local runtime environment and the databricks runtime environment. You can check this page for a list of all library versions included in DBR 8.2. In particular, I would check your local environment to make sure you are running the same version of jackson (2.10.0).
Second, this could be an interaction between how Databricks implements their notebook and a limitation of jackson. Each command of a databricks notebooks is wrapped in a package object that is given a random name. For example, I can tell from the exception that the package object which holds your Test class definition was called $line23851bc084ae4df7a16bf9c475868d9265.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw. (I say that it was named that because each time you re-run the command it will generate a new package object). This implies that all classes (and other kinds of type definitions) are actually path dependent types when put inside a notebook. More specifically for this scenario, your Test class is an inner-class to the package object. based on the error message and some brief reading of docs, I suspect that Jackson cannot serialize path dependent types.