How to covert a Dataframe to a Dataset,having a object reference of the parent class as a composition inside another class? - apache-spark

Question

I am trying to convert a Dataframe to a Dataset, and the java classes structure is as follows:
class A:
public class A {
private int a;
public int getA() {
return a;
}
public void setA(int a) {
this.a = a;
}
}
class B:
public class B extends A {
private int b;
public int getB() {
return b;
}
public void setB(int b) {
this.b = b;
}
}
and class C
public class C {
private A a;
public A getA() {
return a;
}
public void setA(A a) {
this.a = a;
}
}
and the data in the dataframe is as follows :
+-----+
| a |
+-----+
|[1,2]|
+-----+
When I am trying to apply Encoders.bean[C](classOf[C]) to the dataframe. The object reference A which is a instance of B in class C is not returning true when I am checking for .isInstanceOf[B], I am getting it as false. The output of Dataset is as follows:
+-----+
| a |
+-----+
|[1,2]|
+-----+
How do we get all the fields of A and B under the C object while iterating over it in foreach?
Code :-
object TestApp extends App {
implicit val sparkSession = SparkSession.builder()
.appName("Test-App")
.config("spark.sql.codegen.wholeStage", value = false)
.master("local[1]")
.getOrCreate()
var schema = new StructType().
add("a", new ArrayType(new StructType().add("a", IntegerType, true).add("b", IntegerType, true), true))
var dd = sparkSession.read.schema(schema).json("Test.txt")
var ff = dd.as(Encoders.bean[C](classOf[C]))
ff.show(truncate = false)
ff.foreach(f => {
println(f.getA.get(0).isInstanceOf[A])//---true
println(f.getA.get(0).isInstanceOf[B])//---false
})
Content of File : {"a":[{"a":1,"b":2}]}

Answer 1

Spark-catalyst uses google reflection to get schema out of java beans.
Please take a look at the JavaTypeInference.scala#inferDataType. This class uses getters to collect the field name and the returnType of getters to compute the SparkType.
Since class C has getter named getA() with returnType as A and A, in turn, has getter as getA() with returnType as int,
Schema will be created as struct<a:struct<a:int>> where struct<a:int> is derived from the getA of class A.
The solution to this problem that I can think of is -
// Modify your class C to have Real class reference rather its super type
public class C {
private B a;
public B getA() {
return a;
}
public void setA(B a) {
this.a = a;
}
}
Output-
root
|-- a: struct (nullable = true)
| |-- a: integer (nullable = false)
| |-- b: integer (nullable = false)
+------+
|a |
+------+
|[1, 2]|
+------+