how to check if an array has colums in a schema? - apache-spark

Question

I have a schema and I would like to check the array if it has columns inside before exploding it. my schema looks like this
|-- CaseNumber: string (nullable = true)
|-- Interactions: struct (nullable = true)
| |-- EmailInteractions: array (nullable = true)
| | |-- element: struct (containsNull = true)
| | | |-- CreatedBy: string (nullable = true)
| | | |-- CreatedOn: string (nullable = true)
| | | |-- Direction: string (nullable = true)
| |-- PhoneInteractions: array (nullable = true)
| | |-- element: string (containsNull = true)
| |-- WebInteractions: array (nullable = true)
| | |-- element: string (containsNull = true)
|-- EntityAction: string (nullable = true)
I would like to check if "EmailInteractions" has elements under it before I run the job that will explode it,
I have edited the question for clarity
1. check if email interactions array exist and check if it has columns, if both true, explode the array and finish, if one of the conditions is false, pass to step 2
2.check if phone interactions array exist and check if it has columns, if both true, explode the array and finish, if one of the conditions is false, pass to step 3
3.check if web interactions exist and check if it has columns, if both true, explode the array and finish, if one of the conditions is false, finish
I am new to coding and data bricks, please help on this.

Answer 1

This is one way of doing it. Unfortunately information under StructField is not searchable easily, I converted it to a string and search the filed by field name or keyword "StructField" to know that it contains a field.
val jsonWithNull = """{"a": 123,"b":null,"EmailInteractions":[{"CreatedBy":"test"}]}"""
val jsonWithoutNull = """{"a": 123,"b":3,"EmailInteractions":[{"CreatedBy":"test1"}]}"""
import spark.implicits._
val df = spark.read.json(Seq(jsonWithNull,jsonWithoutNull).toDS)
df.printSchema()
val field = df.schema.filter { f =>
if (f.dataType.typeName == "array" && f.toString().contains("CreatedBy")){
true
}
else{
false
}
}
println(field)
// check for field value being not null then explode
Result List(StructField(EmailInteractions,ArrayType(StructType(StructField(CreatedBy,StringType,true)),true),true))