Can we automate spark SQL query generation from AVRO schema? - apache-spark

Question

I'm working on project where everyday I need to deal with tons of AVRO files. To extract the data from AVRO I use sparkSQL. To achieve this first I need to printSchema and then I need to select the fields to see the data. I want to automate this process. Given any input AVRO I want to write a script which will automatically generated SparkSQL query(considering the struct and arrays in avsc file). I'm okay to write a script in Java or Python.
-- Sample input AVRO
root
|-- identifier: struct (nullable = true)
| |-- domain: string (nullable = true)
| |-- id: string (nullable = true)
| |-- version: long (nullable = true)
alternativeIdentifiers: array (nullable = true)
| | |-- element: struct (containsNull = true)
| | | |-- identifier: struct (nullable = true)
| | | | |-- domain: string (nullable = true)
| | | | |-- id: string (nullable = true)
-- Output I'm expecting
SELECT identifier.domain, identifier.id, identifier.version

Answer 1

You can use something like this to generate list of columns based on the schema:
import org.apache.spark.sql.types.{StructField, StructType}
def getStructFieldName(f: StructField, baseName: String = ""): Seq[String] = {
val bname = if (baseName.isEmpty) "" else baseName + "."
f.dataType match {
case StructType(s) =>
s.flatMap(x => getStructFieldName(x, bname + f.name))
case _ => Seq(bname + f.name)
}
}
Then it could be used on the real dataframe, like this:
val data = spark.read.json("some_data.json")
val cols = data.schema.flatMap(x => getStructFieldName(x))
as result, we're getting the sequence of strings, that we can use either to do a select:
import org.apache.spark.sql.functions.col
data.select(cols.map(col): _*)
or we can generate a comma-separated list that we can use in the spark.sql:
spark.sql(s"select ${cols.mkString(", ")} from table")