Spark: Converting nested JSON DataFrame to Parquet crashes with NPE - apache-spark

Question

Spark 1.5.0. My source data is lots of JSON files containing nested JSON structures, which I want to turn into Parquet. However, this crashes with what appears to be a Spark bug, and I'm struggling to find a way around it.
I start by loading:
val df = sqlContext.read.json("s3://...")
This has a schema of 7 columns, but that's just the top-level. Some of those columns are JSON objects, and if we traverse the entire thing we end up with something like 125 columns. However, Parquet doesn't understand this nesting, so we need to flatten it.
Therefore I do like this:
val flat = flattenSchema(df.schema.toArray) // Array[StructField]
val goodfields = flat filter(isgood) // gets rid of problematic columns
I have a function that flattens a row the same way we flattened the schema, and which also filters on the way. So this means that goodfields(x) has the definition for row(x) for all x.
Anyway, we now make an RDD of flattened rows, then turn it into a DataFrame with the flattened schema:
val rows = df.map{ row => Row.fromSeq(smartFlattenRow(row, df.schema, isgood)) }
val newdf = sqlContext.createDataFrame(rows, StructType(goodfields))
So far, everything looks great. Then I try to save to Parquet and it goes kaboom:
newdf.saveAsParquetFile("s3://whatever.parquet")
It runs a little while, then fails with
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
at org.apache.spark.sql.DataFrame.schema(DataFrame.scala:290)
at $line122.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1.apply(<console>:27)
The line that fails is this one:
def schema: StructType = queryExecution.analyzed.schema
However, newdf.schema works just fine, so it's not my new DataFrame that's causing the problem. What's confusing is that rows is an RDD, so that's not the cause either. And we started out by using df.schema, so it's not that DataFrame, either. So where is this coming from?
Am I somehow building this new structure the wrong way? I based the entire thing on the Spark programming guide, so seems like it should be legit.
Additional info
To provide a little more detail, here is some example data and the supporting functions.
The actual data is sensitive, so I can't share that, but to give you an idea it's basically W3C Activity Streams 2.0 with some custom extensions. Example 6 looks a lot like our real data. Posting the entire thing is too much as it's literally 127 fields.
The code to flattenSchema:
def flattenSchema(fields: Array[StructField]) : Array[StructField] = {
if (fields isEmpty)
fields
else {
val first = fields head
val rest = fields tail
first.dataType match {
case subfields: StructType =>
flattenSchema(subfields.fields).map { (f : StructField) =>
StructField(first.name + "." + f.name,
f.dataType, f.nullable, f.metadata)
} ++ flattenSchema(rest)
case _ => first +: flattenSchema(rest)
}
}
}
And the code to smartFlattenRow:
def smartFlattenRow(row : Row, schema : StructType,
pred : StructField => Boolean) : Seq[Any] = {
val cols = schema.toArray
(0 until row.size).flatMap { ix =>
cols(ix).dataType match {
case subfields: StructType =>
val subrow = if (row.isNullAt(ix))
Row.fromSeq(Seq.fill(subfields.fields.size)(null))
else
row(ix).asInstanceOf[Row]
smartFlattenRow(subrow, subfields, pred )
case _ => if (pred(cols(ix))) {
if (row.isNullAt(ix))
Seq(null)
else
Seq(row(ix))
} else
Seq()
}
}
}