Creating a CSV file header dynamically in spark scala - apache-spark

Question

We receive 2 files (data file and metadata file) from Vendors for data ingestion.
Vendor 1
data file format
user_id has_insurance postal_code city
101 Y 20001 Newyork
102 N 40001 Boston
metadata file format
user_id,String
has_insurance,Boolean
postal_code,String
city, String
we will receive same data fields from another vendor but fields order in data file might be different as below
Vendor 2
data file format
user_id postal_code city has_insurance
101 20001 Newyork Y
102 40001 Boston N
metadata file format
user_id,String
postal_code,String
city, String
has_insurance,Boolean
The metadata file will contain the fields order. Is it possible to assign schema dynamically based on the metadata file while reading CSV file?
//function to derive spark datatype for the given field data type
def strToDataType(str: String): DataType = {
| if (str == "String") StringType
| else if (str == "Boolean") BooleanType
| else StringType
| }
val metadataDf = spark.sqlContext.textFile("metadata_folder")
val headerSchema = StructType(metadataDf.map(_.split(",")).map(x => StructField(x(0),strToDataType(x(1)),true)))
val df = sqlContext.read
.format("com.databricks.spark.csv")
.option("header", "true") // Use first line of all files as header
.schema(headerSchema) // defining based on the custom schema
.load("data_file.csv")
val headerSchema =
StructType(bxfd.map(_.split(",")).map(x => StructField(x(0),strToDataType(x(1)),true)))
when I tried to create schema dynamically using the above command, getting below error. Could you please advise.
<console>:34: error: overloaded method value apply with alternatives:
(fields: Array[org.apache.spark.sql.types.StructField])org.apache.spark.sql.types.StructType <and>
(fields: java.util.List[org.apache.spark.sql.types.StructField])org.apache.spark.sql.types.StructType <and>
(fields: Seq[org.apache.spark.sql.types.StructField])org.apache.spark.sql.types.StructType
cannot be applied to (org.apache.spark.rdd.RDD[org.apache.spark.sql.types.StructField])

Answer 1

You can't use the CSV data source in this case, but you can parse it yourself without too much difficult. The basic steps are read the metadata file as one text blob, then the data file separating data lines from header lines and apply the schema.
I made some assumptions about your separators. One thing to remember is you shouldn't assume any ordering to the lines in your DataFrame once you load it, because you don't know what data is being loaded on what worker. This tries to identify data vs metadata lines based upon their content, so you may need to adjust those rules.
def loadVendor(dataPath: String, metaPath: String): DataFrame = {
val df = spark.read.text(path)
// First read the metadata file, wholeTextFiles lets us get it all
// as a single string we so we can parse locally
val metaText = sc.wholeTextFiles(metaPath).first._2
val metaLines = metaText.
split("\n").
map(_.split(","))
// Identify header as line that has all the field names
val fields = Seq("user_id", "has_insurance", "postal_code", "city")
val headerDf = fields.foldLeft(df)((df2, fname) => df2.filter($"value".contains(fname)))
val headerLine = headerDf.first.getString(0)
val header = headerLine.split(" ")
// Identify data rows as ones that aren't special lines or the header
var data = df.
filter($"value" =!= headerLine).
filter(!$"value".startsWith("Vendor 1")).
filter(!$"value".startsWith("data file format"))
// Split the data fields on separator and assign column names. Assumed any whitespace is your separator
val rows = data.select(split($"value", raw"\W+") as "fields")
val named = header.zipWithIndex.map( { case (f, idx) => $"fields".getItem(idx).alias(f)} )
val table = rows.select(named:_*)
// Cast to the right types
val castCols = metaLines.map { case Array(cname, ctype) => col(cname).cast(ctype) }
val typed = table.select(castCols:_*)
// Return the columns sorted by name, so union'ing multiple DF's will line up
typed.select(table.columns.sorted.map(col):_*)
}
Here is the data printed
scala> df.show
+-------+-------------+-----------+-------+
| city|has_insurance|postal_code|user_id|
+-------+-------------+-----------+-------+
|Newyork| true| 20001| 101|
| Boston| false| 40001| 102|
+-------+-------------+-----------+-------+
And here is the schema printed
scala> df.printSchema
root
|-- city: string (nullable = true)
|-- has_insurance: boolean (nullable = true)
|-- postal_code: string (nullable = true)
|-- user_id: string (nullable = true)