Spark: Conditionally Joining/Concatting Columns Based Leading Characters - apache-spark

Question

I've got a data set that has unclean data that has been split incorrectly. This results in an uneven number of columns - the number of columns per row depends on the number of errors arising from one field. You know if the column is incorrect if it has 3 leading double quotes. If the column has 3 leading double quotes then you want to join it with the previous column and shift left.
I import the csv of the data into a dataframe, which creates something similar to the example below.
Example:
INPUT:
`+--+--------+----------+----------+---------+
|id | detail | context | _c3 | _c4|
+---+--------+----------+----------+---------+
| 1 | {blah} | service | null | null |
| 2 | { blah | """ blah | """blah} | service |
| 3 | { blah | """blah} | service | null |
+---+--------+----------+----------+---------+`
DESIRED OUTPUT:
`+--+------------------------+---------+
|id | detail | context |
+---+------------------------+---------+
| 1 | {blah} | service |
| 2 | { blah""" blah"""blah} | service |
| 3 | { blah"""blah} | service |
+---+------------------------+---------+`
I've tried something like - as well as a bunch of other approaches:
`df.filter(col("context").startsWith("\"\"\"")).select($"detail", lit(" "), $"context").collect()`
This doesn't works, and doesn't fully do what I need it to do. Any ideas? Help is much appreciated :)
Thanks!

Answer 1

I think the easiest way to fix this, would be to put the columns back together, and then parse them correctly. One way to do this is use concat to combine all the columns, then use regexp_extract to pull out the pieces you want as individual columns. For example:
case class MyRow(id: Int, detail: String, context: String, _c3: String, _c4: String)
val data = Seq(
MyRow(1, "{blah}", "service", "", ""),
MyRow(2, "{ blah", " \"\"\" blah", " \"\"\"blah}", "service"),
MyRow(3, "{ blah", "\"\"\"blah}", "service", "")
)
val df = sc.parallelize(data).toDF
val columns = df.columns.filterNot(_ == "id")
val nonulls = df.na.fill("")
val combined = nonulls.select($"id", concat(columns.map(col):_*) as "data")
val fixed = combined.withColumn("left", regexp_extract($"data", "(\\{.*\\})", 1)).
withColumn("right", regexp_extract($"data", "([^}]+$)", 1))
fixed.show(10, false)
Which should output:
+---+-------------------------------+------------------------+-------+
|id |data |left |right |
+---+-------------------------------+------------------------+-------+
|1 |{blah}service |{blah} |service|
|2 |{ blah """ blah """blah}service|{ blah """ blah """blah}|service|
|3 |{ blah"""blah}service |{ blah"""blah} |service|
+---+-------------------------------+------------------------+-------+
In the code above I'm assuming that the columns are already in the right order.
This is just splitting on the last }. If you need more complicated parsing, you can write a UDF that parses it however you want and returns a tuple of fields.