Spark text parsing with dynamic delimiter - apache-spark

Question

I have text file which looks like:
:1: some first row of first attribute
second row of first attribute
:55: first row of fifty fifth
:100: some other text
also other
another one
I would like to parse it such manner:
+----------+-----------------------------------+
| AttrNr | Row |
+----------+-----------------------------------+
| 1 | some first row of first attribute |
+----------+-----------------------------------+
| 1 | second row of first attribute |
+----------+-----------------------------------+
| 1 | 3rd value with test: 1,2,3 |
+----------+-----------------------------------+
| 55 | first row of fifty fifth |
+----------+-----------------------------------+
| 100 | some other text |
+----------+-----------------------------------+
| 100 | also other |
+----------+-----------------------------------+
| 100 | another one |
+----------+-----------------------------------+
Parsing should be done according :n: delimeter. ":" symbol might appear in values.

Answer 1

The final output can be achieved by using a set of Window functions that are available on Spark but your data lacks a lot of essential details like a partitioning column, a column with which we can order the data to know which row comes after the first one.
Assuming you are working on a distributed system here the following answer might not work at all. It works for the provided example but things will be different when you are in a distributed environment with a huge file.
Creating a DataFrame from the text file:
Reading the text file as an RDD:
val rdd = sc.parallelize(Seq(
(":1: some first row of first attribute"),
("second row of first attribute"),
(":55: first row of fifty fifth"),
(":100: some other text"),
("also other"),
("another one")
))
// Or use spark.sparkContext.textFile if you are reading from a file
Iterate over the RDD to split the columns in the required format and generate a DataFrame
val df = rdd.map{ c=>
if(c.startsWith(":")) (c.split(" ", 2)(0), c.split(" ", 2)(1))
else (null.asInstanceOf[String], c )
}.toDF("AttrNr", "Row")
//df: org.apache.spark.sql.DataFrame = [AttrNr: string, Row: string]
df.show(false)
// +------+---------------------------------+
// |AttrNr|Row |
// +------+---------------------------------+
// |:1: |some first row of first attribute|
// |null |second row of first attribute |
// |:55: |first row of fifty fifth |
// |:100: |some other text |
// |null |also other |
// |null |another one |
// +------+---------------------------------+
The following set of commands are just a hack and are not performance effective at all and shouldn't be used in a production-like environment. last provides your the last not null column. Partition and ordering is done manually here because your data does not provide those set of columns.
df.withColumn("p", lit(1))
.withColumn("AttrNr",
last($"AttrNr", true).over(Window.partitionBy($"p").orderBy(lit(1)).rowsBetween(Window.unboundedPreceding, 0) ) )
// +------+---------------------------------+
// |AttrNr|Row |
// +------+---------------------------------+
// |:1: |some first row of first attribute|
// |:1: |second row of first attribute |
// |:55: |first row of fifty fifth |
// |:100: |some other text |
// |:100: |also other |
// |:100: |another one |
// +------+---------------------------------+

Answer 2

Actually I solved it with SQL. But I was wondering maybe it's more simple way. I'm using spark 2.3 without high order functions.
import org.apache.spark.sql.expressions.Window
val df = Seq((":1: some first row of first attribute"),
("second row of first attribute"),
("3rd value with test: 1,2,3"),
(":55: first row of fifty fifth"),
(":100: some other text"),
("also other"),
("another one")).toDF("_c0")
df.createOrReplaceTempView("test1")
spark.sql("""select _c0, split(_c0, ":") arr from test1""").createOrReplaceTempView("test2")
val testDF = spark.sql("""
select arr[1] t0,
cast(arr[1] as int) t1,
case when arr[1] = cast(arr[1] as int)
then replace(concat_ws(":",arr),concat(concat(":",arr[1]),":"),"")
else concat_ws(":",arr)
end Row
,monotonically_increasing_id() mrn
from test2""")
val fnc = Window.orderBy("mrn")
val testDF2 = testDF.withColumn("AttrNr", last('t1,true).over(fnc))
testDF2.drop("t0","t1","mrn").show(false)
+----------------------------------+------+
|Row |AttrNr|
+----------------------------------+------+
| some first row of first attribute|1 |
|second row of first attribute |1 |
|3rd value with test: 1,2,3 |1 |
| first row of fifty fifth |55 |
| some other text |100 |
|also other |100 |
|another one |100 |
+----------------------------------+------+

Answer 3

Column "AttrNr" can be received with "regexp_extract" function:
df
.withColumn("AttrNr", regexp_extract($"_c0", "^:([\\d].*):", 0))
.withColumn("Row", when(length($"AttrNr") === lit(0), $"_c0").otherwise(expr("substring(_c0, length(AttrNr) + 2)")))
.withColumn("AttrNr", when(length($"AttrNr") === lit(0), null.asInstanceOf[String]).otherwise(expr("substring(_c0, 2, length(AttrNr) - 2)")))
// Window with no partitioning, bad for performance
.withColumn("AttrNr", last($"AttrNr", true).over(Window.orderBy(lit(1)).rowsBetween(Window.unboundedPreceding, 0)))
.drop("_c0")