Columns value comparison in Spark data frame - apache-spark

Question

I have a data frame which contains huge number of records. In that DF a record can be repeated multiple times and every time when it got updated the last updated field will have the date on which the record modified.
We have a group of columns on which we want to compare the rows of similar ids. During this comparison we want to capture what are the fields/columns has changed from previous record to current record and capture that in a "updated_columns" column of the updated record. Compare this second record to third record and identify the updated columns and capture that in "updated_columns" field of third record, continue the same till the last record of that id and do the same thing for each and every id which has more than one entry.
Initially we grouped the columns and created a hash out of that group of columns and compare against hash values of next row,this way it is helping me to identify records which has updates, but want the columns which got updated.
Here I am sharing some data, which is expected outcome and that is how the final data should like look post adding updated columns (here I can say, use columns Col1, Col2, Col3, col4 and Col5 for comparison between two rows):
Want to do this in a efficient way. Any one tried some thing like this.
Looking for a help!
~Krish.

Answer 1

A window can be used.
The idea is to group the data by ID, sort it by LAST-UPDATED, copy the values of the previous row (if it exists) into the current row and then compare the copied data with the current values.
val data = ... //the dataframe has the columns ID,Col1,Col2,Col3,Col4,Col5,LAST_UPDATED,IS_DELETED
val fieldNames = data.schema.fieldNames.dropRight(1) //1
val columns = fieldNames.map(f => col(f))
val windowspec = Window.partitionBy("ID").orderBy("LAST_UPDATED") //2
def compareArrayUdf() = ... //3
val result = data
.withColumn("cur", array(columns: _*)) //4
.withColumn("prev", lag($"cur", 1).over(windowspec)) //5
.withColumn("updated_columns", compareArrayUdf()($"cur", $"prev")) //6
.drop("cur", "prev") //7
.orderBy("LAST_UPDATED")
Remarks:
create a list of all fields to compare. All fields but the last one (LAST-UPDATED) are used
create a window that is partitioned by ID and each partition is sorted by LAST-UPDATED
create a udf that compares two arrays and maps the discovered differences to the field names, code see below
create a new column that contains all values that should be compared
create a new column that contains all values of the previous row (by using the lag-function) that should be compared. The previous row is the row with the same ID and the biggest LAST-UPDATED that is smaller than the current one. This field can be null
compare the two new columns and put the result into updated-columns
drop the two intermediate columns created in step 3 and 4
The compareArraysUdf is
def compareArray(cur: mutable.WrappedArray[String], prev: mutable.WrappedArray[String]): String = {
if (prev == null || cur == null) return ""
val res = new StringBuilder
for (i <- cur.indices) {
if (!cur(i).contentEquals(prev(i))) {
if (res.nonEmpty) res.append(",")
res.append(fieldNames(i))
}
}
res.toString()
}
def compareArrayUdf() = udf[String, mutable.WrappedArray[String], mutable.WrappedArray[String]](compareArray)

Answer 2

You can join your DataFrame or DataSet to itself, joining the rows where the id is the same in both rows and where the version of the left row is i and the version of the right row is i+1. Here's an example
case class T(id: String, version: Int, data: String)
val data = Seq(T("1", 1, "d1-1"), T("1", 2, "d1-2"), T("2", 1, "d2-1"), T("2", 2, "d2-2"), T("2", 3, "d2-3"), T("3", 1, "d3-1"))
data: Seq[T] = List(T(1,1,d1-1), T(1,2,d1-2), T(2,1,d2-1), T(2,2,d2-2), T(2,3,d2-3), T(3,1,d3-1))
val ds = data.toDS
val joined = ds.as("ds1").join(ds.as("ds2"), $"ds1.id" === $"ds2.id" && (($"ds1.version"+1) === $"ds2.version"))
And then you can reference the columns in the new DataFrame/DataSet like $"ds1.data and $"ds2.data etc.
To find the rows where the data changed from one version to another, you can do
joined.filter($"ds1.data" !== $"ds2.data")