Generate a report of mismatch Columns between 2 Pyspark dataframes - apache-spark

Question

Team, we have a requirement to generate a report of mismatched columns based on key field between 2 Pyspark dataframes of exactly same structure.
Here is first dataframe-
>>> df.show()
+--------+----+----+----+----+----+----+----+----+
| key|col1|col2|col3|col4|col5|col6|col7|col8|
+--------+----+----+----+----+----+----+----+----+
| abcd| 123| xyz| a| ab| abc| def| qew| uvw|
| abcd1| 123| xyz| a| ab| abc| def| qew| uvw|
| abcd12| 123| xyz| a| ab| abc| def| qew| uvw|
| abcd123| 123| xyz| a| ab| abc| def| qew| uvw|
|abcd1234| 123| xyz| a| ab| abc| def| qew| uvw|
+--------+----+----+----+----+----+----+----+----+
And here is 2nd dataframe-
>>> df1.show()
+--------+----+----+----+----+----+----+----+----+
| key|col1|col2|col3|col4|col5|col6|col7|col8|
+--------+----+----+----+----+----+----+----+----+
| abcd| 123| xyz| a| ab| abc| def| qew| uvw|
| abcdx| 123| xyz| a| ab| abc| def| qew| uvw|
| abcd12| 123| xyz| a| abx| abc|defg| qew| uvw|
| abcd123| 123| xyz| a| ab| abc|defg| qew| uvw|
|abcd1234| 123| xyz| a| ab|abcd|defg| qew| uvw|
+--------+----+----+----+----+----+----+----+----+
Full Outer Join gives me this-
>>> dfFull=df.join(df1,'key','outer')
>>> dfFull.show()
+--------+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+
| key|col1|col2|col3|col4|col5|col6|col7|col8|col1|col2|col3|col4|col5|col6|col7|col8|
+--------+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+
| abcd12| 123| xyz| a| ab| abc| def| qew| uvw| 123| xyz| a| abx| abc|defg| qew| uvw|
| abcd1| 123| xyz| a| ab| abc| def| qew| uvw|null|null|null|null|null|null|null|null|
|abcd1234| 123| xyz| a| ab| abc| def| qew| uvw| 123| xyz| a| ab|abcd|defg| qew| uvw|
| abcd123| 123| xyz| a| ab| abc| def| qew| uvw| 123| xyz| a| ab| abc|defg| qew| uvw|
| abcdx|null|null|null|null|null|null|null|null| 123| xyz| a| ab| abc| def| qew| uvw|
| abcd| 123| xyz| a| ab| abc| def| qew| uvw| 123| xyz| a| ab| abc| def| qew| uvw|
+--------+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+----+
if i just look at col6, there are 5 values which mismatched for the "key" field (only value match is for last record).
>>> dfFull.select('key',df['col6'],df1['col6']).show()
+--------+----+----+
| key|col6|col6|
+--------+----+----+
| abcd12| def|defg|
| abcd1| def|null|
|abcd1234| def|defg|
| abcd123| def|defg|
| abcdx|null| def|
| abcd| def| def|
+--------+----+----+
I need to generate a report of something like this for all the columns. The mismatch sample can be any record's value from dataframes.
colName,NumofMismatch,mismatchSampleFromDf,misMatchSamplefromDf1
col6,5,def,defg
col7,2,null,qew
col8,2,null,uvw
col5,3,null,abc
It is a column wise summary based on key, saying how many values are mismatch between 2 dataframes.
Sid

Answer 1

Assume the two dataframes are df1 and df2, you can try the following:
from pyspark.sql.functions import when, array, count, first
# list of columns to be compared
cols = df1.columns[1:]
df_new = (df1.join(df2, "key", "outer")
.select([ when(~df1[c].eqNullSafe(df2[c]), array(df1[c], df2[c])).alias(c) for c in cols ])
.selectExpr('stack({},{}) as (colName, mismatch)'.format(len(cols), ','.join('"{0}",`{0}`'.format(c) for c in cols)))
.filter('mismatch is not NULL'))
df_new.show(10)
+-------+-----------+
|colName| mismatch|
+-------+-----------+
| col4| [ab, abx]|
| col6|[def, defg]|
| col6|[def, defg]|
| col5|[abc, abcd]|
| col6|[def, defg]|
| col1| [, 123]|
| col2| [, xyz]|
| col3| [, a]|
| col4| [, ab]|
| col5| [, abc]|
+-------+-----------+
Notes: (1) the condition ~df1[c].eqNullSafe(df2[c]) used to find the mismatches satisfies either of the following:
+ df1[c] != df2[c]
+ df1[c] is NULL or df2[c] is NULL but not both
(2) The mismatches if exist are saved as ArrayType column with the first item from df1 and 2nd item from df2. NULL is returned if no mismatch and later filtered out.
(3) the stack() function dynamically generated by Python format functions is as follows:
stack(8,"col1",`col1`,"col2",`col2`,"col3",`col3`,"col4",`col4`,"col5",`col5`,"col6",`col6`,"col7",`col7`,"col8",`col8`) as (colName, mismatch)
After we have df_new, then we can do the groupby + aggregation:
df_new.groupby('colName') \
.agg(count('mismatch').alias('NumOfMismatch'), first('mismatch').alias('mismatch')) \
.selectExpr('colName', 'NumOfMismatch', 'mismatch[0] as misMatchFromdf1', 'mismatch[1] as misMatchFromdf2')
.show()
+-------+-------------+---------------+---------------+
|colName|NumOfMismatch|misMatchFromdf1|misMatchFromdf2|
+-------+-------------+---------------+---------------+
| col8| 2| null| uvw|
| col3| 2| null| a|
| col4| 3| ab| abx|
| col1| 2| null| 123|
| col6| 5| def| defg|
| col5| 3| abc| abcd|
| col2| 2| null| xyz|
| col7| 2| null| qew|
+-------+-------------+---------------+---------------+