Compare two DataFrames and check for changes - apache-spark

Question

I have 2 similar Spark Dataframes df1 and df2 that I want to compare for changes:
df1 and df2 share the same columns
df2 can have more rows than df1 but any additional rows in df2 which is not in df1 can be ignored when comparing
Comparaison key columns are PROGRAM_NAME and ACTION
df1 = spark.createDataFrame([
["PROG1","ACTION1","10","NEW"],
["PROG2","ACTION2","12","NEW"],
["PROG3","ACTION1","14","NEW"],
["PROG4","ACTION4","16","NEW"]
],["PROGRAM_NAME", "ACTION", "VALUE1", "STATUS"])
df2 = spark.createDataFrame([
["PROG1","ACTION1","11","IN PROGRESS"],
["PROG2","ACTION2","12","NEW"],
["PROG3","ACTION1","20","FINISHED"],
["PROG4","ACTION4","14","IN PROGRESS"],
["PROG5","ACTION1","20","NEW"]
],["PROGRAM_NAME", "ACTION", "VALUE1", "STATUS"])
Showing below in order df1, df2 and the expected result I want after comparing the 2 DataFrames.

Answer 1

Similar questions have been asked multiple times here in SO.
Use a simple join to get rows that are in df1 and df2 and filter on those that have different values for the 2 other columns:
from pyspark.sql.functions import col
df_final = df2.alias("new").join(
df1.alias("old"),
(col("new.PROGRAM_NAME") == col("old.PROGRAM_NAME")) & (col("new.ACTION") == col("old.ACTION"))
).filter(
(col("new.VALUE1") != col("old.VALUE1")) | (col("new.STATUS") != col("old.STATUS"))
).select("new.*")
df_final.show()
#+------------+-------+------+-----------+
#|PROGRAM_NAME| ACTION|VALUE1| STATUS|
#+------------+-------+------+-----------+
#| PROG3|ACTION1| 20| FINISHED|
#| PROG4|ACTION4| 14|IN PROGRESS|
#| PROG1|ACTION1| 11|IN PROGRESS|
#+------------+-------+------+-----------+
You can also add the filter condition directly to the join condition

Answer 2

You can achieve the result like this:
import pandas as pd
dict1 = {"PROGRAM_NAME":["PROG1","PROG2","PROG3","PROG4"],
"ACTION":["ACTION1","ACTION2","ACTION1","ACTION4"],
"Value1":[10,12,14,16],
"Status":["NEW","NEW","NEW","NEW"]}
dict2 = {"PROGRAM_NAME":["PROG1","PROG2","PROG3","PROG4","PROG5"],
"ACTION":["ACTION1","ACTION2","ACTION1","ACTION4","ACTION1"],
"Value1":[11,12,20,14,20],
"Status":["IN PROGRES","NEW","FINISHED","IN PROGRES","NEW"]}
DF1 = pd.DataFrame(dict1)
DF2 = pd.DataFrame(dict2)
DF3 = DF2.copy()
DF3 = DF3[DF3["PROGRAM_NAME"].isin(DF1["PROGRAM_NAME"])]
Output:

Answer 3

You can merge df1 and df2 and retain VALUE1 and STATUS just of df2
df1
df2
Keeping suffixes of columns of df1 as _x and that of df2 as blank and then retaining just the columns of df2
df1.merge(df2, on=['PROGRAM_NAME', 'ACTION'], suffixes=('_x', ''))[df2.columns]

Answer 4

Here's the solution in Spark:
import pyspark.sql.types as T
import pyspark.sql.functions as F
df1 = spark.createDataFrame(
[
('PROG1', 'ACTION1', 10, 'NEW'),
('PROG2', 'ACTION2', 12, 'NEW'),
('PROG3', 'ACTION1', 14, 'NEW'),
('PROG4', 'ACTION4', 16, 'NEW'),
],
['PROGRAM_NAME', 'ACTION', 'Value1', 'Status']
)
df2 = spark.createDataFrame(
[
('PROG1', 'ACTION1', 11, 'IN PROGRESS'),
('PROG2', 'ACTION2', 12, 'NEW'),
('PROG3', 'ACTION1', 20, 'FINISHED'),
('PROG4', 'ACTION4', 14, 'IN PROGRESS'),
('PROG5', 'ACTION1', 20, 'NEW'),
],
['PROGRAM_NAME', 'ACTION', 'Value1', 'Status']
)
df1 = df1.alias('df1')
df2 = df2.alias('df2')
df = df1.join(df2, on=['PROGRAM_NAME', 'ACTION'], how='inner')
df = df.filter(F.col('df1.Status') != F.col('df2.Status'))
df.select(
F.col('PROGRAM_NAME'),
F.col('ACTION'),
*[F.col(f'df2.{col}') for col in df2.columns[2:]]
)