Merge pandas dataframes based on several conditions for a large dataset - python-3.x

Question

I want to merge two data frames based on certain conditions. First, I want to match the Full name only then for the mismatch entries, I would like to consider First and Last names as a matching condition. I have two Dataframes as follows:
df1
first_name last_name full_name
John Shoeb John Shoeb
John Shumon John Md Shumon
Abu Babu Abu A Babu
William Curl William Curl
df2
givenName surName displayName
John Shoeb John Shoeb
John Shumon John M Shumon
Abu Babu Abu Babu
Raju Kaju Raju Kaju
Bill Curl Bill Curl
I first merge them based on full name:
df3 = pd.merge(df1, df2, left_on=df1['full_name'].str.lower(), right_on=df2['displayName'].str.lower(), how='left')
And add a status and log columns:
df3.loc[ (df3.full_name.str.lower()==df3.displayName.str.lower()), 'status'] = True
df3.loc[ (df3.full_name.str.lower()==df3.displayName.str.lower()), 'log'] = 'Full Name Matching'
So the resultant dataframe df3 now looks like:
first_name last_name full_name givenName surName displayName status log
John Shoeb John Shoeb John Shoeb John Shoeb True Full Name Matching
John Shumon John Md Shumon NaN NaN NaN NaN NaN
Abu Babu Abu A Babu NaN NaN NaN NaN NaN
William Curl William Curl NaN NaN NaN False NaN
Expected Results
Now I want to apply matching condition based on df1 (First Name and Last Name) and df2 (givenName and surName). The final dataframe should look like as follows:
first_name last_name full_name givenName surName displayName status log
John Shoeb John Shoeb John Shoeb John Shoeb True Full Name Matching
John Shumon John Md Shumon John Shumon John Shumon True FN LN Matching
Abu Babu Abu A Babu Abu Babu Abu Babu True FN LN Matching
William Curl William Curl NaN NaN NaN False NaN
Question For the second part i.e. First Name and Last Name matching, I was able to get it done using the dataframe's itertuples(). However, when the same operations are applied to a huge dataset it keeps running forever. I'm looking for efficient ways so it can be applied to a big chunk of data.

Answer 1

You can use indicator=True in your merges. Then compare if first merge and second merge were "both" (for example with np.where):
df3 = (
pd.merge(
df1,
df2,
left_on=df1["full_name"].str.lower(),
right_on=df2["displayName"].str.lower(),
how="left",
indicator=True,
)
.drop(columns="key_0")
.rename(columns={"_merge": "first_merge"})
)
df3 = pd.merge(
df3,
df2,
left_on=df1["first_name"].str.lower() + " " + df1["last_name"].str.lower(),
right_on=df2["givenName"].str.lower() + " " + df2["surName"].str.lower(),
how="left",
indicator=True,
)
df3["log"] = np.where(
(df3["first_merge"] == "both"),
"Full Name Matching",
np.where(df3["_merge"] == "both", "FN LN Matching", None),
)
df3["status"] = df3["log"].notna()
df3 = df3[
[
"first_name",
"last_name",
"full_name",
"givenName_y",
"surName_y",
"displayName_y",
"status",
"log",
]
].rename(
columns={
"givenName_y": "givenName",
"surName_y": "surName",
"displayName_y": "displayName",
}
)
print(df3)
Prints:
first_name last_name full_name givenName surName displayName status log
0 John Shoeb John Shoeb John Shoeb John Shoeb True Full Name Matching
1 John Shumon John Md Shumon John Shumon John M Shumon True FN LN Matching
2 Abu Babu Abu A Babu Abu Babu Abu Babu True FN LN Matching
3 William Curl William Curl NaN NaN NaN False None