How to remove duplicate values in dataframe while preserving the rest of the row in Pandas? - python-3.x

Question

I am working on some gross profit reports in a jupyter notebook. I have exported the data out of our CRM as a csv and am using Pandas to with with the data. Some of the data is being duplicated in a couple of columns. I need to remove those duplicate values in those columns, but preserve the rest of the row.
I have tried to drop_duplicates on a subset of the two columns, but it removes the entire row.
INV INV SUB PO Number PO Subtotal \
0 INV-002504 USD 350.00 PO-03977 240
1 INV-002507 USD 1,400.00 PO-03846 603.56
2 NaN NaN PO-03847 295
3 INV-002489 USD 891.25 PO-03861 658.31
4 INV-002453 USD 3,132.50 PO-03889 4751.19
5 INV-002537 USD 3,856.29 PO-03889 4751.19
6 INV-002420 USD 592.43 PO-03577 1188.46
7 INV-002415 USD 10,779.00 PO-03727 5389.21
Rows 4 & 5 are an example being duplicated in the PO Number & PO Subtotal columns.
I expect the output to remove the duplicate so the value is only shown once in all cases.
INV INV SUB PO Number PO Subtotal \
0 INV-002504 USD 350.00 PO-03977 240
1 INV-002507 USD 1,400.00 PO-03846 603.56
2 NaN NaN PO-03847 295
3 INV-002489 USD 891.25 PO-03861 658.31
4 INV-002453 USD 3,132.50 PO-03889 4751.19
5 INV-002537 USD 3,856.29
6 INV-002420 USD 592.43 PO-03577 1188.46
7 INV-002415 USD 10,779.00 PO-03727 5389.21

Answer 1

Use DataFrame.duplicated to check which rows contain duplicates based on PO Number & PO Subtotal. Then conditionally replace the value by '' with np.where:
m = df.duplicated(['PO Number', 'PO Subtotal'])
df['PO Number'] = np.where(m, '', df['PO Number'])
df['PO Subtotal'] = np.where(m, '', df['PO Subtotal'])
Or using .loc to select the correct rows and columns and replace those rows with '':
m = df.duplicated(['PO Number', 'PO Subtotal'])
df.loc[m, ['PO Number', 'PO Subtotal']] = ''
Output
INV INV SUB PO Number PO Subtotal
0 INV-002504 USD 350.00 PO-03977 240.0
1 INV-002507 USD 1,400.00 PO-03846 603.56
2 NaN NaN PO-03847 295.0
3 INV-002489 USD 891.25 PO-03861 658.31
4 INV-002453 USD 3,132.50 PO-03889 4751.19
5 INV-002537 USD 3,856.29
6 INV-002420 USD 592.43 PO-03577 1188.46
7 INV-002415 USD 10,779.00 PO-03727 5389.21