Change values in pandas dataframe based on values in certain columns - python-3.x

Question

How can I convert this first dataframe to the one below it? Based on different scenarios of the first three columns matching, I want to change the values in the rest of the columns.
import pandas as pd
df = pd.DataFrame([['foo', 'foo', 'bar', 'a', 'b', 'c', 'd'], ['bar', 'foo', 'bar', 'a', 'b', 'c', 'd'],
['spa', 'foo', 'bar', 'a', 'b', 'c', 'd']], columns=['col1', 'col2', 'col3', 's1', 's2', 's3', 's4'])
col1 col2 col3 s1 s2 s3 s4
0 foo foo bar a b c d
1 bar foo bar a b c d
2 spa foo bar a b c d
If col1 = col2, I want to change all a's to 2, all b's and c's to 1, and all d's to 0. This is row 1 in my example df.
If col1 = col3, I want to change all a's to 0, all b's and c's to 1, and all d's to 2. This is row 2 in my example df.
If col1 != col2/col3, I want to delete the row and add 1 to a counter so I have a total of deleted rows. This is row 3 in my example df.
So my final dataframe would look like this, with counter = 1:
df = pd.DataFrame([['foo', 'foo', 'bar', '2', '1', '1', '0'], ['bar', 'foo', 'bar', '0', '1', '1', '2']],
columns=['col1', 'col2', 'col3', 's1', 's2', 's3', 's4'])
col1 col2 col3 s1 s2 s3 s4
0 foo foo bar 2 1 1 0
1 bar foo bar 0 1 1 2
I was reading that using df.iterrows is slow and so there must be a way to do this on the whole df at once, but my original idea was:
for row in df.iterrows:
if (row["col1"] == row["col2"]):
df.replace(to_replace=['a'], value='2', inplace=True)
df.replace(to_replace=['b', 'c'], value='1', inplace=True)
df.replace(to_replace=['d'], value='0', inplace=True)
elif (row["col1"] == row["col3"]):
df.replace(to_replace=['a'], value='0', inplace=True)
df.replace(to_replace=['b', 'c'], value='1', inplace=True)
df.replace(to_replace=['d'], value='2', inplace=True)
else:
(delete row, add 1 to counter)
The original df is massive, so speed is important to me. I'm hoping it's possible to do the conversions on the whole dataframe without iterrows. Even if it's not possible, I could use help getting the syntax right for the iterrows.

Answer 1

You can remove rows by boolean indexing first:
m1 = df["col1"] == df["col2"]
m2 = df["col1"] == df["col3"]
m = m1 | m2
Get number of removed rows by sum of chained condition m1 and m2 with inverting by ~:
counter = (~m).sum()
print (counter)
1
df = df[m].copy()
print (df)
col1 col2 col3 s1 s2 s3 s4
0 foo foo bar a b c d
1 bar foo bar a b c d
and then replace with dictionary by condition:
d1 = {'a':2,'b':1,'c':1,'d':0}
d2 = {'a':0,'b':1,'c':1,'d':2}
m1 = df["col1"] == df["col2"]
#replace all columns without col1-col3
cols = df.columns.difference(['col1','col2','col3'])
df.loc[m1, cols] = df.loc[m1, cols].replace(d1)
df.loc[~m1, cols] = df.loc[~m1, cols].replace(d2)
print (df)
col1 col2 col3 s1 s2 s3 s4
0 foo foo bar 2 1 1 0
1 bar foo bar 0 1 1 2
Timings:
In [138]: %timeit (jez(df))
872 ms ± 6.94 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
In [139]: %timeit (hb(df))
1.33 s ± 9.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
Setup:
np.random.seed(456)
a = ['foo','bar', 'spa']
b = list('abcd')
N = 100000
df1 = pd.DataFrame(np.random.choice(a, size=(N, 3))).rename(columns=lambda x: 'col{}'.format(x+1))
df2 = pd.DataFrame(np.random.choice(b, size=(N, 20))).rename(columns=lambda x: 's{}'.format(x+1))
df = df1.join(df2)
#print (df.head())
def jez(df):
m1 = df["col1"] == df["col2"]
m2 = df["col1"] == df["col3"]
m = m1 | m2
counter = (~m).sum()
df = df[m].copy()
d1 = {'a':2,'b':1,'c':1,'d':0}
d2 = {'a':0,'b':1,'c':1,'d':2}
m1 = df["col1"] == df["col2"]
cols = df.columns.difference(['col1','col2','col3'])
df.loc[m1, cols] = df.loc[m1, cols].replace(d1)
df.loc[~m1, cols] = df.loc[~m1, cols].replace(d2)
return df
def hb(df):
counter = 0
df[df.col1 == df.col2] = df[df.col1 == df.col2].replace(['a', 'b', 'c', 'd'], [2,1,1,0])
df[df.col1 == df.col3] = df[df.col1 == df.col3].replace(['a', 'b', 'c', 'd'], [0,1,1,2])
index_drop =df[((df.col1 != df.col3) & (df.col1 != df.col2))].index
counter = counter + len(index_drop)
df = df.drop(index_drop)
return df

Answer 2

You can use:
import pandas as pd
df = pd.DataFrame([['foo', 'foo', 'bar', 'a', 'b', 'c', 'd'], ['bar', 'foo', 'bar', 'a', 'b', 'c', 'd'],
['spa', 'foo', 'bar', 'a', 'b', 'c', 'd']], columns=['col1', 'col2', 'col3', 's1', 's2', 's3', 's4'])
counter = 0
#
df[df.col1 == df.col2] = df[df.col1 == df.col2].replace(['a', 'b', 'c', 'd'], [2,1,1,0])
df[df.col1 == df.col3] = df[df.col1 == df.col3].replace(['a', 'b', 'c', 'd'], [0,1,1,2])
index_drop =df[((df.col1 != df.col3) & (df.col1 != df.col2))].index
counter = counter + len(index_drop)
df = df.drop(index_drop)
print(df)
print(counter)
Output:
col1 col2 col3 s1 s2 s3 s4
0 foo foo bar 2 1 1 0
1 bar foo bar 0 1 1 2
1 # counter