How do i remove outliers using multiple columns pandas? - python-3.x

Question

Out of my entire dataframe i have two columns price and quantity. These both contain outliers. How can i remove the outliers in both these columns such that the dataframe returned excludes outliers from both these columns? I can apply it to one but not sure how i can apply it to both columns.
I've tried the below
def make_mask(df, column):
standardized = (df[column] - df[column].mean())/df[column].std()
return standardized.abs() >= 2
def filter_outliers(df, columns):
print(columns)
masks = (make_mask(df, column) for column in columns)
print(masks)
full_mask = np.logical_or.reduce(masks)
print(full_mask)
return df[full_mask]
outliersremoved_df=filter_outliers(df,['price','qty'])
I have used this but i can only apply it to one column at a time:
def remove_outlier(df_in, col_name):
q1 = df_in[col_name].quantile(0.25)
q3 = df_in[col_name].quantile(0.75)
iqr = q3-q1 #Interquartile range
fence_low = q1-1.5*iqr
fence_high = q3+1.5*iqr
df_out = df_in.loc[(df_in[col_name] > fence_low) & (df_in[col_name] < fence_high)]
return df_out
error with the top 2 functions:
ValueError: too many values to unpack (expected 1)

Answer 1

Please use the below function which would apply on all the columns you have in #df
def cap_data(df):
for col in df.columns:
print("capping the ",col)
if (((df[col].dtype)=='float64') | ((df[col].dtype)=='int64')):
percentiles = df[col].quantile([0.01,0.99]).values
df[col][df[col] <= percentiles[0]] = percentiles[0]
df[col][df[col] >= percentiles[1]] = percentiles[1]
else:
df[col]=df[col]
return df
final_df=cap_data(df)