How do I conditionally aggregate values in projection part of pandas query? - python-3.x

Question

I currently have a csv file with this content:
ID PRODUCT_ID NAME STOCK SELL_COUNT DELIVERED_BY
1 P1 PRODUCT_P1 12 15 UPS
2 P2 PRODUCT_P2 4 3 DHL
3 P3 PRODUCT_P3 120 22 DHL
4 P1 PRODUCT_P1 423 18 UPS
5 P2 PRODUCT_P2 0 5 GLS
6 P3 PRODUCT_P3 53 10 DHL
7 P4 PRODUCT_P4 22 0 UPS
8 P1 PRODUCT_P1 94 56 GLS
9 P1 PRODUCT_P1 9 24 GLS
When I execute this SQL query:
SELECT
PRODUCT_ID,
MIN(CASE WHEN DELIVERED_BY = 'UPS' THEN STOCK END) as STOCK,
SUM(CASE WHEN ID > 6 THEN SELL_COUNT END) as TOTAL_SELL_COUNT,
SUM(CASE WHEN SELL_COUNT * 100 > 1000 THEN SELL_COUNT END) as COND_SELL_COUNT
FROM products
GROUP BY PRODUCT_ID;
I get the desired result:
PRODUCT_ID STOCK TOTAL_SELL_COUNT COND_SELL_COUNT
P1 12 80 113
P2 null null null
P3 null null 22
P4 22 0 null
Now I'm trying to somehow get the same result on that dataset using pandas, and that's what I'm struggling with.
I imported the csv file to da DataFrame called df_products.
Then I tried this:
def custom_aggregate(grouped):
data = {
'STOCK': np.where(grouped['DELIVERED_BY'] == 'UPS', grouped['STOCK'].min(), np.nan) # [grouped['STOCK'].min() if grouped['DELIVERED_BY'] == 'UPS' else None]
}
d_series = pd.Series(data)
return d_series
result = df_products.groupby('PRODUCT_ID').apply(custom_aggregate)
print(result)
As you can see I'm nowhere near the expected result as I'm already having problems getting the conditional STOCK aggregration to work depending on the DELIVERED_BY values.
This outputs:
STOCK
PRODUCT_ID
P1 [9.0, 9.0, nan, nan]
P2 [nan, nan]
P3 [nan, nan]
P4 [22.0]
which is not even in the correct format, but I'd be happy if I could get the expected 12.0 instead of 9.0 for P1.
Thanks
I just wanted to add that I got near the result by creating additional columns:
df_products['COND_STOCK'] = df_products[df_products['DELIVERED_BY'] == 'UPS']['STOCK']
df_products['SELL_COUNT_ID_GT6'] = df_products[df_products['ID'] > 6]['SELL_COUNT']
df_products['SELL_COUNT_GT1000'] = df_products[(df_products['SELL_COUNT'] * 100) > 1000]['SELL_COUNT']
The function would then look like this:
def custom_aggregate(grouped):
data = {
'STOCK': grouped['COND_STOCK'].min(),
'TOTAL_SELL_COUNT': grouped['SELL_COUNT_ID_GT6'].sum(),
'COND_SELL_COUNT': grouped['SELL_COUNT_GT1000'].sum(),
}
d_series = pd.Series(data)
return d_series
result = df_products.groupby('PRODUCT_ID').apply(custom_aggregate)
This is the 'almost' desired result:
STOCK TOTAL_SELL_COUNT COND_SELL_COUNT
PRODUCT_ID
P1 12.0 80.0 113.0
P2 NaN 0.0 0.0
P3 NaN 0.0 22.0
P4 22.0 0.0 0.0

Answer 1

Usually we can write the pandas as below
df.groupby('PRODUCT_ID').apply(lambda x : pd.Series({'STOCK':x.loc[x.DELIVERED_BY =='UPS','STOCK'].min(),
'TOTAL_SELL_COUNT': x.loc[x.ID>6,'SELL_COUNT'].sum(min_count=1),
'COND_SELL_COUNT':x.loc[x.SELL_COUNT>10,'SELL_COUNT'].sum(min_count=1)}))
Out[105]:
STOCK TOTAL_SELL_COUNT COND_SELL_COUNT
PRODUCT_ID
P1 12.0 80.0 113.0
P2 NaN NaN NaN
P3 NaN NaN 22.0
P4 22.0 0.0 NaN