DataFrameGroupby.agg NamedAgg on same column errors out on custom function, but works on bult-in function - python-3.x

Question

Setup
np.random.seed(0)
df = pd.DataFrame(zip([1, 1, 2, 2, 2, 3, 7, 7, 9, 10],
*np.random.randint(1, 100, 20).reshape(-1,10)),
columns=['A','B', 'C'])
Out[127]:
A B C
0 1 45 71
1 1 48 89
2 2 65 89
3 2 68 13
4 2 68 59
5 3 10 66
6 7 84 40
7 7 22 88
8 9 37 47
9 10 88 89
f = lambda x: x.max()
NamedAgg on built-in function works fine
df.groupby('A').agg(B_min=('B', 'min'), B_max=('B', 'max'), C_max=('C', 'max'))
Out[133]:
B_min B_max C_max
A
1 45 48 89
2 65 68 89
3 10 10 66
7 22 84 88
9 37 37 47
10 88 88 89
NamedAgg on custom function f errors out
df.groupby('A').agg(B_min=('B', 'min'), B_max=('B', f), C_max=('C', 'max'))
KeyError: "[('B', '<lambda>')] not in index"
Is there any explanation for this error? is this error an intentional restriction?

Answer 1

The issue is because of _mangle_lambda_list, which gets called at some point. There seems to be a mismatch where the resulting aggregation gets renamed but the list of output columns, ordered which are then used here, doesn't get changed. Since that function specifically checks for if com.get_callable_name(aggfunc) == "<lambda>" any name other than '<lambda>' will work without issue:
Sample data
import pandas as pd
import numpy as np
np.random.seed(0)
df = pd.DataFrame(zip([1, 1, 2, 2, 2, 3, 7, 7, 9, 10],
*np.random.randint(1, 100, 20).reshape(-1,10)),
columns=['A','B', 'C'])
f = lambda x: x.max()
kwargs = {'B_min': ('B', 'min'), 'B_max':('B', f), 'C_max':('C', 'max')}
Here are the most relevant major steps that get called when you aggregate, and we can see where the KeyError comes from.
func, columns, order = pd.core.groupby.generic._normalize_keyword_aggregation(kwargs)
print(order)
#[('B', 'min'), ('B', '<lambda>'), ('C', 'max')]
func = pd.core.groupby.generic._maybe_mangle_lambdas(func)
df.groupby('A')._aggregate(func)
# B C
# min <lambda_0> max # _0 ruins indexing with ('B', '<lambda>')
#A
#1 45 48 89
#2 65 68 89
#3 10 10 66
#7 22 84 88
#9 37 37 47
#10 88 88 89
Because _mangle_lambda_list is only called when there are multiple aggregations for the same column, you can get away with the '<lambda>' name, so long as it is the only aggregation for that column.
df.groupby('A').agg(A_min=('A', 'min'), B_max=('B', f))
# A_min B_max
#A
#1 1 48
#2 2 68
#3 3 10
#7 7 84
#9 9 37
#10 10 88