Pandas summing rows grouped by another column - pandas-groupby

Question

I have attached dataset
Time podId Batt (avg) Temp (avg)
0 2019-10-07 9999 6.1 71.271053
1 2019-10-08 9999 6.0 71.208285
2 2019-10-09 9999 5.9 77.896628
3 2019-10-10 9999 5.8 78.709279
4 2019-10-11 9999 5.7 71.849283
59 2019-12-05 8888 5.5 76.548780
60 2019-12-06 8888 5.4 73.975295
61 2019-12-07 8888 5.3 76.209434
62 2019-12-08 8888 5.2 76.717481
63 2019-12-09 8888 5.1 70.433920
I imported it using- batt2 = pd.read_csv('battV2.csv')
I need to determine when battery change occurs, i.e. when Batt (avg) increases from previous row. I am able to do this by using the 'diff' in this manner batt2['Vdiff']=batt2['Batt (avg)'].diff(-1)
Now for each podId I need to sum the Vdiff column between battery changes, i.e. between two negative Vdiff values
Also I need to average Temp (avg) over the same range
Count Time to determine the number of days between battery changes
Thanks.

Answer 1

There are a couple of steps involved:
Import data
Be aware that I have changed your dataset a bit to provide a valid test case for your requirements (in your given dataset, Batt_avg never increases).
from io import StringIO
import pandas as pd
data = StringIO('''Time podId Batt_avg Temp_avg
0 2019-10-07 9999 6.1 71.271053
1 2019-10-08 9999 6.0 71.208285
2 2019-10-09 9999 5.9 77.896628
3 2019-10-10 9999 5.8 78.709279
4 2019-10-11 9999 5.7 71.849283
5 2019-10-12 9999 6.0 71.208285
6 2019-10-13 9999 5.9 77.896628
7 2019-10-14 9999 5.8 78.709279
8 2019-10-15 9999 5.7 71.849283
59 2019-12-05 8888 5.5 76.548780
60 2019-12-06 8888 5.4 73.975295
61 2019-12-07 8888 5.3 76.209434
62 2019-12-08 8888 5.2 76.717481
63 2019-12-09 8888 5.1 70.433920''')
df = pd.read_csv(data, delim_whitespace=True)
Determine changes in battery voltage
As you have already found out, you can do this with diff(). I am not certain that the code you have given with df.Batt_avg.diff(-1) satisfies your requirement of: "i.e. when Batt (avg) increases from previous row". Instead, for a given row, this shows how the value will change in the next row (multiplied by -1). If you need the negative change to the previous row, you can instead use -df.Batt_avg.diff().
df['Batt_avg_diff'] = df.Batt_avg.diff(-1)
Group data and apply the aggregation functions
You can express your grouping conditions as df.podId.diff().fillna(0.0) != 0 for the podIds and df.Batt_avg_diff.fillna(0.0) < 0 for the condition "between battery changes, i.e. between two negative Vdiff values" - either of these will trigger a new group. Use cumsum() on the triggers to create the groups. Then you can use groupby() to act on these groups and transform() to expand the results to the dimensions of the original dataframe.
df['group'] = ((df.podId.diff().fillna(0.0) != 0) | (df.Batt_avg_diff.fillna(0.0) < 0)).cumsum()
df['Batt_avg_diff_sum'] = df.Batt_avg_diff.groupby(df.group).transform('sum')
df['Temp_avg_mean'] = df.Temp_avg.groupby(df.group).transform('mean')
Datetime calculations
For the final step, you need to first convert the string to datetime to allow date operations. Then you can use groupby operations to get the max and min in each group, and take the delta.
df.Time = pd.to_datetime(df.Time)
df['Time_days'] = df.Time.groupby(df.group).transform('max') - df.Time.groupby(df.group).transform('min')
Note: if you do not need or want the aggregate data in the original dataframe, just apply the functions directly (without transform):
df_group = pd.DataFrame()
df_group['Batt_avg_diff_sum'] = df.Batt_avg_diff.groupby(df.group).sum()
df_group['Temp_avg_mean'] = df.Temp_avg.groupby(df.group).mean()
df_group['Time_days'] = df.Time.groupby(df.group).max() - df.Time.groupby(df.group).min()