pandas aggregation based on timestamp threshold - python-3.x

Question

I hope somebody can help me to solve this issue.
I have a csv file structured as follow:
I am trying to group the events based on message, name, userID if the events manifests in a 10min threshold starting from the first event matched.
the output I am expecting from the csv, is to see only 3 rows, because the second and third (as they are in 10min threshold and the message and name and ID are the same, they should be grouped) and have an extra columns name event_count that report how many time that event occurred.like this
I start working on this and my script looks like this:
import csv
import pandas as pd
# 0. sort data by timestamp if not already sorted
file_csv = 'test.csv'
f = pd.read_csv(file_csv)
f['#timestamp'] = pd.to_datetime(f['#timestamp'])
f = f.sort_values('#timestamp')
# lazy groupby
groups = f.groupby(['message','name','userID'])
# 1. compute the time differences `timediff` and compare to threshold
f['timediff'] = groups['#timestamp'].diff() < pd.Timedelta(minutes=10)
# 2. find the blocks with cumsum
f['event_count'] = groups['timediff'].cumsum()
# 3. groupby the blocks
out = (f.groupby(['message','name', 'userID'])
.agg({'#timestamp':'first', 'timediff':'count'})
)
keep_col = ['#timestamp', 'message', 'name', 'userID', 'event_count']
new_f = f[keep_col]
new_f.to_csv("aggregationtest.csv", index=False)
But the aggregation is totally wrong because is grouping all the event together even if they don't fall in the 10min threshold.
I am really struggling to understand what I am doing wrong if somebody can help me to understand the issue
UPDATE:
After some testing I managed to get a closer output to what I am expecting but still wrong.
I did some updated on the out variable as follow
out = (f.groupby(['message','name', 'userID', 'timediff']).agg({'#timestamp':'first','message': 'unique','name': 'unique', 'userID': 'unique', 'timediff': 'count'}))
This bit of code now produce an output that looks like:
But even if its grouping now, the count is wrong. Having this csv file
#timestamp,message,name,userID
2021-07-13 21:36:18,Failed to download file,Failed to download file,admin
2021-07-14 03:46:16,Successful Logon for user "user1",Logon Attempt,1
2021-07-14 03:51:16,Successful Logon for user "user1",Logon Attempt,1
2021-07-14 03:54:16,Successful Logon for user "user1",Logon Attempt,1
2021-07-14 04:55:16,Successful Logon for user "user1",Logon Attempt,1
I am expecting to have the following event_count
1
3
1
But I am getting different out come.

Answer 1

You'll have to somehow identify the different periods within the groups. The solution below gives each period within the group a name, which can then be included in the groupby that generates the count:
import pandas as pd
file_csv = 'test.csv'
f = pd.read_csv(file_csv)
f['#timestamp'] = pd.to_datetime(f['#timestamp'])
f = f.sort_values('#timestamp')
def check(item): #taken from https://stackoverflow.com/a/53189777/11380795
diffs = item - item.shift()
laps = diffs > pd.Timedelta('10 min')
periods = laps.cumsum().apply(lambda x: 'period_{}'.format(x+1))
return periods
#create period names
f['period'] = f.groupby(['message','name','userID'])['#timestamp'].transform(check)
#groupby and count
(f.groupby(['message','name', 'userID', 'period']).agg({'#timestamp':'first', 'period': 'count'})).rename(columns={"period": "timediff"}).reset_index()
Output:
message
name
userID
period
#timestamp
timediff
0
Failed to download file
Failed to download file
admin
period_1
2021-07-13 21:36:18
1
1
Successful Logon for user "user1"
Logon Attempt
1
period_1
2021-07-14 03:46:16
3
2
Successful Logon for user "user1"
Logon Attempt
1
period_2
2021-07-14 04:55:16
1