Pandas : How to iterate over only back dates over two ID columns to give unique IDs and count? - python-3.x

Question

I have a dataset which has data for over a month. I need to count the players who have played on the previous day. So if I am looking at date of 5th June users I need to find the number of users who were there in dates preceding 5th June even once.
Dataset is something like this:-
Day pid1 pid2
1 1a 1b
1 1c 2e
1 1d 2w
1 1e 2q
2 1f 4r
2 1g 5t
2 2e 7u
2 2w 8i
2 2q 9o
3 4r 0yu
3 5t 5t
3 6t 1w
4 1a 2e
4 1f 9o
4 7u 6h
5 8i 4f
5 9o 3d
5 0yu 5g
5 5t 6h
I have tried iterating over days then pid1 and pid2 but to no avail and it is computationally expensive as I have over 5 million data points.
I really do not know how to approach this and the only thing I have tried is this:-
for x in range(1, 31):
for i in ids.iterrows():
if i['Ids'] == zip(df4['pid1'], df['pid2']):
print(x, i.count())
But it still doesn't let me iterate over only previous days and not next days.
I need answer that looks something like this (results are not accurate) but I need unique count of users of previous days on a given day:-
Day Previous day users
1 0
2 2
3 2
4 5
5 5

Answer 1

According to me you want to count the number of player Ids which has appeared on all days before the given Day. You can try the below:
m=(df.melt('Day').sort_values('Day').drop_duplicates(['Day','value'])
.reset_index(drop=True).drop('variable',1))
m.assign(k=m.groupby('value').cumcount()).groupby('Day')['k'].sum() #assign it back
Day
1 0
2 3
3 2
4 6
5 7
If cumulative counts are not required instead 1 appearance per cumulative day is required, use:
m.assign(k=m.groupby('value').cumcount().ne(0)).groupby('Day')['k'].sum() #.astype(int)
Day
1 0
2 3
3 2
4 5
5 5

Answer 2

Edit: After op's comment, I am providing both answers:
Solution for checking only one day before:
Instead of using two for loops and one if statement I used more pandas operations to increase computational speed
df.head()
Day pid1 pid2
0 1 1a 1b
1 1 1c 2e
2 1 1d 2w
3 1 1e 2q
4 2 1f 4r
Then groupby Day to get players in list:
tmp = df.groupby("Day").agg(list)
tmp
Day pid1 pid2
1 [1a, 1c, 1d, 1e] [1b, 2e, 2w, 2q]
2 [1f, 1g, 2e, 2w, 2q] [4r, 5t, 7u, 8i, 9o]
3 [4r, 5t, 6t] [0yu, 5t, 1w]
4 [1a, 1f, 7u] [2e, 9o, 6h]
5 [8i, 9o, 0yu, 5t] [4f, 3d, 5g, 6h]
Then concat ith day players and (i-1)th day players:
tmp2 = pd.DataFrame(tmp["pid1"] + tmp["pid2"], columns = ["current_day"])
tmp2["previous_day"] = tmp2.shift()
tmp2 = tmp2.fillna("nan")
tmp2
Day current_day previous_day
1 [1a, 1c, 1d, 1e, 1b, 2e, 2w, 2q] nan
2 [1f, 1g, 2e, 2w, 2q, 4r, 5t, 7u, 8i, 9o] [1a, 1c, 1d, 1e, 1b, 2e, 2w, 2q]
3 [4r, 5t, 6t, 0yu, 5t, 1w] [1f, 1g, 2e, 2w, 2q, 4r, 5t, 7u, 8i, 9o]
4 [1a, 1f, 7u, 2e, 9o, 6h] [4r, 5t, 6t, 0yu, 5t, 1w]
5 [8i, 9o, 0yu, 5t, 4f, 3d, 5g, 6h] [1a, 1f, 7u, 2e, 9o, 6h]
And finally finding length of intersection of days that is number of players that played at the current day and previous day.
tmp2.apply(lambda x: len(list(set(x["current_day"]) & set(x["previous_day"]))), axis = 1)
Day
1 0
2 3
3 2
4 0
5 2
dtype: int64
Solution for checking all previous days:
res = pd.DataFrame()
for day_num in df["Day"].unique():
tmp = df[df["Day"] == day_num]
tmp2 = pd.DataFrame(pd.concat([tmp["pid1"], tmp["pid2"]]).unique(), columns = ["players"])
tmp2["day"] = day_num
res = pd.concat([res, tmp2])
res = res.reset_index(drop = True)
This combines all pid1 and pid2 to players
res.head()
players day
0 1a 1
1 1c 1
2 1d 1
3 1e 1
4 1b 1
Then calculating all previous day players for a current day:
result = []
for day_num in df["Day"].unique():
current_players = pd.Series(res[res["day"] == day_num].players.unique())
previous_players = pd.Series(res[res["day"] < day_num].players.unique())
result.append(len(current_players[current_players.isin(previous_players)]))
result = pd.Series(result, index = df["Day"].unique())
result
The result in pd.Series format:
1 0
2 3
3 2
4 5
5 5
dtype: int64
Hope it works!