How to efficiently append running sum in Python? - python-3.x

Question

I'm writing a python script that uses a model to predict a large number of values by groupID, where efficiency is important (N on the order of 10^8). I initialize a results matrix and am trying to sequentially update a running sum of values in the results matrix.
Trying to be efficient, in my current method I use groupID as row numbers of the results matrix to avoid merging (merging is expensive, as far as I understand).
My attempt:
import numpy as np
# Initialize results matrix
results = np.zeros((5,3)) # dimension: number of groups x timestep
# Now I loop over batches, with batch size 4. Here's an example of one iteration:
batch_groupIDs = [3,1,0,1] # Note that multiple values can be generated for same groupID
batch_results = np.ones((4,3))
# My attempt at appending the results (low dimension example):
results[batch_groupIDs] += batch_results
print(results)
This outputs:
[[1. 1. 1.]
[1. 1. 1.]
[0. 0. 0.]
[1. 1. 1.]
[0. 0. 0.]]
My desired output is the following (since group 1 shows up twice, and should be appended twice):
[[1. 1. 1.]
[2. 2. 2.]
[0. 0. 0.]
[1. 1. 1.]
[0. 0. 0.]]
The actual dimensions of my problem are approximately 100 timesteps x a batch size of 1 million+ and 2000 groupIDs

Answer 1

Here is my understanding, and please correct me if I'm wrong:
We want a resultant matrix that has the shape number of groups x timestep which in this case would be 2000 x 100. This matrix needs to be efficiently updated sequentially for a batch size of 10^6.
If the summary is correct, here is my approach. We divide the 10^6 sequences into let's say "mini-batches" of 10^4 each. Another assumption is that the timesteps are weighted equally across all groups. :
Convert batch_groupIDs to a numpy array of frequencies. For batch_groupIDs = [3,1,0,1], the array would look like [1,2,0,1,0].
Convert the timestep values to numpy array. For batch_results = np.ones((4,3)), timestep array would look like [1,1,1] since 3 steps add equal weight
Find the outer product and add it to the result.
Repeat until all sequences are generated.
Here is the equivalent function:
import random
GROUPIDs = 2000
MINIBATCH_SIZE = 10000
TIMESTEPS = 100
def getMatrixOuter():
result = np.zeros((GROUPIDs,TIMESTEPS))
for x in range(100): #100 mini-batches each of size 10000 to replicate a sequence
batch_groupIDs = [random.randrange(GROUPIDs) for i in range(MINIBATCH_SIZE)] #10000 random sequences in range(0,GROUPIDs)
counts = {k: 0 for k in range(GROUPIDs)}
for i in batch_groupIDs:
counts[i] = counts.get(i, 0) + 1
vals = np.fromiter(counts.values(), dtype=int) #array of frequencies of GROUPIDs
steps=np.ones((TIMESTEPS,)) #weight of each timestep
np.add(np.outer(vals, steps), result, out=result)
return result
Here is how it fared when timed:
%timeit getMatrixOuter()
1 loop, best of 3: 704 ms per loop
A possible benchmark could be evaluating it against a sequential add using np.add.at()
Here is a possible benchmark function:
GROUPIDs = 2000
MINIBATCH_SIZE = 10000
TIMESTEPS = 100
def getMatrixnp():
result = np.zeros((GROUPIDs,TIMESTEPS))
for i in range(int(1e6)):
np.add.at(result, [random.randrange(2000) for i in range(1)], 1)
return result
And here is how it fared on the same system:
%timeit getMatrixnp()
1 loop, best of 3: 10.8 s per loop
The solution certainly rests on some assumptions that may very well turn out to be false. Nevertheless, my 2 cents.

Answer 2

Using pandas, you can count the occurrence of each batch_groupID using the groupby method. Once performed, you can simply add this result to your initial matrix using the add() method (you need to ensure the axis is here is set to 0). If you specifically need a numpy.array, then you can just use the .to_numpy() method on the DataFrame.
import pandas as pd
s = pd.Series(batch_groupIDs)
array = np.zeros((5,3))
#create a new series counting the occurrence of each index
group = s.groupby(s).count()
# leverage pandas add each count occurrence
results = pd.DataFrame(array).add(group, 0).fillna(0).to_numpy()
which gives
array([[1., 1., 1.],
[2., 2., 2.],
[0., 0., 0.],
[1., 1., 1.],
[0., 0., 0.]])