Task.WhenAll inside For Loop - multithreading

Question

I'm using CosmosDB along with Azure Functions. Here I've one long-running activity, processing Store's transactions. There are around 3000 stores and each store is having 27 million transactional records.
I'm reading the data store-by-store, performing some arithmetic operations over data, and saving the calculated data (result) in another Cosmos container.
Here's my code to :
var distinctStores = _storeContainer
.Where(d => d.CreatedDate >= _yesterday.Date && d.CreatedDate < _today.Date)
.DistinctBy(x => x.LocationId)
.Select(s => s.LocationId);
try
{
foreach (var store in distinctStores) // looping 3000 Stores here
{
var data = store.Transactions // It has around 27 millions data collection.
// Here I'm doing all Calculations over data.
// Getting the results from the calculation
var result = getCalculatedData(data);
// Save the result to the container.
var Save(result)
}
}
private bool Save(List<MyModel> list)
{
_storeDataRepo.AddBulkAsync(list);
}
Here's my Repository Service to store the data in Cosmos DB Container.
public async Task AddBulkAsync(List<TEntity> documents)
{
List<Task> concurrentTasks = new();
foreach (var item in documents)
{
concurrentTasks.Add(_container.CreateItemAsync<TEntity>(item);
}
await Task.WhenAll(concurrentTasks);
}
I bit worried about calling _storeDataRepo.AddBulkAsync(list) inside the loop of stores. Which in turn executes await Task.WhenAll(concurrentTasks) for every loop.
Is it a better solution? please advise and help me with better approaches.

Answer 1

private bool Save(List<MyModel> list)
{
_storeDataRepo.AddBulkAsync(list);
}
AddBulkAsync is an async method, you should not do this. This is firing the Async task in the background (Fire and Forget pattern) and not waiting for it.
private Task SaveAsync(List<MyModel> list)
{
return _storeDataRepo.AddBulkAsync(list);
}
or:
private async Task SaveAsync(List<MyModel> list)
{
await _storeDataRepo.AddBulkAsync(list);
}
And called as:
foreach (var store in distinctStores) // looping 3000 Stores here
{
var data = store.Transactions // It has around 27 millions data collection.
// Here I'm doing all Calculations over data.
// Getting the results from the calculation
var result = getCalculatedData(data);
// Save the result to the container.
await SaveAsync(result)
}
Because the Bulk write does not really care if the operations are for the same store, you could just have 1 SaveAsync call with all the data from all stores:
List<TEntity> documents = new List<TEntity>();
foreach (var store in distinctStores) // looping 3000 Stores here
{
var data = store.Transactions // It has around 27 millions data collection.
// Here I'm doing all Calculations over data.
// Getting the results from the calculation
var result = getCalculatedData(data);
// Add to the documents mean to be saved
documents.AddRange(result);
}
await SaveAsync(documents);