EF Core - why Explicit Loading is terribly slow? - domain-driven-design

Question

I am working on microservices in .Net Core with the domain driven design. The infrastructure layer has EF Core DbContext to access the database and in my repositories, I have async methods to retrieve data.
Because Include/ThenInclude does not support filtering (at least not up to Ef Core 2.1), I have tried all the possible approaches I found when googling on how to replace Include. I watched the Pluralsight videos about Ef Core too and when I saw the Explicit Loading option I was really happy due to its ability to filter related objects, but when I rewrote one of the methods into Explicit version, the query that ran for a couple of milliseconds went up to a couple of minutes!
In my entity configurations I set up all the navigations and foreign keys, but I am not sure whether explicit loading requires any additional setup or not? Before recommending to use global filters, please note that the Where clause is usually longer, so the below example is just a shortened version of the actual filtering!
This is how my methods look like (TransferService serves as the aggregate, TransferServiceDetail and any other classes are just entities within the TransferService domain):
public async Task<IEnumerable<TransferService>> GetAllAsync(
TransferServiceFilter transferServiceFilter)
{
int? pageIndex = null;
int? itemsPerPage = null;
IEnumerable<TransferService> filteredList = DBContext.TransferServices.Where(
ts => !ts.IsDeleted); //This one itself is quick.
//This is just our filtering, it does not affect performance.
if (transferServiceFilter != null)
{
pageIndex = transferServiceFilter.PageIndex;
itemsPerPage = transferServiceFilter.ItemsPerPage;
filteredList = filteredList.Where(f =>
(transferServiceFilter.TransferSupplierId == null ||
f.TransferSupplierId == transferServiceFilter.TransferSupplierId) &&
(transferServiceFilter.TransferDestinationId == null ||
f.TransferDestinationId == transferServiceFilter.TransferDestinationId) &&
(transferServiceFilter.TransferSupplierId == null ||
f.TransferSupplierId == transferServiceFilter.TransferSupplierId) &&
(string.IsNullOrEmpty(transferServiceFilter.TransportHubRef) ||
f.NormalizeReference(f.TransportHubRef) ==
f.NormalizeReference(transferServiceFilter.TransportHubRef)));
}
//This is just for paging and again, this is quick.
return await FilterList(filteredList.AsQueryable(), pageIndex, itemsPerPage);
}
public async Task<IEnumerable<TransferService>> GetAllWithServiceDetailsAsync(
TransferServiceFilter transferServiceFilter)
{
IEnumerable<TransferService> returnList = await GetAllAsync(
transferServiceFilter);
//This might be the problem as I need to iterate through my TransferServices
//to be able to load all TransferServiceDetails that belong to each individual
//Service.
foreach (TransferService service in returnList)
{
await DBContext.Entry<TransferService>(service)
.Collection(ts => ts.TransferServiceDetails.Where(
tsd => !tsd.IsDeleted)).LoadAsync();
}
return returnList;
}
In my repository I have other methods as well, similarly referring to a previous GetAllXY... method (TransferServiceDetails have Rates, Rates have Periods, etc...).
My idea was to simply call GetAllAsync when I only need TransferService data (and alone this method is lightning quick), or call GetAllWithServiceDetailsAsync when I also need the Details of the selected Services, etc, but the lower I go in this parent-child hierarchy, the slower the execution becomes and I am talking about minutes, not just a couple of extra milliseconds, or in worst case seconds.
So my question again: is there any additional setting that I might have missed from the entity configurations that explicit loading requires, or simply my queries are incorrect? Or maybe explicit loading is only good when there is only one TransferService as a parent instead of a list of TransferServices (50-100 in my case) and also there are only just a few children related entities (in my case I usually have 5-10 Details, each Detail has 2-3 Rates, each Rate has exactly 1 Period, etc...)?

Answer 1

I guess your filtering can't be converted to SQL Where and all filtering happens client-side (EF loads ALL TransferServices entities into memory, filters in-memory and drops mismatched).
I may check this by enabling detailed (debug) logging - EF will dump SQLs into log.
After you confirm, your should make improvements:
First, put ifs out of Where. Instead of:
filteredList = filteredList.Where(f => transferServiceFilter.TransferSupplierId == null ||
f.TransferSupplierId == transferServiceFilter.TransferSupplierId)
use
if (transferServiceFilter.TransferSupplierId != null)
{
filteredList = filteredList.Where(f => f.TransferSupplierId == transferServiceFilter.TransferSupplierId)
}
Second, you should re-think NormalizeReference. This can't be executed server-side, because SQL server doesn't know about this implementation. You should pre-normalize TransportHubRef, save it in DB (say, NormalizedTransportHubRef) and use Where with simple equality.
(Also, don't forget about indexes).