mongodb caching with aggregation pipeline - pagination

Question

mongodb aggregate pipeline like
db.testing.aggregate(
{
$match : {hosting : "aws.amazon.com"}
},
{
$group : { _id : "$hosting", total : { $sum : 1 } }
},
{
$project : { title : 1 , author : 1, <few other transformations> }
{$sort : {total : -1}}
);
Now I want to enable paging. I have 2 options.
Use skip and limit in the pipeline.
{ $skip : pageNumber * pageSize }
{ $limit : pageSize }
External API level caching for each page can be used which will reduce time for repeated loading of same pages, but the first loading of each page will be painful because of the linear scan due to sorting.
Handle pagination in application.
Cache the findAll result i.e for List findAll();
Now pagination will be handled at the service layer and result will be published
From next request onward you will be referring to the cached result and send the desired set of records from the cache.
Question: 2nd approach seems better if database is not doing some magical optimizations. In 1st, my view is that since the pipeline involves sorting, hence every page request will do a scan of the full table, which will be sub-optimal. What are your views? Which one should be done? What would you choose? What is the good practice(Is moving some db logic to service layer for optimizations advisable)?

Answer 1

It depends on your data.
MongoDB does not cache the query results in order to return the cached results for identical queries. https://docs.mongodb.com/manual/faq/fundamentals/#does-mongodb-handle-caching
However, you may create View (from source + pipeline) and update it on-demand. This will allow you to have aggregated data with good performance for paging and update the content periodically. You may create indexes for better performance (No need to develop in service layer extra logic)
Also, if you always filter and $group by hosting field, you may benefit MongoDB index swapping last $sort next ot $match stage. In this case, MongoDB will use index for filter + sort and paging are done in memory.
db.testing.createIndex({hosting:-1})
db.collection.aggregate([
{
$match: {
hosting: "aws.amazon.com"
}
},
{
$sort: {
hosting: -1
}
},
{
$group: {
_id: "$hosting",
title: {
$first: "$title"
},
author: {
$first: "$author"
},
total: {
$sum: 1
}
}
},
{
$project: {
title: 1,
author: 1,
total: 1
}
},
{ $skip : pageNumber * pageSize },
{ $limit : pageSize }
])