TYPO3: Performance issue with pagination - pagination

Question

I am currently building some kind of video channel based on an extension I created. It consists of videos and playlist that contains videos (obviously).
I have to create a page which contains a list of videos AND playlist by category. You also can sort those items by date. Finally, the page is paginated with an infinite scrolling that should load items 21 by 21.
To do so, I created on both Video and Playlist repositories a "findByCategory" function which is really simple :
$query = $this->createQuery();
return $query->matching($query->equals('categorie.uid',$categoryUid))->execute()->toArray();
Once I requested the items I need, I merge them in one array and do my sorting stuff. Here is my controller show action :
if ($this->request->hasArgument('sort'))
$sort = $this->request->getArgument('sort');
else
$sort = 'antechrono';
//Get videos in repositories
$videos = $this->videoRepository->findByCategorie($categorie->getUid());
$playlists = $this->playlistRepository->findByCategorie($categorie->getUid());
//Merging arrays then sort it
if ($videos && $playlists)
$result = array_merge($videos, $playlists);
else if ($videos)
$result = $videos;
else if ($playlists)
$result = $playlists;
if ($sort == "chrono")
usort($result, array($this, "sortChrono"));
else if ($sort == "antechrono" || $sort == null)
{
usort($result, array($this, "sortAnteChrono"));
$sort="antechrono";
}
$this->view->assignMultiple(array('categorie' => $categorie, 'list' => $result, 'sort' => $sort));
Here is my view :
<f:widget.paginate objects="{list}" as="paginatedList" configuration="{addQueryString: 'true', addQueryStringMethod: 'GET,POST', itemsPerPage: 21}">
<div class="videos row">
<f:for each="{paginatedList}" as="element">
<f:render partial="Show/ItemCat" arguments="{item: element}"/>
</f:for>
</div>
</f:widget.paginate>
The partial render shows stuff including a picture used as a cover. So I need at least this relation in the view.
This works fine and shows only the items from the category that is requested. Unfortunatly I have a huge performance issue : I tried to show a category that contains more than 3000 records and It takes about one minute to load. It's a little bit long.
By f:debugging my list variable, I see that it contains every records even through it shouldn't be the case (that's the point of pagination...). So the first question is : is there something wrong in the way I did my pagination ?
I tried to simplify my requests by enabling the rawQuery thing ($query->execute(true)) : I get way better performance, but I can't get the link for the pictures (in my view, I get 1 or 0 but not the picture's uid...). Second question : is there a way to fix this issue ?
I hope my description is clear enough. Thanks for your help :-)

Answer 1

When you execute a query, it will not actually fetch the data from the database until the results are accessed. If the paginate widget gets a query result it will add limits and offset to the query and then fetch the data from the database, so you will only get the records that are shown on a page.
In your case you added toArray() after execute(), which accesses the results, so the data is fetched from the database and you get all records. The best solution I can think of is to combine the 2 tables into 1 so you can do it with a single query and don't have to merge and order them in PHP.

Answer 2

As long as you sort the data after the query you have to handle all data (request all records and especially resolve all relations).
Try to sort the data in the query itself (order by), so you could restrict the data to only those records which are needed for the current 'page' (limit <offset>,<number>).
Here a complex query with join and limit could be faster than a full query and filtering in PHP.