Server side pagination with limited number of requests - node.js

Question

My front-end is Angular5. Back-end is REST API NODE.JS and the database is POSTGRES.
Table named organization in my postgres contain 100k ( 100 000) rows.
I have table that contains list of all organizations with information such as name of the organization,location etc.
At this moment my frontend only displays 50 organization. I have console logged this and my backend is only sending array of 50.
My goal is to display all organizations( 100 000 of them) in the frontend table using server-side pagination.
Pagination is already made for the current values that are displayed and thats 50 of them. Code is below
Frontend table below that works ( currently displaying only 50 organization sent from backend)
<div class="card-body">
<div class="row">
<div class="table-col">
<div class="card table">
<div class="row header">
<div class="col col-md-6">Name</div>
<div class="col col-md-2">ID</div>
<div class="col col-md-2">Location</div>
<div class="col col-md-2">Person in charge</div>
</div>
<div *ngFor="let group of organizations">
<div *ngIf="group.name == 'all'">
<div class="row content-row" *ngFor="let organization of group.array | search: term | orderBy: order | paginate: {itemsPerPage: 13, currentPage:page1, id: '1'}"
[routerLink]="['/organization/edit', organization.id]">
<div class="col col-md-6">{{organization.name}}</div>
<div class="col col-md-2">{{organization.id}}</div>
<div class="col col-md-2" *ngIf="organization.place?.fullName != null">{{organization.place?.fullName}}</div>
<div class="col col-md-2" *ngIf="organization.place?.fullName == null"><span class="badge badge-danger">-- N/A -- </span></div>
<div *ngIf="organization.person?.fullname != null" class="col col-md-2">{{organization.person?.fullname}}
</div>
<div *ngIf="organization.person?.fullname == null" class="col col-md-2"><span class="badge badge-danger">-- N/A -- </span></div>
</div>
</div>
<div *ngIf="group.name != 'all'">
<div class="row content-row">
<div class="col col-md-12 group-name" (click)="toggleGroup(group.name)">{{group.name + ':'}}</div>
</div>
<div *ngIf="showGroup == group.name">
<div class="row content-row" *ngFor="let organization of group.array | search: term | paginate: {itemsPerPage: 5, currentPage:page2, id: '2'}"
[routerLink]="['/organization/edit', organization.id]">
<div class="col col-md-6" style="padding-left:23px;">{{organization.name}}</div>
<div class="col col-md-2">{{organization.id}}</div>
<div class="col col-md-2">{{organization.place?.fullName}}</div>
<div class="col col-md-2">{{organization.person?.fullname}}
</div>
</div>
</div>
</div>
</div>
<div *ngIf="!(organizations[0].array.length > 0)" class="row noResults">
<label>No results..</label>
</div>
<div *ngIf="organizations[0].name == 'all'" class="paggination-row">
<div class="paggination">
<pagination-controls (pageChange)="page1 = $event" id="1" maxSize="5" directionLinks="true" autoHide="true" previousLabel="Previous"
nextLabel="Next">
</pagination-controls>
</div>
</div>
<div *ngIf="organizations[0].name != 'all' && showGroup != ''" class="paggination-row">
<div class="paggination">
<pagination-controls (pageChange)="page2 = $event" id="2" maxSize="5" directionLinks="true" autoHide="true" previousLabel="Previous"
nextLabel="Next">
</pagination-controls>
</div>
</div>
</div>
</div>
</div>
</div>
Now on backend in my config I have the following code:
"host": "0.0.0.0",
"port": 3030,
"public": "../public/",
"paginate": {
"default": 50,
"max": 50
},
Note: host ip changed on purpose.
Paginate is the reason why my front is only receiving 50 values when there is 100 000 of them.
Now if I go on about and change default and max value to 100k my frontend should show all organizations but will probably crash the client and the server in process of doing this.
If i do paginate false that won't be much of a help either.
Now I am wondering how can I keep having limited requests like 50 or 100(so my server is alive) but still show all organizations in my front-end using server side pagination or whatever necessary.
Any help is appreciated and thank you so much for taking your time to read this.
edit1: service to load all organizations:
import { Injectable } from '#angular/core';
import { Http, Headers } from "#angular/http";
import 'rxjs/add/operator/map';
import { AppSettings } from '../../../../../../app.settings';
import { BehaviorSubject } from 'rxjs/BehaviorSubject';
//service made to comunicate with backend
#Injectable()
export class CreateOrganizacijeService {
private dataSource = new BehaviorSubject<any>("default")
currentData = this.dataSource.asObservable();
constructor(
private http: Http
) { }
updateData(data) {
this.dataSource.next(data);
}
getAll() {
let headers = new Headers();
let token = localStorage.getItem('feathers-jwt');
headers.append('Content-Type', 'application/json');
headers.append('Authorization', 'Bearer ' + token)
return this.http.get(AppSettings.API_ENDPOINT + '/organization', { headers: headers }).map(res => res.json())
}
getOne(id) {
let headers = new Headers();
let token = localStorage.getItem('feathers-jwt');
headers.append('Content-Type', 'application/json');
headers.append('Authorization', 'Bearer ' + token)
return this.http.get(AppSettings.API_ENDPOINT + '/organization?id=' + id, { headers: headers }).map(res => res.json())
}
createNew(organization) {
let headers = new Headers();
let token = localStorage.getItem('feathers-jwt');
headers.append('Content-Type', 'application/json');
headers.append('Authorization', 'Bearer ' + token)
return this.http.post(AppSettings.API_ENDPOINT + '/organization', organization, { headers: headers }).map(res => res.json())
}
delete(id) {
let headers = new Headers();
let token = localStorage.getItem('feathers-jwt');
headers.append('Content-Type', 'application/json');
headers.append('Authorization', 'Bearer ' + token)
return this.http.delete(AppSettings.API_ENDPOINT + '/organization?id=' + id, { headers: headers }).map(res => res.json())
}
edit(id, organization) {
let headers = new Headers();
let token = localStorage.getItem('feathers-jwt');
headers.append('Content-Type', 'application/json');
headers.append('Authorization', 'Bearer ' + token)
return this.http.patch(AppSettings.API_ENDPOINT + '/organization?id=' + id, organization, { headers: headers }).map(res => res.json())
}
}
component ngonit ( where I load organizations)
ngOnInit() {
this.currentUser = JSON.parse(localStorage.getItem('user'));
if(this.currentUser.roleId == '2') {
this.isAdmin = true; //check if user is admin
}
if(JSON.parse(localStorage.getItem('user')).roleId == '2') {
this.isAdmin = true; //check if user is admin
}
this.organizacijeService.currentData.subscribe(res => {
this.loadOrgs();
})
}
loadOrgs() {
this.organizacijeService.getAll().subscribe(res => {
this.organizations[0].array = res.data;
console.log(this.organizations[0])
}, err => {
this.toast.error('Organization: ' + JSON.parse(err._body).message, 'Failed')
})
}

Answer 1

You can load subsequent pages by increasing the $skip query parameter in increments of 50 until the page.total - 50.
I would keep in mind that showing 100k records at once will probably still lead to performance problems on the frontend.

Answer 2

You can use a proxy service on the server side that has pagination disabled:
app.use('organizations-unpaginated', {
find(params){
const _params = Object.assign({}, params, { paginate: false });
return app.service('organizations').find(_params);
}
})
As Daff said, you will run into performance issues and I'd generally say that rendering 100k DOM elements (almost all of which are invisible anyway) is bad practice.