RxJS: Parallel http call to paginated API using NestJS HttpService - node.js

Question

Im using NestJS, and this is my current implementation to make parallel http request:
#Injectable()
export class AppService {
constructor(private readonly http: HttpService) {}
private fetchData(index: number) {
Logger.log(`Collect index: ${index}`);
return this.http
.get<Passenger>(
`https://api.instantwebtools.net/v1/passenger?page=${index}&size=100`,
{ validateStatus: null },
)
.pipe(concatMap(response => of(response.data)));
}
async getAllData() {
let a = 0;
const collect: Passenger[] = [];
const $subject = new BehaviorSubject(a);
await $subject
.pipe(
flatMap(index =>
forkJoin([
this.fetchData(index),
this.fetchData(index + 1),
this.fetchData(index + 2),
]).pipe(mergeAll()),
),
tap(data => {
collect.push(data);
if (data?.data?.length === 0) {
$subject.complete(); // stop the stream
} else {
a += 3; // increment by 3, because the request is 3 times at a time
$subject.next(a);
}
}),
)
.toPromise();
return collect;
}
}
This service is to collect the 3rd party data. As for now, the fetchData() function is called multiple times according to how many parallel requests I want at a time. I use a dummy API for the test, but in the real scenario the API endpoint size limit is 100 and it doesn't return the meta information about how much the totalPage. It just returns the empty data when the last page is reached.
The goal is to make a parallel request and combine the result at the end. I'm doing this to keep the request time as minimum as possible and because the API itself has a rate limit of 50 requests per second. How to optimize this code?

Answer 1

To fetch all pages in one go you can use expand to recursively subscribe to an observable that fetches some pages. End the recursion by returning EMPTY when the last page you received is empty.
function fetchAllPages(batchSize: number = 3): Observable<any[][]> {
let index = 0;
return fetchPages(index, batchSize).pipe(
// if the last page isn't empty fetch the next pages, otherwise end the recursion
expand(pages => pages[pages.length - 1].length > 0
? fetchPages((index += batchSize), batchSize)
: EMPTY
),
// accumulate all pages in one array, filter out any trailing empty pages
reduce((acc, curr) => acc.concat(curr.filter(page => page.length)), [])
);
}
// fetch a given number of pages starting from 'index' as parallel requests
function fetchPages(index: number, numberOfPages: number): Observable<any[][]> {
const requests = Array.from({ length: numberOfPages }, (_, i) =>
fetchData(index + i)
);
return forkJoin(requests);
}
https://stackblitz.com/edit/rxjs-vkad5h?file=index.ts
This will obviously send a few unnecessary requests in the last batch if
(totalNumberOfPages + 1) % batchSize != 0.