Puppeteer iterates through all pages but browser won't close after - node.js

Question

After the program iterates through all pages, it doesn't break out of the while loop and closes the browser. Instead, it ran through the while loop one extra time and gave me an error: "TimeoutError: waiting for selector `.pager .next a` failed: timeout 30000ms exceeded" What went wrong?
const puppeteer = require("puppeteer");
(async () => {
const browser = await puppeteer.launch({ headless: false });
const page = await browser.newPage();
await page.goto("http://books.toscrape.com/");
let isLastPage = false;
while (!isLastPage) {
await page.waitForSelector(".pager .next a");
isLastPage = (await page.$(".pager .next a")) === null;
await page.click(".pager .next a");
}
console.log("done");
await browser.close();
})();

Answer 1

Your last page detection logic is just flawed. While you're on a page, you're trying to both see if ".pager .next a" exists AND you're trying to click that. Obviously if it doesn't exist, you can't click it.
What you want to do is make sure the page is loaded by waiting for .pager .current which is a part of the navigation footer that will be there on every page. Then, check if .pager .next a is there BEFORE you click and if it's not there, then you can just break out of the while loop. If the page is dynamic and you need to use puppeteer, then you can do something like this:
const puppeteer = require("puppeteer");
(async () => {
const browser = await puppeteer.launch({ headless: false });
try {
const page = await browser.newPage();
await page.goto("http://books.toscrape.com/");
let cntr = 0;
while (true) {
await page.waitForSelector(".pager .current");
console.log(`page ${++cntr}`);
// process the page content here
if ((await page.$(".pager .next a")) === null) {
break;
}
await page.click(".pager .next a");
}
console.log("done");
} finally {
await browser.close();
}
})();
And, to make sure that you always close the browser even upon errors, you need to catch any errors and make sure you close the browser in those conditions. In this case, you can use try/finally.
If the page is not dynamic, then you can also just use plain GET requests and use cheerio to examine what's in the page which is simpler and doesn't involve loading the whole chromium browser engine.