Puppeteer timeout and connect issues - node.js

Question

Disclaimer: I'm not a Node-pro. I've read so many tickets and sites today, trying to solve my issues. But when one problem is solved, another occured and vice versa.
Currently, Puppeteer is used in the following way:
const browser = await puppeteer.launch({
headless: true,
ignoreHTTPSErrors: true
});
const page = await browser.newPage();
const response = await page.goto(targetUrl, {waitUntil: 'load'});
const cdp = await page.target().createCDPSession();
const cookies = (await cdp.send('Network.getAllCookies')).cookies;
const localStorage = await page.evaluate(() => Object.assign({}, window.localStorage));
const sessionStorage = await page.evaluate(() => Object.assign({}, window.sessionStorage));
This works for most pages, but when trying to grab https://cioudways.com for example, I get Execution context was destroyed, most likely because of a navigation.
Replacing {waitUntil: 'load'} with {waitUntil: 'networkidle2'} it will only fail randomly then. But when trying to grab https://github.com using networkidle2, the whole process will timeout, resulting in Navigation timeout of 30000 ms exceeded (but with load instead of networkidle2 it works).
How can I solve this to get a stable script, that is able to work with nearly every URL?

Answer 1

The answer to the error in the case of the first URL is in the error message literally: Execution context was destroyed, most likely because of a navigation. When you open https://cioudways.com it will immediately replace location.href to https://www.cloudways.com/en/?id={foo}&data1={bar}&data2=in (note: it is not a regular HTTP 301 redirection, both are HTTP 200. HTTP 30x are handled by puppeteer) so your page is immediately destroyed before you'd have the chance evaluate it.
For this specific URL awaiting a new load event - right after page.goto() - would solve your issue:
await page.goto('https://cioudways.com', { waitUntil: 'load' })
await page.waitForNavigation({ waitUntil: 'load' })
Of course, this would break the script for any other target URL (as it is unusual to have additional navigation on site launch). So you can't apply it as a general solution.
You could use (for this specific site) the redirected page https://www.cloudways.com/ to avoid this issue.
The second case has a different cause. The https://github.com page seems to have an issue with its resources. If I log all network calls:
await page.setRequestInterception(true)
page.on('request', request => {
console.log(request.url())
request.continue()
})
await page.goto('https://github.com', { waitUntil: 'networkidle2' })
It always stops at https://github.githubassets.com/images/modules/site/home/globe/flag.obj. I have no answer for this, the login page is full of canvas-es and animations that may affect networkidle2 (that means there are no more than 2 network calls in progress). Can be caused by a bug on Github's side. Maybe it is worth its own question.
Suggestion
As your problem lies in the unreliability of page loads I suggest using { waitUntil: 'load' } (as this is the default you can omit this argument completely) and I'd pause the page (page.waitForTimeout()) for a short while to give time for localStorage etc. to be filled in case of Angular/React apps too. This is only a workaround, pausing script execution for a huge amount of URLs is not a good thing, for slower pages maybe the hardcoded pause won't be enough while for others it will be unnecessarily long wait.
await page.goto(targetUrl)
await page.waitForTimeout(4000)