Puppeteer Cluster Inconsistent Results - node.js

Question

I've recently converted a script over from Puppeteer to Puppeteer Cluster and during testing I've observed some odd results when testing multiple pages concurrently.
Effectively I'm loading a single page and then iterating over the product options on the page and gathering the price for any product variants.
One particular product has around 9 product variants, sometimes I will accurately capture all 9 variants, whereas on the next testing cycle it may only return 2 or 3 variants.
Any help would be greatly appreciated!
const puppeteer = require('puppeteer');
const { Cluster } = require('puppeteer-cluster');
const Product = require('../utils/product')
const config = require('../config/config.json')
const selectors = config.productData;
(async () => {
const urls = [
{link: ...},
{link: ...},
{link: ...}
]
const cluster = await Cluster.launch({
concurrency: Cluster.CONCURRENCY_CONTEXT,
maxConcurrency: 5,
puppeteerOptions: {
headless: false
},
});
await cluster.task(async ({ page, data: url }) => {
//instantiate a new product object
const product = new Product();
await page.goto(url, { waitUntil: 'load' });
const skuprice = await page.$eval(selectors.price, element => element.innerText);
console.log('Sku Price:' + skuprice)
//deal with variants
const options = await page.$$eval(selectors.variant, elements => elements.map(element=>element.id))
if (options.length > 0) {
//set up a variants array
for (let index = 0; index < options.length; index++) {
const element = options[index];
await page.waitForSelector(`#${element}`);
await page.$eval(`#${element}`, radio => radio.click());
await page.waitForTimeout(500);
const variantprice = await page.$eval(selectors.price, element => element.innerText);
console.log('Variant Price:' + variantprice)
}
}
});
urls.forEach(url => {
cluster.queue(url.link);
})
// many more pages
await cluster.idle();
await cluster.close();
})();

Answer 1

Dynamic javascript page should be scraped when all of the element is visible.
You can do following tricks:
[1] wait until selector is visible, check withawait page.waitForSelector(selector, {visible: true, timeout: 0})
[2] wait for desired time, but this is more flaky and prone to resulting error.
You can simplify and rewrite your code, like this:
await page.waitForSelector(`#${element}`, {visible: true, timeout: 0})
await page.click(`#${element}`)
/* await page.waitForTimeout(500) <= prone to error, use line below */
await page.waitForSelector(selectors.price, {visible: true, timeout: 0})
const variantprice = await page.$eval(selectors.price, element => element.innerText)

Answer 2

For anyone else searching for an answer, it looked like some of my CSS selectors weren't working on page refresh.
Re-reading the project documentation its worth including the following:
// Event handler to be called in case of problems
cluster.on('taskerror', (err, data) => {
console.log(`Error crawling ${data}: ${err.message}`);
});