sHow to fetch elements while scrolling puppeteer - node.js

Question

I'm trying to scroll an auto loading page , and while doing it I want to fetch the appearing (and disapearing elements).
My code looks like that , the scrolling works great, but I'm not able to make my puppeteer code work in order to detect the elements and save their values (the code does work outside the scroll functions)
async function autoScroll(page) {
await page.evaluate(async () => {
await new Promise((resolve, reject) => {
let totalHeight = 0;
let distance = 100;
let timer = setInterval(async () => {
let scrollHeight = document.body.scrollHeight;
window.scrollBy(0, distance);
totalHeight += distance;
console.log("scrolling"); // That one never shows up
await getUsers(); // Trying to fetch elements on every scroll
if (totalHeight >= scrollHeight) {
clearInterval(timer);
resolve();
}
}, 100);
});
});
}
async function getUsers() {
let hrefs = await page.$$('div > a');
for (let i = 0; i < hrefs.length; i = i++) { adding each link to database }
-- What I want to achieve is , that every time I scroll to the bottom of the page , the getUsers functions will fetch all the links in spesific div and add them to the DB if they they don't exist yet but calling the function from the SetInterval doesn't seem to work
How can I Include my puppeteer async function while scrolling through the page?

Answer 1

the code does work outside the scroll functions
The getUsers function is defined in the main node.js script, but in autoScroll it is used inside of page.evaluate function, and the code inside page.evaluate runs in the browser context (as if we run it in DevTools console) where there is no getUsers function.
Since getUsers works with a database it can only work on node.js side, not in page.evaluate, you should rewrite the scraping code.
I'd suggest first getting userdata inside of page.evaluate and only after the page doesn't scroll anymore return the data to the main context and then save to the databaswe.
console.log from page.evaluate is not shown becase you need to specifically subscribe to it to see console messages.