Puppeteer click parent node of a element with no id - node.js

I'm trying to select a certain size on this website, I have tried multiple approaches that have worked for me so far in puppeteer but none of them seems to work on this instance. I can get the size tab open but cannot figure how to select a specific size.
my code:
await page.goto(data[ii][0]), { //the website link
waitUntil: 'load',
timeout: 0
};
//part 1
await page.click('span[class="default-text__21bVM"]'); //opens size menu
let size = data[ii][1]; //gets size from an array, for example 9
// const xp = `//div[contains(#class, "col-3") and text()="${size}"]`;
// await page.waitForXPath(xp);
// const [sizeButton] = await page.$x(xp);
// await sizeButton.evaluate(btn => {
// btn.parentNode.dispatchEvent(new Event("mousedown"));
// });
await delay(1500);
await page.evaluate((size) => {
document.querySelector(`div > div[class="col-3"][text="${size}"]`).parentElement.click()
});
await page.click('span[class="text__1S19c"]'); // click on submit button
Neither of my approaches worked. I get Error: Evaluation failed: TypeError: Cannot read property 'parentElement' of null meaning the div wasn't found for whatever reason
this is the html of the div I'm trying to click on:
I tried different variations of the querySelector but none of them worked so I'm posting the problem here to see if this is even possible, or if I just made a mistake along the way

This seems working:
import puppeteer from 'puppeteer';
const browser = await puppeteer.launch({ headless: false, defaultViewport: null });
try {
const [page] = await browser.pages();
await page.goto('https://releases.footshop.com/nike-air-force-1-07-lx-wmns-5iBRxXsBHBhvh4GFc9ge');
await page.click('span[class="default-text__21bVM"]');
const size = 9;
const xp = `//div[contains(#class, "col-3") and text()="${size}"]`;
await page.waitForXPath(xp);
const [sizeButton] = await page.$x(xp);
await sizeButton.click();
await page.click('span[class="text__1S19c"]');
} catch (err) { console.error(err); }

Related

Can't click link using puppeteer - Thingiverse

I'm trying to automate away downloading multiple files on thingiverse. I choose an object at random. But I'm having a hard time locating the link I need, clicking and then downloading. Has someone run into this before can I get some help?
I've tried several other variations.
import puppeteer from 'puppeteer';
async function main() {
const browser = await puppeteer.launch({
headless: true,
});
const page = await browser.newPage();
const response = await page.goto('https://www.thingiverse.com/thing:2033856/files');
const buttons = await page.$x(`//a[contains(text(), 'Download')]`);
if(buttons.length > 0){
console.log(buttons.length);
} else {
console.log('no buttons');
}
await wait(5000);
await browser.close();
return 'Finish';
}
async function wait(time: number) {
return new Promise(function (resolve) {
setTimeout(resolve, time);
});
}
function start() {
main()
.then((test) => console.log('DONE'))
.catch((reason) => console.log('Error: ', reason));
}
start();
Download Page
Code
I was able to get it to work.
The selector is: a[class^="ThingFile__download"]
Puppeteer is: const puppeteer = require('puppeteer-extra');
Before the await page.goto() I always recommend setting the viewport:
await page.setViewport({width: 1920, height: 720});
After that is set, change the await page.goto() to have a waitUntil option:
const response = await page.goto('https://www.thingiverse.com/thing:2033856/files', { waitUntil: 'networkidle0' }); // wait until page load
Next, this is a very important part. You have to do what is called waitForSelector() or waitForFunction().
I added both of these lines of code after the const response:
await page.waitForSelector('a[class^="ThingFile__download"]', {visible: true})
await page.waitForFunction("document.querySelector('a[class^=\"ThingFile__download\"]') && document.querySelector('a[class^=\"ThingFile__download\"]').clientHeight != 0");
Next, get the buttons. For my testing I just grabbed the button href.
const buttons = await page.$eval('a[class^="ThingFile__download"]', anchor => anchor.getAttribute('href'));
Lastly, do not check the .length of this variable. In this case we are just returning the href value which is a string. You will get a Promise of an ElementHandle when you try getting just the button:
const button = await page.$('a[class^="ThingFile__download"]');
console.log(button)
if (button) { ... }
Now if you change that page.$ to be page.$$, you will be getting a Promise of an Array<ElementHandle>, and will be able to use .length there.
const buttonsAll = await page.$$('a[class^="ThingFile__download"]');
console.log(buttonsAll)
if (buttons.length > 0) { ... }
Hopefully this helps, and if you can't figure it out I can post my full source later if I have time to make it look better.

puppeteer / node.js - enter page, click load more until all comments load, save page as mhtml

What i'm trying to accomplish is enter this site https://www.discoverpermaculture.com/permaculture-masterclass-video-1 wait until it loads, load all comments from disqus (click 'Load more comments' button until it's no longer present) and save page as mhtml for offline use.
I found similar question here Puppeteer / Node.js to click a button as long as it exists -- and when it no longer exists, commence action but unfortunately trying to detect the "Load more comments" button doesn't work for some reason.
Seems like WaitForSelector('a.load-more__button') is not working because all it prints out is "not visible".
Here's my code
const puppeteer = require('puppeteer');
const url = "https://www.discoverpermaculture.com/permaculture-masterclass-video-1";
const isElementVisible = async (page, cssSelector) => {
let visible = true;
await page
.waitForSelector(cssSelector, { visible: true, timeout: 4000 })
.catch(() => {
console.log('not visible');
visible = false;
});
return visible;
};
async function run () {
let browser = await puppeteer.launch({
headless: true,
defaultViewport: null,
args: [
'--window-size=1920,10000',
],
});
const page = await browser.newPage();
const fs = require('fs');
await page.goto(url);
await page.waitForNavigation();
await page.waitForTimeout(4000)
const selectorForLoadMoreButton = 'a.load-more__button';
let loadMoreVisible = await isElementVisible(page, selectorForLoadMoreButton);
while (loadMoreVisible) {
console.log('load more visible');
await page
.click(selectorForLoadMoreButton)
.catch(() => {});
await page.waitForTimeout(4000);
loadMoreVisible = await isElementVisible(page, selectorForLoadMoreButton);
}
const cdp = await page.target().createCDPSession();
const { data } = await cdp.send('Page.captureSnapshot', { format: 'mhtml' });
fs.writeFileSync('page.mhtml', data);
browser.close();
}
run();
You're just waiting for an ajax request to be processed. You could simply save the total number of comments (top left of the DISQUS plugin) and compare it to an array of comments once the array is equal to the total then you've retrieved every comments.
I've posted something a while back on waiting for ajax request you can see it here: https://stackoverflow.com/a/66092889/3645650.
Alternatively, a simpler approach would be to just use the DISQUS api.
Comments are publicly accessible. You can just use the api key from the website:
https://disqus.com/api/3.0/threads/listPostsThreaded?limit=50&thread=7187962034&forum=pdc2018&order=popular&cursor=1%3A0%3A0&api_key=E8Uh5l5fHZ6gD8U3KycjAIAk46f68Zw7C6eW8WSjZvCLXebZ7p0r1yrYDrLilk2F
parameter
options
limit
Default to 50. Maximum is 100.
thread
Thread number. eg: 7187962034.
forum
Forum id. eg: pdc2018.
order
desc, asc, popular.
cursor
Probably the page number. Format is 1:0:0. eg: Page 2 would be 2:0:0.
api_key
The platform api key. Here the api key is E8Uh5l5fHZ6gD8U3KycjAIAk46f68Zw7C6eW8WSjZvCLXebZ7p0r1yrYDrLilk2F.
If you have to iterate through different pages you would need to intercept the xhr responses to retrieve the thread number.
It turned out the problem was that disqus comments were inside of an iframe
//needed to add those 2 lines
const elementHandle = await page.waitForSelector('iframe');
const frame = await elementHandle.contentFrame();
//and change 'page' to 'frame' below
let loadMoreVisible = await isElementVisible(frame, selectorForLoadMoreButton);
while (loadMoreVisible) {
console.log('load more visible');
await frame
.click(selectorForLoadMoreButton)
.catch(() => {});
await frame.waitForTimeout(4000);
loadMoreVisible = await isElementVisible(frame, selectorForLoadMoreButton);
}
After this change it works perfect

Puppeteer unable to find element using xPath contains(text()) until after element has been selected in chrome dev tools

I am trying to click the "Create File" button on fakebook's download your information page. I am currently able to goto the page, and I wait for the login process to finish. However, when I try to detect the button using
page.$x("//div[contains(text(),'Create File')]")
nothing is found. The same thing occurs when I try to find it in the chrome dev tools console, both in a puppeteer window and in a regular window outside of the instance of chrome puppeteer is controlling:
This is the html info for the element:
I am able to find the element however after I have clicked on it using the chrome dev tools inspector tool:
(the second print statement is from after I have clicked on it with the element inspector tool)
How should I select this element? I am new to puppeteer and to xpath so I apologize if I just missed something obvious.
A small few links I currently remember looking at previously:
Puppeteer can't find selector
puppeteer cannot find element
puppeteer: how to wait until an element is visible?
My Code:
const StealthPlugin = require("puppeteer-extra-plugin-stealth");
(async () => {
let browser;
try {
puppeteer.use(StealthPlugin());
browser = await puppeteer.launch({
headless: false,
// path: "C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe",
args: ["--disable-notifications"],
});
const pages = await browser.pages();
const page = pages[0];
const url = "https://www.facebook.com/dyi?referrer=yfi_settings";
await page.goto(url);
//Handle the login process. Since the login page is different from the url we want, I am going to assume the user
//has logged in if they return to the desired page.
//Wait for the login page to process
await page.waitForFunction(
(args) => {
return window.location.href !== args[0];
},
{ polling: "mutation", timeout: 0 },
[url]
);
//Since multifactor auth can resend the user temporarly to the desired url, use a little debouncing to make sure the user is completely done signing in
// make sure there is no redirect for mfa
await page.waitForFunction(
async (args) => {
// function to make sure there is a debouncing delay between checking the url
// Taken from: https://stackoverflow.com/a/49813472/11072972
function delay(delayInms) {
return new Promise((resolve) => {
setTimeout(() => {
resolve(2);
}, delayInms);
});
}
if (window.location.href === args[0]) {
await delay(2000);
return window.location.href === args[0];
}
return false;
},
{ polling: "mutation", timeout: 0 },
[url]
);
// await page.waitForRequest(url, { timeout: 100000 });
const requestArchiveXpath = "//div[contains(text(),'Create File')]";
await page.waitForXPath(requestArchiveXpath);
const [requestArchiveSelector] = await page.$x(requestArchiveXpath);
await page.click(requestArchiveSelector);
page.waitForTimeout(3000);
} catch (e) {
console.log("End Error: ", e);
} finally {
if (browser) {
await browser.close();
}
}
})();
Resolved using the comment above by #vsemozhebuty and source. Only the last few lines inside the try must change:
const iframeXpath = "//iframe[not(#hidden)]";
const requestArchiveXpath = "//div[contains(text(),'Create File')]";
//Wait for and get iframe
await page.waitForXPath(iframeXpath);
const [iframeHandle] = await page.$x(iframeXpath);
//content frame for iframe => https://devdocs.io/puppeteer/index#elementhandlecontentframe
const frame = await iframeHandle.contentFrame();
//Wait for and get button
await frame.waitForXPath(requestArchiveXpath);
const [requestArchiveSelector] = await frame.$x(requestArchiveXpath);
//click button
await requestArchiveSelector.click();
await page.waitForTimeout(3000);

Puppeteer waitForSelector not working as expected

I've this simple code chunk:
const BUSINESS = 'lalala';
await page.waitForSelector('#searchboxinput').then(
page.evaluate( (BUSINESS) => {
document.querySelector('#searchboxinput').value = BUSINESS
}, BUSINESS)
),
If I set wait for selector -> then, I would expect then to be executed when the selector exists, but I'm getting a Cannot set property value of 'null'.
So I uderstand that document.querySelector('#searchboxinput') == undefined while I suppose that it cannot be possible as it's executed when waitForSelector promise is finished...
is that a bug or I'm missing something?
Not sure if I understand correctly as your chunk is syntactically not complete and not reprducible. But what if you use the returned value of page.waitForSelector()? This seems working:
import puppeteer from 'puppeteer';
const browser = await puppeteer.launch({ headless: false, defaultViewport: null });
try {
const [page] = await browser.pages();
await page.goto('https://example.org/');
const BUSINESS = 'lalala';
await page.waitForSelector('a').then((element) => {
page.evaluate((element, BUSINESS) => {
element.textContent = BUSINESS;
}, element, BUSINESS);
});
} catch (err) { console.error(err); }

Puppeteer doesn't recognize selector with just type and class but accepts full selector

I'm trying to click on a cookiewall on a webpage, but Puppeteer refuses to recognize the short selector with just the type and class selector (button.button-action). Changing this to the full CSS selector fixes the problem but isn't a viable solution since any chance in parent elements can break the selector. As far as I know this shouldn't be a problem because on the page in question using document.querySelector("button.button-action") also returns the element I'm trying to click.
The code that doesn't work:
const puppeteer = require('puppeteer');
const main = async () => {
const browser = await puppeteer.launch({headless: false,});
const page = await browser.newPage();
await page.goto("https://www.euclaim.nl/check-uw-vlucht#/problem", { waitUntil: 'networkidle2' });
const cookiewall = await page.waitForSelector("button.button-action", {visible: true});
await cookiewall.click();
};
main();
The code that does work:
const puppeteer = require('puppeteer');
const main = async () => {
const browser = await puppeteer.launch({headless: false,});
const page = await browser.newPage();
await page.goto("https://www.euclaim.nl/check-uw-vlucht#/problem", { waitUntil: 'networkidle2' });
const cookiewall = await page.waitForSelector("#InfoPopupContainer > div.ipBody > div > div > div.row.actionButtonContainer.mobileText > button", {visible: true});
await cookiewall.click();
};
main();
The problem is that you have three button.button-action there. And the first match is not visible.
One thing you could do is waitForSelector but without the visible bit (because it will check the first button).
And then iterate through all items checking which item is clickable.
await page.waitForSelector("button.button-action");
const actions = await page.$$("button.button-action");
for(let action of actions) {
if(await action.boundingBox()){
await action.click();
break;
}
}

Resources