I need to read data on https://www.cmegroup.com/tools-information/quikstrike/options-calendar.html
I tried to click on FX tab from page.click in puppeteer, but the page remains on the default.
Any help welcome
const puppeteer = require('puppeteer');
let scrape = async () => {
const browser = await puppeteer.launch({headless: false});
const page = await browser.newPage();
await page.goto('https://www.cmegroup.com/tools-information/quikstrike/options-calendar.html');
await page.waitFor(1000);
//div select FX
await page.click('#ctl00_MainContent_ucViewControl_IntegratedCMEOptionExpirationCalendar_ucViewControl_ucProductSelector_lvGroups_ctrl3_lbProductGroup');
//browser.close();
return result;
};
scrape().then((value) => {
console.log(value); // Success!
});
I couldn't find the element you're looking for on that page. However, this might be helpful:
Wait for the selector to appear on the page before clicking on it:
await page.waitForSelector(selector);
If still facing the issue, try using Javascript click method:
await page.$eval(selector, elem => elem.click());
Related
I am trying to fetch data and trigger some automatic buying process with the following website. https://www.klwines.com/
Was using "puppeteer" methods with NodeJS to process the script. According to the following screenshot provided, I got stuck with an issue where I cannot select one of the a radio button from the list since all radio buttons having the same id. What I am trying to do is just trying to select the last radio button from the following list and then trigger he button shown in the image. I was using the following NodeJS code with the help of puppeteer.
await page.waitForNavigation();
await page.waitForSelector('[name="continue"]');
const radio = await page.evaluate("table tr:nth-child(4) > td > input[type=radio]")
radio.click()
Please note that the page variable is defined as the following.
const browser = await puppeteer.launch({ headless: false });
const page = await browser.newPage();
If someone can help with this to find a way that would be really great full.
You can try this way;
const puppeteer = require('puppeteer');
exports.yourStatus = async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('https://www.klwines.com/');
const data = await page.evaluate(() => {
function cleanData(element) {
const items = element.getElementById('Shepmente_0__shepmentewayCode');
return [...items].map(item => {
console.log(item)
});
}
return data;
};
I'm trying to click on a cookiewall on a webpage, but Puppeteer refuses to recognize the short selector with just the type and class selector (button.button-action). Changing this to the full CSS selector fixes the problem but isn't a viable solution since any chance in parent elements can break the selector. As far as I know this shouldn't be a problem because on the page in question using document.querySelector("button.button-action") also returns the element I'm trying to click.
The code that doesn't work:
const puppeteer = require('puppeteer');
const main = async () => {
const browser = await puppeteer.launch({headless: false,});
const page = await browser.newPage();
await page.goto("https://www.euclaim.nl/check-uw-vlucht#/problem", { waitUntil: 'networkidle2' });
const cookiewall = await page.waitForSelector("button.button-action", {visible: true});
await cookiewall.click();
};
main();
The code that does work:
const puppeteer = require('puppeteer');
const main = async () => {
const browser = await puppeteer.launch({headless: false,});
const page = await browser.newPage();
await page.goto("https://www.euclaim.nl/check-uw-vlucht#/problem", { waitUntil: 'networkidle2' });
const cookiewall = await page.waitForSelector("#InfoPopupContainer > div.ipBody > div > div > div.row.actionButtonContainer.mobileText > button", {visible: true});
await cookiewall.click();
};
main();
The problem is that you have three button.button-action there. And the first match is not visible.
One thing you could do is waitForSelector but without the visible bit (because it will check the first button).
And then iterate through all items checking which item is clickable.
await page.waitForSelector("button.button-action");
const actions = await page.$$("button.button-action");
for(let action of actions) {
if(await action.boundingBox()){
await action.click();
break;
}
}
i have been working in a web scraping code in node.js using the npm puppeteer to get the url, image and titles from each news in the page but the only thing i was able to get the url, image and title from the first news.
const puppeteer = require('puppeteer');
(async () => {
const brower = await puppeteer.launch();
const page = await brower.newPage();
const url = 'https://es.cointelegraph.com/category/latest';
await page.goto(url, { waitUntil: 'load' });
const datos = await page.evaluate(() => Array.from(document.querySelectorAll('.categories-page__list'))
.map( info => ({
titulo: info.querySelector('.post-preview-item-inline__title').innerText.trim(),
link: info.querySelector('.post-preview-item-inline__title-link').href,
imagen: info.querySelector('.post-preview-item-inline__figure .lazy-image__wrp img ').src
}))
)
console.log(datos);
await page.close();
await brower.close();
})()
Because there is just one .categories-page__list in the page while there are a lot of .post-preview-list-inline__item elements.
You map over an array returned from document.querySelectorAll('.categories-page__list') but the array has just one element, it's right that it run the map closure just once.
So, replace
document.querySelectorAll('.categories-page__list')
with
document.querySelectorAll('.post-preview-list-inline__item')
and everything works.
Here you can find a working example.
Let me know if you need some more help 😉
So I'm trying to add a value to simple input box using puppeteer. The problem is that it will truncate the first letter or first few letters. To my understanding, the await page.waitFor won't result until that selector is visible on the DOM.
function test(){
(async () => {
const browser = await puppeteer.launch({headless: false});
const page = await browser.newPage();
await page.goto('https://sadfasdfasdf.com');
await page.click('#signup_forms_submit');
await page.waitFor('#signup_email');
await page.type('#signup_email','test#gmail.com',{delay: 120});
console.log('test');
})();
}
I want to load a page, and then wait for the text (or class in this case) to be rendered before I get the content.
This example works.
async function test() {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('https://sunnythailand.com');
// Wait until the page is fully rendered
while (content.indexOf("scrapebot_description") < 0) {
console.log("looking for scrapebot_description")
await new Promise((resolve)=>setTimeout(()=> resolve() ,1000));
content = await page.content();
}
console.log("scrapebot_description FOUND!!!")
await browser.close();
}
My question is, can I do this easier with puppeteer?
I tried this:
await page.waitForFunction('document.querySelector("scrapebot_description")');
But that just hangs there forever, nothing ever happens...
(to be honest I dont understand what querySelector is, so perhaps the problem is there)
I also tried this:
var checkText = "scrapebot_description"
await page.evaluate((checkText) => {
console.log("scrapebot_description FOUND IT!!");
},{checkText});
This also does not work.
This is the last element to render on the page what im waiting for....
<span class="hide scrapebot_description ng-binding" ng-bind="'transFrontDescription' | translate">
You can do this:
async function test() {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('https://sunnythailand.com');
const selector = '.scrapebot_description' // or #scrapebot_description
await page.waitForSelector(selector)
console.log("scrapebot_description FOUND!!!")
await browser.close();
}