I'm running puppeteer-extra-plugin-stealth on my device localhost and it threw error RequestError: self signed certificate after the chromium opened.
Previously when I faced this error I solved it by adding --ignore-certificate-errors in args array but now it doesn't work anymore. I saw some discussion asking to add --ignore-certificate-errors-spki-list too but still not working. Making npm to ignore ca also doesn't help.
const StealthPlugin = require('puppeteer-extra-plugin-stealth')();
const puppeteer = require('puppeteer-extra');
StealthPlugin.onBrowser = () => { };
StealthPlugin.enabledEvasions.delete('chrome.runtime')
StealthPlugin.enabledEvasions.delete('iframe.contentWindow')
puppeteer.use(StealthPlugin);
puppeteer.launch({
headless: false,
userDataDir: '/',
args: [
'--no-sandbox',
'--proxy-server="direct://"',
'--proxy-bypass-list=*',
'--start-fullscreen',
'--ignore-certificate-errors',
'--ignore-certificate-errors-spki-list'
]
}).then(async browser => {})
Any way I can ignore this certificate checking? Thank you.
Related
so I have a puppeteer script to watch TikTok live streams and when I run it locally it works as expected, but in Ubuntu 20.04 LTS Server the page loads for the live stream, but the live stream never starts and it requires me to log in, which doesn't have locally either. Any ideas to bypass that detection?
Settings
const puppeteer = require("puppeteer-extra");
const { Cluster } = require("puppeteer-cluster");
// Use stealth plugin to bypass bot detection
const StealthPlugin = require("puppeteer-extra-plugin-stealth");
const AnonymizeUA = require("puppeteer-extra-plugin-anonymize-ua");
const AdblockerPlugin = require('puppeteer-extra-plugin-adblocker');
puppeteer.use(StealthPlugin());
puppeteer.use(AnonymizeUA());
puppeteer.use(AdblockerPlugin({ blockTrackers: true }));
(async () => {
const cluster = await Cluster.launch({
concurrency: Cluster.CONCURRENCY_CONTEXT,
maxConcurrency: 60000,
timeout: 86400000,
puppeteer: puppeteer,
retryLimit: 10,
retryDelay: 1000,
puppeteerOptions: {
headless: true,
timeout: 120000, //360000
args: [
"--start-maximized",
"--no-sandbox",
"--disable-setuid-sandbox",
"--disable-dev-shm-usage",
"--disable-accelerated-2d-canvas",
"--no-first-run",
"--no-zygote",
"--disable-gpu",
],
executablePath: "/snap/bin/chromium",
defaultViewport: null,
},
});
Even when I install GUI on the machine, and visit the live stream manually I still cant view it, so it has something to do with the server getting detected as a server maybe?
Thanks a lot!
I have a nodejs project running puppeteer v13.5.1 which does some webscraping.
After some time (mostly 40-80 minutes) the process freezes without throwing any error. It just stops.
I've added some logs and the strange thing is it happens on different executions.
Sometimes it freezes on
const refreshedHtml = await page.evaluate(() => document.documentElement.innerHTML);
sometimes on
await page.click('button.swiper-button-next');
I've tried many different variations, last one being:
const browser = await puppeteer.launch({
headless: true,
devtools: true,
args: [
'--ignore-certificate-errors',
'--no-sandbox',
'--disable-setuid-sandbox',
'--disable-accelerated-2d-canvas',
'--disable-gpu'
]
});
Any help appriecated
I am using an Ubuntu server 18.04.5 LTS and Puppeteer 10.0.0. My Problem is that the browser.newPage() function never resolves. So basicly in the console it alsways loggs Start but never 1 nor 2. I have tried a different puppeteer version or puppeteer-core with my own chromium version. I even installed an VM on my Pc and it works there but not on my Server.
var puppeteer = require('puppeteer')
var adresse = "https://www.google.de/"
async function test() {
try {
const browser = await puppeteer.launch({
"headless": true,
"args": [
'--disable-setuid-sandbox',
'--no-sandbox',
'--disable-gpu',
]
})
console.log("Start")
const page = await browser.newPage()
console.log("1")
await page.goto(adresse)
console.log("2")
console.log(page)
} catch (error) {
console.log(error)
}
}
test()
when I run puppeteer on Ubuntu I get this error:
UnhandledPromiseRejectionWarning: Error: Unable to launch browser, error message: Failed to launch the browser process!
[2098647:2098647:0520/162023.317120:ERROR:vaapi_wrapper.cc(594)] Could not get a valid VA display
[2098647:2098647:0520/162023.317252:ERROR:gpu_init.cc(426)] Passthrough is not supported, GL is egl
TROUBLESHOOTING: https://github.com/puppeteer/puppeteer/blob/main/docs/troubleshooting.md
at Cluster.<anonymous> (/root/Desktop/Copart/node_modules/puppeteer-cluster/dist/Cluster.js:119:23)
at Generator.throw (<anonymous>)
at rejected (/root/Desktop/Copart/node_modules/puppeteer-cluster/dist/Cluster.js:6:65)
at process._tickCallback (internal/process/next_tick.js:68:7)
Here are my puppeteer options:
pupOptions: {
headless: false,
args: [
"--incognito",
"--disable-gpu",
"--disable-dev-shm-usage",
"--disable-setuid-sandbox",
"--no-first-run",
"--no-sandbox",
"--no-zygote",
],
defaultViewport: null,
slowMo: 10,
sameDomainDelay: 1000,
retryDelay: 3000,
workerCreationDelay: 3000,
timeout: 30000000,
userDataDir: "/root/.config/google-chrome",
executablePath: "/opt/google/chrome/google-chrome",
}
Also, here is the plugins that I use:
const puppeteer = require("puppeteer-extra");
const RecaptchaPlugin = require("puppeteer-extra-plugin-recaptcha");
I tried killing google instance before running code but still didn't work
Also, I would like to mention that it works when using "puppeteer-cluster"
Anyone have any idea or solution for this? Thanks a lot for the help!
I had to remove "--disable-gpu", from args
If you are running puppeteer on Ubuntu server, you should try turning
headless: false
to
headless: true
if there is no GUI on your system, then it can't show you the browser
on my linux server I running my nodejs project which should crawl single page app by puppeteer npm module.
Here is an example of the code I use:
const puppeteer = require('puppeteer');
(async () => {
try {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('https://bvopen.abrickis.me/#/standings');
await page.waitForSelector('.category', { timeout: 1000 });
const body = await page.evaluate(() => {
return document.querySelector('body').innerHTML;
});
console.log(body);
await browser.close();
} catch (error) {
console.log(error);
}
})();
But I've got the next error:
0|www | Error: Failed to launch the browser process!
0|www | [5642:5642:0511/154701.856738:ERROR:browser_main_loop.cc(1485)] Unable to open X display.
0|www | [0511/154701.863486:ERROR:nacl_helper_linux.cc(308)] NaCl helper process running without a sandbox!
0|www | Most likely you need to configure your SUID sandbox correctly
0|www | TROUBLESHOOTING: https://github.com/puppeteer/puppeteer/blob/master/docs/troubleshooting.md
I searched a lot how to install X server and tried a lot of things like sudo apt-get install xorg openbox but it doesn't helps.
Looks like puppeteer wants to start the browser in a non-headless mode but as you don't have xorg installed it failed. But I would say that's not what you want when it's running on a server anyways. So there is no need to install xorg or anything.
Maybe try to launch the puppeteer browser with following options:
await puppeteer.launch({
headless: true,
args: [
"--disable-gpu",
"--disable-dev-shm-usage",
"--no-sandbox",
"--disable-setuid-sandbox"
]
});