I have a nodejs project running puppeteer v13.5.1 which does some webscraping.
After some time (mostly 40-80 minutes) the process freezes without throwing any error. It just stops.
I've added some logs and the strange thing is it happens on different executions.
Sometimes it freezes on
const refreshedHtml = await page.evaluate(() => document.documentElement.innerHTML);
sometimes on
await page.click('button.swiper-button-next');
I've tried many different variations, last one being:
const browser = await puppeteer.launch({
headless: true,
devtools: true,
args: [
'--ignore-certificate-errors',
'--no-sandbox',
'--disable-setuid-sandbox',
'--disable-accelerated-2d-canvas',
'--disable-gpu'
]
});
Any help appriecated
Related
so I have a puppeteer script to watch TikTok live streams and when I run it locally it works as expected, but in Ubuntu 20.04 LTS Server the page loads for the live stream, but the live stream never starts and it requires me to log in, which doesn't have locally either. Any ideas to bypass that detection?
Settings
const puppeteer = require("puppeteer-extra");
const { Cluster } = require("puppeteer-cluster");
// Use stealth plugin to bypass bot detection
const StealthPlugin = require("puppeteer-extra-plugin-stealth");
const AnonymizeUA = require("puppeteer-extra-plugin-anonymize-ua");
const AdblockerPlugin = require('puppeteer-extra-plugin-adblocker');
puppeteer.use(StealthPlugin());
puppeteer.use(AnonymizeUA());
puppeteer.use(AdblockerPlugin({ blockTrackers: true }));
(async () => {
const cluster = await Cluster.launch({
concurrency: Cluster.CONCURRENCY_CONTEXT,
maxConcurrency: 60000,
timeout: 86400000,
puppeteer: puppeteer,
retryLimit: 10,
retryDelay: 1000,
puppeteerOptions: {
headless: true,
timeout: 120000, //360000
args: [
"--start-maximized",
"--no-sandbox",
"--disable-setuid-sandbox",
"--disable-dev-shm-usage",
"--disable-accelerated-2d-canvas",
"--no-first-run",
"--no-zygote",
"--disable-gpu",
],
executablePath: "/snap/bin/chromium",
defaultViewport: null,
},
});
Even when I install GUI on the machine, and visit the live stream manually I still cant view it, so it has something to do with the server getting detected as a server maybe?
Thanks a lot!
I am using an Ubuntu server 18.04.5 LTS and Puppeteer 10.0.0. My Problem is that the browser.newPage() function never resolves. So basicly in the console it alsways loggs Start but never 1 nor 2. I have tried a different puppeteer version or puppeteer-core with my own chromium version. I even installed an VM on my Pc and it works there but not on my Server.
var puppeteer = require('puppeteer')
var adresse = "https://www.google.de/"
async function test() {
try {
const browser = await puppeteer.launch({
"headless": true,
"args": [
'--disable-setuid-sandbox',
'--no-sandbox',
'--disable-gpu',
]
})
console.log("Start")
const page = await browser.newPage()
console.log("1")
await page.goto(adresse)
console.log("2")
console.log(page)
} catch (error) {
console.log(error)
}
}
test()
when I run puppeteer on Ubuntu I get this error:
UnhandledPromiseRejectionWarning: Error: Unable to launch browser, error message: Failed to launch the browser process!
[2098647:2098647:0520/162023.317120:ERROR:vaapi_wrapper.cc(594)] Could not get a valid VA display
[2098647:2098647:0520/162023.317252:ERROR:gpu_init.cc(426)] Passthrough is not supported, GL is egl
TROUBLESHOOTING: https://github.com/puppeteer/puppeteer/blob/main/docs/troubleshooting.md
at Cluster.<anonymous> (/root/Desktop/Copart/node_modules/puppeteer-cluster/dist/Cluster.js:119:23)
at Generator.throw (<anonymous>)
at rejected (/root/Desktop/Copart/node_modules/puppeteer-cluster/dist/Cluster.js:6:65)
at process._tickCallback (internal/process/next_tick.js:68:7)
Here are my puppeteer options:
pupOptions: {
headless: false,
args: [
"--incognito",
"--disable-gpu",
"--disable-dev-shm-usage",
"--disable-setuid-sandbox",
"--no-first-run",
"--no-sandbox",
"--no-zygote",
],
defaultViewport: null,
slowMo: 10,
sameDomainDelay: 1000,
retryDelay: 3000,
workerCreationDelay: 3000,
timeout: 30000000,
userDataDir: "/root/.config/google-chrome",
executablePath: "/opt/google/chrome/google-chrome",
}
Also, here is the plugins that I use:
const puppeteer = require("puppeteer-extra");
const RecaptchaPlugin = require("puppeteer-extra-plugin-recaptcha");
I tried killing google instance before running code but still didn't work
Also, I would like to mention that it works when using "puppeteer-cluster"
Anyone have any idea or solution for this? Thanks a lot for the help!
I had to remove "--disable-gpu", from args
If you are running puppeteer on Ubuntu server, you should try turning
headless: false
to
headless: true
if there is no GUI on your system, then it can't show you the browser
I am using puppeteer on Google App Engine with Node.JS
whenever I run puppeteer on app engine, I encounter an error saying
Navigation failed because browser has disconnected!
This works fine in local environment, so I am guessing it is a problem with app engine.
const browser = await puppeteer.launch({
ignoreHTTPSErrors: true,
headless: true,
args: ["--disable-setuid-sandbox", "--no-sandbox"],
});
This is my app engine's app.yaml file
runtime: nodejs12
env: standard
handlers:
- url: /.*
secure: always
script: auto
-- EDIT--
It works when I add --disable-dev-shm-usage argument, but then it always timeouts. Here are my codes.
const browser = await puppeteer.launch({
ignoreHTTPSErrors: true,
headless: true,
args: [
"--disable-gpu",
"--disable-dev-shm-usage",
"--no-sandbox",
"--disable-setuid-sandbox",
"--no-first-run",
"--no-zygote",
"--single-process",
],
});
const page = await browser.newPage();
try {
const url = "https://seekingalpha.com/market-news/1";
const pageOption = {
waitUntil: "networkidle2",
timeout: 20000,
};
await page.goto(url, pageOption);
} catch (e) {
console.log(e);
await page.close();
await browser.close();
return resolve("error at 1");
}
try {
const ulSelector = "#latest-news-list";
await page.waitForSelector(ulSelector, { timeout: 30000 });
} catch (e) {
// ALWAYS TIMEOUTS HERE!
console.log(e);
await page.close();
await browser.close();
return resolve("error at 2");
}
...
It seems the problem was app engine's memory capacity.
When memory is not enough to deal with puppeteer crawling,
It automatically generates another instance.
However, newly created instance has a different puppeteer browser.
Therefore, it results in Navigation failed because browser has disconnected.
The solution is simply upgrade the app engine instance so it can deal with the crawling job by a single instance.
default instance is F1, which has 256M of memory, so I upgraded to F4, which has 1GB of memery, then it doesn't show an error message anymore.
runtime: nodejs12
instance_class: F4
handlers:
- url: /.*
secure: always
script: auto
For me the error was solved when I stopped using the --use-gl=swiftshader arg.
It is used by default if you use args: chromium.args from chrome-aws-lambda
I was having that error in a deploy, the solution for this problem is change some parameters in waitForNavigation:
{ waitUntil: "domcontentloaded" , timeout: 60000 }
I've written a small crawler with the help of Puppeteer.
Now I'm facing the challenge that my tests are rather slowly (> 3 seconds for each test). I've been able to track it down to the launch function of Puppeteer and the usage of Istanbul/nyc.
If I run the test just with mocha, the tests are finished under 400 ms.
But if I additionally use nyc, the duration of the tests exceeds 3000 ms
All that I'm using is
'use strict';
const puppeteer = require('puppeteer');
module.exports = async function startBrowser() {
const options = {
args: [
// '--no-sandbox',
// '--disable-setuid-sandbox',
// '--disable-dev-shm-usage',
// '--disable-accelerated-2d-canvas',
// '--disable-gpu'
],
headless: false // true
};
return await puppeteer.launch(options);
};
Here is the test I'm using:
'use strict';
/* global describe: false, before: false, it: false,
beforeEach: false, afterEach: false, after: false, window: false, document: false */
const assert = require('assert').strict;
const startBrowser = require('../');
const util = require('util');
describe('Puppeteer', function() {
let pageManager;
it('start the browser', async function() {
this.timeout(10000);
console.time('startBrowser');
const browser = await startBrowser();
console.timeEnd('startBrowser');
assert(browser);
console.time('closeBrowser');
await browser.close();
console.timeEnd('closeBrowser');
});
});
I've created a repository with this code and test here.
nyc _mocha ./test/*.test.js runs in ~3500ms, mocha ./test/*.test.js takes only 130ms.
What I've tried so far:
different combination of include/exclude nyc flags
updating to latest versions of Puppeteer, nyc and mocha
removing my Puppeteer arguments
searching for Puppeteer & Istanbul related issues (with not much success)
trying headless: true
bypassing all proxies, see this puppeteer issue
What can I do to have tests with coverage be as fast as the tests alone?
Using:
Ubuntu 19.04
node.js 10.15.3
I've started to debug Puppeteer and these are my findings:
Puppeteer is unsurprisingly using child_process.spawn() to spawn a new browser
nyc is using spawn-wrap for such child processes
spawn-wrap is reading the whole executable (./node_modules/puppeteer/.local-chromium/linux-686378/chrome-linux/chrome) into memory with fs.readFileSync which is taking an unusually long time to finish
spawn-wraps README delivers some kind of an explanation:
The initial wrap call uses synchronous I/O. Probably you should not be using this script in any production environments anyway. Also, this will slow down child process execution by a lot, since we're adding a few layers of indirection.
For me personally the answer is that I cannot get the same performance for running tests with and without code coverage as long as I use nyc/istanbul.
I've given c8 a shot, and the performance is nearly the same and I can still have code coverage.
Please try this also.
'use strict'
const puppeteer = require('puppeteer')
module.exports = async function startBrowser() {
const options = {
args: [
'--no-sandbox',
'--disable-setuid-sandbox',
'--disable-dev-shm-usage',
'--disable-accelerated-2d-canvas',
'--no-first-run',
'--no-zygote',
'--single-process', // <- this one doesn't works in Windows
'--disable-gpu'
],
headless: true
}
return await puppeteer.launch(options)
}
./Chromium --headless --disable-gpu --remote-debugging-port=9222 --devtools=false
browser = await puppeteer.connect({
browserWSEndpoint: 'ws://127.0.0.1:9222/devtools/browser/........xxxxxx..',
});
use express to hold it
PS: But ,I can't specify the browserWSEndpoint
the url change every time Chromium --headless restart