Running Puppeteer with xfvb headless : false - node.js

I am running Puppeteer in a headless Ubuntu 16.04 AWS EC2 instance and would like to run it with a virtual display through xfvb. whenever I try to run it I continue to get the error:
/home/ubuntu/node_modules/xvfb/index.js:84
throw new Error('Could not start Xvfb.');
Error: Could not start Xvfb.
at Xvfb.startSync (/home/ubuntu/node_modules/xvfb/index.js:84:17)
at Object.<anonymous> (/home/ubuntu/puppeteer-works.js:39:6)
at Module._compile (internal/modules/cjs/loader.js:689:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:700:10)
at Module.load (internal/modules/cjs/loader.js:599:32)
at tryModuleLoad (internal/modules/cjs/loader.js:538:12)
at Function.Module._load (internal/modules/cjs/loader.js:530:3)
at Function.Module.runMain (internal/modules/cjs/loader.js:742:12)
at startup (internal/bootstrap/node.js:266:19)
at bootstrapNodeJSCore (internal/bootstrap/node.js:596:3)
My code is below:
const puppeteer = require('puppeteer');
const fs = require("fs");
const Xvfb = require('xvfb');
var xvfb = new Xvfb();
var text = fs.readFileSync("proxy.txt").toString('utf-8');
const textByLine = text.split(" ");
const preparePageForTests = async (page) => {
const userAgent = 'Mozilla/5.0 (X11; Linux x86_64)' +
'AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.39
Safari/537.36';
await page.setUserAgent(userAgent);
await page.evaluateOnNewDocument(() => {
Object.defineProperty(navigator, 'webdriver', {
get: () => false,
});
});
await page.evaluateOnNewDocument(() => {
window.chrome = {
runtime: {},
};
});
await page.evaluateOnNewDocument(() => {
const originalQuery = window.navigator.permissions.query;
return window.navigator.permissions.query = (parameters) => (
parameters.name === 'notifications' ?
Promise.resolve({ state: Notification.permission }) :
originalQuery(parameters)
);
});
await page.evaluateOnNewDocument(() => {
Object.defineProperty(navigator, 'plugins', {
get: () => [1, 2, 3, 4, 5],
});
});
}
xvfb.startSync();
(async () => {
const browser = await puppeteer.launch({
args: ['--no-sandbox', '--proxy-server='+textByLine[0]],
headless: true, });
const page = await browser.newPage();
page.authenticate({
username: textByLine[1],
password: textByLine[2]
});
await preparePageForTests(page);
const testUrl ="https://publicindex.sccourts.org/abbeville/publicindex/";
await page.goto(testUrl);
const html = await page.content();
await page.screenshot({path: 'result.png'});
await browser.close()
console.log(html)
})();
xvfb.stopSync();
I appreciate any help, am pretty new to node.js so I apologize in advance for any format errors. I am not being allowed to post this due to it being mainly code, so I am adding this extra sentence.

You seem to be trying to use the Xvfb node module. While the other answers definitely work, here's a snippet that works fully within nodejs
const puppeteer = require('puppeteer')
const Xvfb = require('xvfb');
(async () => {
var xvfb = new Xvfb({
silent: true,
xvfb_args: ["-screen", "0", '1280x720x24', "-ac"],
});
xvfb.start((err)=>{if (err) console.error(err)})
const browser = await puppeteer.launch({
headless: false,
defaultViewport: null, //otherwise it defaults to 800x600
args: ['--no-sandbox', '--start-fullscreen', '--display='+xvfb._display]
});
const page = await browser.newPage();
await page.goto(`https://wikipedia.org`,{waitUntil: 'networkidle2'});
await page.screenshot({path: 'result.png'});
await browser.close()
xvfb.stop();
})()
This isn't perfect in terms of handling errors (and possible race conditions) in xvfb.start(), but it should get you started, and it works pretty consistently for me.
Edit: Remember to install Xvfb first: sudo apt-get install xvfb (Thanks, #iamfrank)

it's seem th xfvb not install properly
you should Install xvfb (the X windows virtual framebuffer) packages for ubuntu OS
$ sudo apt-get update
$ sudo apt-get install xvfb
Run Xvfb in the background and specify a display number (10 in my example)
$ Xvfb :10 -ac &
Set the DISPLAY variable to the number you chose
$ export DISPLAY=:10

If other required packages and xvfb are properly installed, then run the following command.
xvfb-run -a --server-args="-screen 0 1280x800x24 -ac -nolisten tcp -dpi 96 +extension RANDR" command-that-runs-chrome
Cheers!!!

Related

SyntaxError: Cannot use import statement outside a module using puppeteer

Hello I keep trying to run the puppeteer test but keep getting this issue. Please help thanks.enter image description here
code:
import puppeteer from 'puppeteer';
(async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('https://developers.google.com/web/');
// Type into search box.
await page.type('.devsite-search-field', 'Headless Chrome');
// Wait for suggest overlay to appear and click "show all results".
const allResultsSelector = '.devsite-suggest-all-results';
await page.waitForSelector(allResultsSelector);
await page.click(allResultsSelector);
// Wait for the results page to load and display the results.
const resultsSelector = '.gsc-results .gs-title';
await page.waitForSelector(resultsSelector);
// Extract the results from the page.
const links = await page.evaluate(resultsSelector => {
return [...document.querySelectorAll(resultsSelector)].map(anchor => {
const title = anchor.textContent.split('|')[0].trim();
return `${title} - ${anchor.href}`;
});
}, resultsSelector);
// Print all the files.
console.log(links.join('\n'));
await browser.close();
})();
error:
SyntaxError: Cannot use import statement outside a module
at internalCompileFunction (node:internal/vm:74:18)
at wrapSafe (node:internal/modules/cjs/loader:1128:20)
at Module._compile (node:internal/modules/cjs/loader:1169:27)
at Module._extensions..js (node:internal/modules/cjs/loader:1259:10)
at Module.load (node:internal/modules/cjs/loader:1068:32)
at Module._load (node:internal/modules/cjs/loader:909:12)
at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:82:12)
at node:internal/main/run_main_module:23:47
I have tried reinstalling node js and npm but nothing has worked.
{
"name": "my-app",
"version": "0.0.0",
"type": "module",
"scripts": { ...
},
...
}
Add type:module in your package.json file or install the latest Nodejs

Invalid chai property: browser when trying to run a test on puppeteer

i'm trying to run a test on a website using puppeteer and i want to use 'expect' assertion from chai but i keep getting this error and that my test failed.
const puppeteer = require('puppeteer')
const expect = require('chai').expect
describe('My First Puppeteer Test ', () => {
it( 'should launch the browser', async function() {
const browser = await puppeteer.launch({
headless: false,
slowMo: 10,
devtools: false,
defaultViewport: null,
args: ['--start-maximized']
})
const page = await browser.newPage()
await page.goto('http://example.com')
const title = await page.title()
const url = page.url()
const text = await page.$eval('h1', Element => Element.textContent)
const count = await page.$$eval('p', Element => Element.length)
expect(title).to.be.a('string', 'Example Domain')
expect(url).to.include('example.com')
expect(text).to.browser.a('string', 'example domain')
expect(count).to.equal(2)
await browser.close()
})
})
what's wrong with it?
do i need to add something in order for this to work?
this is the message i get from terminal
My First Puppeteer Test
1) should launch the browser
0 passing (1s)
1 failing
1) My First Puppeteer Test
should launch the browser:
Error: Invalid Chai property: browser
at Object.proxyGetter [as get] (node_modules\chai\lib\chai\utils\proxify.js:78:17)
at Context.<anonymous> (tests\example.test.js:25:24)
at runNextTicks (node:internal/process/task_queues:61:5)
at listOnTimeout (node:internal/timers:528:9)
at processTimers (node:internal/timers:502:7)

Puppeteer newPage() freezes/ never resolves nor rejects Ubuntu Server

I am using an Ubuntu server 18.04.5 LTS and Puppeteer 10.0.0. My Problem is that the browser.newPage() function never resolves. So basicly in the console it alsways loggs Start but never 1 nor 2. I have tried a different puppeteer version or puppeteer-core with my own chromium version. I even installed an VM on my Pc and it works there but not on my Server.
var puppeteer = require('puppeteer')
var adresse = "https://www.google.de/"
async function test() {
try {
const browser = await puppeteer.launch({
"headless": true,
"args": [
'--disable-setuid-sandbox',
'--no-sandbox',
'--disable-gpu',
]
})
console.log("Start")
const page = await browser.newPage()
console.log("1")
await page.goto(adresse)
console.log("2")
console.log(page)
} catch (error) {
console.log(error)
}
}
test()

Firebase puppeteer PDF function times out due to Chrome revision

I have a Firebase function to create a PDF file. Lately, it times out due to a "Chrome revision"? Neither do I understand the error message, nor do I understand what is wrong. The function works, when I deploy it locally under MacOS.
TimeoutError: Timed out after 30000 ms while trying to connect to the browser! Only Chrome at revision r818858 is guaranteed to work.
at Timeout.onTimeout (/workspace/node_modules/puppeteer/lib/cjs/puppeteer/node/BrowserRunner.js:204:20)
at listOnTimeout (internal/timers.js:549:17)
at processTimers (internal/timers.js:492:7)
The function:
const puppeteer = require('puppeteer');
const createPDF = async (html, outputPath) => {
let pdf;
try {
const browser = await puppeteer.launch({
args: ['--no-sandbox']
});
const page = await browser.newPage();
await page.emulateMediaType('screen');
await page.setContent(html, {
waitUntil: 'networkidle0'
});
pdf = await page.pdf({
// path: outputPath,
format: 'A4',
printBackground: true,
margin: {
top: "50px",
bottom: "50px"
}
});
await browser.close();
} catch (e) {
console.error(e);
}
return pdf;
};
TimeoutError: Timed out after 30000 ms while trying to connect to the browser!
The aforementioned error is coming from the fact that as mentioned in the documentation:
When you install Puppeteer, it downloads a recent version of Chromium
Everytime you're executing Puppeteer you're running a Chromium in the backend to which Puppeteer will try to connect, hence when it can't connect to the browser this errors raises.
After doing multiple test I was able to execute the Cloud Function by adding the parameter headless on the launch option, since the documentation mentions that it should be true by default I don't quite understand why setting it manually allows the Cloud Function to finish correctly.
At the beginning I was trying with the timeout set to 0 to disable the error due to timeout, however it seems that it's not required, since by only adding headless it finished correctly, but if you find the same problem with the timeouts you can add it.
At the end my code looks like this:
const createPDF = async (html, outputPath) => {
let pdf;
try {
const browser = await puppeteer.launch({
args: ['--no-sandbox'],
headless: true,
timeout: 0
});
const page = await browser.newPage();
await page.emulateMediaType('screen');
await page.setContent(html, {
waitUntil: 'networkidle0'
});
pdf = await page.pdf({
// path: outputPath,
format: 'A4',
printBackground: true,
margin: {
top: "50px",
bottom: "50px"
}
});
await browser.close();
console.log("Download finished"); //Added this to debug that it finishes correctly
} catch (e) {
console.error(e);
}
return pdf;
};

Run and install X server to make puppeteer works

on my linux server I running my nodejs project which should crawl single page app by puppeteer npm module.
Here is an example of the code I use:
const puppeteer = require('puppeteer');
(async () => {
try {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('https://bvopen.abrickis.me/#/standings');
await page.waitForSelector('.category', { timeout: 1000 });
const body = await page.evaluate(() => {
return document.querySelector('body').innerHTML;
});
console.log(body);
await browser.close();
} catch (error) {
console.log(error);
}
})();
But I've got the next error:
0|www | Error: Failed to launch the browser process!
0|www | [5642:5642:0511/154701.856738:ERROR:browser_main_loop.cc(1485)] Unable to open X display.
0|www | [0511/154701.863486:ERROR:nacl_helper_linux.cc(308)] NaCl helper process running without a sandbox!
0|www | Most likely you need to configure your SUID sandbox correctly
0|www | TROUBLESHOOTING: https://github.com/puppeteer/puppeteer/blob/master/docs/troubleshooting.md
I searched a lot how to install X server and tried a lot of things like sudo apt-get install xorg openbox but it doesn't helps.
Looks like puppeteer wants to start the browser in a non-headless mode but as you don't have xorg installed it failed. But I would say that's not what you want when it's running on a server anyways. So there is no need to install xorg or anything.
Maybe try to launch the puppeteer browser with following options:
await puppeteer.launch({
headless: true,
args: [
"--disable-gpu",
"--disable-dev-shm-usage",
"--no-sandbox",
"--disable-setuid-sandbox"
]
});

Resources