I use puppeteer for my project. I always installed both puppeteer library and the browser itself with the npm i puppeteer command. Now I want to install the browser separately.
The puppeteer version I use is 13.5.2. In the documentation said, that it's guaranted to work only with specific browser version. The compatibility list can be found here: https://github.com/puppeteer/puppeteer/blob/main/versions.js
So, chromium 100.0.4889.0 matches my puppeteer 13.5.2. I try to install this version in Docker. Here is my Dockerfile:
FROM node:10-slim
ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true
RUN apt-get update
# for https
RUN apt-get install -yyq ca-certificates
# install libraries
RUN apt-get install -yyq libappindicator1 libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libnss3 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6
# tools
RUN apt-get install -yyq gconf-service lsb-release wget xdg-utils
# and fonts
RUN apt-get install -yyq fonts-liberation
RUN apt-get install -yyq chromium=100.0.4889.0
There are more commands, where I install npm dependencies, but it doesn't matter, since I have an error on the last command:
> [ 7/16] RUN apt-get install -yyq chromium=100.0.4889.0:
#10 0.212 Reading package lists...
#10 0.600 Building dependency tree...
#10 0.662 Reading state information...
#10 0.673 E: Version '100.0.4889.0' for 'chromium' was not found
Looks like only versions 73.X and 75.X could be installed:
root#077e9dcf4544:/# apt-cache policy chromium
chromium:
Installed: (none)
Candidate: 73.0.3683.75-1~deb9u1
Version table:
73.0.3683.75-1~deb9u1 500
500 http://deb.debian.org/debian stretch/main arm64 Packages
500 http://security.debian.org/debian-security stretch/updates/main arm64 Packages
How can I solve this problem?
The Chromium releases that Puppeteer downloads are not necessarily published to the package managers of operating systems. So it's likely that the exact same version is not available via the package manager. The same major versions usually have very few differences so if you download M100 via the package manager, it might work. To download the exact version you'd need to use the Puppeteer package (i.e., npm i puppeteer#version in your Dockerfile) or replicate the logic that Puppeteer uses to download the binary (although, that is not recommended).
Related
I wrote a little puppeteer program that let me log into twitter and check a few things. Locally on mac OS Catalina, it is working but on VPS ubuntu 18.04 lts not working. And shows me a log at the start:
/root/retwiter/node_modules/puppeteer/.local-chromium/linux-800071/chrome-linux/chrome: error while loading shared libraries: libnss3.so: cannot open shared object file: No such file or directory
I have 2 running configs for browser:
local:
{
headless: false,
defaultViewport: null,
args: [
'--window-size=1920,1080'
]
}
vps:
{
headless: true,
defaultViewport: null,
args: [
'--no-sandbox',
'--disable-setuid-sandbox'
]
}
i think you are missing the library libnss3 , try to install it using
sudo apt-get install libnss3-dev
and if that's not enough install all the deps for puppeteer
sudo apt-get install -y gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget libgbm-dev
You can add AWS Elastic Beanstalk configuration files (.ebextensions) to your web application's source code to configure your environment and customize the AWS resources that it contains.
In our case, if we don't enable EPEL and if we continue installing Chromium as part of npm install, Puppeteer cannot launch Chromium due to unavailability of libatk-1.0.so.0.
Source # Running Puppeteer on AWS EC2 instance running Amazon-Linux
Hotfix
Add the following to your app root:
Create a .npmrc file.
Add the following to your .npmrc file:
unsafe-perm=true
The .npmrc file defines how npm should behave when running commands. Setting unsafe-perm to true suppress the UID/GID switching when running package scripts. Set the unsafe-perm flag to run scripts with root privileges.
Source # unsafe-perm
Create a .ebextensions folder.
Inside .ebextensions folder, create a 01_enableEPEL.config file and add the following.
commands:
01_enableEPEL:
command: sudo amazon-linux-extras install epel -y
Inside .ebextensions folder, create a 02_installEPELPackages.config file and add the following
packages:
yum:
chromium: []
You can view the same guide on my GitHub # https://github.com/amarinediary/Marionette
If on CentOS, you can use this command with the correct dependencies:
sudo yum install -y alsa-lib.x86_64 atk.x86_64 cups-libs.x86_64 gtk3.x86_64 ipa-gothic-fonts libXcomposite.x86_64 libXcursor.x86_64 libXdamage.x86_64 libXext.x86_64 libXi.x86_64 libXrandr.x86_64 libXScrnSaver.x86_64 libXtst.x86_64 pango.x86_64 xorg-x11-fonts-100dpi xorg-x11-fonts-75dpi xorg-x11-fonts-cyrillic xorg-x11-fonts-misc xorg-x11-fonts-Type1 xorg-x11-utils
sudo yum update nss -y
This is the error I get. This occurs when the test code is run in the Gitlab CI.
However, it works fine locally. Someone please help?
2036 $ npm test
2037 > 10_auth#1.1.3 test /builds/brhanuh/domestic-repair
2038 > jest --coverage 'unitTests/'
2039 Error: Failed to launch chrome!
2040
/builds/brhanuh/domestic-repair/node_modules/puppeteer/.local-chromium/linux-686378/chrome-linux/chrome:
error while loading shared libraries: libX11-xcb.so.1: cannot open
shared object file: No such file or directory
Install the missing packages using
sudo apt install -y gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget
Credits to #coldner on the puppetteer issue tracker for assembling the required pkgs.
If you encounter E: Unable to locate package errors, run sudo apt-get update.
I just wanted to add to #mohammad's answer, that now Puppetteer now has an official troublshooting guide that helps you determine what to install.
Also, they recommend running the following command to tell which particular packages need installing on *nix systems:
ldd chrome | grep not
Or, if you have Chrome installed via Puppetteer via NPM like me:
ldd ./node_modules/puppeteer/.local-chromium/linux-756035/chrome-linux/chrome | grep not
Then, I was able to see what libraries were still missing, and install accordingly.
you need to install required dependencies
here
https://github.com/puppeteer/puppeteer/blob/master/docs/troubleshooting.md
under Chrome headless doesn't launch on UNIX you can find
Debian (e.g. Ubuntu) Dependencies
CentOS Dependencies
For those that are still having this problem, you can try this.
sudo apt update && sudo apt install -y ca-certificates fonts-liberation libappindicator3-1 libasound2 libatk-bridge2.0-0 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgbm1 libgcc1 libglib2.0-0 libgtk-3-0 libnspr4 libnss3 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 lsb-release wget xdg-utils
I'm setting up the CI job for automated testing in selenium inside Gitlab CI, but the test is failing due to the issue.
019-09-27T11:03:17.404Z INFO #wdio/cli:Launcher: Run onPrepare hook
/builds/shauryav/test-react-ci-cd/node_modules/chromedriver/lib/chromedriver/chromedriver: error while loading shared libraries: libnss3.so: cannot open shared object file: No such file or directory
I have tried numbers of solutions like installing package "#wdio/cli": "^5.13.2", "webdriverio": "^5.13.2" but nothing works. For the note, I'm not using any docker setup
/builds/shauryav/test-react-ci-cd/node_modules/chromedriver/lib/chromedriver/chromedriver: error while loading shared libraries: libnss3.so: cannot open shared object file: No such file or directory
2019-09-27T11:03:27.415Z ERROR #wdio/cli:utils: A service failed in the 'onPrepare' hook
Error: timeout
at Timeout.timeoutFunc (/builds/shauryav/test-react-ci-cd/node_modules/tcp-port-used/index.js:204:25)
at listOnTimeout (internal/timers.js:531:17)
at processTimers (internal/timers.js:475:7)```
try these command
apt install libnss
apt install libnss3-dev libgdk-pixbuf2.0-dev libgtk-3-dev libxss-dev
if the above commands didn't work then go for the below one
sudo apt install libgconf-2-4 libatk1.0-0 libatk-bridge2.0-0 libgdk-pixbuf2.0-0 libgtk-3-0 libgbm-dev libnss3-dev libxss-dev
When you see this kind of thing, it means your OS is missing low-level libraries that are expected. This happens a bit with WSL because aside from WSLg it doesn't have a graphical environment and also the OS distributions are very minimal.
To resolve this (Debian/Ubuntu example),
Search the package manager's packages for the one that is missing
apt search libnss3
Determine a suitable candidate from the search results
Sorting... Done
Full Text Search... Done
libnss3/oldstable,now 2:3.42.1-1+deb10u5 amd64
Network Security Service libraries
libnss3-dev/oldstable 2:3.42.1-1+deb10u5 amd64
Development files for the Network Security Service libraries
libnss3-tools/oldstable 2:3.42.1-1+deb10u5 amd64
Network Security Service tools
nss-passwords/oldstable 0.2-2+b2 amd64
read passwords from a Mozilla keyring
Install the candidate
sudo apt install -y libnss3
Run your command again
If the error message goes away => success
If the error message changes => repeat for the new error message
If the error message does not change => the library you installed didn't have the files you need, so uninstall and try a different one
sudo apt purge -y libnss3 && sudo apt autoremove
You need to satisfy the libraries dependencies to run chrome in a headless mode:
sudo apt-get update && sudo apt-get install -y gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates fonts-liberation libnss3 lsb-release xdg-utils wget ca-certificates
For more
I'm using Browsershot (which uses Puppeteer) with Laravel. I have it running perfectly in my local dev environment (Mac, Laravel Valet), but when running it on a Forge provisioned server, I get the following error in my log file (I've replaced sensitive URLs with placeholders):
local.ERROR: The command "PATH=$PATH:/usr/local/bin NODE_PATH=`/usr/bin/node /usr/bin/npm root -g` /usr/bin/node '/home/forge/my-app.co.uk/vendor/spatie/browsershot/src/../bin/browser.js' '{"url":"https:\/\/my-test-site.com\/about.html","action":"screenshot","options":{"type":"png","path":"\/home\/forge\/my-app.com\/storage\/screenshots\/13\/full\/21.jpg","args":[],"viewport":{"width":1400,"height":1080},"fullPage":true,"waitUntil":"networkidle0"}}'" failed.
Exit Code: 1(General error)
I've followed the instructions from the readme for installing the latest version of stable Chrome:
curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -
sudo apt-get install -y nodejs gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget
sudo npm install --global --unsafe-perm puppeteer
sudo chmod -R o+rx /usr/lib/node_modules/puppeteer/.local-chromium
My code looks like this:
I've also tried specifying custom node and npm binaries. I've also tried a custom include path. When doing so, the error I get may vary slightly, but remains essentially the same.
I'm actually installing Puppeteer as part of my build process in Forge. I set that up before I noticed the instructions for installing Chrome in the Browsershot readme.
I'm just not sure how to interpret that Node error. I've tried everything I can think of. So I'm hoping someone can point me in the right direction.
Thanks!
I'm trying to run node apps that use the npm package Puppeteer to scrape websites, using VPSes that use root as the default user I login as.
It never works. I always have to spend half an hour googling to find the solution... well no more.
Do this:
0. Run sudo apt update
1. Install libx stuff
sudo apt-get install libpangocairo-1.0-0 libx11-xcb1 libxcomposite1 libxcursor1 libxdamage1 libxi6 libxtst6 libnss3 libcups2 libxss1 libxrandr2 libgconf2-4 libasound2 libatk1.0-0 libgtk-3-0
2. Set --no-sandbox argument in the launch() part of your Puppeteer app (only required if you are running as root/root's cron)
const browser = await puppeteer.launch({headless: true, args: ['--no-sandbox']})
E: as Niko pointed out, --no-sandbox opens your server to a malicious attack from a website the scraper visits. It looks like there are some SO questions elsewhere that post alternatives that aren't awful to implement. I like David's answer here which seems to suggest that even in 2021 --no-sandbox is still a necessary evil in some cases.
E2: See answer below. It looks like in 2021 we also have to install libgbm-dev
Connect to server by SSH
Open your terminal
Type: ssh root#[server ip]
If it is the first time you connect to the server you will get a security prompt. Type 'yes'
Fill in your password: [password]
You should now be connected to your server
Install app
sudo apt-get update
sudo apt-get install -y libgbm-dev
sudo apt install -y gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget
sudo apt install curl
curl -sL https://deb.nodesource.com/setup_14.x | sudo -E bash -
sudo apt install nodejs
git config --global user.name "your name"
git config --global user.email "your email"
git clone "your project git url"
cd "your project folder"
npm i
npm start
If you need to run multiple apps on one VPS server - google tool "Screen"
After I installed libgbm, everything works fine for me!
hope that help you out
sudo apt-get update
sudo apt-get install -y libgbm-dev