How to Fix Could not find browser revision 756035 - node.js

I am using puppeteer to execute some test cases in Docker and I get this below error:
"before all" hook: codeceptjs.beforeSuite for "Validate_onbord_broker_business":
Could not find browser revision 756035. Run "npm install" or "yarn install" to download a browser binary.
at ChromeLauncher.launch (node_modules/puppeteer/lib/Launcher.js:59:23)
at async Puppeteer._startBrowser (node_modules/codeceptjs/lib/helper/Puppeteer.js:512:22)
And this is the Dockerfile I am using:
# Use whatever version you are running locally (see node -v)
FROM node:12.18
WORKDIR /app
RUN apt-get update \
&& apt-get install -y wget gnupg ca-certificates \
&& wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
&& sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' \
&& apt-get update \
&& apt-get install -y google-chrome-stable \
&& rm -rf /var/lib/apt/lists/* \
&& wget --quiet https://raw.githubusercontent.com/vishnubob/wait-for-it/master/wait-for-it.sh -O /usr/sbin/wait-for-it.sh \
&& chmod +x /usr/sbin/wait-for-it.sh
# Install dependencies (you are already in /app)
COPY package.json package-lock.json ./
# RUN npm ci
RUN npm install
ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD true
RUN npm install codeceptjs puppeteer
COPY . /app
RUN pwd
RUN ls
# RUN npx codeceptjs init
RUN npx codeceptjs run
# CMD ["npm", "start"]
Can someone please help me what is going wrong?

I think the problem is that you're skipping downloading a version of chromium when you install your dependencies: ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD true. Puppeteer is only guaranteed to work with the bundled version of chromium and it's generally a bad idea to use some other version of chromium.
Based on your comments on your post, I'm guessing that the reason you're not using the bundled version of Chromium is because you're having "fun" with dependencies. These can be manually installed using apt - here are the dependencies which are installed by the official puppeteer dockerfile:
apt-get -y install xvfb gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 \
libdbus-1-3 libexpat1 libfontconfig1 libgbm1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 \
libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 \
libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 \
libxtst6 ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget
I think this is the updated Dockerfile - let me know if it works because I can't test it without the rest of your code!
# Use whatever version you are running locally (see node -v)
FROM node:12.18
WORKDIR /app
RUN apt-get update \
&& apt-get -y install xvfb gconf-service gnupg libasound2 libatk1.0-0 libc6 libcairo2 libcups2 \
libdbus-1-3 libexpat1 libfontconfig1 libgbm1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 \
libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 \
libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 \
libxtst6 ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget \
&& wget --quiet https://raw.githubusercontent.com/vishnubob/wait-for-it/master/wait-for-it.sh -O /usr/sbin/wait-for-it.sh \
&& chmod +x /usr/sbin/wait-for-it.sh
# Install dependencies (you are already in /app)
COPY package.json package-lock.json ./
# RUN npm ci
RUN npm install
RUN npm install codeceptjs puppeteer
COPY . /app
RUN pwd
RUN ls
# RUN npx codeceptjs init
RUN npx codeceptjs run
# CMD ["npm", "start"]

Related

Node.js application doesn't start on Azure Web App with custom container

I deployed node.js app to App Service.
Initially I adopted node14 runtime App Service provided.
However, puppeteer doesn't work on node14 runtime, I created custom container to use puppeteer/Headless Chrome.
I can successfully deploy, but server doesn't open.
Would anyone let me know what did I miss?
FROM node:14
# RUN apt-get update && apt-get install -yq libgconf-2-4
# Install latest chrome dev package and fonts to support major charsets (Chinese, Japanese, Arabic, Hebrew, Thai and a few others)
# Note: this installs the necessary libs to make the bundled version of Chromium that Puppeteer
# installs, work.
RUN apt-get update && apt-get install -yq libgconf-2-4 \
&& apt-get update && apt-get install -y wget --no-install-recommends \
&& wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
&& sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' \
&& apt-get update \
&& apt-get install -y google-chrome-unstable fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst ttf-freefont \
--no-install-recommends \
&& rm -rf /var/lib/apt/lists/* \
&& apt-get purge --auto-remove -y curl \
&& rm -rf /src/*.deb
# It's a good idea to use dumb-init to help prevent zombie chrome processes.
ADD https://github.com/Yelp/dumb-init/releases/download/v1.2.0/dumb-init_1.2.0_amd64 /usr/local/bin/dumb-init
RUN chmod +x /usr/local/bin/dumb-init
# Uncomment to skip the chromium download when installing puppeteer. If you do,
# you'll need to launch puppeteer with:
# browser.launch({executablePath: 'google-chrome-unstable'})
# ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD true
# Headfull mode
# Install dependencies
# To run Headful mode, you will need to have a display, which is not present in a server.
# To avoid this, we will use Xvfb, and create a fake display, so the chrome will think there is a display and run properly.
# So we just need to install Xvfb and Puppeteer related dependencies.
RUN apt-get update && apt-get install -yq gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget x11vnc x11-xkb-utils xfonts-100dpi xfonts-75dpi xfonts-scalable xfonts-cyrillic x11-apps xvfb
# Install puppeteer so it's available in the container.
RUN npm i puppeteer
# # Add user so we don't need --no-sandbox.
# RUN groupadd -r pptruser && useradd -r -g pptruser -G audio,video pptruser \
# && mkdir -p /home/pptruser/Downloads \
# && chown -R pptruser:pptruser /home/pptruser \
# && chown -R pptruser:pptruser /node_modules
RUN apt-get update && \
apt-get -y install sudo curl
RUN useradd -m docker && echo "docker:docker" | chpasswd && adduser docker sudo
USER docker
You need to add the CMD and ENTRYPOINT to start your application in the Dockerfile. You missed it. So When you create the image from the Dockerfile and deploy it to the Web App, there is no application run as you expect.

Install a Firefox browser binary for Puppeteer in Docker

I'm trying to run a script with a dependency of https://github.com/pevers/images-scraper in a Docker container. I can run the script on its own (on my machine which has Node 14.15.4, doing node src/index.js which will yield the console outout of A), but in the context of docker it is not working - I am getting an error message about a missing Firefox binary.
Dockerfile:
FROM node:14
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install
RUN PUPPETEER_PRODUCT=firefox npm install puppeteer
COPY . .
CMD [ "node", "src/index.js" ]
src/index.js:
let Scraper = require("images-scraper");
const google = new Scraper({
puppeteer: {
userAgent:
"Mozilla/5.0 (X11; Linux i686; rv:64.0) Gecko/20100101 Firefox/64.0",
headless: true,
safe: true,
},
});
(async () => {
const A = await google.scrape("bananas", 200);
console.log(A)
})();
package.json
{
"name": "test",
"version": "1.0.0",
"scripts": {
"test": "node src/index.js"
},
"dependencies": {
"images-scraper": "^6.2.1"
}
}
When I /bin/bash into my container and do node src/index.js I get the error:
at ChromeLauncher.launch (/usr/src/app/node_modules/puppeteer/lib/cjs/puppeteer/node/Launcher.js:79:23)
at async GoogleScraper.scrape (/usr/src/app/node_modules/images-scraper/src/google/scraper.js:53:21)
at async /usr/src/app/src/index.js:19:13
New to using Node with Docker - I was wondering where I am going wrong here. Thanks
Install firefox execution for your container:
FROM node:14
RUN apt-get update \
&& apt-get install -y wget gnupg fonts-ipafont-gothic fonts-freefont-ttf firefox-esr --no-install-recommends \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install
RUN PUPPETEER_PRODUCT=firefox npm install puppeteer
COPY . .
CMD [ "node", "src/index.js" ]
the error state that there is errors by running chrome not Firefox ....
so make sure that you have all chrome dependencies installed in your Container:
ca-certificates
fonts-liberation
libappindicator3-1
libasound2
libatk-bridge2.0-0
libatk1.0-0
libc6
libcairo2
libcups2
libdbus-1-3
libexpat1
libfontconfig1
libgbm1
libgcc1
libglib2.0-0
libgtk-3-0
libnspr4
libnss3
libpango-1.0-0
libpangocairo-1.0-0
libstdc++6
libx11-6
libx11-xcb1
libxcb1
libxcomposite1
libxcursor1
libxdamage1
libxext6
libxfixes3
libxi6
libxrandr2
libxrender1
libxss1
libxtst6
lsb-release
wget
xdg-utils
source : Here
Following on from #LinPy's comment, adapting my Dockerfile per the puppeteer documentation worked:
FROM alpine:edge
# Installs latest Chromium (89) package.
RUN apk add --no-cache \
chromium \
nss \
freetype \
freetype-dev \
harfbuzz \
ca-certificates \
ttf-freefont \
nodejs \
nodejs-npm \
yarn
# Tell Puppeteer to skip installing Chrome. We'll be using the installed package.
ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true \
PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium-browser
# Puppeteer v6.0.0 works with Chromium 89.
RUN yarn add puppeteer#6.0.0
...
As well I added args: ['--no-sandbox', '--disable-setuid-sandbox'] to the Scraper constructor. Thanks

Heroku Exec on Docker container not installing

Im trying to install Heroku Exec on a Docker container so that I can run scripts through a shell. I cannot access a shell on the container. Ive followed the below linked instructions on using heroku ps:exec with Docker, but get this error when running heroku:ps exec . You can see Ive installed the packages in my dockerfile, and I created a /.profile.d folder in my base /app directory, and added the heroku-exec.sh file and pasted the line of code they wanted. seems like Heroku Exec is not installed or working though.
establishing credentials... error
Heroku Exec is not running!
▸ Could not connect to dyno!
▸ Check if the dyno is running with heroku ps
Heroku Exec with Docker:
https://devcenter.heroku.com/articles/exec#using-with-docker
Dockerfile
FROM node:latest
# update and add all the steps for running with xvfb
RUN apt-get update &&\
apt-get install -yq gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 \
libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 \
libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 \
libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 \
ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget \
xvfb x11vnc x11-xkb-utils xfonts-100dpi xfonts-75dpi xfonts-scalable xfonts-cyrillic x11-apps \
&& apt-get install -y wget gnupg \
&& wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
&& sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' \
&& apt-get update \
&& apt-get install -y google-chrome-stable fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst fonts-freefont-ttf libxss1 \
--no-install-recommends \
&& rm -rf /var/lib/apt/lists/*
RUN apt-get install bash
RUN apt-get install python
RUN apt-get install curl
RUN apt-get install openssh-client
#remove zombie images
ADD https://github.com/Yelp/dumb-init/releases/download/v1.2.2/dumb-init_1.2.2_x86_64 /usr/local/bin/dumb-init
RUN chmod +x /usr/local/bin/dumb-init
ENTRYPOINT ["dumb-init", "--"]
# WORKDIR /app
# using /tmp in order to read/write to container fs as non-root user
WORKDIR /tmp
# add the required dependencies - this breaks the build process for some reason, manually npm install below works
# COPY node_modules /app/node_modules
RUN npm install puppeteer \
# Add user so we don't need --no-sandbox.
# same layer as npm install to keep re-chowned files from using up several hundred MBs more space
&& groupadd -r pptruser && useradd -r -g pptruser -G audio,video pptruser \
&& mkdir -p /home/pptruser/Downloads \
&& chown -R pptruser:pptruser /home/pptruser \
&& npm install express \
&& npm install basic-ftp
# && chown -R pptruser:pptruser /node_modules
# For Heroku exec
RUN rm /bin/sh && ln -s /bin/bash /bin/sh
ADD ./.profile.d /app/.profile.d
EXPOSE 3000
# Run everything after as non-privileged user.
USER pptruser
# USER root
# Finally copy the build application
COPY . .
# make sure we can run without a UI
ENV DISPLAY :99
CMD Xvfb :99 -screen 0 1024x768x16 -nolisten unix & node server.js

Running Headless Chrome Puppeteer and Xfvb in a Node.js Docker Container, trouble running image

Im trying to run a Puppeteer script in a docker container with Xfvb so that I can run headless: false on my production app, which is the only way my script will get the required output from the site I am scraping. Im having trouble getting the docker image to run after Ive built it. I originally following this article's process: http://www.smartjava.org/content/using-puppeteer-in-docker-copy-2/
but I was getting the error
Error: Could not find browser revision 818858 Run "PUPPETEER_PRODUCT=firefox npm install"
once I tried to run the image, which didnt make much sense to me but it seems like the bundled version of Chromium didnt have proper dependencies according to this article
Running Puppeteer in Docker:
https://github.com/puppeteer/puppeteer/blob/main/docs/troubleshooting.md#running-puppeteer-in-docker
So I modified my dockerfile with the ideas they used in their example, which allowed me to build my container with no errors. But when I run the image I get an error that Im stumped on. I believe it has to do with Xfvb.
_XSERVTransmkdir: ERROR: euid != 0,directory /tmp/.X11-unix will not be created.
Dockerfile:
FROM node:latest
# update and add all the steps for running with xvfb
RUN apt-get update &&\
apt-get install -yq gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 \
libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 \
libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 \
libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 \
ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget \
xvfb x11vnc x11-xkb-utils xfonts-100dpi xfonts-75dpi xfonts-scalable xfonts-cyrillic x11-apps
# this installs the necessary libs to make the bundled version of Chromium
# that Puppeteer installs, work.
RUN apt-get update \
&& apt-get install -y wget gnupg \
&& wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
&& sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' \
&& apt-get update \
&& apt-get install -y google-chrome-stable fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst fonts-freefont-ttf libxss1 \
--no-install-recommends \
&& rm -rf /var/lib/apt/lists/*
ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD true
# add the required dependencies
WORKDIR /app
COPY node_modules /app/node_modules
RUN npm install puppeteer \
# Add user so we don't need --no-sandbox.
# same layer as npm install to keep re-chowned files from using up several hundred MBs more space
&& groupadd -r pptruser && useradd -r -g pptruser -G audio,video pptruser \
&& mkdir -p /home/pptruser/Downloads \
&& chown -R pptruser:pptruser /home/pptruser
# && chown -R pptruser:pptruser /node_modules
# Run everything after as non-privileged user.
USER pptruser
# Finally copy the build application
COPY . .
# make sure we can run without a UI
ENV DISPLAY :99
CMD Xvfb :99 -screen 0 1024x768x16 & node ./src/export.js
Package.json
{
"name": "puppeteer-headless",
"version": "1.0.0",
"description": "Headless crawler with simulated UI",
"devDependencies": {
"#types/node": "^14.4.0",
"#types/puppeteer": "^5.4.0"
},
"dependencies": {
"puppeteer": "^5.5.0"
}
}
launch method in my script:
const browser = await puppeteer.launch({
headless: false,
executablePath: 'google-chrome-stable'
});

Deploying a Dockerized node app onto Heroku and using Xvfb with puppeteer, cannot launch browser

Puppeteer version: 5.5
Platform / OS version: Docker container deployed on Heroku
Node.js version: latest
I have set up puppeteer scripts that live in a docker container, so that they can be run with Xvfb, a simulated UI so that the scripts can be run in headless:false mode. The site they crawl causes this to be a requirement.
The docker container builds fine, and the image runs locally. I can run the scripts and the Xfvb simulated display launches and allows the browser to launch as intended. The scripts perform as expected in my local docker image.
Ive deployed this to Heroku and spun up a dyno, and have tried to run the scripts on the dyno. The dyno runs fine, the server launches and listens for requests. When I try to run the node scripts, It seems that the browser isnt able to connect to or launch the simulated display.
Xvfb:
https://www.x.org/releases/X11R7.6/doc/man/man1/Xvfb.1.xhtml
Error when trying to run scripts on the heroku dyno
steveszumski#Steves-MacBook-Pro crawler % heroku run node crawler.js
Running node crawler.js on ⬢ suresale-crawler... up, run.3678 (Free)
Running getReport Headless Script
Error: Failed to launch the browser process!
Fontconfig warning: "/etc/fonts/fonts.conf", line 100: unknown element "blank"
[15:15:1211/184429.509801:ERROR:browser_main_loop.cc(1439)] Unable to open X display.
[1211/184429.534086:ERROR:nacl_helper_linux.cc(307)] NaCl helper process running without a sandbox!
Most likely you need to configure your SUID sandbox correctly
TROUBLESHOOTING: https://github.com/puppeteer/puppeteer/blob/main/docs/troubleshooting.md
at onClose (/app/node_modules/puppeteer/lib/cjs/puppeteer/node/BrowserRunner.js:193:20)
at ChildProcess.<anonymous> (/app/node_modules/puppeteer/lib/cjs/puppeteer/node/BrowserRunner.js:184:79)
at ChildProcess.emit (node:events:388:22)
at Process.ChildProcess._handle.onexit (node:internal/child_process:284:12)
Section of crawler.js that errors
try {
const browser = await puppeteer.launch({
headless: false,
args: [
'--no-sandbox'
]
});
const page = await browser.newPage();
Dockerfile
FROM node:latest
# update and add all the steps for running with xvfb
RUN apt-get update &&\
apt-get install -yq gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 \
libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 \
libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 \
libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 \
ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget \
xvfb x11vnc x11-xkb-utils xfonts-100dpi xfonts-75dpi xfonts-scalable xfonts-cyrillic x11-apps \
&& apt-get install -y wget gnupg \
&& wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
&& sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' \
&& apt-get update \
&& apt-get install -y google-chrome-stable fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst fonts-freefont-ttf libxss1 \
--no-install-recommends \
&& rm -rf /var/lib/apt/lists/*
RUN apt-get install bash
RUN apt-get install python
RUN apt-get install curl
RUN apt-get install openssh-client
#remove zombie images
ADD https://github.com/Yelp/dumb-init/releases/download/v1.2.2/dumb-init_1.2.2_x86_64 /usr/local/bin/dumb-init
RUN chmod +x /usr/local/bin/dumb-init
ENTRYPOINT ["dumb-init", "--"]
WORKDIR /app
# using /tmp in order to read/write to container fs as non-root user
# WORKDIR /tmp
# add the required dependencies - this breaks the build process for some reason, manually npm install below works
# COPY node_modules /app/node_modules
RUN npm install puppeteer \
# Add user so we don't need --no-sandbox.
# same layer as npm install to keep re-chowned files from using up several hundred MBs more space
&& groupadd -r pptruser && useradd -r -g pptruser -G audio,video pptruser \
&& mkdir -p /home/pptruser/Downloads \
&& chown -R pptruser:pptruser /home/pptruser \
&& npm install express \
&& npm install basic-ftp
# && chown -R pptruser:pptruser /node_modules
# For Heroku exec
RUN rm /bin/sh && ln -s /bin/bash /bin/sh
ADD ./.profile.d /app/.profile.d
EXPOSE 3000
# Run everything after as non-privileged user.
USER pptruser
# USER root
# Finally copy the build application
COPY . .
# make sure we can run without a UI
ENV DISPLAY :99
CMD Xvfb :99 -screen 0 1024x768x16 -nolisten unix & node server.js

Resources