I have created an azure pipeline to deploy a custom docker image with an azure function with the following config. Though the config works in terms of build but none of the functions are running on the server and I notice that in the logs that none of the functions are being detected.
pool:
name: Azure Pipelines
steps:
- task: Docker#1
displayName: 'build: build the docker image'
inputs:
azureSubscriptionEndpoint: 'Azure subscription 1 (3fc75102-5159-4491-b61c-b3fcb845f648)'
azureContainerRegistry: whaiiapp.azurecr.io
imageName: 'azureapp.azurecr.io/puppeteer:$(Build.BuildId)'
useDefaultContext: false
- task: Docker#1
displayName: 'push the image'
inputs:
azureSubscriptionEndpoint: 'Azure subscription 1 (3fc75102-5159-4491-b61c-b3fcb845f648)'
azureContainerRegistry: azureapp.azurecr.io
command: 'Push an image'
imageName: 'azureap.azurecr.io/puppeteer:$(Build.BuildId)'
- task: FuncToolsInstaller#0
displayName: 'Install func tools - latest'
- task: AzureFunctionAppContainer#1
displayName: 'Start Azure Function App on Container Deploy: generate-report-dev'
inputs:
azureSubscription: 'Azure subscription 1 (3fc75102-5159-4491-b61c-b3fcb845f648)'
appName: 'generate-report-dev'
imageName: azureapp.azurecr.io/puppeteer
containerCommand: 'npm start'
The following is my Dockerfile config
FROM mcr.microsoft.com/azure-functions/node:3.0-appservice
# Chromium dependencies
RUN apt-get update -yq \
&& apt-get install -yq git git-extras gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 \
libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 \
libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 \
libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 \
fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst libgbm-dev \
ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget && \
wget https://github.com/Yelp/dumb-init/releases/download/v1.2.2/dumb-init_1.2.2_amd64.deb && \
dpkg -i dumb-init_*.deb && rm -f dumb-init_*.deb && \
apt-get clean && apt-get autoremove -y && rm -rf /var/lib/apt/lists/*
WORKDIR /home/site/wwwroot
RUN npm install --no-save puppeteer
ENV AzureWebJobsScriptRoot=/home/site/wwwroot \
AzureFunctionsJobHost__Logging__Console__IsEnabled=true \
FUNCTIONS_WORKER_RUNTIME=node \
COPY . /home/site/wwwroot
RUN cd /home/site/wwwroot && npm install
CMD [ "npm start" ]
I am not sure how I could go about debugging this. I tried logging into the server via ssh and running func start to test it and it shows func command is not available.
I believe your azure function is unable to pull the image from your registry, I did faced the same issue, so I simply created new function app, then selected the repository in deployment process, so it is going to create a webhook behind the scene, that will pull the latest image from a registry, every time whenever it is updated.
Related
I deployed node.js app to App Service.
Initially I adopted node14 runtime App Service provided.
However, puppeteer doesn't work on node14 runtime, I created custom container to use puppeteer/Headless Chrome.
I can successfully deploy, but server doesn't open.
Would anyone let me know what did I miss?
FROM node:14
# RUN apt-get update && apt-get install -yq libgconf-2-4
# Install latest chrome dev package and fonts to support major charsets (Chinese, Japanese, Arabic, Hebrew, Thai and a few others)
# Note: this installs the necessary libs to make the bundled version of Chromium that Puppeteer
# installs, work.
RUN apt-get update && apt-get install -yq libgconf-2-4 \
&& apt-get update && apt-get install -y wget --no-install-recommends \
&& wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
&& sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' \
&& apt-get update \
&& apt-get install -y google-chrome-unstable fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst ttf-freefont \
--no-install-recommends \
&& rm -rf /var/lib/apt/lists/* \
&& apt-get purge --auto-remove -y curl \
&& rm -rf /src/*.deb
# It's a good idea to use dumb-init to help prevent zombie chrome processes.
ADD https://github.com/Yelp/dumb-init/releases/download/v1.2.0/dumb-init_1.2.0_amd64 /usr/local/bin/dumb-init
RUN chmod +x /usr/local/bin/dumb-init
# Uncomment to skip the chromium download when installing puppeteer. If you do,
# you'll need to launch puppeteer with:
# browser.launch({executablePath: 'google-chrome-unstable'})
# ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD true
# Headfull mode
# Install dependencies
# To run Headful mode, you will need to have a display, which is not present in a server.
# To avoid this, we will use Xvfb, and create a fake display, so the chrome will think there is a display and run properly.
# So we just need to install Xvfb and Puppeteer related dependencies.
RUN apt-get update && apt-get install -yq gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget x11vnc x11-xkb-utils xfonts-100dpi xfonts-75dpi xfonts-scalable xfonts-cyrillic x11-apps xvfb
# Install puppeteer so it's available in the container.
RUN npm i puppeteer
# # Add user so we don't need --no-sandbox.
# RUN groupadd -r pptruser && useradd -r -g pptruser -G audio,video pptruser \
# && mkdir -p /home/pptruser/Downloads \
# && chown -R pptruser:pptruser /home/pptruser \
# && chown -R pptruser:pptruser /node_modules
RUN apt-get update && \
apt-get -y install sudo curl
RUN useradd -m docker && echo "docker:docker" | chpasswd && adduser docker sudo
USER docker
You need to add the CMD and ENTRYPOINT to start your application in the Dockerfile. You missed it. So When you create the image from the Dockerfile and deploy it to the Web App, there is no application run as you expect.
I am using puppeteer to execute some test cases in Docker and I get this below error:
"before all" hook: codeceptjs.beforeSuite for "Validate_onbord_broker_business":
Could not find browser revision 756035. Run "npm install" or "yarn install" to download a browser binary.
at ChromeLauncher.launch (node_modules/puppeteer/lib/Launcher.js:59:23)
at async Puppeteer._startBrowser (node_modules/codeceptjs/lib/helper/Puppeteer.js:512:22)
And this is the Dockerfile I am using:
# Use whatever version you are running locally (see node -v)
FROM node:12.18
WORKDIR /app
RUN apt-get update \
&& apt-get install -y wget gnupg ca-certificates \
&& wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
&& sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' \
&& apt-get update \
&& apt-get install -y google-chrome-stable \
&& rm -rf /var/lib/apt/lists/* \
&& wget --quiet https://raw.githubusercontent.com/vishnubob/wait-for-it/master/wait-for-it.sh -O /usr/sbin/wait-for-it.sh \
&& chmod +x /usr/sbin/wait-for-it.sh
# Install dependencies (you are already in /app)
COPY package.json package-lock.json ./
# RUN npm ci
RUN npm install
ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD true
RUN npm install codeceptjs puppeteer
COPY . /app
RUN pwd
RUN ls
# RUN npx codeceptjs init
RUN npx codeceptjs run
# CMD ["npm", "start"]
Can someone please help me what is going wrong?
I think the problem is that you're skipping downloading a version of chromium when you install your dependencies: ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD true. Puppeteer is only guaranteed to work with the bundled version of chromium and it's generally a bad idea to use some other version of chromium.
Based on your comments on your post, I'm guessing that the reason you're not using the bundled version of Chromium is because you're having "fun" with dependencies. These can be manually installed using apt - here are the dependencies which are installed by the official puppeteer dockerfile:
apt-get -y install xvfb gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 \
libdbus-1-3 libexpat1 libfontconfig1 libgbm1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 \
libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 \
libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 \
libxtst6 ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget
I think this is the updated Dockerfile - let me know if it works because I can't test it without the rest of your code!
# Use whatever version you are running locally (see node -v)
FROM node:12.18
WORKDIR /app
RUN apt-get update \
&& apt-get -y install xvfb gconf-service gnupg libasound2 libatk1.0-0 libc6 libcairo2 libcups2 \
libdbus-1-3 libexpat1 libfontconfig1 libgbm1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 \
libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 \
libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 \
libxtst6 ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget \
&& wget --quiet https://raw.githubusercontent.com/vishnubob/wait-for-it/master/wait-for-it.sh -O /usr/sbin/wait-for-it.sh \
&& chmod +x /usr/sbin/wait-for-it.sh
# Install dependencies (you are already in /app)
COPY package.json package-lock.json ./
# RUN npm ci
RUN npm install
RUN npm install codeceptjs puppeteer
COPY . /app
RUN pwd
RUN ls
# RUN npx codeceptjs init
RUN npx codeceptjs run
# CMD ["npm", "start"]
Im trying to install Heroku Exec on a Docker container so that I can run scripts through a shell. I cannot access a shell on the container. Ive followed the below linked instructions on using heroku ps:exec with Docker, but get this error when running heroku:ps exec . You can see Ive installed the packages in my dockerfile, and I created a /.profile.d folder in my base /app directory, and added the heroku-exec.sh file and pasted the line of code they wanted. seems like Heroku Exec is not installed or working though.
establishing credentials... error
Heroku Exec is not running!
▸ Could not connect to dyno!
▸ Check if the dyno is running with heroku ps
Heroku Exec with Docker:
https://devcenter.heroku.com/articles/exec#using-with-docker
Dockerfile
FROM node:latest
# update and add all the steps for running with xvfb
RUN apt-get update &&\
apt-get install -yq gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 \
libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 \
libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 \
libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 \
ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget \
xvfb x11vnc x11-xkb-utils xfonts-100dpi xfonts-75dpi xfonts-scalable xfonts-cyrillic x11-apps \
&& apt-get install -y wget gnupg \
&& wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
&& sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' \
&& apt-get update \
&& apt-get install -y google-chrome-stable fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst fonts-freefont-ttf libxss1 \
--no-install-recommends \
&& rm -rf /var/lib/apt/lists/*
RUN apt-get install bash
RUN apt-get install python
RUN apt-get install curl
RUN apt-get install openssh-client
#remove zombie images
ADD https://github.com/Yelp/dumb-init/releases/download/v1.2.2/dumb-init_1.2.2_x86_64 /usr/local/bin/dumb-init
RUN chmod +x /usr/local/bin/dumb-init
ENTRYPOINT ["dumb-init", "--"]
# WORKDIR /app
# using /tmp in order to read/write to container fs as non-root user
WORKDIR /tmp
# add the required dependencies - this breaks the build process for some reason, manually npm install below works
# COPY node_modules /app/node_modules
RUN npm install puppeteer \
# Add user so we don't need --no-sandbox.
# same layer as npm install to keep re-chowned files from using up several hundred MBs more space
&& groupadd -r pptruser && useradd -r -g pptruser -G audio,video pptruser \
&& mkdir -p /home/pptruser/Downloads \
&& chown -R pptruser:pptruser /home/pptruser \
&& npm install express \
&& npm install basic-ftp
# && chown -R pptruser:pptruser /node_modules
# For Heroku exec
RUN rm /bin/sh && ln -s /bin/bash /bin/sh
ADD ./.profile.d /app/.profile.d
EXPOSE 3000
# Run everything after as non-privileged user.
USER pptruser
# USER root
# Finally copy the build application
COPY . .
# make sure we can run without a UI
ENV DISPLAY :99
CMD Xvfb :99 -screen 0 1024x768x16 -nolisten unix & node server.js
Im trying to run a Puppeteer script in a docker container with Xfvb so that I can run headless: false on my production app, which is the only way my script will get the required output from the site I am scraping. Im having trouble getting the docker image to run after Ive built it. I originally following this article's process: http://www.smartjava.org/content/using-puppeteer-in-docker-copy-2/
but I was getting the error
Error: Could not find browser revision 818858 Run "PUPPETEER_PRODUCT=firefox npm install"
once I tried to run the image, which didnt make much sense to me but it seems like the bundled version of Chromium didnt have proper dependencies according to this article
Running Puppeteer in Docker:
https://github.com/puppeteer/puppeteer/blob/main/docs/troubleshooting.md#running-puppeteer-in-docker
So I modified my dockerfile with the ideas they used in their example, which allowed me to build my container with no errors. But when I run the image I get an error that Im stumped on. I believe it has to do with Xfvb.
_XSERVTransmkdir: ERROR: euid != 0,directory /tmp/.X11-unix will not be created.
Dockerfile:
FROM node:latest
# update and add all the steps for running with xvfb
RUN apt-get update &&\
apt-get install -yq gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 \
libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 \
libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 \
libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 \
ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget \
xvfb x11vnc x11-xkb-utils xfonts-100dpi xfonts-75dpi xfonts-scalable xfonts-cyrillic x11-apps
# this installs the necessary libs to make the bundled version of Chromium
# that Puppeteer installs, work.
RUN apt-get update \
&& apt-get install -y wget gnupg \
&& wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
&& sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' \
&& apt-get update \
&& apt-get install -y google-chrome-stable fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst fonts-freefont-ttf libxss1 \
--no-install-recommends \
&& rm -rf /var/lib/apt/lists/*
ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD true
# add the required dependencies
WORKDIR /app
COPY node_modules /app/node_modules
RUN npm install puppeteer \
# Add user so we don't need --no-sandbox.
# same layer as npm install to keep re-chowned files from using up several hundred MBs more space
&& groupadd -r pptruser && useradd -r -g pptruser -G audio,video pptruser \
&& mkdir -p /home/pptruser/Downloads \
&& chown -R pptruser:pptruser /home/pptruser
# && chown -R pptruser:pptruser /node_modules
# Run everything after as non-privileged user.
USER pptruser
# Finally copy the build application
COPY . .
# make sure we can run without a UI
ENV DISPLAY :99
CMD Xvfb :99 -screen 0 1024x768x16 & node ./src/export.js
Package.json
{
"name": "puppeteer-headless",
"version": "1.0.0",
"description": "Headless crawler with simulated UI",
"devDependencies": {
"#types/node": "^14.4.0",
"#types/puppeteer": "^5.4.0"
},
"dependencies": {
"puppeteer": "^5.5.0"
}
}
launch method in my script:
const browser = await puppeteer.launch({
headless: false,
executablePath: 'google-chrome-stable'
});
Hello I am new to gitlab, and so far i succeded in doing everything (runners, pipelines, etc...)
But I now want to set automatic tests build only once per day, triggered by a CRON but NOT TRIGGERED by each push made on the git repo.
I am using a 8.14.2 gitlab without the new SCHEDULE feature
So I used this gitlab.ci.yml:
stages:
- test
test-karma:
stage: test
image: node:8.9.4
before_script:
- apt-get update
- apt-get --yes install npm
- apt-get install -yq gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3
libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4
libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1
libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6
ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget
- npm install #angular/cli#1.4.5
- npm install
environment: dev
script:
- ./node_modules/.bin/ng test --single-run=true --watch=false
only:
- triggers
except:
- pushes
But despite the only / except part, the job is triggered :
() For each git push (which i don't want)
() By the CRON which I want
So I found another way
1- Set a variable SHOULD_EXECUTE to false on gitlab
2- Protect every script line by an if statement
3- Use the following CRON:
00 4 * * * curl -X POST -F token=MY_TOKEN -F ref=master -F "variables[SOULD_EXECUTE]=true" http://MY_GITLAB_URL/api/v3/projects/30/trigger/builds
4- And changed the gitlab.ci.yml to
stages:
- test
test-karma:
stage: test
image: node:8.9.4
before_script:
- if [ ${SOULD_EXECUTE} == "true" ]; then apt-get update;fi
- if [ ${SOULD_EXECUTE} == "true" ]; then apt-get --yes install npm ;fi
- if [ ${SOULD_EXECUTE} == "true" ]; then apt-get install -yq gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3
libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4
libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1
libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6
ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget;fi
- if [ ${SOULD_EXECUTE} == "true" ]; then npm install #angular/cli#1.4.5;fi
- if [ ${SOULD_EXECUTE} == "true" ]; then npm install; else echo "TEST JOB SKIPPED";fi
- echo 'SOULD_EXECUTE => ' + ${SOULD_EXECUTE}
environment: dev
script:
- if [ ${SOULD_EXECUTE} == "true" ]; then ./node_modules/.bin/ng test --single-run=true --watch=false;fi
only:
- triggers
It's Ugly... Could some one help me to set this up ? Thank you so much.
regards.
The below also works without using except
only:
- schedules
I fixed my issue without knowing how or why. I had to move the repo to another group in gitlab. And now it's working perfectly, the only : triggers (without except : pushes) enable me to NOT run a pipeline for each pushes, and I can run this with the CRON post call.