Gitlab CI - Run pipeline ONLY when triggered by CRON - gitlab

Hello I am new to gitlab, and so far i succeded in doing everything (runners, pipelines, etc...)
But I now want to set automatic tests build only once per day, triggered by a CRON but NOT TRIGGERED by each push made on the git repo.
I am using a 8.14.2 gitlab without the new SCHEDULE feature
So I used this gitlab.ci.yml:
stages:
- test
test-karma:
stage: test
image: node:8.9.4
before_script:
- apt-get update
- apt-get --yes install npm
- apt-get install -yq gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3
libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4
libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1
libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6
ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget
- npm install #angular/cli#1.4.5
- npm install
environment: dev
script:
- ./node_modules/.bin/ng test --single-run=true --watch=false
only:
- triggers
except:
- pushes
But despite the only / except part, the job is triggered :
() For each git push (which i don't want)
() By the CRON which I want
So I found another way
1- Set a variable SHOULD_EXECUTE to false on gitlab
2- Protect every script line by an if statement
3- Use the following CRON:
00 4 * * * curl -X POST -F token=MY_TOKEN -F ref=master -F "variables[SOULD_EXECUTE]=true" http://MY_GITLAB_URL/api/v3/projects/30/trigger/builds
4- And changed the gitlab.ci.yml to
stages:
- test
test-karma:
stage: test
image: node:8.9.4
before_script:
- if [ ${SOULD_EXECUTE} == "true" ]; then apt-get update;fi
- if [ ${SOULD_EXECUTE} == "true" ]; then apt-get --yes install npm ;fi
- if [ ${SOULD_EXECUTE} == "true" ]; then apt-get install -yq gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3
libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4
libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1
libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6
ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget;fi
- if [ ${SOULD_EXECUTE} == "true" ]; then npm install #angular/cli#1.4.5;fi
- if [ ${SOULD_EXECUTE} == "true" ]; then npm install; else echo "TEST JOB SKIPPED";fi
- echo 'SOULD_EXECUTE => ' + ${SOULD_EXECUTE}
environment: dev
script:
- if [ ${SOULD_EXECUTE} == "true" ]; then ./node_modules/.bin/ng test --single-run=true --watch=false;fi
only:
- triggers
It's Ugly... Could some one help me to set this up ? Thank you so much.
regards.

The below also works without using except
only:
- schedules

I fixed my issue without knowing how or why. I had to move the repo to another group in gitlab. And now it's working perfectly, the only : triggers (without except : pushes) enable me to NOT run a pipeline for each pushes, and I can run this with the CRON post call.

Related

How to Fix Could not find browser revision 756035

I am using puppeteer to execute some test cases in Docker and I get this below error:
"before all" hook: codeceptjs.beforeSuite for "Validate_onbord_broker_business":
Could not find browser revision 756035. Run "npm install" or "yarn install" to download a browser binary.
at ChromeLauncher.launch (node_modules/puppeteer/lib/Launcher.js:59:23)
at async Puppeteer._startBrowser (node_modules/codeceptjs/lib/helper/Puppeteer.js:512:22)
And this is the Dockerfile I am using:
# Use whatever version you are running locally (see node -v)
FROM node:12.18
WORKDIR /app
RUN apt-get update \
&& apt-get install -y wget gnupg ca-certificates \
&& wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
&& sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' \
&& apt-get update \
&& apt-get install -y google-chrome-stable \
&& rm -rf /var/lib/apt/lists/* \
&& wget --quiet https://raw.githubusercontent.com/vishnubob/wait-for-it/master/wait-for-it.sh -O /usr/sbin/wait-for-it.sh \
&& chmod +x /usr/sbin/wait-for-it.sh
# Install dependencies (you are already in /app)
COPY package.json package-lock.json ./
# RUN npm ci
RUN npm install
ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD true
RUN npm install codeceptjs puppeteer
COPY . /app
RUN pwd
RUN ls
# RUN npx codeceptjs init
RUN npx codeceptjs run
# CMD ["npm", "start"]
Can someone please help me what is going wrong?
I think the problem is that you're skipping downloading a version of chromium when you install your dependencies: ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD true. Puppeteer is only guaranteed to work with the bundled version of chromium and it's generally a bad idea to use some other version of chromium.
Based on your comments on your post, I'm guessing that the reason you're not using the bundled version of Chromium is because you're having "fun" with dependencies. These can be manually installed using apt - here are the dependencies which are installed by the official puppeteer dockerfile:
apt-get -y install xvfb gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 \
libdbus-1-3 libexpat1 libfontconfig1 libgbm1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 \
libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 \
libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 \
libxtst6 ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget
I think this is the updated Dockerfile - let me know if it works because I can't test it without the rest of your code!
# Use whatever version you are running locally (see node -v)
FROM node:12.18
WORKDIR /app
RUN apt-get update \
&& apt-get -y install xvfb gconf-service gnupg libasound2 libatk1.0-0 libc6 libcairo2 libcups2 \
libdbus-1-3 libexpat1 libfontconfig1 libgbm1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 \
libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 \
libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 \
libxtst6 ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget \
&& wget --quiet https://raw.githubusercontent.com/vishnubob/wait-for-it/master/wait-for-it.sh -O /usr/sbin/wait-for-it.sh \
&& chmod +x /usr/sbin/wait-for-it.sh
# Install dependencies (you are already in /app)
COPY package.json package-lock.json ./
# RUN npm ci
RUN npm install
RUN npm install codeceptjs puppeteer
COPY . /app
RUN pwd
RUN ls
# RUN npx codeceptjs init
RUN npx codeceptjs run
# CMD ["npm", "start"]

Azure Function deployed via Azure DevOps pipeline not executing

I have created an azure pipeline to deploy a custom docker image with an azure function with the following config. Though the config works in terms of build but none of the functions are running on the server and I notice that in the logs that none of the functions are being detected.
pool:
name: Azure Pipelines
steps:
- task: Docker#1
displayName: 'build: build the docker image'
inputs:
azureSubscriptionEndpoint: 'Azure subscription 1 (3fc75102-5159-4491-b61c-b3fcb845f648)'
azureContainerRegistry: whaiiapp.azurecr.io
imageName: 'azureapp.azurecr.io/puppeteer:$(Build.BuildId)'
useDefaultContext: false
- task: Docker#1
displayName: 'push the image'
inputs:
azureSubscriptionEndpoint: 'Azure subscription 1 (3fc75102-5159-4491-b61c-b3fcb845f648)'
azureContainerRegistry: azureapp.azurecr.io
command: 'Push an image'
imageName: 'azureap.azurecr.io/puppeteer:$(Build.BuildId)'
- task: FuncToolsInstaller#0
displayName: 'Install func tools - latest'
- task: AzureFunctionAppContainer#1
displayName: 'Start Azure Function App on Container Deploy: generate-report-dev'
inputs:
azureSubscription: 'Azure subscription 1 (3fc75102-5159-4491-b61c-b3fcb845f648)'
appName: 'generate-report-dev'
imageName: azureapp.azurecr.io/puppeteer
containerCommand: 'npm start'
The following is my Dockerfile config
FROM mcr.microsoft.com/azure-functions/node:3.0-appservice
# Chromium dependencies
RUN apt-get update -yq \
&& apt-get install -yq git git-extras gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 \
libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 \
libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 \
libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 \
fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-kacst libgbm-dev \
ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget && \
wget https://github.com/Yelp/dumb-init/releases/download/v1.2.2/dumb-init_1.2.2_amd64.deb && \
dpkg -i dumb-init_*.deb && rm -f dumb-init_*.deb && \
apt-get clean && apt-get autoremove -y && rm -rf /var/lib/apt/lists/*
WORKDIR /home/site/wwwroot
RUN npm install --no-save puppeteer
ENV AzureWebJobsScriptRoot=/home/site/wwwroot \
AzureFunctionsJobHost__Logging__Console__IsEnabled=true \
FUNCTIONS_WORKER_RUNTIME=node \
COPY . /home/site/wwwroot
RUN cd /home/site/wwwroot && npm install
CMD [ "npm start" ]
I am not sure how I could go about debugging this. I tried logging into the server via ssh and running func start to test it and it shows func command is not available.
I believe your azure function is unable to pull the image from your registry, I did faced the same issue, so I simply created new function app, then selected the repository in deployment process, so it is going to create a webhook behind the scene, that will pull the latest image from a registry, every time whenever it is updated.

Can't Use Puppeteer -- Error: Failed to launch chrome

This is the error I get. This occurs when the test code is run in the Gitlab CI.
However, it works fine locally. Someone please help?
2036 $ npm test
2037 > 10_auth#1.1.3 test /builds/brhanuh/domestic-repair
2038 > jest --coverage 'unitTests/'
2039 Error: Failed to launch chrome!
2040
/builds/brhanuh/domestic-repair/node_modules/puppeteer/.local-chromium/linux-686378/chrome-linux/chrome:
error while loading shared libraries: libX11-xcb.so.1: cannot open
shared object file: No such file or directory
Install the missing packages using
sudo apt install -y gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget
Credits to #coldner on the puppetteer issue tracker for assembling the required pkgs.
If you encounter E: Unable to locate package errors, run sudo apt-get update.
I just wanted to add to #mohammad's answer, that now Puppetteer now has an official troublshooting guide that helps you determine what to install.
Also, they recommend running the following command to tell which particular packages need installing on *nix systems:
ldd chrome | grep not
Or, if you have Chrome installed via Puppetteer via NPM like me:
ldd ./node_modules/puppeteer/.local-chromium/linux-756035/chrome-linux/chrome | grep not
Then, I was able to see what libraries were still missing, and install accordingly.
you need to install required dependencies
here
https://github.com/puppeteer/puppeteer/blob/master/docs/troubleshooting.md
under Chrome headless doesn't launch on UNIX you can find
Debian (e.g. Ubuntu) Dependencies
CentOS Dependencies
For those that are still having this problem, you can try this.
sudo apt update && sudo apt install -y ca-certificates fonts-liberation libappindicator3-1 libasound2 libatk-bridge2.0-0 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgbm1 libgcc1 libglib2.0-0 libgtk-3-0 libnspr4 libnss3 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 lsb-release wget xdg-utils

Not able to run Puppeteer in headless mode on shared runner on Gitlab.com

I have a private repo which I have set up on Gitlab.com . I have written some test cases for my code including E2E tests which runs on browser. For this I am using Puppeteer. My test cases are written in node.js and I am using Mocha and Nyc for testing and coverage part.
Here is my sample .gitlab-ci.yml file.
image: node:10.14.2
cache:
paths:
- node_modules/
stages:
- test
test_async:
stage: test
before_script:
- apt-get update
- apt-get install -yyq ca-certificates
- apt-get install -yyq libappindicator1 libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libnss3 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6
- apt-get install -yyq gconf-service lsb-release wget xdg-utils
- apt-get install -yyq fonts-liberation
script:
- npm install
- npm i mocha
- npm i puppeteer
- npm i nyc
- npm test
Update :
While running the test now I get the following error on my pipeline build :
Failed to launch chrome! spawn chromium ENOENT
Browser initialization script :
browser = await puppeteer.launch({
headless: true,
executablePath: 'chromium',
args: ['--no-sandbox', '--disable-setuid-sandbox'],
slowMo: 100,
timeout: 10000
});
Any help will be appreciated. Regards
Thanks #nurettin . That doc certainly helped me a lot. Finally by setting my initialization as :
browser = await puppeteer.launch({
headless: true,
slowMo: 100,
timeout: 10000,
args: ['--no-sandbox'],
});
Did the trick for me along with the dependencies defined in .gitlab-ci.yml .

How to run a node Puppeteer app on Ubuntu Server as root

I'm trying to run node apps that use the npm package Puppeteer to scrape websites, using VPSes that use root as the default user I login as.
It never works. I always have to spend half an hour googling to find the solution... well no more.
Do this:
0. Run sudo apt update
1. Install libx stuff
sudo apt-get install libpangocairo-1.0-0 libx11-xcb1 libxcomposite1 libxcursor1 libxdamage1 libxi6 libxtst6 libnss3 libcups2 libxss1 libxrandr2 libgconf2-4 libasound2 libatk1.0-0 libgtk-3-0
2. Set --no-sandbox argument in the launch() part of your Puppeteer app (only required if you are running as root/root's cron)
const browser = await puppeteer.launch({headless: true, args: ['--no-sandbox']})
E: as Niko pointed out, --no-sandbox opens your server to a malicious attack from a website the scraper visits. It looks like there are some SO questions elsewhere that post alternatives that aren't awful to implement. I like David's answer here which seems to suggest that even in 2021 --no-sandbox is still a necessary evil in some cases.
E2: See answer below. It looks like in 2021 we also have to install libgbm-dev
Connect to server by SSH
Open your terminal
Type: ssh root#[server ip]
If it is the first time you connect to the server you will get a security prompt. Type 'yes'
Fill in your password:  [password]
You should now be connected to your server
Install app
sudo apt-get update
sudo apt-get install -y libgbm-dev
sudo apt install -y gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget
sudo apt install curl
curl -sL https://deb.nodesource.com/setup_14.x | sudo -E bash -
sudo apt install nodejs
git config --global user.name "your name"
git config --global user.email "your email"
git clone "your project git url"
cd "your project folder"
npm i
npm start
If you need to run multiple apps on one VPS server - google tool "Screen"
After I installed libgbm, everything works fine for me!
hope that help you out
sudo apt-get update
sudo apt-get install -y libgbm-dev

Resources