I have a simple .gitlab-ci.yml:
stages:
- test
- build
.docker_image: &docker_image
image:
name: foo
entrypoint: [""]
tags:
- deploy
test:
stage: test
<<: *docker_image
script:
- 'echo $CI_MERGE_REQUEST_IID'
- 'echo $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME'
only:
- merge_requests
build:
stage: build
<<: *docker_image
script:
- export
only:
- stage
How can I use CI_MERGE_REQUEST_IID and CI_MERGE_REQUEST_SOURCE_BRANCH_NAME in a build job?
I need this to add information to the commit message.
Related
Currently I have this script in my .gitlab-ci.yml file:
image: node:16
cache:
paths:
- .npm
- cache/Cypress
- node_modules
stages:
- build
- deploy
- test
install:dependencies:
stage: build
script:
- yarn install
artifacts:
paths:
- node_modules/
only:
- merge_requests
test:unit:
stage: test
script: yarn test --ci --coverage
needs: ["install:dependencies"]
artifacts:
when: always
paths:
- coverage
expire_in: 30 days
only:
- merge_requests
deploy-to-vercel:
stage: deploy
image: node:16
script:
- npm i -g vercel
- DEPLOYMENT_URL=$(vercel -t $VERCEL_TOKEN --confirm)
- echo $DEPLOYMENT_URL > vercel_deployment_url.txt
- cat vercel_deployment_url.txt
artifacts:
when: on_success
paths:
- vercel_deployment_url.txt
only:
- merge_requests
I need to trigger a pipeline to an environment called playground but only when a pipeline from test enviroment is finished, when a pipeline to master happens, I don't to mirror to the playground environment.
Everything is deployed to vercel, and the project is powered by Next JS.
Committing my code to its Gitlab repository triggers the pipeline (as it should). However, I have an instruction telling it not to run the verify job unless the commit message contains the phrase 'trigger ci'.
However, it runs the job even when the phrase isn't in the commit message. Where am I going wrong?
include:
- '/gitlab-ci/includes.yml'
- '/idt-test-stub/gitlab-ci.yml'
variables:
ALPINE_VERSION: "3.16"
NODE_VERSION: "14-alpine"
ESLINT_CONFIG_PATH: "./.eslintrc.js"
SOURCE_DIR: "."
RULESET: MAVEN_CI
BUILD_TYPE: MAVEN
MVN_CLI_OPTS: "--batch-mode"
MVN_OPTS: "-Dmaven.repo.local=.m2-local -f wiremock-java/pom.xml"
MAVEN_IMAGE: "maven:3-jdk-11"
stages:
- test
- code-quality
- code-test
- code-analysis
- verify
- transform
- application-build
- image-build
- move-container-tests
- container-image-test
- image-push
.branches: &branches
only:
- branches
todo-check:
<<: *branches
shell-check:
<<: *branches
docker-lint:
<<: *branches
unit-test:
<<: *branches
artifacts:
expire_in: 20 mins
paths:
- ./coverage
verify:
only:
variables:
- $CI_COMMIT_MESSAGE =- /trigger ci/
<<: *branches
stage: verify
image: node:$NODE_VERSION
script:
- npm install
- ./run-verify.sh
tags:
- docker-in-docker-privileged
The correct syntax to check against a regular expression pattern is =~ instead of =-.
Welcome
As you can see from the image, the CI for the experiment consists of three stages, the first stage is just "prepare" nothing special)). The second stage is the TEST itself, which consists of four jobs, is started manual depending on what is needed, usually only one of them is started. The question is in the last stage, it was thought that the notification in the telegram (bash script) should go if ONE OF the jobs of the previous stage was completed successfully. But there is no way to do this, HOW TO MAKE IT SO THAT IF ONE OF THE JOBS OF THE SECOND STAGE IS COMPLETED SUCCESSFULLY, THE JOB FROM THE THIRD STAGE IS LAUNCHED?
Here's what it looks like now:
include:
- project: "devops/public-resources/cicd-templates/telegram-notification"
ref: v1.0.0
file: "template.yml"
stages:
- prepare
- test
- notificatoin
.common_tags: &common_tags
tags:
- infra-runners
Prepare_job:
stage: prepare
image: alpine:latest
script:
- echo "prepare"
<<: *common_tags
test-one:
stage: test
image: alpine:latest
script:
- echo "test-one"
needs:
- Prepare_job
variables:
JOB_NAME: $CI_JOB_NAME
when: manual
<<: *common_tags
test-two:
stage: test
image: alpine:latest
script:
- echo "test-two"
needs:
- Prepare_job
variables:
JOB_NAME: $CI_JOB_NAME
when: manual
<<: *common_tags
test-three:
stage: test
image: alpine:latest
script:
- echo "test-three"
needs:
- Prepare_job
variables:
JOB_NAME: $CI_JOB_NAME
when: manual
<<: *common_tags
test-four:
stage: test
image: alpine:latest
script:
- echo "test-four"
needs:
- Prepare_job
when: manual
variables:
JOB_NAME: $CI_JOB_NAME
<<: *common_tags
# Notification jobs
Notify Success:
needs:
- test-one
- test-two
- test-three
- test-four
stage: notificatoin
extends: .telegram_notify
variables:
MESSAGE: |
✅ Successfully deployed to Production
test: $JOB_NAME
Text: THIS IS TEST PIPELINE
when: on_success
Notify Failure:
stage: notificatoin
extends: .telegram_notify
variables:
MESSAGE: |
Text: THIS IS TEST PIPELINE
when: on_failure
Notify Success:
needs:
test-one
test-two
test-three
test-four
This works if all jobs of the second stage were completed successfully, we need that if at least one completed successfully, Notify Success was launched)
I have 3 projects in Gitlab.
Frontend
Backend
Deployment
Each project has separate pipeline definition which has CI pipeline and using Multi-project pipeline concept, it will invoke the deployment project pipeline for deploying each module.
Frontend
image: frontend:runner1.1
# Cache modules in between jobs
cache:
key: ${CI_COMMIT_REF_SLUG}
paths:
- .npm/
variables:
GIT_SUBMODULE_STRATEGY: recursive
JAVA_OPTS: "Dlog4j.formatMsgNoLookups=true"
LOG4J_FORMAT_MSG_NO_LOOKUPS: "true"
stages:
- VersionCheck
- Static Analysis
- Test
- SonarQube
- Tag
- Version
- Build
- Deploy
.build:
stage: Build
image: google/cloud-sdk
services:
- docker:dind
before_script:
- mkdir -p $HOME/.docker && echo $DOCKER_AUTH_CONFIG > $HOME/.docker/config.json
script:
- echo $CI_REGISTRY_USER
- echo $CI_REGISTRY
- echo ${IMAGE_TAG}
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
- docker build -t $IMAGE_TAG . --build-arg REACT_APP_ENV=${CI_COMMIT_BRANCH} --build-arg REACT_APP_BACKEND_API=${REACT_APP_BACKEND_API} --build-arg REACT_APP_GOOGLE_CLIENT_ID=${REACT_APP_GOOGLE_CLIENT_ID}
- docker push $IMAGE_TAG
VersionCheck:
stage: VersionCheck
allow_failure: false
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "sandbox"'
before_script:
- git fetch origin $CI_MERGE_REQUEST_TARGET_BRANCH_NAME:$CI_MERGE_REQUEST_TARGET_BRANCH_NAME
- git fetch origin $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME:$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME
script:
- deployed_version=`git show $CI_MERGE_REQUEST_TARGET_BRANCH_NAME:package.json | sed -nE 's/^\\s*\"version\"\:\ \"(.*?)\",$/\\1/p'`
- new_version=`git show $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME:package.json | sed -nE 's/^\\s*\"version\"\:\ \"(.*?)\",$/\\1/p'`
- >
echo "sandbox version: $deployed_version"
echo "feature version: $new_version"
if [ "$(printf '%s\n' "$deployed_version" "$new_version" | sort -V | head -n1)" = "$deployed_version" ]; then
echo "version is incremented"
else
echo "Version need to be incremented on the feature branch. See the README.md file"
exit 1
fi
eslint:
stage: Static Analysis
allow_failure: false
before_script:
- npm ci --cache .npm --prefer-offline
script:
- echo "Start building App"
- npm install
- npm run eslint-report
- echo "Build successfully!"
artifacts:
reports:
junit: coverage/eslint-report.xml
paths:
- coverage/eslint-report.json
test:
stage: Test
allow_failure: false
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
when: always
- when: always
before_script:
- npm ci --cache .npm --prefer-offline
script:
- echo "Testing App"
- npm install
- npm run generate-tests-report
- echo "Test successfully!"
artifacts:
reports:
junit: junit.xml
paths:
- test-report.xml
- coverage/lcov.info
sonarqube-check:
stage: SonarQube
allow_failure: true
image:
name: sonarsource/sonar-scanner-cli:4.6
entrypoint: [""]
variables:
SONAR_USER_HOME: "${CI_PROJECT_DIR}/.sonar" # Defines the location of the analysis task cache
GIT_DEPTH: "0" # Tells git to fetch all the branches of the project, required by the analysis task
cache:
key: "${CI_JOB_NAME}"
paths:
- .sonar/cache
script:
# wait for the quality results, true/false
- sonar-scanner -X -Dsonar.qualitygate.wait=false -Dsonar.branch.name=$CI_COMMIT_BRANCH -Dsonar.login=$SONAR_TOKEN -Dsonar.projectVersion=$(npm run print-version --silent)
only:
- merge_requests
- release
- master
- develop
- sandbox
Version:
stage: Version
allow_failure: false
only:
- sandbox
- develop
- release
- master
script:
- VERSION=`sed -nE 's/^\\s*\"version\"\:\ \"(.*?)\",$/\\1/p' package.json`
- echo "VERSION=$CI_COMMIT_REF_SLUG$VERSION" >> build.env
artifacts:
reports:
dotenv: build.env
build:sb:
stage: Build
allow_failure: false
environment:
name: sb
variables:
IMAGE_TAG: $CI_REGISTRY_IMAGE:$VERSION
TF_ENV: "sb"
extends:
- .build
only:
- sandbox
dependencies:
- Version
build:dev:
stage: Build
allow_failure: false
environment:
name: dev
variables:
IMAGE_TAG: $CI_REGISTRY_IMAGE:$VERSION
TF_ENV: "dev"
extends:
- .build
only:
- develop
dependencies:
- Version
build:qa:
stage: Build
allow_failure: false
environment:
name: qa
variables:
IMAGE_TAG: $CI_REGISTRY_IMAGE:$VERSION
TF_ENV: "qa"
extends:
- .build
only:
- release
dependencies:
- Version
build:prod:
stage: Build
allow_failure: false
environment:
name: prod
variables:
IMAGE_TAG: $CI_REGISTRY_IMAGE:$VERSION
TF_ENV: "prod"
extends:
- .build
only:
- master
dependencies:
- Version
deployment:sandbox:
rules:
- if: "$CI_COMMIT_BRANCH =~ /^feature/"
when: never
- if: $CI_COMMIT_BRANCH == "sandbox"
variables:
TF_ENV: "sb"
MODULE: "frontend"
VERSION: $VERSION
stage: Deploy
allow_failure: false
trigger:
project: in-silico-prediction/isp/isp-deployment
strategy: depend
needs:
- job: Version
artifacts: true
- job: build:sb
artifacts: false
deployment:dev:
rules:
- if: "$CI_COMMIT_BRANCH =~ /^feature/"
when: never
- if: $CI_COMMIT_BRANCH == "develop"
variables:
TF_ENV: "dev"
MODULE: "frontend"
VERSION: $VERSION
stage: Deploy
allow_failure: false
trigger:
project: deployment
strategy: depend
needs:
- job: Version
artifacts: true
- job: build:dev
artifacts: false
deployment:qa:
rules:
- if: "$CI_COMMIT_BRANCH =~ /^feature/"
when: never
- if: $CI_COMMIT_BRANCH == "release"
variables:
TF_ENV: "qa"
MODULE: "frontend"
VERSION: $VERSION
stage: Deploy
allow_failure: false
trigger:
project: deployment
strategy: depend
needs:
- job: Version
artifacts: true
- job: build:qa
artifacts: false
deployment:prod:
rules:
- if: "$CI_COMMIT_BRANCH =~ /^feature/"
when: never
- if: $CI_COMMIT_BRANCH == "master"
variables:
TF_ENV: "prod"
MODULE: "frontend"
VERSION: $VERSION
stage: Deploy
allow_failure: false
trigger:
project: deployment
strategy: depend
needs:
- job: Version
artifacts: true
- job: build:prod
artifacts: false
The deployment stage will invoke downstream project. The backend project also have same pipeline definition. Now both Frontend and Backend project will trigger the deployment project independently.
The deployment project should wait for the trigger from both project and run only 1 time which deploys both frontend and backend in single run into the environment.
For Merge train, is it possible to configure 2 different project merge request
As long as those MRs are from the same project, yes, the all idea of merge train is to, as in this example, list multiple MRs and combine them.
However, that would trigger the FE (FrontEnd) deployment, then it would trigger FE and BE deployments, which is not what you want.
I would rather use a scheduled cron job which detects when both FE and BE have been deployed, for instance by query the the date of their respective published images (latest FE and BE in the registry): if that date is more recent than the latest completed deployment, both for FE and BE build, then the deployment cron job would trigger an actual and full deployment.
Is there a way to configure multiple specifically-named environments (specifically, test, stage, and prod)?
In their documentation (https://docs.gitlab.com/ce/ci/environments.html) they talk about dynamically-created environments, but they are all commit based.
My build steps are the same for all of them, save for swapping out the slug:
deploy_to_test:
environment:
name: test
url: ${CI_ENVIRONMENT_SLUG}.mydomain.com
scripts:
- deploy ${CI_ENVIRONMENT_SLUG}
deploy_to_stage:
environment:
name: stage
url: ${CI_ENVIRONMENT_SLUG}.mydomain.com
scripts:
- deploy ${CI_ENVIRONMENT_SLUG}
deploy_to_prod:
environment:
name: prod
url: ${CI_ENVIRONMENT_SLUG}.mydomain.com
scripts:
- deploy ${CI_ENVIRONMENT_SLUG}
Is there any way to compress this down into one set of instructions? Something like:
deploy:
environment:
url: ${CI_ENVIRONMENT_SLUG}.mydomain.com
scripts:
- deploy ${CI_ENVIRONMENT_SLUG}
Yes, you can use anchors. If I follow the documentation properly, you would rewrite it using a hidden key .XX and then apply it with <<: *X.
For example this to define the key:
.job_template: &deploy_definition
environment:
url: ${CI_ENVIRONMENT_SLUG}.mydomain.com
scripts:
- deploy ${CI_ENVIRONMENT_SLUG}
And then all blocks can be writen using <<: *job_template. I assume environment will merge the name with the predefined URL.
deploy_to_test:
<<: *deploy_definition
environment:
name: test
deploy_to_stage:
<<: *deploy_definition
environment:
name: stage
deploy_to_prod:
<<: *deploy_definition
environment:
name: prod
Full docs section from the link above:
YAML has a handy feature called 'anchors', which lets you easily duplicate content across your document. Anchors can be used to duplicate/inherit properties, and is a perfect example to be used with hidden keys to provide templates for your jobs.
The following example uses anchors and map merging. It will create two jobs, test1 and test2, that will inherit the parameters of .job_template, each having their own custom script defined:
.job_template: &job_definition # Hidden key that defines an anchor named 'job_definition'
image: ruby:2.1
services:
- postgres
- redis
test1:
<<: *job_definition # Merge the contents of the 'job_definition' alias
script:
- test1 project
test2:
<<: *job_definition # Merge the contents of the 'job_definition' alias
script:
- test2 project
& sets up the name of the anchor (job_definition), << means "merge the given hash into the current one", and * includes the named anchor (job_definition again). The expanded version looks like this:
.job_template:
image: ruby:2.1
services:
- postgres
- redis
test1:
image: ruby:2.1
services:
- postgres
- redis
script:
- test1 project
test2:
image: ruby:2.1
services:
- postgres
- redis
script:
- test2 project
Besides what the answer offered, I'd like to add another similar way to achieve kind of the same thing but it's more flexible rather than to use a template and then merge it in a stage.
What you can do is to create a hidden key as well, but in this format, e.g.,
.login: &login |
cmd1
cmd2
cmd3
...
And then you can apply it to different stages by using the '*', the asterisk, like:
deploy:
stage: deploy
script:
- ...
- *login
- ...
bake:
stage: bake
script:
- ...
- *login
- ...
And the result would be equivalent to:
deploy:
stage: deploy
script:
- ...
- cmd1
- cmd2
- cmd3
- ...
bake:
stage: bake
script:
- ...
- cmd1
- cmd2
- cmd3
- ...
Based on the resource of:
https://gitlab.com/gitlab-org/gitlab-ce/issues/19677#note_13008199
As for the template implementation, it's "merged". With my own experience, if you append more scripts after merging a template, the template scripts will be overwritten. And you cannot apply multiple templates at a time. Only the last template scripts will be executed. For example:
.tmp1: &tmp1
script:
- a
- b
.tmp2: &tmp2
script:
- c
- d
job1:
<<: *tmp1
<<: *tmp2
stage: xxx
job2:
<<: *tmp2
stage: yyy
script:
- e
- f
The equivalent result would be:
job1:
stage: xxx
script:
- c
- d
job2:
stage: yyy
script:
- e
- f
If not sure about the syntax correctness, just copy and paste your .gitlab.yml file content to "CI Lint" to validate. The button is in the tab of Pipelines.
gitlab gitlab-ci yaml
Just in case: Gitlab offers (since 11.3) an extends keyword, which can be used to "templates" yaml entries (so far as I understand it):
See the official doc
Have you tried implementing variables for various environments and using different jobs for various environments? I've come up with a solution for you.
image: node:latest
variables:
GIT_DEPTH: '0'
stages:
- build
- deploy
workflow:
rules:
- if: $CI_COMMIT_REF_NAME == "develop"
variables:
DEVELOP: "true"
ENVIRONMENT_NAME: Develop
WEBSITE_URL: DEVELOP_WEBSITE_URL
S3_BUCKET: (develop-s3-bucket-name)
AWS_REGION: ************** develop
AWS_ACCOUNT: ********develop
- if: $CI_COMMIT_REF_NAME == "main"
variables:
PRODUCTION: "true"
ENVIRONMENT_NAME: PRODUCTION
WEBSITE_URL: $PROD_WEBSITE_URL
S3_BUCKET: $PROD-S3-BUCKET-NAME
AWS_REGION: ************** (prod-region)
AWS_ACCOUNT: ***********(prod-acct)
- when: always
build-app:
stage: build
script:
#build-script
environment:
name: $ENVIRONMENT_NAME
deploy-app:
stage: deploy
script:
#deploy-script
environment:
name: $ENVIRONMENT_NAME