Deploy for different environments from same manual job depending on commit message - gitlab

I have 10 environments. Each environment should have same manual jobs to deploy 5 services (5 buttons). The only difference between environemnt-1 jobs and environment-2 jobs are tags, variables.only check for commit message and environment.name.
E.g. I should be able to specify [staging] in commit message and have 5 separate buttons that deploy 5 services to staging. Same for [production], etc...
staging-deploy-service1:
<<: *deploy
environment:
name: staging
variables:
ServiceName: service2
tags:
- dev-staging
only:
variables:
- $CI_COMMIT_MESSAGE =~ /\[staging\]/
staging-deploy-service2:
<<: *deploy
environment:
name: staging
variables:
ServiceName = service2
tags:
- dev-staging
only:
variables:
- $CI_COMMIT_MESSAGE =~ /\[staging\]/
staging-deploy-service3:
<<: *deploy
environment:
name: staging
variables:
ServiceName: service3
tags:
- dev-staging
only:
variables:
- $CI_COMMIT_MESSAGE =~ /\[staging\]/
staging-deploy-service4:
<<: *deploy
environment:
name: staging
variables:
ServiceName: service4
tags:
- dev-staging
only:
variables:
- $CI_COMMIT_MESSAGE =~ /\[staging\]/
staging-deploy-service5:
<<: *deploy
environment:
name: staging
variables:
ServiceName: service5
tags:
- dev-staging
only:
variables:
- $CI_COMMIT_MESSAGE =~ /\[staging\]/
Currently I have to repeat this block for each environment - 10 times. So it's 50 jobs for just 5 services.
How can I simplify this?

I don't have the perfect solution because having variable substitution in CI tags is still not possible but if we forget the tags, using parallel matrix can do the trick :
.service_name:
parallel:
matrix:
- SERVICE_NAME: ['service1', 'service2', 'service3', 'service4', 'service5']
ENV: ['env1', 'env2', 'env3', 'env4', 'env5']
deploy_service:
<<: *deploy
<<: *service_name
environment:
name: $ENV
script:
- echo $SERVICE_NAME
rules:
- if: '$CI_COMMIT_MESSAGE =~ $ENV'
when: manual
If you really need to keep one tag per environment, you can still simplify your current config using one-dimensional matrix and using more anchors, ex for staging for all services :
.service_name: &service_name
parallel:
matrix:
- SERVICE_NAME: ['service1', 'service2', 'service3', 'service4', 'service5']
.staging_common: &staging_common
environment:
name: $staging
rules:
- if: '$CI_COMMIT_MESSAGE =~ /\[staging\]/'
when: manual
.dev_staging_tags: &dev_staging_tags
tags:
- dev-staging
staging-deploy:
<<: *deploy
<<: *dev_staging_tags
<<: *staging_common
<<: *service_name
script:
- echo $SERVICE_NAME

Related

Why is the Gitlab pipeline running a job when I my gitlab-ci.yml says it shouldn't (by means of change only)?

This is the relevant job in the yml file:
#container structure test for check service stub
container-test-command-check:
only:
changes:
- stub-services/check-service/**/*
extends: container-test-command
<<: *branches
allow_failure: true
variables
CI_REGISTRY_IMAGE: $CI_REGISTRY_IMAGE/check-stub
dependencies:
- move-container-tests-check
My intention is that this job should only run if any files in the folder 'stub-services/check-service' have been altered. Unfortunately, it runs regardless of whether there have been any changes or not.
Here's the full yml file with non-relevant parts redacted for the sake of clarity and commercial confidentiality:
include:
- '/gitlab-ci/includes.yml'
- '/idt-test-stub/gitlab-ci.yml'
variables:
ALPINE_VERSION: "3.16"
NODE_VERSION: "14-alpine"
ESLINT_CONFIG_PATH: "./.eslintrc.js"
SOURCE_DIR: "."
RULESET: MAVEN_CI
BUILD_TYPE: MAVEN
MVN_CLI_OPTS: "--batch-mode"
MVN_OPTS: "-Dmaven.repo.local=.m2-local -f wiremock-java/pom.xml"
MAVEN_IMAGE: "maven:3-jdk-11"
stages:
- test
- code-quality
- code-test
- code-analysis
- verify
- transform
- application-build
- image-build
- move-container-tests
- container-image-test
- image-push
.branches: &branches
only:
- branches
todo-check:
<<: *branches
shell-check:
<<: *branches
lint-source:
<<: *branches
docker-lint:
<<: *branches
unit-test:
<<: *branches
artifacts:
expire_in: 20 mins
paths:
- ./coverage
sonar-scanner:
extends: .sonar-scanner-base
<<: *branches
stage: code-analysis
node-audit:
<<: *branches
node-outdated:
<<: *branches
allow_failure: true
verify:
<<: *branches
stage: verify
image: node:$NODE_VERSION
script:
- npm install
- ./run-verify.sh
tags:
- docker-in-docker-privileged
transform:
<<: *branches
stage: transform
image: node:$NODE_VERSION
script:
- npm install
- ./run-transform.sh
artifacts:
name: "wiremock-json"
paths:
- stub-services/check-service/wiremock/stubs
- stub-services/kgb-service/wiremock/stubs
tags:
- docker-in-docker-privileged
maven-package:
<<: *branches
artifacts:
paths:
- ./wiremock-java/target
expire_in: 1 hour
docker-build:
extends: .docker-build-template
<<: *branches
parallel:
matrix:
- DOCKERFILE_PATH: "./Dockerfiles/Dockerfile.check-service"
CI_REGISTRY_IMAGE: $CI_REGISTRY_IMAGE/check-stub
ADD_LATEST_TAG: "true"
ADD_GIT_TAGS: "true"
CI_COMMIT_TAG: "${CI_COMMIT_REF_SLUG}"
- DOCKERFILE_PATH: "./Dockerfiles/Dockerfile.kgb-service"
CI_REGISTRY_IMAGE: $CI_REGISTRY_IMAGE/kgb-service-stub
ADD_LATEST_TAG: "true"
ADD_GIT_TAGS: "true"
CI_COMMIT_TAG: "${CI_COMMIT_REF_SLUG}"
except:
variables:
- $PROD_AWS_ACCESS_KEY_ID
- $PROD_AWS_SECRET_ACCESS_KEY
- $PROD_AWS_SESSION_TOKEN
tags:
- docker-in-docker
dependencies:
- transform
- maven-package
#sast container test for service stubs
sast-container:
<<: *branches
allow_failure: true
parallel:
matrix:
- CI_REGISTRY_IMAGE: $CI_REGISTRY_IMAGE/check-stub
- CI_REGISTRY_IMAGE: $CI_REGISTRY_IMAGE/kgb-service-stub
trivy-container:
extends: .trivy-container-base
<<: *branches
parallel:
matrix:
- CI_REGISTRY_IMAGE: $CI_REGISTRY_IMAGE/check-stub
- CI_REGISTRY_IMAGE: $CI_REGISTRY_IMAGE/kgb-service-stub
allow_failure: true
#move container test json into container-tests, for check-service stub
#so container-tests folder will be picked up by container-test-command below
move-container-tests-check:
<<: *branches
stage: move-container-tests
image: alpine:$ALPINE_VERSION
script:
- cp -R container-test-folders/check-service container-tests
tags:
- docker-in-docker-privileged
artifacts:
paths:
- container-tests
#container structure test for check service stub
container-test-command-check:
only:
changes:
- stub-services/check-service/**/*
extends: container-test-command
<<: *branches
allow_failure: true
variables:
CI_REGISTRY_IMAGE: $CI_REGISTRY_IMAGE/check-stub
dependencies:
- move-container-tests-check
#move container test json into container-tests, for kgb-service stub
#so container-tests folder will be picked up by container-test-command below
move-container-tests-kgb:
<<: *branches
stage: move-container-tests
image: alpine:$ALPINE_VERSION
script:
- cp -R container-test-folders/kgb-service container-tests
tags:
- docker-in-docker-privileged
artifacts:
paths:
- container-tests
#container structure test for kgb service stub
container-test-command-kgb:
extends: container-test-command
only:
changes:
- stub-services/kgb-service/**/*
<<: *branches
allow_failure: true
variables:
CI_REGISTRY_IMAGE: $CI_REGISTRY_IMAGE/kgb-service-stub
dependencies:
- move-container-tests-kgb

Reference a different Variable group per stage - stage per environment in environments - Azure Devops, template, variable groups

I want to get the environments and the associated variableGroup - loop thru the environments array in Stages.yml and create a stage per environment... insert correct variableGroup per stage... use variablesGroup values to perform jobs in jobs.yml - each variable group contains the same vars.. with different values.
``
I have a main.yml file in which I reference my environments and the associated group variable
main.yaml :
name: $(BuildID)
trigger: none
pool:
vmImage: 'leetchi-agentspool-windows'
variables:
- template: vars/variables-api.yaml
extends:
template: deploy-stages.yaml
parameters:
environements:
- environement: 'DV1'
variableGroupName: tf-dv1
- environement: 'IN1'
variableGroupName: tf-in1
A stage template which is supposed to apply with the environment parameters and the group variable.
stage.yaml :
---
parameters:
- name: environements
type: object
default: []
stages:
- ${{ each environement in parameters.environements }}:
- stage: 'Deploy to ${{ environement.environements }}'
variables:
- group: ${{ environement.variableGroup }}
jobs:
- template: template-terraform-deployment.yml
parameters:
environement: ${{environement.environements}}
project: $(project)
definition: $(def)
project1: $(project1)
definition1: $(def1)
stateKey: ${{environement.environement}}$(tfkey)
TerraformDirectory: $(TfDirectory)
azureSubscription: $(service_connection_name)
TfStateResourceGroupName: $(tf_resource_group)
TfStateStorageAccountName: $(tf_storage_account)
TStateContainerName: $(tf_container)
commandOptions: -out=$(System.DefaultWorkingDirectory)/terraform.tfplan -detailed-exitcode
I start in YAML. Could you help me please? Thank you very much in advance
You can modify your main.yaml as below:
trigger: none
pool:
vmImage: 'leetchi-agentspool-windows'
variables:
- template: vars/variables-api.yaml
extends:
template: deploy-stages.yaml
parameters:
- name: environment
values:
- DV1
- IN1
- name: variableGroupName
values:
- tf-dv1
- tf-in1
stage.yaml :
stages:
- stage: 'Deploy to ${{ parameters.environment }}'
variables:
- group: ${{ parameters.variableGroupName}}
jobs:
- template: template-terraform-deployment.yml
parameters:
environment: ${{ parameters.environment }}
project: $(project)
definition: $(def)
project1: $(project1)
definition1: $(def1)
stateKey: ${{ parameters.environment }} + $(tfkey)
TerraformDirectory: $(TfDirectory)
azureSubscription: $(service_connection_name)
TfStateResourceGroupName: $(tf_resource_group)
TfStateStorageAccountName: $(tf_storage_account)
TStateContainerName: $(tf_container)
commandOptions: -out=$(System.DefaultWorkingDirectory)/terraform.tfplan -detailed-exitcode

Gitlab CI needs for an abstract job

Can we make a job depend to an abstract job ?
I'm trying to manage my jobs avoiding writing the same jobs twice by using abstract jobs.
For example, I would like make the deployment jobs to depend on their respective packagings :
.package:...
.package_qual:
extends: .package
script: ...
except: [master]
tags: [...]
.package_prod:
extends: .package
script: ...
only: [master]
tags: [...]
.deploy:...
.deploy_qual:
extends: .deploy
script: ...
environment:
name: qual
only: [develop]
tags: [...]
needs: [.package_qual] #there
.deploy_prod:
extends: .deploy
script: ...
environment:
name: prod
only: [master]
tags: [...]
needs: [.package_prod] #there
include:
- '/A/.gitlab-ci.yml'
- '/B/.gitlab-ci.yml'
A/gitlab.yml
A_package_qual:
extends: .package_qual
variables:
CONTEXT: A
A_package_prod:
extends: .package_prod
variables:
CONTEXT: A
A_deploy_qual:
extends: .deploy_qual
variables:
CONTEXT: A
A_deploy_prod:
extends: .deploy_prod
variables:
CONTEXT: A
B/gitlab.yml
B_package_qual:
extends: .package_qual
variables:
CONTEXT: B
B_package_prod:
extends: .package_prod
variables:
CONTEXT: B
B_deploy_qual:
extends: .deploy_qual
variables:
CONTEXT: A
B_deploy_prod:
extends: .deploy_prod
variables:
CONTEXT: B
When I try this, I have this error A_deploy_qual job: undefined need: .package_qual

$CI_ENVIRONMENT_NAME remains empty after set in previous stage

I have the following GitLab pipeline. For brevity I have removed some code. In deploy-dev stage I am creating environment dev. Then in the next stage test-postdeploy I am using $CI_ENVIRONMENT_NAME to set the base_url variable value. However, in test-postdeploy stage the $CI_ENVIRONMENT_NAME always empty
stages:
deploy:dev
deploy:qa
deploy:prod
test:postdeploy
deploy-dev:
image: node
stage: deploy:dev
environment:
name: dev
script:
- . deploy/scripts/deploy-dev.sh
rules:
- if: $CI_COMMIT_REF_NAME == "dev"
deploy-qa:
image: node
stage: deploy:qa
environment:
name: qa
script:
- . deploy/scripts/deploy-qa.sh
rules:
- if: $CI_COMMIT_REF_NAME == "qa"
deploy-prod:
image: node
stage: deploy:prod
environment:
name: prod
script:
- . deploy/scripts/deploy-prod.sh
rules:
- if: $CI_COMMIT_REF_NAME == "release"
test-postdeploy:
tags:
- xlarge
image: node
stage: test:postdeploy
variables:
base_url: ""
script:
- echo "$CI_ENVIRONMENT_NAME" // this is empty
- script/postdeploy-test.sh
rules:
- if: $CI_ENVIRONMENT_NAME == "dev"
variables:
base_url: "https://dev.example.com"
- if: $CI_ENVIRONMENT_NAME == "qa"
variables:
base_url: "https://qa.example.com"
- if: $CI_ENVIRONMENT_NAME == "prod"
variables:
base_url: "https://prod.example.com"
UPDATE 1
I can probably pass environment name to another stage using https://docs.gitlab.com/ee/ci/variables/#pass-an-environment-variable-to-another-job
However, this does not make sense. Environment Name is most basic value that any ci/cd pipeline would need to know. The CI_ENVIRONMENT_NAME
variable should be available in all stages once its set without doing some extra work.

how to create and extend reusable gitlab pipeline?

I have a pipeline like below:
include:
- project: "some-project"
ref: 0.5.0
file: reusable-jobs.yml
variables:
VAR1: ""
stages:
- stage-1
- stage-2
job1:
extends: .reusable-job-1
stage: stage-1
variables:
SOME_VAR: "$VAR1"
job2:
extends: .reusable-job-2
stage: stage-2
variables:
SOME_VAR: "$VAR1"
I want to be able to save this an imported into other projects. with my reusable jobs I create one like this
.some-reusable-job:
image:
name: alpine
script:
- echo "hello"
and then I extended it when I want to use it
ex:
reuse_the_job:
extends: .some-reusable-job
stage: some-stage
but I cannot figure out how to do the same thing with the entire pipeline including the stages
I want to be able to call the pipeline like so:
reuse_the_pipeline:
extends: .my-reusable-pipeline
variables:
VAR1: "hello"
and have the pipeline be created with both stage_1 and stage_2
I tried to create a definition like this in reusable-flow-file.yml
.reusable-flow
include:
- project: "some-project"
ref: 0.5.0
file: reusable-jobs.yml
variables:
VAR1: ""
stages:
- stage-1
- stage-2
job1:
extends: .reusable-job-1
stage: stage-1
variables:
SOME_VAR: "$VAR1"
job2:
extends: .reusable-job-2
stage: stage-2
variables:
SOME_VAR: "$VAR1"
and then use it like this (.gitlab-ci.yml)
include:
- local: "reusable-flow-file.yml"
dev_na:
extends: .reusable-flow
variables:
VAR1: "hello"
but when I try to run it and get lab I get this error
config should implement a script: or a trigger: keyword
not really sure what that try
It's because you're extending the ".reusable-flow" on a job-level. And what you're actually getting is (the following) which is invalid, because the job dev_na has no script property.
dev_na:
include:
- project: "some-project"
ref: 0.5.0
file: reusable-jobs.yml
variables:
VAR1: ""
stages:
- stage-1
- stage-2
job1:
extends: .reusable-job-1
stage: stage-1
variables:
SOME_VAR: "$VAR1"
job2:
extends: .reusable-job-2
stage: stage-2
variables:
SOME_VAR: "$VAR1"
variables:
VAR1: "hello"
You should just define the jobs (and other stuff) on the root level of your template. And play around a bit in the Project > CI/CD > editor where you can get a life preview of the templated configuration.

Resources