Gitlab - Separating CI from Deployment - gitlab

Question

We are currently using Jenkins, and planning to migrate to Gitlab. We actually have 2 Jenkinsfiles in each repo, 1 is setup as a Multibranch pipeline and runs on all changes. Its is the merge check, that runs all the various linting, tests, building the docker containers etc. The second Jenkinsfile is only ran manually from Jenkins, it takes in all the various input parameters and it deploys the code. Which is mostly coming in from say, the linted Ansible/Terraform and selecting a docker image that would have already been built via the CI side of things.
I know gitlab doesnt support this model, but this project is already MVP'd so re-working how the dev's combined their logic and deployment code together is probably not going to happen.
Is it possibly, in 1 gitlab-ci.yml file to say run these jobs on merge/pushes and only run this on manual deployment .
e.g.
workflow:
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
- if: '$CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS'
when: never
- if: '$CI_COMMIT_BRANCH'
stages:
- test
- test
- deploy
- destroy
test-python-job:
stage: test
rules:
- if: '$CI_PIPELINE_SOURCE == "push"'
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
script:
- echo "Test Python"
- black
- bandit
- flake8
- tox
test-terraform-job:
stage: test
rules:
- if: '$CI_PIPELINE_SOURCE == "push"'
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
script:
- echo "Test Terraform"
- terraform validate --yadda
test-ansible-job:
stage: test
rules:
- if: '$CI_PIPELINE_SOURCE == "push"'
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
script:
- echo "Test Ansible"
- ansible-lint --yadda
deploy-job:
stage: deploy
variables:
DEPLOYMENT_ID: "Only deploy-job can use this variable's value"
secrets:
DATABASE_PASSWORD:
vault: production/db/password#ops
rules:
- when: manual
script:
- echo "Terraform Deploy"
- terraform deploy
- ansible-playbook yaddas
destroy-job:
stage: destroy
variables:
DEPLOYMENT_ID: "Only destroy-job can use this variable's value"
secrets:
DATABASE_PASSWORD:
vault: production/db/password#ops
rules:
- when: manual
script:
- terraform destroy
We have not even deployed gitlab yet, so im writing that off the top of my head, but I want to know what level of pain I am in for.

Answer 1

There are multiple options to achieve your goal with minimized configuration effort:
Working with private jobs and use inheritance or references for easier configuration - doable in one file
Extract parts into child pipelines for easier usage
reduced Configuration in one File
I assume you most hate that you have to redefine the rules for your jobs. There are two ways how you can reduce those duplication.
Inheritance
Can reduce a lot of duplication, but can also cause problems, with unintended side behaviour.
.test:
stage: test
rules:
- if: '$CI_PIPELINE_SOURCE == "push"'
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
test-python-job:
extends: .test
script:
- echo "Test Python"
- black
- bandit
- flake8
- tox
test-terraform-job:
extends: .test
script:
- echo "Test Terraform"
- terraform validate --yadda
test-ansible-job:
extends: .test
script:
- echo "Test Ansible"
- ansible-lint --yadda
Composition
By using !reference you can combine certain aspects of jobs see https://docs.gitlab.com/ee/ci/yaml/#reference-tags
.test:
rules:
- if: '$CI_PIPELINE_SOURCE == "push"'
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
test-python-job:
stage: test
rules:
- !reference [.test, rules]
script:
- echo "Test Python"
- black
- bandit
- flake8
- tox
test-terraform-job:
stage: test
rules:
- !reference [.test, rules]
script:
- echo "Test Terraform"
- terraform validate --yadda
stage: test
rules:
- !reference [.test, rules]
script:
- echo "Test Ansible"
- ansible-lint --yadda
Parent Child pipelines
Sometimes it might be also suitable to extract functionality into child pipelines. You can easier control what is happening at one stage, when you call the child pipeline and gain overview due to fewer lines of codes. It adds complexity to your builds. But generally it will generate a cleaner and easier to maintain ci structure (my opinion).
This approach will only add the child pipeline when needed - furthermore, you could also centralize this file, if it is similar for the deployments
.gitlab-ci.yml
deploy:
stage: deploy
trigger:
include:
- local: Deploy.gitlab-ci.yml
strategy: depend
rules:
- if: $CI_PIPELINE_SOURCE == 'web' #maybe also useful, as it will only happen on a web interaction
when: manual
- if: $CI_PIPELINE_SOURCE == 'schedule' #maybe also useful, for schedules
Deploy.gitlab-ci.yml
deploy-job:
stage: deploy
variables:
DEPLOYMENT_ID: "Only deploy-job can use this variable's value"
secrets:
DATABASE_PASSWORD:
vault: production/db/password#ops
script:
- echo "Terraform Deploy"
- terraform deploy
- ansible-playbook yaddas
destroy-job:
stage: destroy
variables:
DEPLOYMENT_ID: "Only destroy-job can use this variable's value"
secrets:
DATABASE_PASSWORD:
vault: production/db/password#ops
script:
- terraform destroy
Sidenotes
This is not 100% answering your question, but it shows you a lot of flexibility, and you will soon realize that mimicking jenkins is not 100% ideal. Eg having deployment jobs directly attached to a commit, and visible on that one, allows for a better overview of what was exactly deployed. If you need to run such things manually, i highly recommend to use schedules with preconfigured values, as they only have a play button. Also you have the artifacts already in place and build from your pipeline, why not add additional steps utilizing them, instead of providing this information
I hope my insights will be useful to you, and happy migration ;)