This is the relevant job in the yml file:
#container structure test for check service stub
container-test-command-check:
only:
changes:
- stub-services/check-service/**/*
extends: container-test-command
<<: *branches
allow_failure: true
variables
CI_REGISTRY_IMAGE: $CI_REGISTRY_IMAGE/check-stub
dependencies:
- move-container-tests-check
My intention is that this job should only run if any files in the folder 'stub-services/check-service' have been altered. Unfortunately, it runs regardless of whether there have been any changes or not.
Here's the full yml file with non-relevant parts redacted for the sake of clarity and commercial confidentiality:
include:
- '/gitlab-ci/includes.yml'
- '/idt-test-stub/gitlab-ci.yml'
variables:
ALPINE_VERSION: "3.16"
NODE_VERSION: "14-alpine"
ESLINT_CONFIG_PATH: "./.eslintrc.js"
SOURCE_DIR: "."
RULESET: MAVEN_CI
BUILD_TYPE: MAVEN
MVN_CLI_OPTS: "--batch-mode"
MVN_OPTS: "-Dmaven.repo.local=.m2-local -f wiremock-java/pom.xml"
MAVEN_IMAGE: "maven:3-jdk-11"
stages:
- test
- code-quality
- code-test
- code-analysis
- verify
- transform
- application-build
- image-build
- move-container-tests
- container-image-test
- image-push
.branches: &branches
only:
- branches
todo-check:
<<: *branches
shell-check:
<<: *branches
lint-source:
<<: *branches
docker-lint:
<<: *branches
unit-test:
<<: *branches
artifacts:
expire_in: 20 mins
paths:
- ./coverage
sonar-scanner:
extends: .sonar-scanner-base
<<: *branches
stage: code-analysis
node-audit:
<<: *branches
node-outdated:
<<: *branches
allow_failure: true
verify:
<<: *branches
stage: verify
image: node:$NODE_VERSION
script:
- npm install
- ./run-verify.sh
tags:
- docker-in-docker-privileged
transform:
<<: *branches
stage: transform
image: node:$NODE_VERSION
script:
- npm install
- ./run-transform.sh
artifacts:
name: "wiremock-json"
paths:
- stub-services/check-service/wiremock/stubs
- stub-services/kgb-service/wiremock/stubs
tags:
- docker-in-docker-privileged
maven-package:
<<: *branches
artifacts:
paths:
- ./wiremock-java/target
expire_in: 1 hour
docker-build:
extends: .docker-build-template
<<: *branches
parallel:
matrix:
- DOCKERFILE_PATH: "./Dockerfiles/Dockerfile.check-service"
CI_REGISTRY_IMAGE: $CI_REGISTRY_IMAGE/check-stub
ADD_LATEST_TAG: "true"
ADD_GIT_TAGS: "true"
CI_COMMIT_TAG: "${CI_COMMIT_REF_SLUG}"
- DOCKERFILE_PATH: "./Dockerfiles/Dockerfile.kgb-service"
CI_REGISTRY_IMAGE: $CI_REGISTRY_IMAGE/kgb-service-stub
ADD_LATEST_TAG: "true"
ADD_GIT_TAGS: "true"
CI_COMMIT_TAG: "${CI_COMMIT_REF_SLUG}"
except:
variables:
- $PROD_AWS_ACCESS_KEY_ID
- $PROD_AWS_SECRET_ACCESS_KEY
- $PROD_AWS_SESSION_TOKEN
tags:
- docker-in-docker
dependencies:
- transform
- maven-package
#sast container test for service stubs
sast-container:
<<: *branches
allow_failure: true
parallel:
matrix:
- CI_REGISTRY_IMAGE: $CI_REGISTRY_IMAGE/check-stub
- CI_REGISTRY_IMAGE: $CI_REGISTRY_IMAGE/kgb-service-stub
trivy-container:
extends: .trivy-container-base
<<: *branches
parallel:
matrix:
- CI_REGISTRY_IMAGE: $CI_REGISTRY_IMAGE/check-stub
- CI_REGISTRY_IMAGE: $CI_REGISTRY_IMAGE/kgb-service-stub
allow_failure: true
#move container test json into container-tests, for check-service stub
#so container-tests folder will be picked up by container-test-command below
move-container-tests-check:
<<: *branches
stage: move-container-tests
image: alpine:$ALPINE_VERSION
script:
- cp -R container-test-folders/check-service container-tests
tags:
- docker-in-docker-privileged
artifacts:
paths:
- container-tests
#container structure test for check service stub
container-test-command-check:
only:
changes:
- stub-services/check-service/**/*
extends: container-test-command
<<: *branches
allow_failure: true
variables:
CI_REGISTRY_IMAGE: $CI_REGISTRY_IMAGE/check-stub
dependencies:
- move-container-tests-check
#move container test json into container-tests, for kgb-service stub
#so container-tests folder will be picked up by container-test-command below
move-container-tests-kgb:
<<: *branches
stage: move-container-tests
image: alpine:$ALPINE_VERSION
script:
- cp -R container-test-folders/kgb-service container-tests
tags:
- docker-in-docker-privileged
artifacts:
paths:
- container-tests
#container structure test for kgb service stub
container-test-command-kgb:
extends: container-test-command
only:
changes:
- stub-services/kgb-service/**/*
<<: *branches
allow_failure: true
variables:
CI_REGISTRY_IMAGE: $CI_REGISTRY_IMAGE/kgb-service-stub
dependencies:
- move-container-tests-kgb
Related
Can we make a job depend to an abstract job ?
I'm trying to manage my jobs avoiding writing the same jobs twice by using abstract jobs.
For example, I would like make the deployment jobs to depend on their respective packagings :
.package:...
.package_qual:
extends: .package
script: ...
except: [master]
tags: [...]
.package_prod:
extends: .package
script: ...
only: [master]
tags: [...]
.deploy:...
.deploy_qual:
extends: .deploy
script: ...
environment:
name: qual
only: [develop]
tags: [...]
needs: [.package_qual] #there
.deploy_prod:
extends: .deploy
script: ...
environment:
name: prod
only: [master]
tags: [...]
needs: [.package_prod] #there
include:
- '/A/.gitlab-ci.yml'
- '/B/.gitlab-ci.yml'
A/gitlab.yml
A_package_qual:
extends: .package_qual
variables:
CONTEXT: A
A_package_prod:
extends: .package_prod
variables:
CONTEXT: A
A_deploy_qual:
extends: .deploy_qual
variables:
CONTEXT: A
A_deploy_prod:
extends: .deploy_prod
variables:
CONTEXT: A
B/gitlab.yml
B_package_qual:
extends: .package_qual
variables:
CONTEXT: B
B_package_prod:
extends: .package_prod
variables:
CONTEXT: B
B_deploy_qual:
extends: .deploy_qual
variables:
CONTEXT: A
B_deploy_prod:
extends: .deploy_prod
variables:
CONTEXT: B
When I try this, I have this error A_deploy_qual job: undefined need: .package_qual
I have the following GitLab pipeline. For brevity I have removed some code. In deploy-dev stage I am creating environment dev. Then in the next stage test-postdeploy I am using $CI_ENVIRONMENT_NAME to set the base_url variable value. However, in test-postdeploy stage the $CI_ENVIRONMENT_NAME always empty
stages:
deploy:dev
deploy:qa
deploy:prod
test:postdeploy
deploy-dev:
image: node
stage: deploy:dev
environment:
name: dev
script:
- . deploy/scripts/deploy-dev.sh
rules:
- if: $CI_COMMIT_REF_NAME == "dev"
deploy-qa:
image: node
stage: deploy:qa
environment:
name: qa
script:
- . deploy/scripts/deploy-qa.sh
rules:
- if: $CI_COMMIT_REF_NAME == "qa"
deploy-prod:
image: node
stage: deploy:prod
environment:
name: prod
script:
- . deploy/scripts/deploy-prod.sh
rules:
- if: $CI_COMMIT_REF_NAME == "release"
test-postdeploy:
tags:
- xlarge
image: node
stage: test:postdeploy
variables:
base_url: ""
script:
- echo "$CI_ENVIRONMENT_NAME" // this is empty
- script/postdeploy-test.sh
rules:
- if: $CI_ENVIRONMENT_NAME == "dev"
variables:
base_url: "https://dev.example.com"
- if: $CI_ENVIRONMENT_NAME == "qa"
variables:
base_url: "https://qa.example.com"
- if: $CI_ENVIRONMENT_NAME == "prod"
variables:
base_url: "https://prod.example.com"
UPDATE 1
I can probably pass environment name to another stage using https://docs.gitlab.com/ee/ci/variables/#pass-an-environment-variable-to-another-job
However, this does not make sense. Environment Name is most basic value that any ci/cd pipeline would need to know. The CI_ENVIRONMENT_NAME
variable should be available in all stages once its set without doing some extra work.
I have a pipeline like below:
include:
- project: "some-project"
ref: 0.5.0
file: reusable-jobs.yml
variables:
VAR1: ""
stages:
- stage-1
- stage-2
job1:
extends: .reusable-job-1
stage: stage-1
variables:
SOME_VAR: "$VAR1"
job2:
extends: .reusable-job-2
stage: stage-2
variables:
SOME_VAR: "$VAR1"
I want to be able to save this an imported into other projects. with my reusable jobs I create one like this
.some-reusable-job:
image:
name: alpine
script:
- echo "hello"
and then I extended it when I want to use it
ex:
reuse_the_job:
extends: .some-reusable-job
stage: some-stage
but I cannot figure out how to do the same thing with the entire pipeline including the stages
I want to be able to call the pipeline like so:
reuse_the_pipeline:
extends: .my-reusable-pipeline
variables:
VAR1: "hello"
and have the pipeline be created with both stage_1 and stage_2
I tried to create a definition like this in reusable-flow-file.yml
.reusable-flow
include:
- project: "some-project"
ref: 0.5.0
file: reusable-jobs.yml
variables:
VAR1: ""
stages:
- stage-1
- stage-2
job1:
extends: .reusable-job-1
stage: stage-1
variables:
SOME_VAR: "$VAR1"
job2:
extends: .reusable-job-2
stage: stage-2
variables:
SOME_VAR: "$VAR1"
and then use it like this (.gitlab-ci.yml)
include:
- local: "reusable-flow-file.yml"
dev_na:
extends: .reusable-flow
variables:
VAR1: "hello"
but when I try to run it and get lab I get this error
config should implement a script: or a trigger: keyword
not really sure what that try
It's because you're extending the ".reusable-flow" on a job-level. And what you're actually getting is (the following) which is invalid, because the job dev_na has no script property.
dev_na:
include:
- project: "some-project"
ref: 0.5.0
file: reusable-jobs.yml
variables:
VAR1: ""
stages:
- stage-1
- stage-2
job1:
extends: .reusable-job-1
stage: stage-1
variables:
SOME_VAR: "$VAR1"
job2:
extends: .reusable-job-2
stage: stage-2
variables:
SOME_VAR: "$VAR1"
variables:
VAR1: "hello"
You should just define the jobs (and other stuff) on the root level of your template. And play around a bit in the Project > CI/CD > editor where you can get a life preview of the templated configuration.
I have 10 environments. Each environment should have same manual jobs to deploy 5 services (5 buttons). The only difference between environemnt-1 jobs and environment-2 jobs are tags, variables.only check for commit message and environment.name.
E.g. I should be able to specify [staging] in commit message and have 5 separate buttons that deploy 5 services to staging. Same for [production], etc...
staging-deploy-service1:
<<: *deploy
environment:
name: staging
variables:
ServiceName: service2
tags:
- dev-staging
only:
variables:
- $CI_COMMIT_MESSAGE =~ /\[staging\]/
staging-deploy-service2:
<<: *deploy
environment:
name: staging
variables:
ServiceName = service2
tags:
- dev-staging
only:
variables:
- $CI_COMMIT_MESSAGE =~ /\[staging\]/
staging-deploy-service3:
<<: *deploy
environment:
name: staging
variables:
ServiceName: service3
tags:
- dev-staging
only:
variables:
- $CI_COMMIT_MESSAGE =~ /\[staging\]/
staging-deploy-service4:
<<: *deploy
environment:
name: staging
variables:
ServiceName: service4
tags:
- dev-staging
only:
variables:
- $CI_COMMIT_MESSAGE =~ /\[staging\]/
staging-deploy-service5:
<<: *deploy
environment:
name: staging
variables:
ServiceName: service5
tags:
- dev-staging
only:
variables:
- $CI_COMMIT_MESSAGE =~ /\[staging\]/
Currently I have to repeat this block for each environment - 10 times. So it's 50 jobs for just 5 services.
How can I simplify this?
I don't have the perfect solution because having variable substitution in CI tags is still not possible but if we forget the tags, using parallel matrix can do the trick :
.service_name:
parallel:
matrix:
- SERVICE_NAME: ['service1', 'service2', 'service3', 'service4', 'service5']
ENV: ['env1', 'env2', 'env3', 'env4', 'env5']
deploy_service:
<<: *deploy
<<: *service_name
environment:
name: $ENV
script:
- echo $SERVICE_NAME
rules:
- if: '$CI_COMMIT_MESSAGE =~ $ENV'
when: manual
If you really need to keep one tag per environment, you can still simplify your current config using one-dimensional matrix and using more anchors, ex for staging for all services :
.service_name: &service_name
parallel:
matrix:
- SERVICE_NAME: ['service1', 'service2', 'service3', 'service4', 'service5']
.staging_common: &staging_common
environment:
name: $staging
rules:
- if: '$CI_COMMIT_MESSAGE =~ /\[staging\]/'
when: manual
.dev_staging_tags: &dev_staging_tags
tags:
- dev-staging
staging-deploy:
<<: *deploy
<<: *dev_staging_tags
<<: *staging_common
<<: *service_name
script:
- echo $SERVICE_NAME
Problem
When the pipeline underneath is triggered, the Dev stage is not being run after Build and push.
The pipeline is triggered with a PR with develop as the target branch.
pipeline.yaml
trigger:
branches:
include:
- master
- develop
pr:
branches:
include:
- develop
stages:
# Frontend tests: Triggered by opening new PRs to develop or updating pr to develop.
- stage: FrontEndTests
displayName: "Frontend tests"
condition: eq( variables['Build.Reason'], 'PullRequest') # Trigger stage only for PullRequests
jobs:
- template: /templates/pipelines/npm-unit-tests.yml#templates
parameters:
triggerType: ${{ variables.triggerType }}
# Common build triggered by push to master or develop
- stage: BuildPush
displayName: "Build and push"
condition: ne(variables['Build.Reason'], 'PullRequest') # Don't perform stage if PR triggered pipeline
variables:
envName: "common"
jobs:
- template: /templates/pipelines/dockerbuild-dashboard-client.yml#templates
parameters:
displayName: "Build docker image"
deploymentName: "docker_build_push"
dependsOn: ""
# Dev deploy stage
- stage: dev
displayName: "Dev"
dependsOn: BuildPush
condition: and(succeeded(), ne(variables['Build.SourceBranch'], 'refs/heads/master'))
variables:
envName: "dev"
jobs:
- template: /templates/pipelines/webapprelease-dashboard-dev-client.yml#templates
parameters:
dependsOn: ""
deploymentName: "publish_container_to_webapp"
# Test deploy stage
- stage: test
displayName: "Test"
dependsOn: BuildPush
condition: and(succeeded(), ne(variables['Build.SourceBranch'], 'refs/heads/develop'))
jobs:
- template: /templates/pipelines/webapprelease-dashboard-test-client.yml#templates
parameters:
dependsOn: ""
deploymentName: "publish_container_to_webapp"
# Prod deploy stage
- stage: prod
displayName: "Prod"
dependsOn: test
condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/master'))
variables:
envName: "prod"
jobs:
- template: /templates/pipelines/webapprelease-dashboard-prod-client.yml#templates
parameters:
dependsOn: ""
deploymentName: "publish_container_to_webapp"
Question
Why is the Dev stage not run? It seems to me that the conditions for the dev stage are not met, but I cannot see why.
In your case succeded is evaluated as
With no arguments, evaluates to True only if all previous jobs in the dependency graph succeeded or partially succeeded.
And FrontEndTests was skipped thus while was evaluated as false.
Please change it to
condition: |
and(
or
(
in(dependencies.FrontEndTests.result, 'Succeeded', 'SucceededWithIssues', 'Skipped'),
in(dependencies.BuildPush.result, 'Succeeded', 'SucceededWithIssues', 'Skipped')
),
ne(variables['Build.SourceBranch'], 'refs/heads/master'))
I tested this on this case and it works as above
pool:
vmImage: 'ubuntu-latest'
stages:
- stage: build
displayName: Build
condition: eq( variables['Build.Reason'], 'PullRequest')
jobs:
- job: Build
steps:
- bash: echo "Build"
- stage: test
displayName: Test
condition: succeeded()
jobs:
- job: Test
steps:
- bash: echo "Test"
- stage: test2
displayName: Test2
condition: |
or(
in(dependencies.build.result, 'Succeeded', 'SucceededWithIssues', 'Skipped'),
in(dependencies.test.result, 'Succeeded', 'SucceededWithIssues', 'Skipped')
)
jobs:
- job: Test2
steps:
- bash: echo "Test2"