Can we make a job depend to an abstract job ?
I'm trying to manage my jobs avoiding writing the same jobs twice by using abstract jobs.
For example, I would like make the deployment jobs to depend on their respective packagings :
.package:...
.package_qual:
extends: .package
script: ...
except: [master]
tags: [...]
.package_prod:
extends: .package
script: ...
only: [master]
tags: [...]
.deploy:...
.deploy_qual:
extends: .deploy
script: ...
environment:
name: qual
only: [develop]
tags: [...]
needs: [.package_qual] #there
.deploy_prod:
extends: .deploy
script: ...
environment:
name: prod
only: [master]
tags: [...]
needs: [.package_prod] #there
include:
- '/A/.gitlab-ci.yml'
- '/B/.gitlab-ci.yml'
A/gitlab.yml
A_package_qual:
extends: .package_qual
variables:
CONTEXT: A
A_package_prod:
extends: .package_prod
variables:
CONTEXT: A
A_deploy_qual:
extends: .deploy_qual
variables:
CONTEXT: A
A_deploy_prod:
extends: .deploy_prod
variables:
CONTEXT: A
B/gitlab.yml
B_package_qual:
extends: .package_qual
variables:
CONTEXT: B
B_package_prod:
extends: .package_prod
variables:
CONTEXT: B
B_deploy_qual:
extends: .deploy_qual
variables:
CONTEXT: A
B_deploy_prod:
extends: .deploy_prod
variables:
CONTEXT: B
When I try this, I have this error A_deploy_qual job: undefined need: .package_qual
Related
This is the relevant job in the yml file:
#container structure test for check service stub
container-test-command-check:
only:
changes:
- stub-services/check-service/**/*
extends: container-test-command
<<: *branches
allow_failure: true
variables
CI_REGISTRY_IMAGE: $CI_REGISTRY_IMAGE/check-stub
dependencies:
- move-container-tests-check
My intention is that this job should only run if any files in the folder 'stub-services/check-service' have been altered. Unfortunately, it runs regardless of whether there have been any changes or not.
Here's the full yml file with non-relevant parts redacted for the sake of clarity and commercial confidentiality:
include:
- '/gitlab-ci/includes.yml'
- '/idt-test-stub/gitlab-ci.yml'
variables:
ALPINE_VERSION: "3.16"
NODE_VERSION: "14-alpine"
ESLINT_CONFIG_PATH: "./.eslintrc.js"
SOURCE_DIR: "."
RULESET: MAVEN_CI
BUILD_TYPE: MAVEN
MVN_CLI_OPTS: "--batch-mode"
MVN_OPTS: "-Dmaven.repo.local=.m2-local -f wiremock-java/pom.xml"
MAVEN_IMAGE: "maven:3-jdk-11"
stages:
- test
- code-quality
- code-test
- code-analysis
- verify
- transform
- application-build
- image-build
- move-container-tests
- container-image-test
- image-push
.branches: &branches
only:
- branches
todo-check:
<<: *branches
shell-check:
<<: *branches
lint-source:
<<: *branches
docker-lint:
<<: *branches
unit-test:
<<: *branches
artifacts:
expire_in: 20 mins
paths:
- ./coverage
sonar-scanner:
extends: .sonar-scanner-base
<<: *branches
stage: code-analysis
node-audit:
<<: *branches
node-outdated:
<<: *branches
allow_failure: true
verify:
<<: *branches
stage: verify
image: node:$NODE_VERSION
script:
- npm install
- ./run-verify.sh
tags:
- docker-in-docker-privileged
transform:
<<: *branches
stage: transform
image: node:$NODE_VERSION
script:
- npm install
- ./run-transform.sh
artifacts:
name: "wiremock-json"
paths:
- stub-services/check-service/wiremock/stubs
- stub-services/kgb-service/wiremock/stubs
tags:
- docker-in-docker-privileged
maven-package:
<<: *branches
artifacts:
paths:
- ./wiremock-java/target
expire_in: 1 hour
docker-build:
extends: .docker-build-template
<<: *branches
parallel:
matrix:
- DOCKERFILE_PATH: "./Dockerfiles/Dockerfile.check-service"
CI_REGISTRY_IMAGE: $CI_REGISTRY_IMAGE/check-stub
ADD_LATEST_TAG: "true"
ADD_GIT_TAGS: "true"
CI_COMMIT_TAG: "${CI_COMMIT_REF_SLUG}"
- DOCKERFILE_PATH: "./Dockerfiles/Dockerfile.kgb-service"
CI_REGISTRY_IMAGE: $CI_REGISTRY_IMAGE/kgb-service-stub
ADD_LATEST_TAG: "true"
ADD_GIT_TAGS: "true"
CI_COMMIT_TAG: "${CI_COMMIT_REF_SLUG}"
except:
variables:
- $PROD_AWS_ACCESS_KEY_ID
- $PROD_AWS_SECRET_ACCESS_KEY
- $PROD_AWS_SESSION_TOKEN
tags:
- docker-in-docker
dependencies:
- transform
- maven-package
#sast container test for service stubs
sast-container:
<<: *branches
allow_failure: true
parallel:
matrix:
- CI_REGISTRY_IMAGE: $CI_REGISTRY_IMAGE/check-stub
- CI_REGISTRY_IMAGE: $CI_REGISTRY_IMAGE/kgb-service-stub
trivy-container:
extends: .trivy-container-base
<<: *branches
parallel:
matrix:
- CI_REGISTRY_IMAGE: $CI_REGISTRY_IMAGE/check-stub
- CI_REGISTRY_IMAGE: $CI_REGISTRY_IMAGE/kgb-service-stub
allow_failure: true
#move container test json into container-tests, for check-service stub
#so container-tests folder will be picked up by container-test-command below
move-container-tests-check:
<<: *branches
stage: move-container-tests
image: alpine:$ALPINE_VERSION
script:
- cp -R container-test-folders/check-service container-tests
tags:
- docker-in-docker-privileged
artifacts:
paths:
- container-tests
#container structure test for check service stub
container-test-command-check:
only:
changes:
- stub-services/check-service/**/*
extends: container-test-command
<<: *branches
allow_failure: true
variables:
CI_REGISTRY_IMAGE: $CI_REGISTRY_IMAGE/check-stub
dependencies:
- move-container-tests-check
#move container test json into container-tests, for kgb-service stub
#so container-tests folder will be picked up by container-test-command below
move-container-tests-kgb:
<<: *branches
stage: move-container-tests
image: alpine:$ALPINE_VERSION
script:
- cp -R container-test-folders/kgb-service container-tests
tags:
- docker-in-docker-privileged
artifacts:
paths:
- container-tests
#container structure test for kgb service stub
container-test-command-kgb:
extends: container-test-command
only:
changes:
- stub-services/kgb-service/**/*
<<: *branches
allow_failure: true
variables:
CI_REGISTRY_IMAGE: $CI_REGISTRY_IMAGE/kgb-service-stub
dependencies:
- move-container-tests-kgb
I have the following GitLab pipeline. For brevity I have removed some code. In deploy-dev stage I am creating environment dev. Then in the next stage test-postdeploy I am using $CI_ENVIRONMENT_NAME to set the base_url variable value. However, in test-postdeploy stage the $CI_ENVIRONMENT_NAME always empty
stages:
deploy:dev
deploy:qa
deploy:prod
test:postdeploy
deploy-dev:
image: node
stage: deploy:dev
environment:
name: dev
script:
- . deploy/scripts/deploy-dev.sh
rules:
- if: $CI_COMMIT_REF_NAME == "dev"
deploy-qa:
image: node
stage: deploy:qa
environment:
name: qa
script:
- . deploy/scripts/deploy-qa.sh
rules:
- if: $CI_COMMIT_REF_NAME == "qa"
deploy-prod:
image: node
stage: deploy:prod
environment:
name: prod
script:
- . deploy/scripts/deploy-prod.sh
rules:
- if: $CI_COMMIT_REF_NAME == "release"
test-postdeploy:
tags:
- xlarge
image: node
stage: test:postdeploy
variables:
base_url: ""
script:
- echo "$CI_ENVIRONMENT_NAME" // this is empty
- script/postdeploy-test.sh
rules:
- if: $CI_ENVIRONMENT_NAME == "dev"
variables:
base_url: "https://dev.example.com"
- if: $CI_ENVIRONMENT_NAME == "qa"
variables:
base_url: "https://qa.example.com"
- if: $CI_ENVIRONMENT_NAME == "prod"
variables:
base_url: "https://prod.example.com"
UPDATE 1
I can probably pass environment name to another stage using https://docs.gitlab.com/ee/ci/variables/#pass-an-environment-variable-to-another-job
However, this does not make sense. Environment Name is most basic value that any ci/cd pipeline would need to know. The CI_ENVIRONMENT_NAME
variable should be available in all stages once its set without doing some extra work.
I have a pipeline like below:
include:
- project: "some-project"
ref: 0.5.0
file: reusable-jobs.yml
variables:
VAR1: ""
stages:
- stage-1
- stage-2
job1:
extends: .reusable-job-1
stage: stage-1
variables:
SOME_VAR: "$VAR1"
job2:
extends: .reusable-job-2
stage: stage-2
variables:
SOME_VAR: "$VAR1"
I want to be able to save this an imported into other projects. with my reusable jobs I create one like this
.some-reusable-job:
image:
name: alpine
script:
- echo "hello"
and then I extended it when I want to use it
ex:
reuse_the_job:
extends: .some-reusable-job
stage: some-stage
but I cannot figure out how to do the same thing with the entire pipeline including the stages
I want to be able to call the pipeline like so:
reuse_the_pipeline:
extends: .my-reusable-pipeline
variables:
VAR1: "hello"
and have the pipeline be created with both stage_1 and stage_2
I tried to create a definition like this in reusable-flow-file.yml
.reusable-flow
include:
- project: "some-project"
ref: 0.5.0
file: reusable-jobs.yml
variables:
VAR1: ""
stages:
- stage-1
- stage-2
job1:
extends: .reusable-job-1
stage: stage-1
variables:
SOME_VAR: "$VAR1"
job2:
extends: .reusable-job-2
stage: stage-2
variables:
SOME_VAR: "$VAR1"
and then use it like this (.gitlab-ci.yml)
include:
- local: "reusable-flow-file.yml"
dev_na:
extends: .reusable-flow
variables:
VAR1: "hello"
but when I try to run it and get lab I get this error
config should implement a script: or a trigger: keyword
not really sure what that try
It's because you're extending the ".reusable-flow" on a job-level. And what you're actually getting is (the following) which is invalid, because the job dev_na has no script property.
dev_na:
include:
- project: "some-project"
ref: 0.5.0
file: reusable-jobs.yml
variables:
VAR1: ""
stages:
- stage-1
- stage-2
job1:
extends: .reusable-job-1
stage: stage-1
variables:
SOME_VAR: "$VAR1"
job2:
extends: .reusable-job-2
stage: stage-2
variables:
SOME_VAR: "$VAR1"
variables:
VAR1: "hello"
You should just define the jobs (and other stuff) on the root level of your template. And play around a bit in the Project > CI/CD > editor where you can get a life preview of the templated configuration.
Need a small help , can someone point me what is the issue here .
What i am trying to achieve , the pipeline has to run checkout irrespective of the branch i am in , but in the build stage if the pipeline run from master it has execute some templet if from other branch different templet , i tried many option no luck .
Any pointer much appreciated .
- name: release
displayName: ReleaseVersion
type: string
default: ' '
- name: Deployment
displayName: DeploymentVersion
type: string
default: ' '
- name: Library
displayName: Library Release
type: boolean
default: True
trigger:
- none
pool:
name: DOTAzure-Ubuntu-20.04
resources:
repositories:
- repository: azureTemplates
type: githubenterprise
name: AAAAA/azure-pipelines
endpoint: BBBBBB
# You should add below variables to work complete pipeline
variables:
- name: servicename
value: XXXXX
- name: countryCode
value: YYYYY
- name: targetPort
value: 8113
stages:
- stage: Checkout
displayName: Micro Service checkout and Secrets
jobs:
- job:
steps:
- template: templates/repocheckout.yml#azureTemplates
- template: templates/retrieving_secrets.yml#azureTemplates
- template: templates/Ingesting_secrets.yml#azureTemplates
- stage: Build
displayName: Microservice Build
jobs:
- job: Master
condition: eq(variables['Build.SourceBranchName'], 'master')
steps:
- template: templates/mavenbuild_new.yml#azureTemplates
- template: templates/push_artifact_jfrog.yml#azureTemplates
parameters:
service_name: '$(countryCode)-$(servicename)'
release_version: $(release)
- job: Develop
dependsOn: Master
condition: eq(variables['Build.SourceBranchName'], 'develop')
steps:
- template: templates/mavenbuild.yml#azureTemplates
- template: templates/push_artifact_jfrog.yml#azureTemplates
parameters:
service_name: '$(countryCode)-$(servicename)'
release_version: $(release)
This doesn't look like valid yaml.
Have a look at this: https://learn.microsoft.com/en-us/azure/devops/pipelines/yaml-schema?view=azure-devops&tabs=schema%2Cparameter-schema
It looks like you are missing this at the top of the file:
stages:
i.e. your yaml should be:
stages:
- stage: Checkout
displayName: Micro Service checkout and Secrets
jobs:
etc
I have 10 environments. Each environment should have same manual jobs to deploy 5 services (5 buttons). The only difference between environemnt-1 jobs and environment-2 jobs are tags, variables.only check for commit message and environment.name.
E.g. I should be able to specify [staging] in commit message and have 5 separate buttons that deploy 5 services to staging. Same for [production], etc...
staging-deploy-service1:
<<: *deploy
environment:
name: staging
variables:
ServiceName: service2
tags:
- dev-staging
only:
variables:
- $CI_COMMIT_MESSAGE =~ /\[staging\]/
staging-deploy-service2:
<<: *deploy
environment:
name: staging
variables:
ServiceName = service2
tags:
- dev-staging
only:
variables:
- $CI_COMMIT_MESSAGE =~ /\[staging\]/
staging-deploy-service3:
<<: *deploy
environment:
name: staging
variables:
ServiceName: service3
tags:
- dev-staging
only:
variables:
- $CI_COMMIT_MESSAGE =~ /\[staging\]/
staging-deploy-service4:
<<: *deploy
environment:
name: staging
variables:
ServiceName: service4
tags:
- dev-staging
only:
variables:
- $CI_COMMIT_MESSAGE =~ /\[staging\]/
staging-deploy-service5:
<<: *deploy
environment:
name: staging
variables:
ServiceName: service5
tags:
- dev-staging
only:
variables:
- $CI_COMMIT_MESSAGE =~ /\[staging\]/
Currently I have to repeat this block for each environment - 10 times. So it's 50 jobs for just 5 services.
How can I simplify this?
I don't have the perfect solution because having variable substitution in CI tags is still not possible but if we forget the tags, using parallel matrix can do the trick :
.service_name:
parallel:
matrix:
- SERVICE_NAME: ['service1', 'service2', 'service3', 'service4', 'service5']
ENV: ['env1', 'env2', 'env3', 'env4', 'env5']
deploy_service:
<<: *deploy
<<: *service_name
environment:
name: $ENV
script:
- echo $SERVICE_NAME
rules:
- if: '$CI_COMMIT_MESSAGE =~ $ENV'
when: manual
If you really need to keep one tag per environment, you can still simplify your current config using one-dimensional matrix and using more anchors, ex for staging for all services :
.service_name: &service_name
parallel:
matrix:
- SERVICE_NAME: ['service1', 'service2', 'service3', 'service4', 'service5']
.staging_common: &staging_common
environment:
name: $staging
rules:
- if: '$CI_COMMIT_MESSAGE =~ /\[staging\]/'
when: manual
.dev_staging_tags: &dev_staging_tags
tags:
- dev-staging
staging-deploy:
<<: *deploy
<<: *dev_staging_tags
<<: *staging_common
<<: *service_name
script:
- echo $SERVICE_NAME