Is it possible the nested use of !reference in gitlab-ci.yml? - reference

rules.yml
.rules:
default:
- .gitlab-ci.yml
- Makefile
- VERSION
a_pybuild_deps:
- !reference [.rules, default]
- foo/a/**
b_pybuild_deps:
- !reference [.rules, default]
- foo/b/**
It works if I am simply referring to default:
gitlab-ci.yml
include:
- local: "./rules.yml"
a_pybuild:
...
stage: py_build
script:
...
only:
changes: !reference [.rules, default]
But I want to do the following:
gitlab-ci.yml
include:
- local: "./rules.yml"
a_pybuild:
...
stage: py_build
script:
...
only:
changes: !reference [.rules, a_pybuild_deps]
b_pybuild:
...
stage: py_build
script:
...
only:
changes: !reference [.rules, b_pybuild_deps]
Like this I will get a lint error: jobs:a_pybuild:only changes should be an array of string
I understand the problem, but is there any proper way to apply this?

As of GitLab 14.8, you can use nested !references up to 10 levels deep, but only for script:, before_script: and after_script:.
Nesting of !reference is not allowed in other keys, like only: which is why you get this error.
In versions of GitLab prior to 14.8, nested !references are prohibited entirely.
In your situation, you would simply have to manually duplicate your default rules into a_pybuild_deps and b_pybuild_deps or omit the default rules from those keys entirely and !reference both the default and respective pybuild keys in the job.
This is probably the most flexible and DRY way to do it is to use rules: instead of only: and arrange your rules like so:
# rules.yml
.rules:
default:
changes:
- .gitlab-ci.yml
- Makefile
- VERSION
a_pybuild_deps:
changes:
- foo/a/**
b_pybuild_deps:
changes:
- foo/b/**
#.gitlab-ci.yml
include:
- local: "./rules.yml"
a_pybuild:
script: '...'
rules:
- !reference [.rules, default]
- !reference [.rules, a_pybuild_deps]
b_pybuild:
script: '...'
rules:
- !reference [.rules, default]
- !reference [.rules, b_pybuild_deps]

Related

Cannot get !reference to work in .gitlab-ci.yaml

I am working with GitLab 14.6.2-ee.
I am trying to use the YAML reference feature to re-use code snippets in my GitLab configuration files. I've boiled it down to what I think is the simplest use case, but cannot get the file to validate.
.gitlab-ci.yaml:
...
.hidden:
.update-static-site:
rules:
if: $CI_COMMIT_BRANCH =~ /(^develop$|^master$|^staging-.*)/
...
trigger-downstream-pipeline:
stage: .post
trigger:
project: project-x/project-x-manifest
branch: master
variables:
SOURCE_COMMIT_SHA: $CI_COMMIT_SHA
SOURCE_COMMIT_REF_NAME: $CI_COMMIT_REF_NAME
APP_NAME: project-x
needs:
- job: push-project-x-to-harbor
artifacts: false
rules:
- !reference [.hidden, .update-static-site, rules]
I cannot get the !reference line to validate. I keep getting Unresolved tag: !reference.
I found a SO posting (GitLab CI - Reuse rules with !reference tag) that addresses exactly what I'm trying to do, but it does not work.
So, the main problem was that I needed to change my VS Code config to accept the YAML reference tags. Once that was done, no more validation errors.
When I committed the changes and watched the pipeline, there were no issues.

!reference in dynamic pipelines .yml is being split and not correct format

I have the following YAML file that is dynamically created.
one:
script:
- !reference [.install_env, before_script]
- python -c 'from integration_tests.staging_test_pipeline import run_algorithm; run_algorithm()'
- !reference [.install_env, after_script]
two:
script:
- !reference [.install_env, before_script]
- python -c 'from integration_tests.staging_test_pipeline import run_algorithm; run_algorithm()'
- !reference [.install_env, after_script]
When the trigger tries to use this file I get this error:
Found errors in your .gitlab-ci.yml:
!reference [".install_env"
"before_script"] could not be found
Why is the yaml linter spliting the line?
I am able to use !reference [".install_env", "before_script"] in my .gitlab-ci.yml file just fine.
Need to have
include:
- local: <file that has .install_env>.yml

How reference variables in job rules in gitlab ci?

I need to reuse variables in a gitlab ci job rules
include:
- template: "Workflows/Branch-Pipelines.gitlab-ci.yml"
.staging_variables:
variables:
CONFIG_NAME: "staging"
.staging_rules:
rules:
- if: $CI_COMMIT_BRANCH == $STAGING_BRANCH
variables: !reference [.staging_variables, variables]
stages:
- staging
staging:
stage: staging
rules:
- !reference [.staging_rules, rules]
script:
- echo $CONFIG_NAME
tags:
- staging
However, I am seeing this Syntax is incorrect linting error:
jobs:staging:rules:rule:variables config should be a hash of key value pairs
I am using the example explained here:
https://docs.gitlab.com/ee/ci/yaml/yaml_optimization.html#reference-tags
Please note that I can do this and it works:
include:
- template: "Workflows/Branch-Pipelines.gitlab-ci.yml"
.staging_rules:
rules:
- if: $CI_COMMIT_BRANCH == $STAGING_BRANCH
variables:
CONFIG_NAME: "staging"
stages:
- staging
staging:
stage: staging
rules:
- !reference [.staging_rules, rules]
script:
- echo $CONFIG_NAME
tags:
- staging
Using !reference-keyword in referenced section with !reference is not possible at this moment.
!reference documentation:
You can’t reuse a section that already includes a !reference tag. Only
one level of nesting is supported.
For your needs you could use YAML-anchors. (not tested)
include:
- template: "Workflows/Branch-Pipelines.gitlab-ci.yml"
.staging_variables: &staging_variables
variables:
CONFIG_NAME: "staging"
.staging_rules: &staging_rules
rules:
- if: $CI_COMMIT_BRANCH == $STAGING_BRANCH
variables: *staging_variables
stages:
- staging
staging:
stage: staging
rules:
- *staging_rules
script:
- echo $CONFIG_NAME
tags:
- staging
As stated in the comment above you produce a syntax error for the GitLab CI linting tool which tries to resolve the variables array from the referenced section.
Change your config to have the !reference tag like below:
staging:
stage: staging
rules: !reference [.staging_rules, rules]
script:
- echo $CONFIG_NAME
tags:
- staging
Note that here the - !reference has changed to rules: !reference […]
This should fix your error

GITLAB CI pipeline, run job only with git tag

need help from GitLab gurus. I have a following pipeline below.
I expect "sync_s3:prod" job will run only when i will push new git tag. But gitlab trigger both
jobs. Why its behaving like this ? I create $git_commit_tag rule only for one job. Any ideas?
stages:
- sync:nonprod
- sync:prod
.sync_s3:
image:
name: image
entrypoint: [""]
script:
- aws configure set region eu-west-1
- aws s3 sync ${FOLDER_ENV} s3://img-${AWS_ENV} --delete
sync_s3:prod:
stage: sync:prod
rules:
- if: $CI_COMMIT_TAG
changes:
- prod/*
extends: .sync_s3
variables:
AWS_ENV: prod
FOLDER_ENV: prod/
tags:
- gaming_prod
sync_s3:nonprod:
stage: sync:nonprod
rules:
- changes:
- pp2/*
extends: .sync_s3
variables:
AWS_ENV: nonprod
FOLDER_ENV: pp2/
tags:
- gaming_nonprod
If I understand the question correctly, you do not want to have the sync_s3:nonprod job run if the sync_s3:prod is run. (?)
To achieve this, on the sync_s3:nonprod job you should be able to copy the same rule from sync_s3:prod together with when: never:
stages:
- sync:nonprod
- sync:prod
.sync_s3:
image:
name: image
entrypoint: [""]
script:
- aws configure set region eu-west-1
- aws s3 sync ${FOLDER_ENV} s3://img-${AWS_ENV} --delete
sync_s3:prod:
stage: sync:prod
rules:
- if: $CI_COMMIT_TAG
changes:
- prod/*
extends: .sync_s3
variables:
AWS_ENV: prod
FOLDER_ENV: prod/
tags:
- gaming_prod
sync_s3:nonprod:
stage: sync:nonprod
rules:
- if: $CI_COMMIT_TAG
changes:
- prod/*
when: never
- changes:
- pp2/*
extends: .sync_s3
variables:
AWS_ENV: nonprod
FOLDER_ENV: pp2/
tags:
- gaming_nonprod
As #slauth already mentions in his answer the rules need to be adjusted per step of the pipeline. I only post this as an answer as an addition to the original answer above.
In order to prevent pipeline steps from running when a git-tag is present you need to explicitly set the rule for the corresponding job.
stages:
- sync:nonprod
- sync:prod
.sync_s3:
image:
name: image
entrypoint: [""]
script:
- aws configure set region eu-west-1
- aws s3 sync ${FOLDER_ENV} s3://img-${AWS_ENV} --delete
sync_s3:prod:
stage: sync:prod
rules:
- if: $CI_COMMIT_TAG
changes:
- prod/*
extends: .sync_s3
variables:
AWS_ENV: prod
FOLDER_ENV: prod/
tags:
- gaming_prod
sync_s3:nonprod:
stage: sync:nonprod
rules:
- changes:
- pp2/*
- if: $CI_COMMIT_TAG
when: never
extends: .sync_s3
variables:
AWS_ENV: nonprod
FOLDER_ENV: pp2/
tags:
- gaming_nonprod
For further clarification:
The following rule will evaluate similar to a logic AND, so this will evaluate to true if there is a $CI_COMMIT_TAG AND there are changes in prod/*. So only when both conditions are met this will be added to the pipeline.
rules:
- if: $CI_COMMIT_TAG
changes:
- prod/*

Gitlab CI parent/child pipelines with complex subfolders

i have a problem with gitlab (community edition, version 14.1.2) CI with complex pipeline on my monorepo.
My structure is client/server:
root/
---- server/
-------- lib/
----------- libA/
----------- libB/
----------- libC/
-------- applications/
----------- appA/
----------- appB/
----------- appC/
---- client/
-------- applications/
------------- appA/
------------- appB/
...
Every folder (root, server, lib, libA, libB, libC etc...) have his own ".gitlab-ci.yml"
Root ".gitlab-ci.yml" is:
stages:
- build
- test
build-server:
stage: build
trigger:
include:
- local: 'server/.gitlab-ci.yml'
rules:
- changes:
- server/**/*
build-client:
stage: build
trigger:
include:
- local: 'client/.gitlab-ci.yml'
rules:
- changes:
- client/**/*
Server ".gitlab-ci.yml" is:
stages:
- build
- test
build-lib:
stage: build
trigger:
include:
- local: 'lib/.gitlab-ci.yml'
rules:
- changes:
- lib/**/*
build-applications:
stage: build
trigger:
include:
- local: 'applications/.gitlab-ci.yml'
rules:
- changes:
- applications/**/*
lib ".gitlab-ci.yml" is:
stages:
- build
- test
build-libA:
stage: build
script:
- echo "Execute libA build!"
rules:
- changes:
- libA/**/*
build-libB:
stage: build
script:
- echo "Execute libB build!"
rules:
- changes:
- libB/**/*
If i change a file inside libA only the ".gitlab-ci.yml" of root folder is triggered, other subfolders not detect file changes and not trigger the build.
The purpose of this configuration is that , for example, when i change a file inside libA, the pipeline detects the changes and build the libA.
Somone can help me to resolve? I hope the structure and the problem is clear. Thanks.
UPDATE
I'm using gitlab 14.1.0
Thanks to DavidC for the answer but with your solution I have not solved my problem, especially with the trigger $CI_PROJECT_PATH seems not to work.
After some time I finally got a solution (which can be evolved with variables)
Root ".gitlab-ci.yml" is:
stages:
- build
- test
build-server:
stage: build
trigger:
include:
- local: '/server/.gitlab-ci.yml'
rules:
- changes:
- server/**/*
build-client:
stage: build
trigger:
include:
- local: '/client/.gitlab-ci.yml'
rules:
- changes:
- client/**/*
Server ".gitlab-ci.yml" is:
stages:
- build
- test
build-lib:
stage: build
trigger:
include:
- local: '/server/lib/.gitlab-ci.yml'
rules:
- changes:
- server/lib/**/*
build-applications:
stage: build
trigger:
include:
- local: '/server/applications/.gitlab-ci.yml'
rules:
- changes:
- server/applications/**/*
lib ".gitlab-ci.yml" is:
stages:
- build
- test
build-libA:
stage: build
script:
- echo "Execute libA build!"
rules:
- changes:
- server/lib/libA/**/*
build-libB:
stage: build
script:
- echo "Execute libB build!"
rules:
- changes:
- server/lib/libB/**/*
Pay attention to this line from the gitlab documentation: "Parent and child pipelines were introduced with a maximum depth of one child pipeline level, which was subsequently increased to two. A parent pipeline can activate many child pipelines and these child pipelines can activate their own child pipelines. It is not possible to activate another level of child pipeline. " refer to: https://docs.gitlab.com/ee/ci/pipelines/parent_child_pipelines.html#nested-child-pipelines
Thanks for help!
It seems like GitLab child-pipeline context execution path is the same as the root directory of your repository, and is not relative to the path of the child-pipeline gitlab-ci.yml file.
The only solution so far seems to either give the path to the directory you want as a variable to your child-pipeline or to directly define it in the .gitlab-ci.yml.
Example :
Root ".gitlab-ci.yml" is:
stages:
- build
- test
build-server:
stage: build
variables:
CI_ROOT_DIR: server # you don't need to provide it, if you define it in the server/.gitlab-ci.yml file
trigger:
include:
- local: '$CI_ROOT_DIR/.gitlab-ci.yml'
rules:
- changes:
- $CI_ROOT_DIR/**/*
build-client:
stage: build
variables:
CI_ROOT_DIR: client
trigger:
include:
- local: '$CI_ROOT_DIR/.gitlab-ci.yml'
rules:
- changes:
- $CI_ROOT_DIR/**/*
Server ".gitlab-ci.yml" is:
stages:
- build
- test
variables:
CI_ROOT_DIR: $CI_PROJECT_PATH/server # default
build-lib:
stage: build
variables:
CI_ROOT_DIR: $CI_ROOT_DIR/lib # you don't need to provide it, if you define it in the server/lib/.gitlab-ci.yml file
trigger:
include:
- local: '$CI_ROOT_DIR/.gitlab-ci.yml'
rules:
- changes:
- $CI_ROOT_DIR/**/*
build-applications:
stage: build
variables:
CI_ROOT_DIR: $CI_ROOT_DIR/applications # you don't need to provide it, if you define it in the server/applications/.gitlab-ci.yml file
trigger:
include:
- local: '$CI_ROOT_DIR/.gitlab-ci.yml'
rules:
- changes:
- $CI_ROOT_DIR/**/*
lib ".gitlab-ci.yml" is:
stages:
- build
- test
variables:
CI_ROOT_DIR: $CI_PROJECT_PATH/server/lib # default
build-libA:
stage: build
script:
- echo "Execute libA build!"
rules:
- changes:
- $CI_ROOT_DIR/libA/**/*
build-libB:
stage: build
script:
- echo "Execute libB build!"
rules:
- changes:
- $CI_ROOT_DIR/libB/**/*
It would be better tho if it was possible to choose the context of the execution of the pipeline when triggered by the parent-pipeline or to have a CI_CHILD_PIPELINE_DIR variable available by Gitlab predefined environment variables

Resources