I'm wondering if using extends: in global variables: is supposed to work.
The documentation here states, that:
Keyword type: Job keyword. You can use it ONLY as part of a job.
(my highlighting).
However, if I use something like
stages:
- test
.common_variables:
__FOO: "foo"
variables:
extends: .common_variables
__BAR: "bar"
test:
stage: test
script:
- env
rules:
- when: always
then I find both __FOO as well as __BAR variables defined on the runner. However, I also find the extends variable (set to .common_variables defined on the runner).
So I wonder if this is the expected behaviour on which one can rely and the presence of extends variable is just a minor side effect or just a bug in the GitLab version that we use and this keyword is supported only inside jobs as the docs state?
P.S. note, that i specifically ask about extends because i plan to off-load the shared variables into a separate file in a separate repo to be able to reuse it in pipelines of multiple repositories
Yes it works like that.
But you can also read in addition how it can be used correctly if you want to run several scripts in one YAML file.
You can read more details here
Related
How can I let gitlab fill a global variable with from CI/CD secret, and then inherit this global variable in other projects?
templates/commons.yml:
variables:
TEST_VAR: $FILLED_FROM_SECRETS
project/.gitlab_ci.yml.
include:
- project: '/templates'
ref: master
file:
- 'commons.yml'
test:
stage: test
script:
- echo $TEST_VAR
Result: the variable is never set. Why?
(of course the FILLED_FORM_SECRETS variable is set in the commons project)
The problem you have is that include: only brings in the contents on the YAML file, not the project level settings or variables.
As possible alternatives, you can:
Set the variable in the template directly (not recommended for sensitive values)
Set variables set on your own self-hosted runners (note variables cannot be masked this way)
Set instance CI/CD variables
Set a required CI configuration to forcibly include a template to all projects (that template can include variables you need) (note variables cannot be masked this way)
Set group CI/CD variables (where all your projects live under the common group)
Retrieve your secrets using the vault integration or as part of your job script
With the include keyword the included files are merged with the .gitlab-ci.yml and then your .gitlab-ci.yml is executed in the repo where the pipeline is triggered. Therefore, only gobal variables in this repo or inherited variables from any parent groups are known. That's why TEST_VAR is not substituted with the value from the secret as the variable is defined in another repository.
According to the doc, the syntax you used require you provide the whole path for your project (all the part after gitlab.com/group/project).
Assuming your project path is gitlab.com/group/my_project, then you choose one of the following
include:
- project: 'group/my_project'
ref: master
file:
- 'templates/commons.yml'
# or simply, if 'templates' folder lives in the same project as your gitlab-ci.yml file
- '/templates/commons.yml'
test:
stage: test
script:
- echo $TEST_VAR
I personally used both ways in my work, but the doc shows other ways to implement this that you can have a look at.
I would like to achieve this Gitlab CI/CD behaviour:
Two jobs defined in the pipeline:
job_beta triggered when a tag is created with a semver number containing a "-rc*" suffix. For example: v1.0.0-rc1.
job_production triggered when a tag is created with a semver number, without any suffix. For example: v1.0.0.
Is there a way to parse the $CI_COMMIT_TAG variable in the .gitlab-ci.yml file of the project? Or any other ideas about how to achieve this?
To achieve this kind of behaviour, you can rely on the only keyword to control which job will be created. Simply specify a regular expression that will either match a semver with the rc suffix or one without any suffix.
You could do it the following way if your semantic versions are prefixed with v:
build-beta:
image: your-build-image:latest
stage: build
only:
- /^v[0-9]+\.[0-9]+\.[0-9]+-rc[0-9]+$/
script:
- ./build-beta.sh # Do something...
build-prod:
image: your-build-image:latest
stage: build
only:
- /^v[0-9]+\.[0-9]+\.[0-9]+$/
script:
- ./build-prod.sh # Do something...
You can also achieve something similar with the rules keyword by matching $CI_COMMIT_REF_NAME with the appropriate regex in an if condition.
When you work on your .gitlab-ci.yml for a big project, for example having a time consuming testing stage causes a lot of delay. Is there an easy way to disable that stage, as just removing it from the stages definition, will make the YAML invalid from Gitlab's point of view (since there's a defined but unused stage), and in my case results in:
test job: chosen stage does not exist; available stages are .pre, build, deploy, .post
Since YAML does not support block comments, you'd need to comment out every line of the offending stage.
Are there quicker ways?
You could disable all the jobs from your stage using this trick of starting the job name with a dot ('.'). See https://docs.gitlab.com/ee/ci/jobs/index.html#hide-jobs for more details.
.hidden_job:
script:
- run test
There is a way to disable individual jobs (but not stages) like this:
test:
stage: test
when: manual
The jobs are skipped by default, but still can be triggered in the UI:
Also possible with rules and when as below:
test:
stage: test
rules:
- when: never
So far, the easiest way I've found is to use a rules definition like so:
test:
stage: test
rules:
- if: '"1" != "1"'
(...)
This still feels a bit odd, so if you have a better solution, I'll gladly accept another answer.
I know that you can reuse blocks of code in a before script using yaml anchors:
.something_before: &something_before
- echo 'something before'
before_script:
- *something_before
- echo "Another script step"
but this doesn't seem to work when the .something_before is declared in a shared .yml file via the include:file. It also does not seem that extends works for before_script. Does anyone know a way of reusing some steps in a before_script from a shared .yml file?
EDIT: My use case is that I have 2 gitlab projects with almost identical before_script steps. I don't want to have to change both projects whenever there's a change, so I have a third, separate Gitlab project that has a .yml template that I am including via include:file in both projects. I want to put all the common code in that shared template, and just have like two lines before_script for the git project that has the two extra steps.
Yaml anchors don't work with included files. You need to use the extends keyword. But what you want to achieve won't work with before_script as code in your template will be overwritten in the job which uses the template if there is a before_script as well.
Do you really need a before_script in your specific job or can you achieve the same with a normal script? If yes you can do something like this:
Template File:
.something_before:
before_script:
- echo 'something before'
- echo 'something more before'
Project Pipeline:
include:
- project: 'my-group/my-project'
file: '/something_before.yml'
stages:
- something
something:
stage: something
extends: .something_before
script:
- echo 'additional stuff to do'
And your before_script section will be merged into the something job and executed before the script part.
See if GitLab 13.6 (November 2020) does make it easier:
Include multiple CI/CD configuration files as a list
Previously, when adding multiple files to your CI/CD configuration using the include:file syntax, you had to specify the project and ref for each file. In this release, you now have the ability to specify the project, ref, and provide a list of files all at once. This prevents you from having to repeat yourself and makes your pipeline configuration less verbose.
See Documentation) and Issue.
And even, with GitLab 14.9 (March 2022):
Include the same CI/CD template multiple times
Previously, trying to have standard CI/CD templates that you reuse in many places was complicated because each template could only be included in a pipeline once.
We dropped this limitation in this release, so you can include the same configuration file as many times as you like.
This makes your CI/CD configuration more flexible as you can define identical includes in multiple nested configurations, and rest assured that there will be no conflicts or duplication.
See Documentation and Issue.
You can use extends without any problem, but you will need to overwrite the entire before_script block.
If you want to change just a piece of your before_script, use a shell script to do it
Set the if condition inside of your template
before_script
- |
if [ condition ]
then
commands here
fi
AFTER EDIT: You can use variables to achieve it
Project 1: VAR = command 1
Project 2: VAR = command 2
You can set the content of env var on the gitlab-ci.yml file or on the CI/CD settings in each project!
I am using Gitlab and added a .gitlab-ci.yml file to trigger my pipeline. What I don't understand is the purpose of refs when used in combination with the only keyword. Here is an example I use:
only:
- /^newlib.*$/
But in another example I found this:
only:
refs:
- master
The documentation only states:
The refs strategy can take the same values as the simplified only/except configuration.
Can anyone share some light on this? What is the difference between the two?
There is no difference between your two examples. As you quoted the docs yourself:
The refs strategy can take the same values as the simplified only/except configuration.
That said, you only need to use refs if you also want to use the other options: changes, kubernetes and variables.
The docs has some examples using multiple options such as:
test:
script: npm run test
only:
refs:
- master
- schedules
variables:
- $CI_COMMIT_MESSAGE =~ /run-end-to-end-tests/
kubernetes: active
If you remove the refs you will get an invalid yml file