I've got something like this in my .gitlab-ci.yml:
.templates:
- &deploy-master-common
script:
- ansible-playbook … --limit=${environment:name}.example.org
environment:
url: https://${environment:name}.example.org
…
deploy-master1:
<<: *deploy-master-common
environment:
name: master1
only:
- master
When running the deploy-master1 job, unfortunately, ${environment:name} is expanded to the empty string. Is this sort of expansion not supported by YAML/GitLab CI?
I can't tell yet whether this is a GitLab or YAML restriction, but it looks like the environment hash is being replaced rather than merged. Moving <<: *deploy-master-common to the bottom of deploy-master1 I get the following error message from the GitLab CI lint API endpoint:
.gitlab-ci.yml is not valid. Errors:
[
"jobs:deploy-master1:environment name can't be blank"
]
I was able to work around this with custom variables because the template doesn't specify any:
.templates:
- &deploy-master-common
script:
- ansible-playbook … --limit=${environment_name}.example.com
environment:
name: $environment_name
url: https://${environment_name}.example.com
…
deploy-master1:
<<: *deploy-master-common
variables:
environment_name: master1
only:
- master
Related
This is my gitlab-ci.yml:
workflow:
rules:
- if: $CI_COMMIT_MESSAGE =~ /-draft$/
when: never
- if: $CI_PIPELINE_SOURCE == "push"
include:
- '/gitlab-ci/includes.yml'
- '/idt-test-stub/gitlab-ci.yml'
variables:
ALPINE_VERSION: "3.16"
NODE_VERSION: "14-alpine"
ESLINT_CONFIG_PATH: "./.eslintrc.js"
SOURCE_DIR: "."
BUILD_TYPE: MAVEN
MVN_CLI_OPTS: "--batch-mode"
MVN_OPTS: "-Dmaven.repo.local=.m2-local -f wiremock-java/pom.xml"
MAVEN_IMAGE: "maven:3-jdk-11"
stages:
- test
- code-quality
- code-test
- code-analysis
- verify
- transform
- application-build
- image-build
- move-container-tests
- container-image-test
- image-push
.branches: &branches
only:
- branches
todo-check:
<<: *branches
shell-check:
<<: *branches
docker-lint:
<<: *branches
unit-test:
<<: *branches
artifacts:
expire_in: 20 mins
paths:
- ./coverage
verify:
<<: *branches
stage: verify
image: node:$NODE_VERSION
script:
- npm install
- ./run-verify.sh
tags:
- docker-in-docker-privilegedx
My understand is that if I commit a change including the word 'draft' in my commit message, when I push it, my workflow rules should stop the pipeline from running and yet it doesn't.
All the jobs in the pipeline get run.
The workflow rules part is copied directly from the Gitlab documentation which says this:
In the following example:
Pipelines run for all push events (changes to branches and new tags).
Pipelines for push events with -draft in the commit message don’t run, because they are set to when: never.
Pipelines for schedules or merge requests don’t run either, because no rules evaluate to true for them.
It's because of the $ at the end of /-draft$/. $ is a special character in regular expressions and matches the end of a piece of text.
Even trying to use $ for its purpose in Gitlab and using a commit message like 'Commit -draft' still fails though. I don't know why. Perhaps they don't support the $ character at all.
If you just want to check whether your commit message contains the word draft or not, you can omit the - and the $ at the end:
workflow:
rules:
- if: $CI_COMMIT_MESSAGE =~ /draft/
when: never
I tested it on Gitlab:
PS. I've created a merge request on Gitlab to fix this part of their docs, so let's see what they say.
Im trying pass a user defined variable from a custom pipeline to a step defined within the definitions section.
My yml snippet is below:
definitions:
steps:
- step: &TagVersion
trigger: manual
script:
- export VERSION=$VERSION
- echo "VERSION $VERSION"
custom:
run_custom:
- variables:
- name: VERSION
- step:
script:
- echo "starting"
- parallel:
- step:
<<: *TagVersion
variables:
VERSION: $VERSION
When I build the pipeline I can see the variable is listed as a pipeline variable when running the step TagVersion, and the correct value is shown there but dont know how to use this within the scripts section where im trying to echo out the value.
thanks
I create template for gitlab-ci with script
.test-script: &test
- echo "hello"
How to use this script in .gitlab-ci.yml? I tried like this
include: '/templates/test-template.yml'
example-stage:
script:
- *test
Error occurs "This GitLab CI configuration is invalid: Unknown alias: test." because there is no alias for this script in the merged YAML.
GitLab version: GitLab Community Edition 14.6.0
Why not use GitLab's extends or reference keywords?
---
include:
- local: '/templates/test-template.yml'
example-stage:
script:
- !reference [.test-script, script]
or
---
include:
- local: '/templates/test-template.yml'
example-stage:
extends: .test-script
Maybe you could try this syntax :
.test-script: &test-script
script:
- echo "hello"
For your example-stage job :
example-stage:
<<: *test-script
We have a collection of Azure DevOps pipeline templates that we re-use across multiple repositories. Therefore we wanted to have a file that contains variables for all of our templates.
The repo structure looks like this
template repo
├── template-1.yml
├── template-2.yml
└── variables.yml
project repo
├── ...
└── azure-pipelines.yml
The variables.yml looks like this
...
variables:
foo: bar
In template-1.yml we are importing the variables.yml as described in here
variables:
- template: variables.yml
In the azure-pipelines.yml we are using the template like this
resources:
repositories:
- repository: build-scripts
type: git
name: project-name/build-scripts
steps:
...
- template: template-1.yml#build-scripts
When we now try to run the pipeline, we get the following error message:
template-1.yml#build-scripts (Line: 10, Col: 1): Unexpected value 'variables'
The issue is because you used variable template at steps scope. And variables simply doesn't exists at that level. This should work for you:
resources:
repositories:
- repository: build-scripts
type: git
name: project-name/build-scripts
variables:
- template: template-1.yml#build-scripts
steps:
...
this is available to use at any place where variables are possible to use. So for instance you can use this in that way:
jobs:
- job: myJob
timeoutInMinutes: 10
variables:
- template: template-1.yml # Template reference
pool:
vmImage: 'ubuntu-16.04'
steps:
- script: echo My favorite vegetable is ${{ variables.favoriteVeggie }}.
If your template file only has variables, you can refer to Krzysztof Madej's answer.
If your template file has both variables and steps as shown below, it can only be used by extends.
# File: template-1.yml
variables: ...
steps: ...
Or you can write them in a stage, as shown below.
# File: template-1.yml
stages:
- stage: {stage}
variables: ...
jobs:
- job: {job}
steps: ...
Then insert it as a separate stage.
# azure-pipelines.yml
stages:
- stage: ...
- template: template-1.yml
Replying as I think there was something missed in the original explanations:
First:
If you are referencing variables in a template then you must ensure that you are extending that template when you call it. Calling a template that defines variables that is not extended will result in a failed pipeline execution.
template-1.yml#build-scripts (Line: 10, Col: 1): Unexpected value 'variables'
Second: When referencing variables in multiple templates do not use:
${{ variables.foo }} #even though this is seen in templates examples.
but rather use the normal variable syntax
$(foo) #this works
So for the originally reference example the following should work.
Variables.yml:
variables:
foo: bar
template-1.yml:
variables:
- template: variables.yml
....
steps:
...
- task: CmdLine#2
displayName: Display Variable
inputs:
script: |
echo $(foo)
azure-pipelines.yml:
resources:
repositories:
- repository: build-scripts
type: git
name: project-name/build-scripts
extends:
template: template-1.yml#build-scripts
template-2.yml
steps:
...
- task: CmdLine#2
displayName: Display Variable
inputs:
script: |
echo $(foo)
This approach can help someone, so I decided to post it here.
One more kind of "workaround" for that case, maybe a bit "dirty" since you need to specify the parameters explicitly each time you execute template, which is not a good idea if you have a lot of parameters to pass. (actually, there is a way, read below improved version) But, if you really want or you have not that much parameters, this should work:
the logic is: you have all your parameters in variables template, like templates/vars.yml:
variables:
- name: myVar1
value: MyValue1
- name: myVar2
value: MyValue2
and since you have everything you need in the variables, probably there is no need to importing variables into template itself, because template will be executed in pipeline, which will have your variables imported and you can substitute it explicitly, like in the example below:
templates/my-template-setup-env.yml's content (without variables inside it):
steps:
- script: |
echo "$(myVar3)"
my-azure-pipeline.yml's content (with importing variables template):
name: my-cute-CI-name
trigger:
- main
pool:
vmImage: ubuntu-18.04
variables:
- template: templates/vars.yml # importing your variables from templates
stages:
- stage: RunTheStage
displayName: 'Run first stage'
jobs:
- job: RunTheJob
displayName: 'Run your job'
steps:
- template: templates/my-template-setup-env.yml # your template
parameters:
myVar3: $(myVar1) # substitute value from vars.yml, so myVar1 will be used in templated and printed
Improved version
But, if you have unique naming of your params and variables across all pipelines and templates, you are in safe to not specify it explicitly during template usage, that will work as well:
edited and shortened version of my-azure-pipeline.yml (in case you have the same name of your variable and parameter in template):
variables:
- template: templates/vars.yml
...
steps:
- template: templates/my-template-setup-env.yml # env preparation
# parameters:
# myVar2: $(myVar2) # you don't need to pass any parameters explicitly to the template since you have same name of variable
templates/my-template-setup-env.yml then should be like this:
steps:
- script: |
echo "$(myVar2)" # not myVar3, since myVar3 is not in initial variables file templates/vars.yml
or you need to add remaining variables (myVar3 in our first case) into templates/vars.yml file as well.
Is there a way to configure multiple specifically-named environments (specifically, test, stage, and prod)?
In their documentation (https://docs.gitlab.com/ce/ci/environments.html) they talk about dynamically-created environments, but they are all commit based.
My build steps are the same for all of them, save for swapping out the slug:
deploy_to_test:
environment:
name: test
url: ${CI_ENVIRONMENT_SLUG}.mydomain.com
scripts:
- deploy ${CI_ENVIRONMENT_SLUG}
deploy_to_stage:
environment:
name: stage
url: ${CI_ENVIRONMENT_SLUG}.mydomain.com
scripts:
- deploy ${CI_ENVIRONMENT_SLUG}
deploy_to_prod:
environment:
name: prod
url: ${CI_ENVIRONMENT_SLUG}.mydomain.com
scripts:
- deploy ${CI_ENVIRONMENT_SLUG}
Is there any way to compress this down into one set of instructions? Something like:
deploy:
environment:
url: ${CI_ENVIRONMENT_SLUG}.mydomain.com
scripts:
- deploy ${CI_ENVIRONMENT_SLUG}
Yes, you can use anchors. If I follow the documentation properly, you would rewrite it using a hidden key .XX and then apply it with <<: *X.
For example this to define the key:
.job_template: &deploy_definition
environment:
url: ${CI_ENVIRONMENT_SLUG}.mydomain.com
scripts:
- deploy ${CI_ENVIRONMENT_SLUG}
And then all blocks can be writen using <<: *job_template. I assume environment will merge the name with the predefined URL.
deploy_to_test:
<<: *deploy_definition
environment:
name: test
deploy_to_stage:
<<: *deploy_definition
environment:
name: stage
deploy_to_prod:
<<: *deploy_definition
environment:
name: prod
Full docs section from the link above:
YAML has a handy feature called 'anchors', which lets you easily duplicate content across your document. Anchors can be used to duplicate/inherit properties, and is a perfect example to be used with hidden keys to provide templates for your jobs.
The following example uses anchors and map merging. It will create two jobs, test1 and test2, that will inherit the parameters of .job_template, each having their own custom script defined:
.job_template: &job_definition # Hidden key that defines an anchor named 'job_definition'
image: ruby:2.1
services:
- postgres
- redis
test1:
<<: *job_definition # Merge the contents of the 'job_definition' alias
script:
- test1 project
test2:
<<: *job_definition # Merge the contents of the 'job_definition' alias
script:
- test2 project
& sets up the name of the anchor (job_definition), << means "merge the given hash into the current one", and * includes the named anchor (job_definition again). The expanded version looks like this:
.job_template:
image: ruby:2.1
services:
- postgres
- redis
test1:
image: ruby:2.1
services:
- postgres
- redis
script:
- test1 project
test2:
image: ruby:2.1
services:
- postgres
- redis
script:
- test2 project
Besides what the answer offered, I'd like to add another similar way to achieve kind of the same thing but it's more flexible rather than to use a template and then merge it in a stage.
What you can do is to create a hidden key as well, but in this format, e.g.,
.login: &login |
cmd1
cmd2
cmd3
...
And then you can apply it to different stages by using the '*', the asterisk, like:
deploy:
stage: deploy
script:
- ...
- *login
- ...
bake:
stage: bake
script:
- ...
- *login
- ...
And the result would be equivalent to:
deploy:
stage: deploy
script:
- ...
- cmd1
- cmd2
- cmd3
- ...
bake:
stage: bake
script:
- ...
- cmd1
- cmd2
- cmd3
- ...
Based on the resource of:
https://gitlab.com/gitlab-org/gitlab-ce/issues/19677#note_13008199
As for the template implementation, it's "merged". With my own experience, if you append more scripts after merging a template, the template scripts will be overwritten. And you cannot apply multiple templates at a time. Only the last template scripts will be executed. For example:
.tmp1: &tmp1
script:
- a
- b
.tmp2: &tmp2
script:
- c
- d
job1:
<<: *tmp1
<<: *tmp2
stage: xxx
job2:
<<: *tmp2
stage: yyy
script:
- e
- f
The equivalent result would be:
job1:
stage: xxx
script:
- c
- d
job2:
stage: yyy
script:
- e
- f
If not sure about the syntax correctness, just copy and paste your .gitlab.yml file content to "CI Lint" to validate. The button is in the tab of Pipelines.
gitlab gitlab-ci yaml
Just in case: Gitlab offers (since 11.3) an extends keyword, which can be used to "templates" yaml entries (so far as I understand it):
See the official doc
Have you tried implementing variables for various environments and using different jobs for various environments? I've come up with a solution for you.
image: node:latest
variables:
GIT_DEPTH: '0'
stages:
- build
- deploy
workflow:
rules:
- if: $CI_COMMIT_REF_NAME == "develop"
variables:
DEVELOP: "true"
ENVIRONMENT_NAME: Develop
WEBSITE_URL: DEVELOP_WEBSITE_URL
S3_BUCKET: (develop-s3-bucket-name)
AWS_REGION: ************** develop
AWS_ACCOUNT: ********develop
- if: $CI_COMMIT_REF_NAME == "main"
variables:
PRODUCTION: "true"
ENVIRONMENT_NAME: PRODUCTION
WEBSITE_URL: $PROD_WEBSITE_URL
S3_BUCKET: $PROD-S3-BUCKET-NAME
AWS_REGION: ************** (prod-region)
AWS_ACCOUNT: ***********(prod-acct)
- when: always
build-app:
stage: build
script:
#build-script
environment:
name: $ENVIRONMENT_NAME
deploy-app:
stage: deploy
script:
#deploy-script
environment:
name: $ENVIRONMENT_NAME