Gitlab only: variables multiple - gitlab

is there anyway to do multiple AND variable expression?
let's say
.template1:
only:
variables:
- $flag1 == "true"
.template2:
only:
variables:
- $flag2 == "true"
job1:
extends:
- .template1
- .template2
script: echo "something"
How will this get evaluated?
Is this going to result in only:variables overwriting each other thus template2 is the final result?
or is this going to result in a combined variables such that it becomes an OR statement
only:
variables:
- $flag1 == "true"
- $flag2 == "true"
Is there anyway to make it as and AND statement instead? keeping the templating system, and without using rules: if since using rules if has its own quirk, triggering multiple pipeline during merge request

Problem
Any time two jobs get "merged", either using extends or anchors, GitLab will overwrite one section with another. The sections don't actually get merged. In your case, you're extending from two jobs, so GitLab will completely overwrite the first variables section with the second.
Solution
One way to achieve your desired result is by defining the variables in your template jobs. The problem you will have then is the two variables sections will overwrite each other. So..
You can use a before_script section to define the variable in the 2nd template. This approach works for your specific case of 2 templates. You can use script and after_script if you need a third template, buy you'd have to use a more advanced approach if you need more templates than that.
.template1:
# We can define a variables section here, no problem
variables:
flag1: "true"
.template2:
# You can't define a second variables section here, since it will overwrite the first
# Instead, define the environment variable directly in a before-script section
before_script:
- export flag2="true"
job1:
extends:
- .template1
- .template2
only:
variables:
- $flag1 == "true"
- $flag2 == "true"
script: echo "something"

Related

Trigger Gitlab-CI Pipeline only when there is a new tag, with some rules

in the answer of this (Trigger Gitlab-CI Pipeline only when there is a new tag ), I understand to set only as filter
But How can I trigger the pipeline only:
there is a new tag created, these tags are semantic versioning tags, how can I filter them? something like
only:
- /[0-9]+\.[0-9]+\.[0-9]+/
but the tags are not always as 1.0.2, sometimes, they have beta or fc, such as 0.2.3-rc.1 or 2.3.5-beta, will the filter to be /[0-9]+\.[0-9]+\.[0-9]+.*/ is fine?
second rule is, the trigger is only on master or main branch
Regarding the Semantic versioning regex, I found this gist
https://gist.github.com/jhorsman/62eeea161a13b80e39f5249281e17c39
the sample is too complex
^([0-9]+)\.([0-9]+)\.([0-9]+)(?:-([0-9A-Za-z-]+(?:\.[0-9A-Za-z-]+)*))?(?:\+[0-9A-Za-z-]+)?$
Using only is deprecated in favor of rules.
rules replaces only/except and they can’t be used together in the same job.
https://docs.gitlab.com/ee/ci/yaml/#rules
Create a rule and check the predefined variable CI_COMMIT_TAG for your pattern.
job:
script: echo "Hello, Rules!"
rules:
- if: $CI_COMMIT_TAG =~ /^\d+\.\d+\.\d+.*/
You can list multiple of these rules. So if you want to execute the job also on every push to the master branch, you'd add another rule:
rules:
- if: ...
- if: $CI_COMMIT_REF_NAME == "master"
The extended pattern with .* at the end would be fine, but you could also be a bit more specific like this (-[0-9a-zA-Z.]+)? if you want to enforce a specific syntax.

How to exclude tags with patterns in gitlab ci config

I have a gitlab repo and I am trying to create some tags on top of required commits. But my CICD library (included) is having a job (job1) in a stage which runs on every tag/branch/commit. I want to exclude it to run on particular formats of tags. (eg. pre_final-2.3.1).
I tried giving the tag pattern in my .gitlab-ci.yaml in the except section of the job as below.
Eg:
job1:
except:
- ^pre_final-\d+\.\d+\.\d+$
It is still adding the job job1 to the pipeline for the CICD builds running on this tag. I believe this pattern is checking with branch name. But is there a way to mention the ref we provide in the except section is branch or tag?
Ref: https://docs.gitlab.com/ee/ci/yaml/#onlyrefs--exceptrefs
You should rules instead as only/except is not developed further. There you can check for the the name of your tag.
job1:
rules:
- if: '$CI_COMMIT_TAG =~ /^pre_final-\d+\.\d+\.\d+$/'
when: never
- when: always
There is no separate classification as branch or tag while using them in only/except.
I realized that I am giving a wrong regex pattern and thus it is not working for the tag pattern I am trying for
^pre_final-\d+\.\d+\.\d+$ --> /^pre_final-\d+\.\d+\.\d+$/
So my .gitlab-ci.yaml contains as below
job1:
except:
# matches with branch or tags names
- /^pre_final-\d+\.\d+\.\d+$/
Since above usage matches both branch and tag names, if we want to match only branch name or match only tag name, then we can use the below format
job1:
except:
variables:
# to match only tag names
- $CI_COMMIT_TAG == /^pre_final-\d+\.\d+\.\d+$/
# to match only branch names
- $CI_COMMIT_BRANCH = /^pre_final-\d+\.\d+\.\d+$/

GitLab: extend `!reference` with extra values in `variables`

How can one use !reference in variables block and then extend the list of entries?
For example, I want to be able to do something like
variables:
!reference [.common_variables, variables]
FOO: "bar"
At the moment I get an error in the GitLab linter:
This GitLab CI configuration is invalid: (): did not find expected key while parsing a block mapping at line 1 column 1
P.S. Note that this question is specifically about !reference.
Yaml anchors are out of scope of this question, as I want to place .common_variables into a separate file that I include in my pipeline.
extends: mechanism is out of scope of this question. It does work, but it has a weird behaviour that one also gets an extends variable in the environment, see GitLab: use `extends` in `variables`
variables: !reference [.common_variables, variables]
Is the correct way to reference, however gitlab yml structure will not support additional definitions after the !reference keyword.
My advice would be to have a more specific definition in your variables file that uses anchors to share the common varaibles.
e.g:
in 'variables.gitlab-ci.yml':
.common_variables: &common_variables
COMMON_VAR: value
.variables:Example-job
variables:
<<: *common_variables
FOO: "bar"
in '.gitlab.ci.yml':
Example-job:
variables: !reference [".variables:Example-job", "variables"]
script:
...
I achieve it by using multiple extends, one of them, .contract-env-vars-default, containing my shared variables.
# base template
.contract-env-vars-default:
variables:
GIT_CURRENT_BRANCH: $CI_COMMIT_BRANCH
PACT_BROKER_CLI: ./pact/bin/pact-broker
…
Then using two extends
# adding language specific variable
.contract-verify-java:
extends:
- .gradle-test-base
- .contract-env-vars-default # < < < < < < < <
variables:
ENVIRONMENT: contract-test # adding language specific variable
And finally
# adding project specific variable
gradle-contract-testing-verify:
extends: .contract-verify-java
variables:
PROJECT_SPECIFIC: foo_bar_biz # add project specific
Thus I got the expected result of:
gradle-contract-testing-verify:
variables:
CACHE_FALLBACK_KEY: main-cache-non_protected # platform?
GIT_CURRENT_BRANCH: "$CI_COMMIT_BRANCH" # base
PACT_BROKER_CLI: "./pact/bin/pact-broker" # base
…
ENVIRONMENT: contract-test # language specific
PROJECT_SPECIFIC: foo_bar_biz # project specific
variables:
!reference [.common_variables, variables]
FOO: "bar"
or if you want to assign the variables to a new variable, you can use this
variables:
MY_VAR: !reference [.common_variables, variables]
For additional information - https://docs.gitlab.com/ee/ci/yaml/yaml_optimization.html#reference-tags

Not getting any values from variables in variable groups in Azure Devops pipeline

I am really struggling with some variables which I have in my variable group named 'android-pipeline'.
Inside this variable group, I have some variables with values.
But when I am running the pipeline it cannot read the values inside my variable group. :(
Example:
Inside the variable group, I have a variable called
$(key.alias)
I am trying to get this value which is behind the variable, see my code below.
I think something is wrong with the syntax (or the way I am using it), but I cannot find the right syntax for using my $(key.alias) variable.
Also, inside the variable group I have made sure that All pipelines have access to this Variable group.
Can someone, please tell me how I can get the value behind the $(key.alias) variable and use this in a task? I tried to follow many guides, but none are clear enough for me or not working
variables:
group: android-pipeline
buildConfiguration: 'Release'
stages:
stage: Publish
dependsOn: Build
displayName: Sign Apps
jobs:
- task: AndroidSigning#3
displayName: Android App signing
inputs:
apkFiles: '**/*.apk'
apksignerKeystoreFile: '$(androidKeyStore)'
apksignerKeystorePassword: '********'
apksignerKeystoreAlias: '$(key.alias)'
apksignerKeyPassword: '*******'
apksignerArguments: --out $(outputDirectory)/app.release.apk
zipalign: true
Since you're mixing groups and inline variables, you may need to change this from a mapping to a sequence, as in:
variables:
- group: android-pipeline
- name: buildConfiguration
value: Release
Normally when you declare variables, you can do them like a mapping, or hashtable, of name/value pairs:
variables
var1: value1 # note there's no dash at the beginning of the line
var2: value2
var3: value3
# etc
When you want to use a group, you have to change your syntax a little, so that the parser doesn't think you want to create a variable named "group" - you turn it into a sequence, or array:
variables
- group: groupname1 # note there's a dash at the beginning of the line
- group: groupname2
# etc
Here's the final wrinkle - once you've gone from the first format to the second (mapping to sequence), you have to declare new variables that are local to your file in the "sequence" style:
variables
- group: groupname1 # note there's a dash at the beginning of the line
- name: varname1
value: value1
- name: varname2
value: value2
# etc
You reference the variable further down in your pipeline the same way, with $(varname1) syntax.
If you're having problems with this, I recommend a couple of things (actually, 3):
Use script or pwsh tasks to echo or Write-Host everything you want to see but aren't, as in "pwsh: Write-Host "My var should be $(varname1)"
Turn on system diagnostics when you run the pipeline and see if the output has any useful details
Edit the pipeline through the portal - Pipelines - select your pipeline -> Edit. Then, from the ellipsis menu in the top right of the page, select "Download full YAML" - this will give download what the compiler would create. Now, it won't give you variable values, but what it can do is give you clues as to possible format or declaration errors.

Howto: Dynamic variable name resolution in Azure DevOps YAML

The consistency of Variable support & the syntax vary wildly in Azure DevOps YAML.
Case in point:
trigger:
- master
# Variable Group has $(testCategory1) with value
# 'TestCategory=bvttestonly | TestCategory=logintest'
variables:
- group: DYNAMIC_VG
jobs:
- job:
pool: 'MyPool' #Has about 10+ self hosted agents
strategy:
parallel: $[ variables['noOfVMsDynamic']]
variables:
indyx: '$(testCategories$(System.JobPositionInPhase))'
indyx2: $[ variables['indyx'] ]
testCategories: $[ variables[ 'indyx2' ] ]
steps:
- script: |
echo "indyx2 - $(indyx2)"
echo "testCategories $(testCategories)"
displayName: 'Display Test Categories'
The step prints:
"indyx2 - $(testCategories1)"
"testCategories $(testCategories1)"
I need to print the value of $(testCategories1) defined in the Variable Group:
'TestCategory=bvttestonly | TestCategory=logintest'
This may work to you:
variables
indyx: $[ variables[format('{0}{1}', 'testCategories', variables['System.JobPositionInPhase'])] ]
It worked for me, in a slightly different situation which also required some dynamic variable names.
Howto: Dynamically resolve a nested variable in Azure DevOps YAML
That because the value of nested variables (like $(testCategories$(System.JobPositionInPhase))) are not yet supported in the build pipelines at this moment.
That the reason why you always get the value $(testCategories1) rather than the real value of variable testCategories1.
I encountered this issue many times in my past posts and we do not have a perfect solution before Azure Devops supports this feature.
For the convenience of testing, I simplified your yaml like following:
jobs:
- job: ExecCRJob
timeoutInMinutes: 800
pool:
name: MyPrivateAgent
displayName: 'Execute CR'
variables:
testCategories1: 123456
testCategoriesSubscripted: $(testCategories$(System.JobPositionInPhase))
strategy:
parallel: $[variables['noOfVMs']]
steps:
- template: execute-cr.yml
parameters:
testCategories: $(testCategoriesSubscripted)
The execute-cr.yml:
steps:
- script: echo ${{ parameters.testCategories }}
We always get the $(testCategories1)NOT the value of it.
If I change the $(testCategories$(System.JobPositionInPhase)) to $(testCategories1), everything work fine.
Since nested variables are not yet supported, As workaround, we need to expand the nested variables for each value of testCategories, like:
- job: B
condition: and(succeeded(), eq(dependencies.A.outputs['printvar.skipsubsequent'], 'Value1'))
dependsOn: A
steps:
- script: echo hello from B
Check the Expressions Dependencies for some more details.
Hope this helps.
If I'm understanding your issue correctly, the problem is that the pipeline evaluates all variables at the runtime of the job. The solution in this scenario is to split your tasks into separate jobs with dependencies.
Have a look at my answer in this post and let me know if it's what you're after : YAML pipeline - Set variable and use in expression for template
I manage to get dynamic name resolution by using get-item to read the corresponding environment variable, allowing construction of the name of the variable and then getting the value.
In our case we save the name of an autogenerated branch into a variable group and each repository will have its own variable.
$branchVarName = "$(Build.Repository.Name).BranchName".replace(".","_")
$branchName = (get-item -Path Env:$branchVarName).value
write-host "/$(System.TeamProject)/_apis/build/builds?&repositoryId=$(Build.Repository.ID)&repositoryType=TFSGit&branchName=refs/heads/$branchName&api-version=6.0"
Notice in the second line that I reference the variable content using .value because get-item returns a name/value key pair.
This extraction has to be done in script but could be exposed as an output variable if needed in another context.

Resources