I'm trying to create a scheduled pipeline which will run every 4 AM but the problem is that I want to run the job only if there is any commits in the previous day. I know I can get the current date and the last commit date by $CI_JOB_STARTED_AT and $CI_COMMIT_TIMESTAMP but I don't know how will compare these two. For example if the last commit date is 2022-01-26 and the current date is 2022-01-27 that means the job should run but how. I've tried comparing them by converting them to strings by "==" that works but I don't need to know if they are equal I need to know that the current date is 1 day ahead of last commit. I've pasted some of my yml script to get an idea of what I'm trying to achieve.
Any help will be highly appreciated.
stage: deploy
script:
- $debug_version = $xmlserver_version + "." + $CI_PIPELINE_ID
- .\CI\DeployToDev "UK1-OFF-DEXM-02" $debug_version
needs:
- Publish_Debug
dependencies:
- Publish_Debug
rules:
- if: '$CURRENT_DATE == $COMMIT_DATE'
when: always
when: manual
tags: *tags
variables:
CURRENT_DATE: ${CI_JOB_STARTED_AT%T*}
COMMIT_DATE: ${CI_COMMIT_TIMESTAMP%T*}```
Related
I struggle with setting up a condition statement to run a specific task only on main and when there is a CI build.
While trying without success, I came across this statement which is an AND of two exactly the same statements:
condition: and(contains(variables['build.sourceBranch'], 'refs/heads/main'), contains(variables['build.sourceBranch'], 'refs/heads/main'))
When running it on my branch, I see this result:
Evaluating: and(contains(variables['build.sourceBranch'], 'refs/heads/main'), contains(variables['build.sourceBranch'], 'refs/heads/main'))
Expanded: and(contains('refs/heads/task/mybranch', 'refs/heads/main'), contains(variables['build.sourceBranch'], 'refs/heads/main'))
Result: False
Why is the second variable not expanded?
To reach my aim of that condition, I'd like to run something like this, but here the expansion does not work as well:
and(endsWith(variables['build.sourceBranch'], 'refs/heads/main'), endsWith(variables['build.reason'], 'CI'))
What am I doing wrong?
Thanks in advance
I want to run pipeline specific branches in GitLab like the branch names are sprint_100, Sprint-1,SPRINT-202.
How should I give commands in rules?
(/^SPRINT_[0-9]+\.[0-9]+$/)||(/^Sprint_[0-9]+\.[0-9]+$/)||(/^Sprint_[0-9]+\.[0-9]+$/)
I used this, but it doesn't work. Kindly help me to sort it out this. Thank you!
This rules clause should work for your example to run a pipeline only branch names like sprint_100, Sprint-1 or SPRINT-202.
The regex will match case-insensitive on branch names starting with 'sprint' followed by either '-' or '_' and an unlimited amount of digits.
I am no regex expert so this regex can likely be improved.
rules:
- if: '$CI_COMMIT_BRANCH =~ /^SPRINT[-_][0-9]+/i'
I have a gitlab repo and I am trying to create some tags on top of required commits. But my CICD library (included) is having a job (job1) in a stage which runs on every tag/branch/commit. I want to exclude it to run on particular formats of tags. (eg. pre_final-2.3.1).
I tried giving the tag pattern in my .gitlab-ci.yaml in the except section of the job as below.
Eg:
job1:
except:
- ^pre_final-\d+\.\d+\.\d+$
It is still adding the job job1 to the pipeline for the CICD builds running on this tag. I believe this pattern is checking with branch name. But is there a way to mention the ref we provide in the except section is branch or tag?
Ref: https://docs.gitlab.com/ee/ci/yaml/#onlyrefs--exceptrefs
You should rules instead as only/except is not developed further. There you can check for the the name of your tag.
job1:
rules:
- if: '$CI_COMMIT_TAG =~ /^pre_final-\d+\.\d+\.\d+$/'
when: never
- when: always
There is no separate classification as branch or tag while using them in only/except.
I realized that I am giving a wrong regex pattern and thus it is not working for the tag pattern I am trying for
^pre_final-\d+\.\d+\.\d+$ --> /^pre_final-\d+\.\d+\.\d+$/
So my .gitlab-ci.yaml contains as below
job1:
except:
# matches with branch or tags names
- /^pre_final-\d+\.\d+\.\d+$/
Since above usage matches both branch and tag names, if we want to match only branch name or match only tag name, then we can use the below format
job1:
except:
variables:
# to match only tag names
- $CI_COMMIT_TAG == /^pre_final-\d+\.\d+\.\d+$/
# to match only branch names
- $CI_COMMIT_BRANCH = /^pre_final-\d+\.\d+\.\d+$/
is there anyway to do multiple AND variable expression?
let's say
.template1:
only:
variables:
- $flag1 == "true"
.template2:
only:
variables:
- $flag2 == "true"
job1:
extends:
- .template1
- .template2
script: echo "something"
How will this get evaluated?
Is this going to result in only:variables overwriting each other thus template2 is the final result?
or is this going to result in a combined variables such that it becomes an OR statement
only:
variables:
- $flag1 == "true"
- $flag2 == "true"
Is there anyway to make it as and AND statement instead? keeping the templating system, and without using rules: if since using rules if has its own quirk, triggering multiple pipeline during merge request
Problem
Any time two jobs get "merged", either using extends or anchors, GitLab will overwrite one section with another. The sections don't actually get merged. In your case, you're extending from two jobs, so GitLab will completely overwrite the first variables section with the second.
Solution
One way to achieve your desired result is by defining the variables in your template jobs. The problem you will have then is the two variables sections will overwrite each other. So..
You can use a before_script section to define the variable in the 2nd template. This approach works for your specific case of 2 templates. You can use script and after_script if you need a third template, buy you'd have to use a more advanced approach if you need more templates than that.
.template1:
# We can define a variables section here, no problem
variables:
flag1: "true"
.template2:
# You can't define a second variables section here, since it will overwrite the first
# Instead, define the environment variable directly in a before-script section
before_script:
- export flag2="true"
job1:
extends:
- .template1
- .template2
only:
variables:
- $flag1 == "true"
- $flag2 == "true"
script: echo "something"
The consistency of Variable support & the syntax vary wildly in Azure DevOps YAML.
Case in point:
trigger:
- master
# Variable Group has $(testCategory1) with value
# 'TestCategory=bvttestonly | TestCategory=logintest'
variables:
- group: DYNAMIC_VG
jobs:
- job:
pool: 'MyPool' #Has about 10+ self hosted agents
strategy:
parallel: $[ variables['noOfVMsDynamic']]
variables:
indyx: '$(testCategories$(System.JobPositionInPhase))'
indyx2: $[ variables['indyx'] ]
testCategories: $[ variables[ 'indyx2' ] ]
steps:
- script: |
echo "indyx2 - $(indyx2)"
echo "testCategories $(testCategories)"
displayName: 'Display Test Categories'
The step prints:
"indyx2 - $(testCategories1)"
"testCategories $(testCategories1)"
I need to print the value of $(testCategories1) defined in the Variable Group:
'TestCategory=bvttestonly | TestCategory=logintest'
This may work to you:
variables
indyx: $[ variables[format('{0}{1}', 'testCategories', variables['System.JobPositionInPhase'])] ]
It worked for me, in a slightly different situation which also required some dynamic variable names.
Howto: Dynamically resolve a nested variable in Azure DevOps YAML
That because the value of nested variables (like $(testCategories$(System.JobPositionInPhase))) are not yet supported in the build pipelines at this moment.
That the reason why you always get the value $(testCategories1) rather than the real value of variable testCategories1.
I encountered this issue many times in my past posts and we do not have a perfect solution before Azure Devops supports this feature.
For the convenience of testing, I simplified your yaml like following:
jobs:
- job: ExecCRJob
timeoutInMinutes: 800
pool:
name: MyPrivateAgent
displayName: 'Execute CR'
variables:
testCategories1: 123456
testCategoriesSubscripted: $(testCategories$(System.JobPositionInPhase))
strategy:
parallel: $[variables['noOfVMs']]
steps:
- template: execute-cr.yml
parameters:
testCategories: $(testCategoriesSubscripted)
The execute-cr.yml:
steps:
- script: echo ${{ parameters.testCategories }}
We always get the $(testCategories1)NOT the value of it.
If I change the $(testCategories$(System.JobPositionInPhase)) to $(testCategories1), everything work fine.
Since nested variables are not yet supported, As workaround, we need to expand the nested variables for each value of testCategories, like:
- job: B
condition: and(succeeded(), eq(dependencies.A.outputs['printvar.skipsubsequent'], 'Value1'))
dependsOn: A
steps:
- script: echo hello from B
Check the Expressions Dependencies for some more details.
Hope this helps.
If I'm understanding your issue correctly, the problem is that the pipeline evaluates all variables at the runtime of the job. The solution in this scenario is to split your tasks into separate jobs with dependencies.
Have a look at my answer in this post and let me know if it's what you're after : YAML pipeline - Set variable and use in expression for template
I manage to get dynamic name resolution by using get-item to read the corresponding environment variable, allowing construction of the name of the variable and then getting the value.
In our case we save the name of an autogenerated branch into a variable group and each repository will have its own variable.
$branchVarName = "$(Build.Repository.Name).BranchName".replace(".","_")
$branchName = (get-item -Path Env:$branchVarName).value
write-host "/$(System.TeamProject)/_apis/build/builds?&repositoryId=$(Build.Repository.ID)&repositoryType=TFSGit&branchName=refs/heads/$branchName&api-version=6.0"
Notice in the second line that I reference the variable content using .value because get-item returns a name/value key pair.
This extraction has to be done in script but could be exposed as an output variable if needed in another context.