I have created a Gitlab Pipeline. I want to use variable for which I can set the value dynamically.
e.g. I have created a variable with default value which will be used in each stage of the pipeline. But for Stage B I want to use different value for that variable. reference code as below.
jobA:
stage: A
allow_failure: true
script:
- echo "$LOG_PATH"
- echo "jobA" > ./completedA
artifacts:
paths:
- ./completedA
jobB:
stage: B
allow_failure: true
script:
- LOG_PATH="/opt/testlogs/"
- echo "${LOG_PATH}"
- exit 1
- echo "jobB" > ./completedB
artifacts:
paths:
- ./completedB
stages:
- A
- B
Variables:
LOG_PATH: "/opt/test/"
Current output for the variable:
For Stage A, Value of LOG_PATH is "/opt/test/"
For Stage B, Value of LOG_PATH is "/opt/test/"
Expected output for the variable:
For Stage A, Value of LOG_PATH is "/opt/test/"
For Stage B, Value of LOG_PATH is "/opt/testlogs/"
Looking at the "Create a custom CI/CD variable in the .gitlab-ci.yml " section, you might need to set the variable in the variables: section of the job
jobB:
variables:
LOG_PATH: "/opt/testlogs/"
stage: B
allow_failure: true
script:
- echo "${LOG_PATH}"
- exit 1
- echo "jobB" > ./completedB
artifacts:
paths:
- ./completedB
Instead of "variables" section, is it possible to set the variable value within "script" section?
In my case, the log file path will get generated dynamically within script section. So, I can't set it variable section.
That is what is tested in "How to set variable within 'script:' section of gitlab-ci.yml file"
Check if the approach illustrated in "How to set gitlab-ci variables dynamically?", using artifacts.
You can set the variable inside the job, this will overwrite the global variable (see docs)
jobB:
stage: B
variables:
LOG_PATH: "/opt/testlogs/"
...
Related
I have the following .gitlab-ci.yml:
image:
name: my.private.registry.com/my-repository/ci-tools:latest
variables:
TF_IN_AUTOMATION: "true"
PLAN: "plan.tfplan"
application_name: "my_app"
variables: &eu-variables
TF_VAR_region: "westeurope"
variables: &us-variables
TF_VAR_region: "eastus"
variables: &dev-variables
TF_VAR_application_name: "$application_name"
stages:
- Validate
- Plan
dev-plan:
variables:
<<: *dev-variables
<<: *eu-variables
stage: Plan
script:
- echo "$PLAN"
- echo "$TF_VAR_application_name"
- echo "$application_name"
artifacts:
paths:
- "$PLAN"
When the pipeline runs, $PLAN, TF_VAR_application_name, and application_name are blank and the output is as follows:
.....
$ echo "$PLAN"
$ echo "$application_name"
$ echo "$TF_VAR_application_name"
..........
What am I missing? Why is the job variable not able to see the global variable?
By using global, just put all your variables in variables:
example
variables:
TF_VAR_region_WE: "westeurope"
TF_VAR_region_EU: "eastus"
.....
also by reading the docs, its not recommend to use " <<: "
see this documentation https://docs.gitlab.com/ee/ci/yaml/yaml_optimization.html#yaml-anchors-for-scripts
How to make Delay task take value from variable instead of hardcoded value for delayForMinutes input?
When i do following, it works fine:
- job: WaitForDeploy
dependsOn: Main
pool: Server
steps:
- task: Delay#1
inputs:
delayForMinutes: '1'
but when i do it like this it does not:
- job: WaitForDeploy
dependsOn: Main
pool: Server
steps:
- task: Delay#1
inputs:
delayForMinutes: '$(SleepCount)'
$(SleepCount) is defined in variables as empty string and later on is passed from python script to pipeline via:
print(f'##vso[task.setvariable variable=SleepCount]{delay_seconds}')
I am printing this out in previos job and it shows intiger correctly:
- script: 'echo $(SleepCount)'
displayName: "print_sleep_count"
it looks like this variable value is not passed beyond job where i return it from python script, how to pass it?
This do not work:
variables:
SleepCount: ""
jobs:
- job: Main
pool:
name: 'CDaaSLinux'
[...]
- task: PythonScript#0
inputs:
scriptSource: 'filePath'
scriptPath: '$(Build.SourcesDirectory)/sleep_count.py'
env:
ACTUAL_START_DATE: $(ActualStartDate_value)
- script: 'echo $(SleepCount)'
name: setVariable
displayName: "print_sleep_count"
- script: 'echo "Waiting for Deploy for $(SleepCount) minutes"'
displayName: "Deploy_message_for_user"
- job: WaitForDeploy
dependsOn: Main
variables:
SleepCount: $[ dependencies.Main.outputs['setVariable.SleepCount']]
pool: Server
steps:
- task: Delay#1
inputs:
delayForMinutes: '$(SleepCount)'
In order to use a variable in a different stage/job you need to set the isoutput flag when setting the variable
print(f'##vso[task.setvariable variable=SleepCount;isoutput=true]{delay_seconds}')
Set an output variable for use in future jobs
I have a .yaml file
variables:
- name: command1
value: none
- scripts: |
echo '##vso[task.setvariable variable=command1]new_value'
- ${{ if ne(variables['command1'], 'none') }}:
- template: templates/run.yml#temp1 # Template reference
parameters:
COMMAND: '$(command1)'
I have created the variable for two reason ,
to be global
I dont want it to be displayed in the variable list for the users
I want the template only to be executed if variable value of 'command1' is not 'none'
Currently it is not skipping , it keeps executing it even if the value inside the variable is not none.
The other if conditions format I have used is
- ${{ if ne(variables['taskName.command1'], 'none') }}:
- ${{ if ne('$(command1)', 'none') }}:
None of the above worked
Please help in resolving this issue.
As it is written here:
The difference between runtime and compile time expression syntaxes is primarily what context is available. In a compile-time expression (${{ }}), you have access to parameters and statically defined variables. In a runtime expression ($[ ]), you have access to more variables but no parameters.
variables:
staticVar: 'my value' # static variable
compileVar: ${{ variables.staticVar }} # compile time expression
isMain: $[eq(variables['Build.SourceBranch'], 'refs/heads/main')] # runtime expression
steps:
- script: |
echo ${{variables.staticVar}} # outputs my value
echo $(compileVar) # outputs my value
echo $(isMain) # outputs True
So it could work for your YAML value:
variables:
- name: command1
value: none
steps:
- scripts: |
echo '##vso[task.setvariable variable=command1]new_value'
- ${{ if ne(variables.command1, 'none') }}:
- template: templates/run.yml#temp1 # Template reference
parameters:
COMMAND: '$(command1)'
However, it will pick this value:
variables:
- name: command1
value: none
There is no chance that it will take this:
- scripts: |
echo '##vso[task.setvariable variable=command1]new_value'
It is because ${{}} expressions are compile time and echo '##vso[task.setvariable variable=command1]new_value' is runtime.
The first stage in my pipeline checks for what services have actually changed. This is in an effort to speed up the pipeline by avoiding rebuilding, retesting, redeploying services if there have been no changes.
This is the changed.yaml for that stage:
parameters:
- name: comparedTo
default: ''
stages:
- stage: Changed
displayName: Check for changes in services and configs...
jobs:
- job: Changes
displayName: Checking for changes in services and configs...
steps:
- bash: |
mapfile -t changed < <(git diff HEAD ${{ parameters.comparedTo }} --name-only | awk -F'/' 'NF!=1{print $1}' | sort -u)
servicesChanged=()
configChanged=()
echo ""
echo "Total Changed: ${#changed[#]}"
for i in "${changed[#]}"
do
echo $i
if [[ $i == 'admin' ]]; then
echo "##vso[task.setvariable variable=adminChanged;isOutput=True]true"
servicesChanged+=("admin")
elif [[ $i == 'admin-v2' ]]; then
echo "##vso[task.setvariable variable=adminV2Changed;isOutput=True]true"
servicesChanged+=("admin-v2")
elif [[ $i == 'api' ]]; then
echo "##vso[task.setvariable variable=apiChanged;isOutput=True]true"
servicesChanged+=("api")
elif [[ $i == 'client' ]]; then
echo "##vso[task.setvariable variable=clientChanged;isOutput=True]true"
servicesChanged+=("client")
elif [[ $i == 'k8s' ]]; then
echo "##vso[task.setvariable variable=k8sChanged;isOutput=True]true"
configsChanged+=("k8s")
elif [[ $i == 'pipelines' ]]; then
echo "##vso[task.setvariable variable=pipelineChanged;isOutput=True]true"
configsChanged+=("pipelines")
fi
done
echo ""
echo "Services Changed: ${#servicesChanged[#]}"
for i in "${servicesChanged[#]}"
do
echo $i
done
echo ""
echo "Configs Changed: ${#configsChanged[#]}"
for i in "${configsChanged[#]}"
do
echo $i
done
if [[ ${#servicesChanged[#]} > 0 ]]; then
echo ""
echo "Any services changed: True"
echo "##vso[task.setvariable variable=anyServicesChanged;isOutput=true]true"
echo "##vso[task.setvariable variable=servicesChanged;isOutput=true]${servicesChanged[#]}"
fi
if [[ ${#configsChanged[#]} > 0 ]]; then
echo ""
echo "Any configs changed: True"
echo "##vso[task.setvariable variable=anyConfigsChanged;isOutput=true]true"
echo "##vso[task.setvariable variable=configsChanged;isOutput=true]${configsChanged[#]}"
fi
echo ""
name: detectChanges
As you can see, it creates a number of task output variables:
# This just indicates if the service has changed: true/false
echo "##vso[task.setvariable variable=<service-name>;isOutput=True]true"
# This should be creating a an output variable that is an array of the services that have changed
echo "##vso[task.setvariable variable=servicesChanged;isOutput=true]${servicesChanged[#]}"
So I gave myself two options: just a true/false for each service or iterating (somewhow) through an array of the services that have changed.
Each stage basically has the following form:
# pr.yaml
...
- template: templates/unitTests.yaml
parameters:
services:
- api
- admin
- admin-v2
- client
...
parameters:
- name: services
type: object
default: []
stages:
- stage: UnitTests
displayName: Run unit tests on service...
dependsOn: Changed
condition: succeeded()
jobs:
- job: UnitTests
condition: or(eq(stageDependencies.Changed.Changes.outputs['detectChanges.anyServicesChanged'], true), eq(variables['Build.Reason'], 'Manual'))
displayName: Running unit tests...
steps:
- ${{ each service in parameters.services }}:
- bash: |
echo "Now running ${{ service }} unit tests..."
Here is what I've tried so far and the errors I get:
Adding each service conditionally to services array or adding the array of changed services:
- template: templates/changed.yaml
parameters:
comparedTo: origin/production
- template: templates/unitTests.yaml
dependsOn: Changed
parameters:
services:
- ${{ if eq(stageDependencies.Changed.Changes.outputs['detectChanges.apiChanged'], true) }}
- api
- ${{ if eq(stageDependencies.Changed.Changes.outputs['detectChanges.adminChanged'], true) }}
- admin
- ${{ if eq(stageDependencies.Changed.Changes.outputs['detectChanges.adminV2Changed'], true) }}
- admin-v2
- ${{ if eq(stageDependencies.Changed.Changes.outputs['detectChanges.clientChanged'], true) }}
- client
Or...
- template: templates/changed.yaml
parameters:
comparedTo: origin/production
- template: templates/unitTests.yaml
dependsOn: Changed
parameters:
services:
- ${{ if eq(dependencies.Changed.outputs['Changes.detectChanges.apiChanged'], true) }}
- api
- ${{ if eq(dependencies.Changed.outputs['Changes.detectChanges.adminChanged'], true) }}
- admin
- ${{ if eq(dependencies.Changed.outputs['Changes.detectChanges.adminV2Changed'], true) }}
- admin-v2
- ${{ if eq(dependencies.Changed.outputs['Changes.detectChanges.clientChanged'], true) }}
- client
Or...
- template: templates/changed.yaml
parameters:
comparedTo: origin/production
- template: templates/unitTests.yaml
dependsOn: Changed
parameters:
services:
- $[ stageDependencies.Changed.Changes.outputs['detectChanges.servicesChanged'] ]
This results in:
An error occurred while loading the YAML build pipeline. Object reference not set to an instance of an object.
I know variables: will only take strings and not arrays.
One solution would be to have a variables: for each of the true/false variables and then conditions based on the parameters.services and whether the task output variables is true.
Any suggestions?
Ref:
Task Output Variables: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch#set-variables-in-scripts
Parameters: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops#parameters
Expressions: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/expressions?view=azure-devops
The template expression ${{}} is evaluated at compile-time(before the jobs run), which means it cannot access to the variables that are dynamically set at the runtime(after the jobs start). So you cannot use template expression ${{}} in above scenario. See below description from here.
Within a template expression, you have access to the parameters context that contains the values of parameters passed in. Additionally, you have access to the variables context that contains all the variables specified in the YAML file plus many of the predefined variables (noted on each variable in that topic). Importantly, it doesn't have runtime variables such as those stored on the pipeline or given when you start a run. Template expansion happens very early in the run, so those variables aren't available
You can use conditions as a workaround. You need to add multiple tasks to be executed on the conditions. See below example:
- template: templates/changed.yaml
parameters:
comparedTo: origin/production
- template: templates/unitTests.yaml
dependsOn: Changed
#unitTests.yaml
stages:
- stage: UnitTests
displayName: Run unit tests on service...
dependsOn: Changed
condition: succeeded()
jobs:
- job: UnitTests
condition: or(eq(stageDependencies.Changed.Changes.outputs['detectChanges.anyServicesChanged'], true), eq(variables['Build.Reason'], 'Manual'))
displayName: Running unit tests...
variables:
changedServices: $[stageDependencies.Changed.Changes.outputs['detectChanges.servicesChanged']]
steps:
- bash: |
echo "Now running api unit tests..."
name: Api-unit-test
conidtion: contains(variables['changedServices'], 'api')
- bash: |
echo "Now running admin unit tests..."
name: admin-unit-test
conidtion: contains(variables['changedServices'], 'admin')
- bash: |
echo "Now running client unit tests..."
name: client-unit-test
conidtion: contains(variables['changedServices'], 'client')
Another workaround is to separate your pipeline into two pipelines. The fist pipeline to run the Changed stage. And Then call rest api in a script task to trigger the second pipeline and pass the variables in the request body. See this similar thread.
I have a yml pipeline with 2 stages:
- stage:
- stage:
My second stage needs to refer to the $(Build.BuildNumber) of the previous stage. How can this be achieved? My understanding is output variables are scoped to the same stage and can't be used across-stages.
Trying to pull from stageDependencies:
stages:
- stage: BuildPublish
displayName: "Build & Publish"
jobs:
- job: BuildPublishJob
displayName: "Build & Publish"
steps:
- script: |
echo "Recording MSI version"
echo "##vso[task.setvariable variable=msiVersion;isOutput=true]$(Build.BuildNumber)"
name: MsiVersion
- script: echo $(MsiVersion.msiVersion)
name: echovar
- stage: DeployInstallerTest
displayName: "Deploy Installer Test"
jobs:
- job:
displayName: "Deploy Installer Test"
steps:
- task: AzurePowerShell#5
inputs:
azureSubscription: 'Spektrix Non-Prod'
ScriptType: 'InlineScript'
Inline: |
$msiVersion = stageDependencies.BuildPublish.BuildPublishJob.outputs['MsiVersion.msiVersion']
azurePowerShellVersion: 'LatestVersion'
Fails with:
##[error]The term 'stageDependencies.BuildPublish.BuildPublishJob.outputs[MsiVersion.msiVersion]' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
It changed recently:
stages:
- stage: A
jobs:
- job: JA
steps:
- script: |
echo "This is job Foo."
echo "##vso[task.setvariable variable=doThing;isOutput=true]Yes" #The variable doThing is set to true
name: DetermineResult
- script: echo $(DetermineResult.doThing)
name: echovar
- job: JA_2
dependsOn: JA
condition: eq(dependencies.JA.outputs['DetermineResult.doThing'], 'Yes')
steps:
- script: |
echo "This is job Bar."
#stage B runs if DetermineResult task set doThing variable n stage A
- stage: B
dependsOn: A
jobs:
- job: JB
condition: eq(stageDependencies.A.JA.outputs['DetermineResult.doThing'], 'Yes') #map doThing and check if true
variables:
varFromStageA: $[ stageDependencies.A.JA.outputs['DetermineResult.doThing'] ]
steps:
- bash: echo "Hello world stage B first job"
- script: echo $(varFromStageA)
However, please be aware that stageDependencies is not available in condition at stage level. Of course you can use stageDependencies not only in condition.