I've found the following block:
stages:
- stage: Print_Params
jobs:
- job: Print_Params
steps:
- ${{ each parameter in parameters }}:
- script: echo ${{ parameter.Key }} ${{ parameter.Value }}
But it invokes CmdLine once for each specified parameter. I'd really like to have a single screen I can look at to review all the parameters that a pipeline was invoked with. Is this built in, and there's a place I can already review it, or is there a way I can invoke the loop within a script to print all of the parameters in a single execution? I've tried a number of different syntaxes and nothing I've tried so far is working.
You can view runtime parameters, queue time variables and job preparation parameters in Azure Pipelines UI:
Related
Surely many of you have encountered this and I would like to share my hacky solution. Essentially during the CI / CD process of a Gitlab pipeline most parameters are passed through "Variables". There are two issues that I've encountered with that.
Those parameters cannot be altered in Realtime - Say I'd want to execute jobs based on information from previous jobs it would always need to be saved in the cache as opposed to written to CI / CD variables
The execution of jobs is evaluated before the script, so the "rules" will only ever apply to the original parameters. Trouble arises when those are only available during runtime.
For complex pipelines one would want to pick and choose the tests automatically without having to respecify parameters every time. In my case I delivered a data product and depending on the content different steps had to be taken. How do we deal with those issues?
Changing parameters real-time:
https://docs.gitlab.com/ee/api/project_level_variables.html This API provides users with a way of interacting with the CI / CD variables. This will not work for any variables defined at the head of the YML files under the Variables tag. Rather this is a way to access the "Custom CI/CD Variables" https://docs.gitlab.com/ee/ci/variables/#custom-cicd-variables found under this link. This way custom variables can be created altered and deleted during running a pipeline. The only thing needed is a PRIVATE-TOKEN that has access rights to the API (I believe my token has all the rights).
job:
stage: example
script:
- 'curl --request PUT --header "PRIVATE-TOKEN: $ACCESS_TOKEN" "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/variables/VARIABLE_NAME" --form "value=abc"'
Onto the next problem. Altering the variables won't let us actually control downstream jobs like this because of the fact that the "rules" block is executed before the pipeline is actually run. Hence it will use the variable before the curl request is sent.
job2:
stage: after_example
rules:
- if: $VARIABLE_NAME == "abc"
script:
- env
The way to avoid that is child pipelines. Child pipelines are initialized inside the parent pipeline and check the environment variables anew. A full example should illustrate my point.
variables:
PARAMETER: "Cant be changed"
stages:
- example
- after_example
- finally
job_1:
# Changing "VARIABLE_NAME" during runtime to "abc", VARIABLE_NAME has to exist
stage: example
script:
- 'curl --request PUT --header "PRIVATE-TOKEN: $ACCESS_TOKEN" "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/variables/VARIABLE_NAME" --form "value=abc"'
job_2.1:
# This wont get triggered as we assume "abc" was not the value in VARIABLE_NAME before job_1
stage: after_example
rules:
- if: $VARIABLE_NAME == "abc"
script:
- env
job_3:
stage: after_example
trigger:
include:
- local: .donwstream.yml
strategy: depend
job_4:
stage: finally
script:
- echo "Make sure to properly clean up your variables to a default value"
# inside downstream.yml
stages:
- downstream
job_2.2:
# This will happen because the pipeline is initialized after job_1
stage: downstream
rules:
- if: $VARIABLE_NAME == "abc"
script:
- env
This coding bit probably won't run, however it exemplifies my point rather nicely. Job 2 should be executed based on an action that happens in Job 1. While the variables will be updated once we reach job 2.1, the rules check happens before so it will never be executed. Child pipelines do the rule check during the runtime of the Parent pipeline, this is why job 2.2 does run.
This variant is quite hacky and probably really inefficient, but for all intents and purposes it gets the job done.
I'm trying to use a variable yaml file where I store a variable that is the cron syntax for a build. I wish to use this variable for multiple build pipelines, and want to be able to change the time/day of the build without having to go into each pipeline and change each schedule within each yaml pipeline.
However, trying this current method hits an error:
variable.yml
variables:
- name: cronSyntax
value: "0 9 * * *"
azure-pipelines.yml
variables:
- template: variable.yml
schedules:
cron: ${{ cronSyntax }}
etc
I have also tried doing $(cronSyntax) but neither seem to work. Is it just a case that I cannot use variables within the schedule task in yaml? Any help greatly appreciated.
Thanks
Looks like you can't use pipeline variables when specifying schedules.
See the official documentation here : https://learn.microsoft.com/en-us/azure/devops/pipelines/process/scheduled-triggers?view=azure-devops&tabs=yaml#scheduled-triggers
I have a yaml pipeline that builds and deploys a project. it deploys to dev & tst. But i dont always want to deploy to tst. at the moment i have solved this with approvals. when the dev stage finishes the pipeline waits for approval.
this introduces some problems:
all developers get an email every time, which can be annoying.
the pipeline waits for deployment to tst, even when deploymen to tst isnt needed. it keeps
waiting
new ci builds dont start when the pipeline is waiting
If there is a well defined way how to determine whether to run the tst stage or not, you can either specify a custom condition on the stage or conditionally include the stage using an expression
Specify a condition
From the docs:
You can specify the conditions under which each stage, job, or step
runs. By default, a job or stage runs if it does not depend on any
other job or stage, or if all of the jobs or stages that it depends on
have completed and succeeded.
There are a number of ways you can customize this behavior, either by built in or custom defined conditions, for example, to run tun the tst stage only if the branch is main:
stages:
- stage: dev
jobs:
- job: dev1
steps:
- script: echo Hello from dev!
- stage: tst
condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
jobs:
- job: tst1
steps:
- script: echo Hello From tst!
The tst stage will still be visible when you open the pipeline run, it will be marked as skipped
The condition under the tst stage can be altered to fit your need (it can include both variables, parameters and even output from previous stages.
Conditional expression
From the docs:
You can use if, elseif, and else clauses to conditionally assign
variable values or set inputs for tasks. You can also conditionally
run a step when a condition is met.
Conditional insertion only works with template syntax, which is evaluated when the pipeline yaml is compiled (i.e before the pipeline starts). Thus it cannot evaluate output of previous steps. In the below example the tst stage is completely removed from the pipeline run when the pipeline is instantiated (and the template compiled) if the branch is not main
stages:
- stage: dev
jobs:
- job: dev1
steps:
- script: echo Hello from dev!
- ${{ if eq(variables['Build.SourceBranch'], 'refs/heads/main') }}:
- stage: tst
jobs:
- job: tst1
steps:
- script: echo Hello From tst!
I have a Pipeline that has a few stages: detect, test, build, deploy
The detect stage detects the type of application and the test and build stages have jobs that are included or excluded based on what is computed in detect. The detect stage writes it's value to a environment variable called BUILD_MODE.
I am using rules like so:
ng-build:
extends:
- '.ng/job/build'
stage: build
rules:
- if: $BUILD_MODE == "ANGULAR"
when: always
npm-build:
extends:
- '.npm/job/build'
stage: build
rules:
- if: $BUILD_MODE == "NPM"
when: always
The problem with this is that the BUILD_MODE variable is evaluated statically when the pipeline is created not after the detect stage runs so the above never works unless I set the variable explicitly in the top level YML file like so:
variables:
BUILD_MODE: "ANGULAR"
What is the best way to solve this problem? The summary of what I want to do is evaluate some condition, either set the stages dynamically or set the variable itself before the stages in the Pipleline are created so they will be created with the rules evaluated correctly.
You could take a look at dynamic child-pipelines. Maybe you could solve your problem by dynamically creating your npm/ng build jobs.
I'm trying to import variable groups depending on the current branch in azure build pipeline, but I get this error: 'Object reference not set to an instance of an object'.
I have simplified the case and I get this error when I have both of the lines (condition and import) in my .yaml file:
variables:
${{ if eq('bbb', 'master') }}:
- group: variables-shared
If I remove condition, everything works as expected. If I remove group importing, I get other errors related to undefined variable below (that is normal).
I am interested why I get this error
I also had this exact issue. The reasoning in the currently accepted answer is not correct, and you can in fact use conditional insertion in your azure-pipelines.yml file to conditionally include a particular variable group.
When the docs say that conditional insertion requires the use of template syntax, they're referring to template expression syntax, not the use of templates directly. Per the previous link, template expression syntax is the ${{ }} syntax for expanding expressions.
As gleaned from this example from the docs, The problem with the example in the question is actually a syntax error.
Incorrect:
variables:
${{ if eq('bbb', 'master') }}:
- group: variables-shared
Correct:
variables:
- ${{ if eq('bbb', 'master') }}:
- group: variables-shared
Note the leading - before the $ char on the second line. I've tested this in my own pipelines and it works, although it does freak out the yaml syntax checker in my IDE.
I actually discovered a hack yesterday for debugging this obnoxiously vague and wide-ranging error message.
For the build that fails with this message, if you click "Run new" and try to run the job manually by clicking "Run", it will typically give you a much more specific error message at that point.
For instance:
If I remove condition, everything works as expected.
I am interested why I get this error
Check the Expressions doc and you will find this: Conditionals only work when using template syntax.
That's why you can not use condition for your variable group.
The workaround is to use the template to store your variables rather than variable groups.
Please refer to Variable reuse:
# File: vars.yml
variables:
favoriteVeggie: 'brussels sprouts'
# File: azure-pipelines.yml
variables:
- template: vars.yml # Template reference
steps:
- script: echo My favorite vegetable is ${{ variables.favoriteVeggie }}.
I found two other potential causes to "An error occurred while loading the yaml build pipeline. Object reference not set to an instance of an object."
"stages:" was missing in stage template before the stage
Template file reference didn't exist (- template: 'template.yml')
I had also an object reference not set to an instance of an object with code 606802. The build pipeline had no errors at all.
The error was caused by a pre-validation build, where 1 parameter value had no default value.
After adding the default value, the PR validation build succeeded.