Gitlab CICD: use functions inside gitlab-ci.yml - gitlab

I have a .gitlab-ci.yml file which allows needs to execute the same function for every step. I have the following and this works.
image:
name: hashicorp/terraform
before_script:
- export MYDATE=$(date "+%d/%m/%y - %H:%M:%S")
stages:
- validate
- plan
validate:
stage: validate
script:
- terraform validate
- 'curl --request POST --header "Authorization: Bearer $bearer" --form "text=$MYDATE $msg" https://api.teams.com/v1/messages'
variables:
msg: "Example1"
plan:
stage: plan
script:
- terraform validate
- 'curl --request POST --header "Authorization: Bearer $bearer" --form "text=$MYDATE $msg" https://api.teams.com/v1/messages'
variables:
msg: "Example2"
Given it is always the same curl command, I wanted to use a function which I declare once and can use in every step. Something along the lines of below snippet.
image:
name: hashicorp/terraform
before_script:
- export MYDATE=$(date "+%d/%m/%y - %H:%M:%S")
.send_message: &send_message
script:
- 'curl --request POST --header "Authorization: Bearer $bearer" --form "text=$MYDATE $msg" https://api.teams.com/v1/messages'
stages:
- validate
- plan
validate:
stage: validate
script:
- terraform validate
- &send_message
variables:
msg: "Example1"
plan:
stage: plan
script:
- terraform validate
- &send_message
variables:
msg: "Example2"
How could I use such a function in a .gitlab-ci.yml file.

you can used include with !reference such as:
functions.yml
.send_message:
script:
- 'curl --request POST --header "Authorization: Bearer $bearer" --form "text=$MYDATE $msg" https://api.teams.com/v1/messages'
.gitlab-ci.yml
include:
- local: functions.yml
default:
image:
name: hashicorp/terraform
before_script:
- export MYDATE=$(date "+%d/%m/%y - %H:%M:%S")
stages:
- validate
- plan
validate:
stage: validate
script:
- terraform validate
- !reference [.send_message, script]
variables:
msg: "Example1"
plan:
stage: plan
script:
- terraform validate
- !reference [.send_message, script]
variables:
msg: "Example2"
ref: https://docs.gitlab.com/ee/ci/yaml/yaml_optimization.html#reference-tags

You can also use a regular old bash function defined at the top:
before_script:
- export MYDATE=$(date "+%d/%m/%y - %H:%M:%S")
- send_bearer () { terraform validate; curl --request POST --header "Authorization: Bearer $bearer" --form "text=$MYDATE $1" https://api.teams.com/v1/messages; }
...
validate:
stage: validate
script:
- send_bearer $msg
variables:
msg: "Example1"
plan:
stage: plan
script:
- send_bearer $msg
variables:
msg: "Example2"

Related

Sending cURL command through Gitlab CICD file

I have a .gitlab-ci.yml file. I need to send a cURL command to a REST API (documentation here). When I hardcode parameters like bearer, roomid it works and the message is received on the client (in this example a Webex Teams bot).
The below works (static entries for bearer, roomid and a message)
before_script:
- export roomid
- export bearer
.send_message:
send:
- |
curl --location --request POST 'https://webexapis.com/v1/messages' \
--header 'Authorization: Bearer M***0f' \
--header 'Content-Type: application/json' \
--data-raw '{
"roomId" : "Y***Ri",
"markdown" : "# Message formatted in markdown",
}'
stages:
- convert
convert:
stage: convert
script:
- python3 convert.py -t dna_sites.xlsx
- !reference [.send_message, send]
variables:
msg: "Conversion complete"
However, when I want to make it more generic by using values stored in Gitlab CICD variable section (bearer, roomid) or elsewhere in the .gitlab-ci..yml file (msg), I cannot get it to work. Therefor I'm referencing them through $ sign ($bearer, $roomid and $msg)
before_script:
- export roomid
- export bearer
.send_message:
send:
- |
curl --location --request POST 'https://webexapis.com/v1/messages' \
--header 'Authorization: Bearer $bearer' \
--header 'Content-Type: application/json' \
--data-raw '{
"roomId" : "$roomid",
"markdown" : "$msg",
}'
stages:
- convert
convert:
stage: convert
script:
- python3 convert.py -t dna_sites.xlsx
- !reference [.send_message, send]
variables:
msg: "Conversion complete"
My guess is I'm doing something wrong with the quotes ' or ".

how to create and extend reusable gitlab pipeline?

I have a pipeline like below:
include:
- project: "some-project"
ref: 0.5.0
file: reusable-jobs.yml
variables:
VAR1: ""
stages:
- stage-1
- stage-2
job1:
extends: .reusable-job-1
stage: stage-1
variables:
SOME_VAR: "$VAR1"
job2:
extends: .reusable-job-2
stage: stage-2
variables:
SOME_VAR: "$VAR1"
I want to be able to save this an imported into other projects. with my reusable jobs I create one like this
.some-reusable-job:
image:
name: alpine
script:
- echo "hello"
and then I extended it when I want to use it
ex:
reuse_the_job:
extends: .some-reusable-job
stage: some-stage
but I cannot figure out how to do the same thing with the entire pipeline including the stages
I want to be able to call the pipeline like so:
reuse_the_pipeline:
extends: .my-reusable-pipeline
variables:
VAR1: "hello"
and have the pipeline be created with both stage_1 and stage_2
I tried to create a definition like this in reusable-flow-file.yml
.reusable-flow
include:
- project: "some-project"
ref: 0.5.0
file: reusable-jobs.yml
variables:
VAR1: ""
stages:
- stage-1
- stage-2
job1:
extends: .reusable-job-1
stage: stage-1
variables:
SOME_VAR: "$VAR1"
job2:
extends: .reusable-job-2
stage: stage-2
variables:
SOME_VAR: "$VAR1"
and then use it like this (.gitlab-ci.yml)
include:
- local: "reusable-flow-file.yml"
dev_na:
extends: .reusable-flow
variables:
VAR1: "hello"
but when I try to run it and get lab I get this error
config should implement a script: or a trigger: keyword
not really sure what that try
It's because you're extending the ".reusable-flow" on a job-level. And what you're actually getting is (the following) which is invalid, because the job dev_na has no script property.
dev_na:
include:
- project: "some-project"
ref: 0.5.0
file: reusable-jobs.yml
variables:
VAR1: ""
stages:
- stage-1
- stage-2
job1:
extends: .reusable-job-1
stage: stage-1
variables:
SOME_VAR: "$VAR1"
job2:
extends: .reusable-job-2
stage: stage-2
variables:
SOME_VAR: "$VAR1"
variables:
VAR1: "hello"
You should just define the jobs (and other stuff) on the root level of your template. And play around a bit in the Project > CI/CD > editor where you can get a life preview of the templated configuration.

Return an object variable from a script - Azure YAML pipelines

Consider the following simplified pipeline:
### template.yml ###
parameters:
- name: "tables"
type: object
default: {}
steps:
- ${{ each table in parameters.tables }}:
- task: BackupTask#0
displayName: "Backup ${{ table }}"
### pipeline.yml ###
- template: template.yml
parameters:
tables:
- "table1"
- "table2"
- "table3"
- "table4"
- "table5"
What I would like is that the list of tables are generated with a bash script instead of having to write them by hand. So every time a new table is created it gets automatically backed up by the pipeline.
As a workaround, we can create another pipeline. In this pipeline, we add two powershell tasks. In the first task, we set a variable with tables as the value.
- task: PowerShell#2
inputs:
targetType: 'inline'
script: 'Write-Host "##vso[task.setvariable variable=list]table1,table2,table3"'
In the second task, we use rest api to trigger the pipeline.yml pipeline. In the request body, we use the variable set in the first task as the value of the template parameter.
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
$token = "PAT"
$url="https://dev.azure.com/{org}/{pro}/_apis/pipelines/{pipelineId}/runs?api-version=5.1-preview"
$token = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($token)"))
$JSON = #'
{
"templateParameters": {
"tab":"[$(list)]"
},
}
'#
$response = Invoke-RestMethod -Uri $url -Headers #{Authorization = "Basic $token"} -Method Post -Body $JSON -ContentType application/json
Below is my test sample:
### pipeline.yml ###
parameters:
- name: tab
type: object
default: {}
pool:
vmImage: 'ubuntu-latest'
steps:
- template: template1.yml
parameters:
tables: ${{ parameters.tab }}
template.yml:
### template1.yml ###
parameters:
- name: "tables"
type: object
default: {}
steps:
- ${{ each table in parameters.tables }}:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: echo "${{ table }}"
Then we run the newly created pipeline to trigger the pipeline.yml pipeline, get the result:

Debugging yaml pipeline script

I'm trying to debug a pipeline script in Azure to see what the variables are, but I'm not sure of the syntax.
I have looked through the documentation
What I have tried so far
parameters:
deploymentName: ""
dependsOn: ""
env: ""
dockerfilePath: ""
buildContext: ""
repository: ""
envGroup: ""
dockerRegistryServiceConnection: ""
tag: ""
token: ""
runTests: ""
jobs:
- deployment: ${{ parameters.deploymentName }}
dependsOn: ${{ parameters.dependsOn }}
pool: "Private Docker"
environment: "${{ parameters.envGroup }}"
strategy:
runOnce:
deploy:
steps:
- checkout: self
- task: Docker#2
displayName: "ACR Login"
inputs:
command: login
containerRegistry: "${{ parameters.dockerRegistryServiceConnection }}"
##[debug]Debug!!!!!1 $runTests
##[debug]Debug!!!!!2 $parameters.runTests
##[debug]Debug!!!!!3 ${{parameters.runTests}}
- script: echo runTests in dockerbuild.yml ${{parameters.runTests}}
- script: echo runTests in dockerbuild.yml $parameters.runTests
- script: echo runTests in dockerbuild.yml $runTests
- task: Docker#2
displayName: Build
inputs:
command: build
repository: "${{ parameters.repository }}"
buildContext: "${{ parameters.buildContext }}"
dockerfile: "${{ parameters.dockerfilePath }}"
arguments: "--build-arg TOKEN=${{ parameters.token }} --build-arg RUNTESTS=${{ parameters.runTests }}"
tags: |
$(Build.BuildId)
latest
Problem
The debug and echo do not print anything in the azure pipeline build:
##[debug]Debug!!!!!1 $runTests
##[debug]Debug!!!!!2 $parameters.runTests
##[debug]Debug!!!!!3 ${{parameters.runTests}}
- script: echo runTests in dockerbuild.yml ${{parameters.runTests}}
- script: echo runTests in dockerbuild.yml $parameters.runTests
- script: echo runTests in dockerbuild.yml $runTests
Question
Is this the correct syntax for printing to the azure bash when running the yaml file?
Debugging yaml pipeline script
It seems that you did not define the parameters correctly. We need provide the name, displayName, type, default, values like following (Not every attribute is required):
parameters:
- name: deploymentName
type: string
default: Product
Then we could use the ${{ parameters.deploymentName }} to get the value.
Please check this document Runtime parameters for some more details.
My test YAML file:
parameters:
- name: deploymentName
type: string
default: Product
- name: dependsOn
type: string
default: Test
- name: envGroup
type: string
default: test
- name: runTests
type: number
default: 123
stages:
- stage: Build
jobs:
- job: Build
displayName: Build
pool:
name: MyPrivateAgent
steps:
- script: echo This is for Build
- stage: Dev
jobs:
- job: Test
displayName: Test
pool:
name: MyPrivateAgent
steps:
- script: echo This is for test
- deployment: ${{ parameters.deploymentName }}
dependsOn: ${{ parameters.dependsOn }}
pool:
name: MyPrivateAgent
environment: "${{ parameters.envGroup }}"
strategy:
runOnce:
deploy:
steps:
- checkout: self
- script: echo runTests in dockerbuild.yml ${{parameters.runTests}}
The test result:
Try this:
${{ parameters.runTests}}

Azure Pipelines YAML: Unexpected value 'variables'

I'm using Azure Pipelines as a part of Azure DevOps.
I'm trying to define variables in my template file, because I need to use the same value multiple times.
This is my stage-template.yml:
parameters:
- name: param1
type: string
- name: param2
type: string
variables:
var1: path/${{ parameters.param2 }}/to-my-file.yml
stages:
- stage: Deploy${{ parameters.param2 }}
displayName: Deploy ${{ parameters.param1 }}
jobs:
...
When trying to use this pipeline, I get an error message:
/stage-template.yml (Line: 7, Col: 1): Unexpected value 'variables'
Why is this not working? What did I do wrong?
You can't have variables in a template that is included as a stage, job or step template (i.e. included below a stages, jobs or steps element in a pipeline). You can only use variables in a variable template.
The documentation sadly is not really clear about that.
Including a stage template
# pipeline-using-stage-template.yml
stages:
- stage: stage1
[...]
# stage template reference, no 'variables' element allowed in stage-template.yml
- template: stage-template.yml
Including a variable template
# pipeline-using-var-template.yml
variables:
# variable template reference, only variables allowed inside the template
- template: variables.yml
steps:
- script: echo A step.
If you are using a template to include variables in a pipeline, the included template can only be used to define variables.
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops#variable-reuse
you cant have parameters in the pipeline, only in the templateReferences:
name: string # build numbering format
resources:
pipelines: [ pipelineResource ]
containers: [ containerResource ]
repositories: [ repositoryResource ]
variables: # several syntaxes, see specific section
trigger: trigger
pr: pr
stages: [ stage | templateReference ]
if you want to use variables in templates you have to use proper syntax:
parameters:
- name: param1
type: string
- name: param2
type: string
stages:
- stage: Deploy${{ parameters.param2 }}
displayName: Deploy ${{ parameters.param1 }}
variables:
var1: path/${{ parameters.param2 }}/to-my-file.yml
jobs:
...
This works for me:
In your parent yaml:
stages:
- stage: stage1
displayName: 'stage from parent'
jobs:
- template: template1.yml
parameters:
param1: 'somevalueforparam1'
inside template1:
parameters:
param1: ''
param2: ''
jobs:
- job: job1
workspace:
clean: all
displayName: 'Install job'
pool:
name: 'yourpool'
variables:
- name: var1
value: 'value1'

Resources