I am trying to execute an azure pipeline. It was executing fine. But when I added a task to conditionally check pom.xml file exists then only I need to execute that particular task, but it fails
Error
Your script needs to be more indented:
- bash: |
if [ -f Maven pom.xml ]; then
echo "##vso[task.setVariable variable=FILEEXISTS]true"
fi
- task: Maven#3
etc
See an example here: https://stackoverflow.com/a/58555822/174843
Related
I have a simple pipeline with one job to test bash scripts. The pipeline as follow:
image: alpine/git
stages:
- test_branching
test_branch:
stage: test_branching
before_script:
- mkdir -p .common
- wget https://x.x.x.x/branching.sh > .common/test.sh && chmod +x .common/test.sh
- source .common/test.sh
script:
- test_pipe
- echo "app version is ${app_version}"
The bash script as follow:
#!/bin/sh
function test_pipe () {
app_version="1.0.0.0-SNAPSHOT"
}
The problem is that the pipeline for whatever reason does not recognize the function inside the script. The logs are:
...
$ test_pipe
/scripts-1050-417479/step_script: eval: line 180: test_pipe: not found
Does anybody know what happend with this?? I miss a lot Jenkins shared libraries, gitlab does not have it, also gitlab does not have the function to include scripts inside yml files.
I dont want to use multiproject pipeline, I need to do it at this way. This is only an example of a more complicated pipeline logic.
Thanks in advance
As the documentation states before_script is just concatenated together with script and run on a single shell. The script you are downloading does not define test_pipe.
... gitlab does not have the function to include scripts inside yml
files.
It does, just use the YAML multiline literal syntax with |, e.g.:
script:
- |
echo "this"
echo "is"
echo "an \
example"
variable big_var_01 defined with value '3q4w#V$X3q4w#V$X' by following azure pipeline yaml file gets corrupted to become value '3q4w#V#V' when read back in azure pipeline template
cat parent_scott.yaml
variables:
- name: big_var_01
value: ${{ parameters.big_var_01 }}
parameters:
- name: big_var_01
displayName: "this var wants to get written to then read by templates"
type: string
default: '3q4w#V$X3q4w#V$X'
# CI Triggers
trigger:
branches:
exclude:
- '*'
pool:
vmImage: 'ubuntu-latest'
# Release Stages
stages:
- template: child_scott_one.yaml
following azure pipeline template variable big_var_01 is read back however its value is corrupted and does not match above assignment
cat child_scott_one.yaml
# Release Stages
stages:
- stage: A
jobs:
- job: JA
steps:
- script: |
echo "here is big_var_01 -->$(big_var_01)<-- "
local_var_01=$(big_var_01)
echo
echo "here is local_var_01 -->$local_var_01<-- "
echo
echo "length of local_var_01 is ${#local_var_01}"
echo
name: DetermineResult
see run of above pipeline
https://dev.azure.com/sekhemrekhutawysobekhotep/public_project/_build/results?buildId=525&view=logs&j=54e3124b-25ae-54f7-b0df-b26e1988012b&t=52fad91f-d6ac-51fb-b63d-00fda7898bb6&l=13
see code at https://github.com/sekhemrekhutawysobekhotep/shared_variables_across_templates
How to make the string variable big_var_01 get treated as a literal evidently its somehow getting evaluated and thus corrupted ... above code is a simplification of my actual azure pipeline where I get same variable corruption issue even when setting up a Key Value secret with value 3q4w#V$X3q4w#V$X which gets corrupted when read back in a pipeline template
here is another pipeline run which explicitly shows this issue https://dev.azure.com/sekhemrekhutawysobekhotep/public_project/_build/results?buildId=530&view=logs&j=ed5db508-d8c1-5154-7d4e-a21cef45e99c&t=a9f49566-82d0-5c0a-2e98-46af3de0d6e9&l=38 on this run I check marked ON pipeline run option: "Enable system diagnostics" ... next I will try to single quote my shell assignment from the azure variable
At some step Azure DevOps or Ubuntu replaced part of your string. So you have:
3q4w#V$X3q4w#V$X = 3q4w#V + $X3q4w + #V + $X
and this part $X3q4w and this $X was replaced with empty string giving you 3q4w#V + #V.
If you run this with \ before $ like here 3q4w#V\$X3q4w#V\$X
This is job Foo.
here is big_var_01 -->3q4w#V$X3q4w#V$X<--
here is local_var_01 -->3q4w#V$X3q4w#V$X<--
length of local_var_01 is 16
I got an error running this on windows-latest however I got correct string:
"This is job Foo."
"here is big_var_01 -->3q4w#V$X3q4w#V$X<-- "
'local_var_01' is not recognized as an internal or external command,
operable program or batch file.
ECHO is off.
"here is local_var_01 -->$local_var_01<-- "
ECHO is off.
"length of local_var_01 is ${#local_var_01}"
ECHO is off.
##[error]Cmd.exe exited with code '9009'.
so it looks like ubuntu replaces it with env variables however, since there is not variables like $X3q4w and $X it replaces with empty string.
Found a solution ... it works if I single quote around the azure variable during bash shell assignment
local_var_01=$(big_var_01) # bad results with value 3q4w#V#V
local_var_02="$(big_var_01)" # bad results with value 3q4w#V#V
local_var_03='$(big_var_01)' # good this gives value 3q4w#V$X3q4w#V$X
my instincts said to not use single quotes knowing its not the bash shell way however once I accepted the fact azure performs some magic ~~helping~~ interstigal layer pre processing in between pipeline source code and fully resolved bash shell execution I took a chance and tried this ... come to find out this is how bash shell blocks variable expansion
I am converting my project from Jenkins to GitLab CI. There is a .sh file which I am executing from .gitlab-ci.yml file where I am extracting the version from the project file using following statement:
VERSION=$(grep -oPm1 "(?<=)[^<]+" /Service.csproj
I am getting the project version and this is working fine.
How can I run the above statement in .gitlab-ci.yml file and assign the version value to a variable?
I tried running statement but I am getting error like invalid option "P"
I use in my example the sed command
stages:
- test
test:
stage: test
script:
- VERSION=$(sed -n "s/<Version>\(.*\)<\/Version>/\1/p" Service.csproj)
- echo $VERSION
Somehow above grep didn't work for me. Below works for me.
$VERSION=[regex]::match((Get-Content .\AssemblyFileVersion.cs), "AssemblyFileVersion\(`"(.*)`"\)").Groups[1].Value'
You need to execute the command in a script block, so your gitlab-ci.yml could look like this:
stages:
- test
test:
stage: test
script:
- VERSION=$(grep -oPm1 "(?<=)[^<]+" Service.csproj | xargs)
- echo $VERSION
I have been trying to define a variable in a lifecycle hook within a deployment job, to then access that variable in a later lifecycle hook. The documentation on deployment jobs references the ability to do so, but offers no actual examples of this case (added emphasis):
Define output variables in a deployment job's lifecycle hooks and consume them in other downstream steps and jobs within the same stage.
The sample pipeline that I've been working with:
stages:
- stage: Pipeline
jobs:
- deployment: Deploy
environment: 'testing'
strategy:
runOnce:
preDeploy:
steps:
- bash: |
echo "##vso[task.setvariable variable=myLocalVar;isOutput=false]local variable"
name: setvarStep
- bash: |
echo "##vso[task.setvariable variable=myOutputVar;isOutput=true]output variable"
name: outvarStep
- bash: |
echo 'Both $(myLocalVar) and $(outvarStep.myOutputVar) are available here'
echo "Both $(myLocalVar) and $(outvarStep.myOutputVar) are available here"
deploy:
steps:
- bash: |
echo 'Neither $(myLocalVar) nor $(outvarStep.myOutputVar) are available here'
echo "Neither $(myLocalVar) nor $(outvarStep.myOutputVar) are available here"
I have tried any number of options, but nothing I've done seems to allow this to actually work - the output of the bash task in the deploy lifecycle hook is:
Neither nor are available here
I've tried wiring in the variables to the bash task via the env: input, both using the macro syntax (eg $(myOutputVar) and the runtime expression syntax, hoping maybe there's a hidden dependency I could find (eg $[ dependencies.Deploy.outputs['preDeploy.outVarStep.myOutputVar'] ], and many, many other syntaxes)
I've tried defining the variables at the job level, hoping they'd be updated by the first preDeploy lifecycle hook and available to the deploy lifecycle hook.
I've tried echo-ing the variables via the runtime expression syntax
Many other combinations of syntax, hoping I'd stumble onto the actual answer
But all to no avail. I will likely resort to a hack of uploading the variable as an artifact and downloading it later, but would really like to find the solution to this issue. Has anyone been able to accomplish this? Many thanks in advance.
Work around:
Firstly, share you the correct sample on this scenario you are looking for:
stages:
- stage: Pipeline
jobs:
- deployment: Deploy
environment: 'testing'
strategy:
runOnce:
preDeploy:
steps:
- bash: |
echo "##vso[task.setvariable variable=myLocalVar;isOutput=false]local variable"
name: setvarStep
- bash: |
echo "##vso[task.setvariable variable=myOutputVar;isOutput=true]output variable"
name: outvarStep
- bash: |
echo 'Both $(myLocalVar) and $(outvarStep.myOutputVar) are available here'
echo "Both $(myLocalVar) and $(outvarStep.myOutputVar) are available here"
mkdir -p $(Pipeline.Workspace)/variables
echo "$(myLocalVar)" > $(Pipeline.Workspace)/variables/myLocalVar
echo "$(outvarStep.myOutputVar)" > $(Pipeline.Workspace)/variables/myOutputVar
- publish: $(Pipeline.Workspace)/variables
artifact: variables
deploy:
steps:
- bash: |
echo 'Neither $(myLocalVar) nor $(outvarStep.myOutputVar) are available here'
echo "Neither $(myLocalVar) nor $(outvarStep.myOutputVar) are available here"
- download: current
artifact: variables
- bash: |
myLocalVar=$(cat $(Pipeline.Workspace)/variables/myLocalVar)
myOutputVar=$(cat $(Pipeline.Workspace)/variables/myOutputVar)
echo "##vso[task.setvariable variable=myLocalVar;isoutput=true]$myLocalVar"
echo "##vso[task.setvariable variable=myOutputVar;isoutput=true]$myOutputVar"
name: output
- bash: |
echo "$(output.myLocalVar)"
echo "$(output.myOutputVar)"
name: SucceedToGet
You will see that the output variables can succeed to printed in SucceedToGet task:
Explanation for why your previous attempts were keeping failed:
For our system, strategy represents one job before it start to running(compile time). It only expanded at running time.
Define output variables in a deployment job's lifecycle hooks and
consume them in other downstream steps and jobs within the same stage.
Here other downstream steps and jobs means a independent job which independent at compile time. That's why we provide that sample YAML under this line.
Is it possible to make a build Pipeline with a file-based trigger?
Let's say I have the following Directory structure.
Microservices/
|_Service A
|_Test_Stage
|_Testing_Config
|_QA_Stage
|_QA_Config
|_Prod_stage
|_Prod_Config
|_Service B
|_Test_Stage
|_Testing_Config
|_QA_Stage
|_QA_Config
|_Prod_stage
|_Prod_Config
I want to have just one single YAML Build Pipeline File.
Based on the Variables $(Project) & $(Stage) different builds are created.
Is it possible to check what directory/file initiated the Trigger and set the variables accordingly?
Additionally it would be great if its possible to use those variables to set the tags to the artifact after the run.
Thanks
KR
Is it possible to check what directory/file initiated the Trigger and
set the variables accordingly?
Of course yes. But there's no direct way since we do not provide any pre-defined variables to store such message, so you need additional complex work around to get that.
#1:
Though there's no variable can direct stores the message like which folder and which file is modified, but you could get it by tracking the commit message Build.SourceVersion via api.
GET https://dev.azure.com/{organization}/{project}/_apis/git/repositories/{repositoryId}/commits/{commitId}/changes?api-version=5.1
From its response body, you can directly know its path and file:
Since the response body is JSON format, you could make use of some JSON function to parsing this path value. See this similar script as a reference.
Then use powershell script to set these value as pipeline variable which the next agent jobs/tasks could use them.
Also, in your scenario, all of these should be finished before all next job started. So, you could consider to create a simple extension with pipeline decorator. Define all above steps in decorator, so that it can be finished in the pre-job of every pipeline.
#2
Think you should feel above method is little complex. So I'd rather suggest you could make use of commit messge. For example, specify project name and file name in commit message, get them by using variable Build.SourceVersionMessage.
Then use the powershell script (I mentioned above) to set them as variable.
This is more convenient than using api to parse commits body.
Hope one of them could help.
Thanks for your reply.
I tried a different approach with a Bash Script.
Because I only use ubuntu Images.
I make "git log" with Filtering for the last commit of the Directory Microservices.
With some awk (not so a satisfying Solution) I get the Project & Stage and write them into Pipeline Variables.
The Pipeline just gets triggered when there is a change to the Microservices/* Path.
trigger:
batch: true
branches:
include:
- master
paths:
include:
- Microservices/*
The first job when the trigger activated, is the Dynamic_variables job.
This Job I only use to set the Variables $(Project) & $(Stage). Also the build tags are set with those Variables, so I'm able do differentiate the Artifacts in the Releases.
jobs:
- job: Dynamic_Variables
pool:
vmImage: 'ubuntu-latest'
steps:
- checkout: self
- task: Bash#3
name: Dynamic_Var
inputs:
filePath: './scripts/multi-usage.sh'
arguments: '$(Build.SourcesDirectory)'
displayName: "Set Dynamic Variables Project"
- task: Bash#3
inputs:
targetType: 'inline'
script: |
set +e
if [ -z $(Dynamic_Var.Dynamic_Project) ]; then
echo "target Project not specified";
exit 1;
fi
echo "Project is:" $(Dynamic_Var.Dynamic_Project)
displayName: 'Verify that the Project parameter has been supplied to pipeline'
- task: Bash#3
inputs:
targetType: 'inline'
script: |
set +e
if [ -z $(Dynamic_Var.Dynamic_Stage) ]; then
echo "target Stage not specified";
exit 1;
fi
echo "Stage is:" $(Dynamic_Var.Dynamic_Stage)
displayName: 'Verify that the Stage parameter has been supplied to pipeline'
The Bash Script I run in this Job looks like this:
#!/usr/bin/env bash
set -euo pipefail
WORKING_DIRECTORY=${1}
cd ${WORKING_DIRECTORY}
CHANGEPATH="$(git log -1 --name-only --pretty='format:' -- Microservices/)"
Project=$(echo $CHANGEPATH | awk -F[/] '{print $2}')
CHANGEFILE=$(echo $CHANGEPATH | awk -F[/] '{print $4}')
Stage=$(echo $CHANGEFILE | awk -F[-] '{print $1}')
echo "##vso[task.setvariable variable=Dynamic_Project;isOutput=true]${Project}"
echo "##vso[task.setvariable variable=Dynamic_Stage;isOutput=true]${Stage}"
echo "##vso[build.addbuildtag]${Project}"
echo "##vso[build.addbuildtag]${Stage}"
If someone has a better solution then the awk commands please let me know.
Thanks a lot.
KR