I have the following DotNet Test task in my pipeline
displayName: 'unit tests'
inputs:
command: test
projects: '**/*Unit*.csproj'
publishTestResults: true
arguments: '/p:CollectCoverage=true /p:CoverletOutputFormat=cobertura /p:CoverletOutput=results/'
How can I fail the pipeline if no files matched the project pattern: '**/*Unit*.csproj'?
Currently, it displays the current error message and moves on to the next task
##[warning]Project file(s) matching the specified pattern were not found.
Use the Visual Studio Test task. It has a minimumExpectedTests parameter, so if you set it to 1, the task will fail if 0 tests are run.
One could also run a bash script checking for the existence of the test result file (and that number of results is greater than 0):
- bash: |
if [ $(test_results_host_folder)**/*.trx ] && [ $(grep -E "<UnitTestResult" $(test_results_host_folder)**/*.trx -c) -gt 0 ]; then
echo "##vso[task.setVariable variable=TESTRESULTFOUND]true"
fi
displayName: Check if test results file & results >= 1
- script: |
echo No test result found
exit 1
displayName: No test result found
condition: ne(variables.TESTRESULTFOUND, 'true')
If you have stages in your yaml pipeline, then you can do something like this, this will not run the next stage if the previous stage has failed
stages:
- stage: unittests
displayName: 'unit tests'
- stage: nextstage
dependsOn: unittests
displayName: 'unit tests'
As far as I know, we cannot set the task itself to make the pipeline fail when it cannot find the file.
For a workaround:
You could use the Build Quality Checks task from Build Quality Checks extension.
This task can scan all set tasks and check warnings. If the number of warnings is greater than the set upper limit, the pipeline will fail.
Result:
Related
I'm following the gitlab docs for multi-project pipelines. I'm running on gitlab.com (not enterprise/self-hosted).
I have successfully set up a multi-project pipeline. My question is - is there a way to pass some but not all variables between stages?
Here's a very simple build script for two projects:
Main project:
variables:
THIS_PROJECT_NAME: trigger-source
SHARED_ARGUMENT: "hello world!"
stages:
- build
- downstream
build-code-job:
stage: build
script:
- echo "${THIS_PROJECT_NAME}"
- echo "${SHARED_ARGUMENT}"
run-trigger-job:
stage: downstream
inherit:
variables: false
variables:
SHARED_ARGUMENT: $SHARED_ARGUMENT
trigger: my-org/triggers_dest
Triggered project:
variables:
THIS_PROJECT_NAME: trigger-dest
SHARED_ARGUMENT: "overwrite me"
stages:
- test
triggered-job:
stage: test
script:
- echo "${THIS_PROJECT_NAME}"
- echo "${SHARED_ARGUMENT}"
only:
- pipelines
when I run this with inherit: variables: false, the output in the triggered project's builds just show the default variables (no variables are passed):
$ echo "${THIS_PROJECT_NAME}"
trigger-dest
$ echo "${SHARED_ARGUMENT}"
overwrite me
However, when I use inherit: variables: true, all variables are passed, except the value of SHARED_ARGUMENT is actually written as the literal "$SHARED_ARGUMENT, which then gets expanded to "overwrite me":
$ echo "${THIS_PROJECT_NAME}"
trigger-source
$ echo "${SHARED_ARGUMENT}"
overwrite me
This is the opposite of what I want! Essentially I want to whitelist variables to pass through, rather than blacklisting them as above. Any way to do this?
Found the answer buried in the docs on the inherit: variables keyword. In addition to true/false, you can specify a list of variables to inherit.
Changing the source project's .gitlab-ci.yml to the following:
variables:
THIS_PROJECT_NAME: trigger-source
SHARED_ARGUMENT: "hello world!"
stages:
- build
- downstream
build-code-job:
stage: build
script:
- echo "${THIS_PROJECT_NAME}"
- echo "${SHARED_ARGUMENT}"
run-trigger-job:
stage: downstream
inherit:
variables:
- SHARED_ARGUMENT
trigger: my-org/triggers_dest
results in the desired output:
$ echo "${THIS_PROJECT_NAME}"
trigger-dest
$ echo "${SHARED_ARGUMENT}"
hello world!
I have same stages :
stage:
- A
- B
job1:
stage: A
job2:
stage: A
job3 :
stage: B
the sequence must be job1 -> job3 -> job2 and each job depend from previous job.
job1 and job3 is working fine but as job2 depend from job3 (previous stage) is not working.
I get an error 'job2 job: need job3 is not defined in current or prior stages'
Any solution for this problem ?
You can use the needs keyword to create an acyclic relationship between one job and another - which also lets you make the job depend on any job from any previous stage - however in your case you don't need to do that.
Use the needs keyword for more advanced scenarios (like ignoring job failures from the proceeding stage).
Keep in mind that all jobs that share a stage will be run (by default) in parallel. This should work just fine:
stages:
- first
- second
job1:
stage: first
script: echo "this job will run"
job2:
stage: first
script: echo "at the same time as this job"
job3:
stage: second
script: echo "this job will run after all jobs in the first stage succeed"
Your question was
"How make a job depend from other job in previous stage"
and the above example answers that question. However in your description you wrote
"...the sequence must be job1 -> job3 -> job2"
Which, based on that description, could be achieved like so:
- first
- second
- third
job1:
stage: first
script: echo "this job will run during the first stage"
job2:
stage: third
script: echo "this job will run during the third stage"
job3:
stage: second
script: echo "this job will run during the second stage"
The above config would guarantee that none of your jobs execute in parallel and proceed in the sequence you described. You still wouldn't need the needs keyword in this case and the config would be easier to read.
There is a typo in your code. It has to be stages and not stage.
Below is the working code.
stages:
- A
- B
job1:
stage: A
script:
- echo "Job 1"
job2:
stage: A
script:
- echo "Job 2"
needs:
- job1
job3 :
stage: B
script:
- echo "Job 3"
needs:
- job2
I had to remove the tags: as it was internal to my runner.
See t he dependency is shown clearly in the Visualize tab.
And I validated that it triggers job1 first and job3 waits for it.
I have a gitlab-ci like this:
stages:
- calculation
- execution
calculation-job:
stage: calculation
script: ./calculate_something_and_output_results.sh
tags:
- my-runner
execution-job:
stage: execution
script: ./execute_something_with_calculation_results.sh foo
tags:
- my-runner
The foo argument in execution-job is base on the results of calculation-job. I want to pass the results from one job to another job via variables. How can I do that?
If you're looking to get the results without storing a file anywhere you can use artifacts: reports: dotenv. This is taken entirely from DarwinJS shared-variables-across-jobs repo.
stages:
- calculation
- execution
calculation-job:
stage: calculation
script: - |
# stores new or updates existing env variables, ex. $OUTPUT_VAR1
./calculate_something_and_output_results.sh >> deploy.env
tags:
- my-runner
artifacts:
reports:
#propagates variables into the pipeline level, but never stores the actual file
dotenv: deploy.env
execution-job:
stage: execution
script: - |
echo "OUTPUT_VAR1: $OUTPUT_VAR1"
./execute_something_with_calculation_results.sh foo
tags:
- my-runner
AFAIK it is not possible to pass a variable directly from one job to another job. Instead you have to write them into a file and pass that as artifact to the receiving job. To make parsing of the file easy, I recommend to create it with bash export statements and source it in the receiving job's script:
calculation-job:
stage: calculation
script:
- ./calculate_something_and_output_results.sh
- echo "export RESULT1=$calculation_result1" > results
- echo "export RESULT2=$calculation_result2" >> results
tags:
- my-runner
artifacts:
name: "Calculation results"
path: results
execution-job:
stage: execution
script:
- source ./results
# You can access $RESULT1 and $RESULT2 now
- ./execute_something_with_calculation_results.sh $RESULT1 $RESULT2 foo
tags:
- my-runner
needs: calculation-job
Note the ./ when sourcing results might be necessary in case of a POSIX compliant shell that does not source files in the current directory directly like, for example, a bash started as sh.
As a simpler version of what #bjhend answered (no need for export or source statements), since GitLab 13.1 the docs. recommend using a dotenv artifact.
stages:
- calculation
- execution
calculation-job:
stage: calculation
script:
# Output format must feature one "VARIABLE=value" statement per line (see docs.)
- ./calculate_something_and_output_results.sh >> calculation.env
tags:
- my-runner
artifacts:
reports:
dotenv: calculation.env
execution-job:
stage: execution
script:
# Any variables created by above are now in the environment
- ./execute_something_with_calculation_results.sh
tags:
- my-runner
# The following is technically not needed, but serves as good documentation
needs:
job: calculation-job
artifacts: true
If you have a job after the calculation stage that you don't want to use the variables, you can add the following to it:
needs:
job: calculation-job
artifacts: false
I have been trying to define a variable in a lifecycle hook within a deployment job, to then access that variable in a later lifecycle hook. The documentation on deployment jobs references the ability to do so, but offers no actual examples of this case (added emphasis):
Define output variables in a deployment job's lifecycle hooks and consume them in other downstream steps and jobs within the same stage.
The sample pipeline that I've been working with:
stages:
- stage: Pipeline
jobs:
- deployment: Deploy
environment: 'testing'
strategy:
runOnce:
preDeploy:
steps:
- bash: |
echo "##vso[task.setvariable variable=myLocalVar;isOutput=false]local variable"
name: setvarStep
- bash: |
echo "##vso[task.setvariable variable=myOutputVar;isOutput=true]output variable"
name: outvarStep
- bash: |
echo 'Both $(myLocalVar) and $(outvarStep.myOutputVar) are available here'
echo "Both $(myLocalVar) and $(outvarStep.myOutputVar) are available here"
deploy:
steps:
- bash: |
echo 'Neither $(myLocalVar) nor $(outvarStep.myOutputVar) are available here'
echo "Neither $(myLocalVar) nor $(outvarStep.myOutputVar) are available here"
I have tried any number of options, but nothing I've done seems to allow this to actually work - the output of the bash task in the deploy lifecycle hook is:
Neither nor are available here
I've tried wiring in the variables to the bash task via the env: input, both using the macro syntax (eg $(myOutputVar) and the runtime expression syntax, hoping maybe there's a hidden dependency I could find (eg $[ dependencies.Deploy.outputs['preDeploy.outVarStep.myOutputVar'] ], and many, many other syntaxes)
I've tried defining the variables at the job level, hoping they'd be updated by the first preDeploy lifecycle hook and available to the deploy lifecycle hook.
I've tried echo-ing the variables via the runtime expression syntax
Many other combinations of syntax, hoping I'd stumble onto the actual answer
But all to no avail. I will likely resort to a hack of uploading the variable as an artifact and downloading it later, but would really like to find the solution to this issue. Has anyone been able to accomplish this? Many thanks in advance.
Work around:
Firstly, share you the correct sample on this scenario you are looking for:
stages:
- stage: Pipeline
jobs:
- deployment: Deploy
environment: 'testing'
strategy:
runOnce:
preDeploy:
steps:
- bash: |
echo "##vso[task.setvariable variable=myLocalVar;isOutput=false]local variable"
name: setvarStep
- bash: |
echo "##vso[task.setvariable variable=myOutputVar;isOutput=true]output variable"
name: outvarStep
- bash: |
echo 'Both $(myLocalVar) and $(outvarStep.myOutputVar) are available here'
echo "Both $(myLocalVar) and $(outvarStep.myOutputVar) are available here"
mkdir -p $(Pipeline.Workspace)/variables
echo "$(myLocalVar)" > $(Pipeline.Workspace)/variables/myLocalVar
echo "$(outvarStep.myOutputVar)" > $(Pipeline.Workspace)/variables/myOutputVar
- publish: $(Pipeline.Workspace)/variables
artifact: variables
deploy:
steps:
- bash: |
echo 'Neither $(myLocalVar) nor $(outvarStep.myOutputVar) are available here'
echo "Neither $(myLocalVar) nor $(outvarStep.myOutputVar) are available here"
- download: current
artifact: variables
- bash: |
myLocalVar=$(cat $(Pipeline.Workspace)/variables/myLocalVar)
myOutputVar=$(cat $(Pipeline.Workspace)/variables/myOutputVar)
echo "##vso[task.setvariable variable=myLocalVar;isoutput=true]$myLocalVar"
echo "##vso[task.setvariable variable=myOutputVar;isoutput=true]$myOutputVar"
name: output
- bash: |
echo "$(output.myLocalVar)"
echo "$(output.myOutputVar)"
name: SucceedToGet
You will see that the output variables can succeed to printed in SucceedToGet task:
Explanation for why your previous attempts were keeping failed:
For our system, strategy represents one job before it start to running(compile time). It only expanded at running time.
Define output variables in a deployment job's lifecycle hooks and
consume them in other downstream steps and jobs within the same stage.
Here other downstream steps and jobs means a independent job which independent at compile time. That's why we provide that sample YAML under this line.
Is it possible to make a build Pipeline with a file-based trigger?
Let's say I have the following Directory structure.
Microservices/
|_Service A
|_Test_Stage
|_Testing_Config
|_QA_Stage
|_QA_Config
|_Prod_stage
|_Prod_Config
|_Service B
|_Test_Stage
|_Testing_Config
|_QA_Stage
|_QA_Config
|_Prod_stage
|_Prod_Config
I want to have just one single YAML Build Pipeline File.
Based on the Variables $(Project) & $(Stage) different builds are created.
Is it possible to check what directory/file initiated the Trigger and set the variables accordingly?
Additionally it would be great if its possible to use those variables to set the tags to the artifact after the run.
Thanks
KR
Is it possible to check what directory/file initiated the Trigger and
set the variables accordingly?
Of course yes. But there's no direct way since we do not provide any pre-defined variables to store such message, so you need additional complex work around to get that.
#1:
Though there's no variable can direct stores the message like which folder and which file is modified, but you could get it by tracking the commit message Build.SourceVersion via api.
GET https://dev.azure.com/{organization}/{project}/_apis/git/repositories/{repositoryId}/commits/{commitId}/changes?api-version=5.1
From its response body, you can directly know its path and file:
Since the response body is JSON format, you could make use of some JSON function to parsing this path value. See this similar script as a reference.
Then use powershell script to set these value as pipeline variable which the next agent jobs/tasks could use them.
Also, in your scenario, all of these should be finished before all next job started. So, you could consider to create a simple extension with pipeline decorator. Define all above steps in decorator, so that it can be finished in the pre-job of every pipeline.
#2
Think you should feel above method is little complex. So I'd rather suggest you could make use of commit messge. For example, specify project name and file name in commit message, get them by using variable Build.SourceVersionMessage.
Then use the powershell script (I mentioned above) to set them as variable.
This is more convenient than using api to parse commits body.
Hope one of them could help.
Thanks for your reply.
I tried a different approach with a Bash Script.
Because I only use ubuntu Images.
I make "git log" with Filtering for the last commit of the Directory Microservices.
With some awk (not so a satisfying Solution) I get the Project & Stage and write them into Pipeline Variables.
The Pipeline just gets triggered when there is a change to the Microservices/* Path.
trigger:
batch: true
branches:
include:
- master
paths:
include:
- Microservices/*
The first job when the trigger activated, is the Dynamic_variables job.
This Job I only use to set the Variables $(Project) & $(Stage). Also the build tags are set with those Variables, so I'm able do differentiate the Artifacts in the Releases.
jobs:
- job: Dynamic_Variables
pool:
vmImage: 'ubuntu-latest'
steps:
- checkout: self
- task: Bash#3
name: Dynamic_Var
inputs:
filePath: './scripts/multi-usage.sh'
arguments: '$(Build.SourcesDirectory)'
displayName: "Set Dynamic Variables Project"
- task: Bash#3
inputs:
targetType: 'inline'
script: |
set +e
if [ -z $(Dynamic_Var.Dynamic_Project) ]; then
echo "target Project not specified";
exit 1;
fi
echo "Project is:" $(Dynamic_Var.Dynamic_Project)
displayName: 'Verify that the Project parameter has been supplied to pipeline'
- task: Bash#3
inputs:
targetType: 'inline'
script: |
set +e
if [ -z $(Dynamic_Var.Dynamic_Stage) ]; then
echo "target Stage not specified";
exit 1;
fi
echo "Stage is:" $(Dynamic_Var.Dynamic_Stage)
displayName: 'Verify that the Stage parameter has been supplied to pipeline'
The Bash Script I run in this Job looks like this:
#!/usr/bin/env bash
set -euo pipefail
WORKING_DIRECTORY=${1}
cd ${WORKING_DIRECTORY}
CHANGEPATH="$(git log -1 --name-only --pretty='format:' -- Microservices/)"
Project=$(echo $CHANGEPATH | awk -F[/] '{print $2}')
CHANGEFILE=$(echo $CHANGEPATH | awk -F[/] '{print $4}')
Stage=$(echo $CHANGEFILE | awk -F[-] '{print $1}')
echo "##vso[task.setvariable variable=Dynamic_Project;isOutput=true]${Project}"
echo "##vso[task.setvariable variable=Dynamic_Stage;isOutput=true]${Stage}"
echo "##vso[build.addbuildtag]${Project}"
echo "##vso[build.addbuildtag]${Stage}"
If someone has a better solution then the awk commands please let me know.
Thanks a lot.
KR