I have a YAML file as below. Let’s say the *.md file is committed, the build does not work, but the test works. Here how can I make the test depend on the build? Like if the build doesn’t work, the test shouldn’t work.
Thanks in advance.
build:
stage: build
script:
- echo "Build is running"
only:
changes:
- Dockerfile
- requirements.txt
- ./configs/*
test:
stage: test
script:
- echo "Test is running"
- echo "$CI_JOB_STAGE"
dependencies:
- build
That should be what stages defines
Use stages to define stages that contain groups of jobs.
stages is defined globally for the pipeline.
Use stage in a job to define which stage the job is part of.
The order of the stages items defines the execution order for jobs:
Jobs in the same stage run in parallel.
Jobs in the next stage run after the jobs from the previous stage complete successfully.
For example:
stages:
- build
- test
- deploy
All jobs in build execute in parallel.
If all jobs in build succeed, the test jobs execute in parallel.
If all jobs in test succeed, the deploy jobs execute in parallel.
If all jobs in deploy succeed, the pipeline is marked as passed.
If any job fails, the pipeline is marked as failed and jobs in later stages do not start.
Jobs in the current stage are not stopped and continue to run.
So, in your case:
stages:
- build
- test
test won't run if build fails.
Related
Within an Azure Pipeline, I have an agent job called "job_3" which I want to run if it fulfills the following conditions:
If Job_1(agent job) is successful
or
If Job_1(agent job) fails but Job_2(agentless job) succeeds. Job_2 runs only if Job_1 fails.
To start Job_3 in the pipeline I used custom variable expressions for it to trigger:
The expression I used:
and( eq(dependencies.Job_1.result,'Succeeded'), eq(dependencies.Job_2.result,'Succeeded') )
I get an error saying that there are cycle dependencies. What can i do to mitigate the issue?
An error saying that there are cycle dependencies.
The cause of this issue could be that there are cycle dependencies in the dependencies set by your jobs.
For example: Job_1 depend on Job_2 , Job2 depend on Job_1.
Based on your requirements, you can reset the following dependencies to fix this issue.
Job2 depend on Job 1 and Job 3 depend on Job 1 and Job 2.
In this case , the job_2 will run depend on the result of Job_1 and the Job_3 will depend on the result of job_1 and job_2.
YAML sample:
jobs:
- job: job1
- job: job2
dependsOn: job1
- job: job3
dependsOn:
- job1
- job2
Classic Sample:
Job_1
Job_2
Job_3
I have 2 stages with multiple jobs and the jobs in the first stage have some rules that tell them if the need to run or not, so what I am trying to do is to tell some of the jobs in the second stage to execute only if the relevant job in the first stage ran.
I don't want to use the same rules I used for the first stage job to prevent conflicts.
Is there a way to do that?
stages:
- build
- deploy
Build0:
stage: build
extends:
- .Build0Rules
- .Build0Make
Build1:
stage: build
extends:
- .Build1Rules
- .Build1Make
Deploy0:
stage: deploy
dependencies:
- Build0
script:
- bash gitlab-ci/deploy0.sh
Deploy1:
stage: deploy
dependencies:
- Build1
script:
- bash gitlab-ci/deploy1.sh
Thank you in advance :)
No you cannot specify that a job should be added to the pipeline if another job was added to the pipeline. Each job can specify whether it is added to the pipeline using only/except conditions or rules, but these are not able to reference other jobs.
It is possible to generate a pipeline yaml file and then trigger it, but I think this would not be ideal because of the amount of work involved.
stages:
- Build
- Deploy
build:
stage: Build
script:
- do something...
artifacts:
paths:
- deploy-pipeline-gitlab-ci.yml
deploy:
stage: Deploy
trigger:
include:
- artifact: deploy-pipeline-gitlab-ci.yml
job: build
strategy: depend
I would recommend using similar only/except conditions or rules on each job to build the pipeline that you want.
Yes you can. You should check the keyword needs that allow to do what you want: execute a job based on the execution of other jobs, ignoring stages order.
The documentation: https://docs.gitlab.com/ee/ci/yaml/#needs
Here is also an exemple of how to build a DAG (direct acrylic graph) using needs: https://about.gitlab.com/blog/2020/12/10/basics-of-gitlab-ci-updated/#directed-acyclic-graphs-get-faster-and-more-flexible-pipelines
In your case:
Deploy0:
stage: deploy
needs: ["Build0"]
script:
- bash gitlab-ci/deploy0.sh
Deploy1:
stage: deploy
needs: ["Build1"]
script:
- bash gitlab-ci/deploy1.sh
Note you can also specify multiple jobs in the needs command:
needs: ["build0", "test0", "test1"]
I would like to know how to manually trigger specific jobs in a project's CI pipeline.
Since there is only one gitlab-ci.yml file, I can define many jobs to be executed one after the other sequentially. But what if I want to start a manual CI pipeline that only carries out one job?
As I understand it, every time the pipeline will run, it will run all jobs, unless I use many only and similar parameters. For instance, when I have this simple pipeline config:
stages:
- build
build:
stage: build
script:
- npm i
- npm run build
- echo "successful build"
What do I do if I want to only run an echo job that runs a simple echo "hello" script, but does only that and only when I manually run it? There are no 'triggers' for a job like that afaik.
Is this even a possibility?
Thanks for the clarification!
Apparently, the solution is pretty simple, just needed to add a when: manual paramater to the job:
echo:
stage: echo
script:
- echo 'this is a manual job'
when: manual
Once that's done, the job can be triggered independently right here:
I'm trying to run 2 pipelines for a project in GitLab, but I can't find any way to do it.
In gitlab CI you can't create multiple pipelines for one project explicitly. There are cases where multiple pipelines will run simultaneously, such as when you have jobs that run only for merge requests and other jobs that do not run on merge requests.
That said, there are ways to obtain the effect of running multiple series of jobs independently from one another.
The hacky way, before gitlab-ce 12.2
If you want to start 2 pipelines for the same project you can use pipeline triggers. This method is limited to 2 pipelines and gitlab CI is not meant to be used this way. Usually, triggers are used to start pipelines on another project.
All in your .gitlab-ci.yml:
stages:
- start
- build
###################################################
# First pipeline #
###################################################
start_other_pipeline:
stage: start
script:
# Trigger a pipeline for current project on current branch
- curl --request POST --form "token=$CI_JOB_TOKEN" --form ref=$CI_COMMIT_REF_NAME $CI_API_V4_URL/projects/$CI_PROJECT_ID/trigger/pipeline
except:
- pipelines
build_first_pipeline:
stage: build
script:
- echo "Building first pipeline"
except:
- pipelines
###################################################
# Second pipeline #
###################################################
# Will run independently of the first pipeline.
build_second_pipeline:
stage: build
script:
- echo "Building second pipeline"
only:
- pipelines
To clean up this mess of a .gitlab-ci.yml, you can use the include keyword:
# .gitlab-ci.yml
include:
- '/first-pipeline.yml'
- '/second-pipeline.yml'
stages:
- start
- build
# This starts the second pipeline. The first pipeline is already running.
start_other_pipeline:
stage: start
script:
# Trigger a pipeline for current project on current branch
- curl --request POST --form "token=$CI_JOB_TOKEN" --form ref=$CI_COMMIT_REF_NAME $CI_API_V4_URL/projects/$CI_PROJECT_ID/trigger/pipeline
except:
- pipelines
# first-pipeline.yml
build_first_pipeline:
stage: build
script:
- echo "Building first pipeline"
except:
- pipelines
# second-pipeline.yml
build_second_pipeline:
stage: build
script:
- echo "Building second pipeline"
only:
- pipelines
The reason this works is the use only and except in the jobs. The jobs marked with
except:
- pipelines
do not run when the pipeline has started because of a trigger coming from another pipeline, so they don't run in the second pipeline. On the other hand,
only:
- pipelines
does the exact opposite, therefore those jobs run only when the pipeline is triggered by another pipeline, so they only run in the second pipeline.
The probably right way, depending on your needs ;)
In gitlab CE 12.2, it is possible to define Directed Acyclic Graphs to specify the order that your jobs run. This way, a job can start as soon as the job it depends on (using needs) finishes.
As of GitLab 12.7 it is also possible to use parent-child pipelines for this:
# .gitlab-ci.yml
trigger_child:
trigger:
include: child.yml
do_some_stuff:
script: echo "doing some stuff"
# child.yml
do_some_more_stuff:
script: echo "doing even more stuff"
The trigger_child job completes successfully once the child pipeline has been created.
In a project I'm running two stages with these jobs:
build
compile & test
generate sonar report
deploy
deploy to staging environment [manual]
deploy to production [manual]
The jobs in the deploy stage depend on the outputs of the compile & test job. However the generate sonar report job is not required to finish before I can start any job in the deploy stage. Nevertheless, GitLab insists that all jobs in the build phase have finished before I can launch any job in the deploy phase.
Is there a way I can tell GitLab that the generate sonar report job should not block subsequent pipeline stages? I already tried allow_failure: true on this job but this does not seem to have the desired effect. This job takes a long time to finish and I really don't want to wait for it all the time before being able to deploy.
We have similar situation, and while we do use allow_failure: true, this does not help when the Sonar job simply takes a long time to run, whether it fails or succeeds.
Since you are not wanting your deploy stage to actually be gated by the outcome of the generate sonar report job, then I suggest moving the generate sonar report job to the deploy stage, so your pipeline would become:
build
compile & test
deploy
deploy to staging environment [manual]
deploy to production [manual]
generate sonar report [allow_failure: true]
This way the generate sonar report job does not delay your deploy stage jobs
The other benefit of running generate sonar report after build & test is that you can save coverage reports from the build & test job as Gitlab job artifacts, and then have generate sonar report job consume them as dependencies, so Sonar can monitor your coverage, too
Finally, we find it useful to separate build & test into build, then test, so we can separate build failures from test failures - and we can then also run multiple test jobs in parallel, all in the same test stage, etc. Note you will need to convey the artifacts from the build job to the test job(s) via Gitlab job artifacts & dependencies if you choose to do this
From my point of view, it depends on your stage semantics. You should try to decide what is mostly important in your pipeline: clarity on stages or get the job done.
GitLab has many handy features like needs keyword you can use it to specify direct edges on the dependency graph.
stages:
- build
- deploy
build:package:
stage: build
script:
- echo "compile and test"
- mkdir -p target && echo "hello" > target/file.txt
artifacts:
paths:
- ./**/target
build:report:
stage: build
script:
- echo "consume the target artifacts"
- echo "waiting for 120 seconds to continue"
- sleep 120
- mkdir -p target/reports && echo "reporting" > target/reports/report.txt
artifacts:
paths:
- ./**/target/reports
deploy:
stage: deploy
needs: ["build:package"]
script:
- echo "deploy your package on remote site"
- cat target/file.txt
Unless I'm mistaken, this is currently not possible, and there is an open feature proposal, and another one similar to add what you are suggesting.