Here is my .gitlab-ci.yml file:
script1:
only:
refs:
- merge_requests
- master
changes:
- script1/**/*
script: echo 'script1 done'
script2:
only:
refs:
- merge_requests
- master
changes:
- script2/**/*
script: echo 'script2 done'
I want script1 to run whenever there is a change in script1 directory; likewise script2.
I tested these with a change in script1, a change in script2, change in both the directories, and no change in either of these directories.
Former 3 cases are passing as expected but 4th case, the one with no change in either directory, is failing.
In the overview, Gitlab gives the message
Could not retrieve the pipeline status. For troubleshooting steps, read thedocumentation.
In the Pipelines tab, I have an option to Run pipeline. Clicking on that gives the error
An error occurred while trying to run a new pipeline for this Merge Request.
If there is no job, I want the pipeline to succeed.
Gitlab pipelines do not have any independent validity outside of jobs. A pipeline, by definition, consists of one or more jobs. In your example 4 above no jobs are created. The simplest hack you can add to your pipeline is a job which always runs:
dummyjob:
script: exit 0
Related
I have some problems with understanding how and why "rules" in GitLab CI work.
I have written a minimal code showing the issue in a GitLab project: https://gitlab.com/daniel.grabowski/gitlab-ci-rules-problems It contains two directories ("files1/" and "files2/") with some files in them and a .gitlab-ci.yml file.
My configuration
Here's the CI configuration:
stages:
- build
workflow:
rules:
- if: $CI_PIPELINE_SOURCE == "push"
.job_tpl:
image: alpine:latest
stage: build
variables:
TARGET_BRANCH: $CI_DEFAULT_BRANCH
rules:
- if: $CI_COMMIT_BRANCH == $TARGET_BRANCH
changes:
- $FILES_DIR/**/*
variables:
JOB_ENV: "prod"
- if: $CI_COMMIT_BRANCH != $TARGET_BRANCH
changes:
- $FILES_DIR/**/*
when: manual
allow_failure: true
variables:
JOB_ENV: "dev"
script:
- echo "CI_COMMIT_BRANCH=$CI_COMMIT_BRANCH"
- echo "TARGET_BRANCH=$TARGET_BRANCH"
- echo "JOB_ENV=$JOB_ENV"
files1 job:
extends: .job_tpl
variables:
FILES_DIR: files1
files2 job:
extends: .job_tpl
variables:
FILES_DIR: files2
As you can see in the above code I'm using workflow to run only "branch pipelines" and have two "twin" jobs configured to watch for changes in one of the project's directories each. The TARGET_BRANCH variable is of course unnecessary in the demo project but i need something like this in the real one and it shows one of my problems. Additionally the jobs behave differently depending on the branch for which they are run.
My expectations
What I want to achieve is:
Each of the jobs should be added to a pipeline only when I push changes to files1/ or files2/ directory respectively.
When I push changes to a branch different then "main" a manual job responsible for the changed directory shoud be added to a pipeline.
When I merge changes to the "main" branch a job responsible for the changed directory shoud be added to a pipeline and it should be automatically started.
Test scenario
I'm creating a new branch from "main", make some change in the file1/test.txt and push the branch to GitLab.
what I expect: a pipeline created with only "files1 job" runnable manually
what I get: a pipeline with both jobs (both manual). Actually I've found explanation of such behaviour here: https://docs.gitlab.com/ee/ci/jobs/job_control.html#jobs-or-pipelines-run-unexpectedly-when-using-changes - "The changes rule always evaluates to true when pushing a new branch or a new tag to GitLab."
On the same branch I make another change in the file1/test.txt and make push.
what I expect: a pipeline created with only "files1 job" runnable manually
what I get: exactly what I expect since the branch isn't a "new" one
I create a Merge Request from my branch to main and make the merge.
what I expect: a pipeline created with only "files1 job" which starts automatically
what I get: a pipeline created with only "files1 job" but a manual one
My questions/problems
Can you suggest me any way to bypass the issue with "changes" evaluating always to "true" on new branches? Actually it behaves exactly as I want it if I don't use "rules" but let's assume I need "rules".
Why the jobs run as "manual" on the main branch in spite of the "if" condition in which both CI_COMMIT_BRANCH and TARGET_BRANCH variables are (or should be) set to "main". To debug it I'm printing those vars in job's "script" and when I run it on "main" pipeline I'm getting:
$ echo "CI_COMMIT_BRANCH=$CI_COMMIT_BRANCH"
CI_COMMIT_BRANCH=main
$ echo "TARGET_BRANCH=$TARGET_BRANCH"
TARGET_BRANCH=main
$ echo "JOB_ENV=$JOB_ENV"
JOB_ENV=dev
so theoretically CI should enter into the "automatic" job path.
Generally I find the CI "rules" quite inconvenient and confusing but as I understand it GitLab prefers them to "only/except" solution so I'm trying to refactor my CI/CD to use them which will fail if I don't find solution for the above difficulties :(
From what I have just ran into, the "needs" line in a gitlab-ci.yml file only checks to see if the job that is defined in the "needs" line is being run - not if it passes or fails.
I ran the below code in my pipeline and the "build-latest" job runs even if the "test-
tag" job fails.
I only want the "build-latest" job to run if the "test-tag" job passes.
How is this achieved?
build-latest:
stage: publish
image:
name: gcr.io/go-containerregistry/crane:debug
entrypoint: [""]
rules:
#- if: $CI_COMMIT_TAG != null
- if: $CI_COMMIT_REF_NAME == "add-latest-tagging"
when: always
needs:
- test-tag
script:
- crane auth login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
The issue lies with the fact that you added
when: always
It is true that since you specified needs, the build-latest job will need the job test-tag to execute first.
After test-tag job concluded it will evaluate if it should execute the build-latest job.
By adding the always clause to the build-latest job will force it to execute, even if the test-tag fails. Provided test-tag job has at least concluded
Long story sort, you should remove the when always clause
If you want a job to run only when one or more previous jobs pass, then you want to put it in a separate stage.
Not sure how you've broken up the jobs without more of the CI file, but assuming:
test-tag job is in stage: test
stage: publish comes after test
Then it should work the way you want simply by removing the needs: option from your build-latest job.
I have a very basic Gitlab CICD yaml file that is supposed to run two jobs on completion of another job. I'd like one job to automatically set a variable, while the other job prompts the user to define one. These jobs are called auto and manual respectively. However, when I run the file below, the manual step just doesn't set anything and continues to run without prompting the user for variables. This does make sense, but how exactly do I force this job to prompt the user for variable definitions?
stages:
- A
- B
.template
script:
- echo ${MY_VARIABLE}
setup:
stage: A
script:
- echo "do nothing yet"
allow_failure: false
when: manual
auto:
stage: B
variables:
MY_VARIABLE: "I am automatically set"
when: on_success
dependencies:
- setup
manual:
stage: B
when: manual
dependencies:
- setup
Currently, the above results in:
auto output
I am automatically set
manual output
What I want:
UPDATE: It doesn't look like there is a way to force the above.
I would like to know how to manually trigger specific jobs in a project's CI pipeline.
Since there is only one gitlab-ci.yml file, I can define many jobs to be executed one after the other sequentially. But what if I want to start a manual CI pipeline that only carries out one job?
As I understand it, every time the pipeline will run, it will run all jobs, unless I use many only and similar parameters. For instance, when I have this simple pipeline config:
stages:
- build
build:
stage: build
script:
- npm i
- npm run build
- echo "successful build"
What do I do if I want to only run an echo job that runs a simple echo "hello" script, but does only that and only when I manually run it? There are no 'triggers' for a job like that afaik.
Is this even a possibility?
Thanks for the clarification!
Apparently, the solution is pretty simple, just needed to add a when: manual paramater to the job:
echo:
stage: echo
script:
- echo 'this is a manual job'
when: manual
Once that's done, the job can be triggered independently right here:
I'm trying to run 2 pipelines for a project in GitLab, but I can't find any way to do it.
In gitlab CI you can't create multiple pipelines for one project explicitly. There are cases where multiple pipelines will run simultaneously, such as when you have jobs that run only for merge requests and other jobs that do not run on merge requests.
That said, there are ways to obtain the effect of running multiple series of jobs independently from one another.
The hacky way, before gitlab-ce 12.2
If you want to start 2 pipelines for the same project you can use pipeline triggers. This method is limited to 2 pipelines and gitlab CI is not meant to be used this way. Usually, triggers are used to start pipelines on another project.
All in your .gitlab-ci.yml:
stages:
- start
- build
###################################################
# First pipeline #
###################################################
start_other_pipeline:
stage: start
script:
# Trigger a pipeline for current project on current branch
- curl --request POST --form "token=$CI_JOB_TOKEN" --form ref=$CI_COMMIT_REF_NAME $CI_API_V4_URL/projects/$CI_PROJECT_ID/trigger/pipeline
except:
- pipelines
build_first_pipeline:
stage: build
script:
- echo "Building first pipeline"
except:
- pipelines
###################################################
# Second pipeline #
###################################################
# Will run independently of the first pipeline.
build_second_pipeline:
stage: build
script:
- echo "Building second pipeline"
only:
- pipelines
To clean up this mess of a .gitlab-ci.yml, you can use the include keyword:
# .gitlab-ci.yml
include:
- '/first-pipeline.yml'
- '/second-pipeline.yml'
stages:
- start
- build
# This starts the second pipeline. The first pipeline is already running.
start_other_pipeline:
stage: start
script:
# Trigger a pipeline for current project on current branch
- curl --request POST --form "token=$CI_JOB_TOKEN" --form ref=$CI_COMMIT_REF_NAME $CI_API_V4_URL/projects/$CI_PROJECT_ID/trigger/pipeline
except:
- pipelines
# first-pipeline.yml
build_first_pipeline:
stage: build
script:
- echo "Building first pipeline"
except:
- pipelines
# second-pipeline.yml
build_second_pipeline:
stage: build
script:
- echo "Building second pipeline"
only:
- pipelines
The reason this works is the use only and except in the jobs. The jobs marked with
except:
- pipelines
do not run when the pipeline has started because of a trigger coming from another pipeline, so they don't run in the second pipeline. On the other hand,
only:
- pipelines
does the exact opposite, therefore those jobs run only when the pipeline is triggered by another pipeline, so they only run in the second pipeline.
The probably right way, depending on your needs ;)
In gitlab CE 12.2, it is possible to define Directed Acyclic Graphs to specify the order that your jobs run. This way, a job can start as soon as the job it depends on (using needs) finishes.
As of GitLab 12.7 it is also possible to use parent-child pipelines for this:
# .gitlab-ci.yml
trigger_child:
trigger:
include: child.yml
do_some_stuff:
script: echo "doing some stuff"
# child.yml
do_some_more_stuff:
script: echo "doing even more stuff"
The trigger_child job completes successfully once the child pipeline has been created.